scadaproj/ZB.MOM.WW.LocalDb/CLAUDE.md

# ZB.MOM.WW.LocalDb

Shared embedded SQLite cache + 2-node LWW sync library for the **ZB.MOM.WW SCADA family** (3 packages; consumed by ScadaBridge and OtOpcUa). Build/test/pack from this directory: `dotnet build ZB.MOM.WW.LocalDb.slnx`, `dotnet test ZB.MOM.WW.LocalDb.slnx` (fully offline), `dotnet pack ZB.MOM.WW.LocalDb.slnx -c Release -o artifacts`. Design: `../docs/plans/2026-07-17-localdb-design.md`; adoption designs `../docs/plans/2026-07-19-scadabridge-localdb-design.md` + `../docs/plans/2026-07-20-otopcua-localdb-design.md`.

## Index history (migrated from the scadaproj root CLAUDE.md)

> Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**

### Component-table row as it read in the index

| Component | Status | Goal | Design | Implementation |
|---|---|---|---|---|
| LocalDb (embedded cache + 2-node sync) | **Adopted by ScadaBridge — Phases 1 AND 2 COMPLETE and MERGED** to `main` (2026-07-20, PR #23, merge `28ca04d7`, pushed to origin; both `feat/localdb-phase*` branches fully contained and kept only as history). Libs **`0.3.4`** on the Gitea feed (3 pkgs; `0.1.0`–`0.3.3` also present). **`0.3.4` (2026-08-20, scadaproj `75c1897` — ScadaBridge register row 106; sender-side, wire-compatible, no schema change):** `SnapshotStreamer` stamps `as_of_seq = max(MAX(seq), last_acked_seq)` instead of bare `MAX(seq)`, which read **0** once ack-pruning had emptied a converged sender's oplog — so a rebuilt peer (watermark 0) was healed by the snapshot but never lifted by it, and the `0.1.2` empty-oplog horizon rule re-streamed the whole database on EVERY reconnect until the sender wrote again (ScadaBridge's site-a-a: 127 k rows per restart, observed on two consecutive restarts). Pinned by `WipedPeer_SnapshotLiftsItsWatermark_…`; 216/216. **ScadaBridge pins `0.3.4`** (2026-08-20; live: first restart sent one last snapshot `as_of 149`, second restart sent none); OtOpcUa remains `0.2.1` — note OtOpcUa's rebuilt-node path has the same loop until it re-pins. **`0.3.3` (2026-08-20, scadaproj `122940c` — ScadaBridge arch-review P4-7; wire-compatible, no proto/schema change)** closes two latent `LwwApplier` hazards: (1) **`RegisterReplicated` refuses a table that is the parent of a row-mutating `ON DELETE` FK** (`CASCADE`/`SET NULL`/`SET DEFAULT`; `RESTRICT`/`NO ACTION` allowed) via `LocalDbRegistrationException` naming `child.column ON DELETE <action>` — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired child actions while capture was suppressed, a silent divergence for any future FK'd table (none of ScadaBridge's ten or OtOpcUa's four has an FK today; registration-time check only, since SQLite can't add an FK to an existing table); (2) the per-entry apply (delete-or-upsert + ledger write) is **atomic under a `SAVEPOINT`** — a fault between the two statements left a deleted row with a live ledger that `SnapshotStreamer` warns-and-skips forever; poison-catch semantics preserved. 213/213 tests (+9, failure injected with a `RAISE(ABORT)` trigger on the ledger, no production seam). **ScadaBridge pins `0.3.3`** (2026-08-20); OtOpcUa remains `0.2.1`. **`0.3.2` (2026-08-19, scadaproj `f62b017` — ScadaBridge arch-review P4-1 + P4-3; wire-compatible, no proto/schema change)**: (1) the sync pump **faults a live session when a cap-prune invalidates its position** (oplog min-seq horizon check before each read batch + on the batch head; new public `LocalDbOplogGapException` rides the 0.3.1 fault teardown) — previously a slow-but-alive peer silently streamed past the pruned gap and the pair diverged until the next reconnect; (2) **client h2 keepalive pings** via new validated `Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (60s/20s, `Always` policy); (3) the server single-stream guard is **newest-wins with an awaited handover** (`TakeoverTimeout` 10s) — a fresh authenticated `Sync` displaces a dead incumbent instead of being refused `AlreadyExists` forever by a half-open link whose zombie side reports connected. 204/204 tests. **ScadaBridge pins `0.3.2`** (2026-08-19, adding Kestrel h2 keepalive on its site gRPC listener as the server half); OtOpcUa remains `0.2.1`. **`0.3.1` (2026-08-18, scadaproj `3e5a5fd`) flushes a faulting session's control lane before its writer is cancelled** — on a handshake-phase fault (i.e. a schema-mismatch rejection) `RunAsync`'s teardown cancelled the writer while the session's OWN Handshake could still be sitting in the lane (the enqueue→receive→validate→fault path has no yields while the writer's wakeup waits in the thread-pool queue; a pinning test lost the race 39/40), and mismatch detection is symmetric — the peer computes its fail-closed verdict FROM that Handshake — so the healthy peer misread a permanent schema mismatch as a generic disconnect and reconnect-looped at the fast backoff instead of the classified `FailedPrecondition`→LogError+max-backoff path. Teardown now completes both lanes so the writer drains what is already enqueued (2 s deferred cancel as the wedged-transport backstop); caller-initiated shutdown is not delayed, **no wire or config change** — an older peer simply receives the Handshake it was always owed. Closes ScadaBridge deferred-register row 41 (its flaky `Handshake_DigestMismatch_FailsClosed` reproduced 2/6 pre-fix under load, 6/6 green post-fix; the other named flake, `BatchBudgetTests.Pump_RowLargerThanTheBudget…`, was a harness defect — session faults hidden as opaque 15 s timeouts — fixed with a run-observing `NextAsync`). **ScadaBridge re-pinned through `0.3.1` to `0.3.2` (2026-08-19); OtOpcUa remains `0.2.1`.** **`0.3.0` (2026-08-15) makes the snapshot-resync flag PER-TABLE** (branch `localdb-per-table-snapshot`; the residual `0.2.1` explicitly deferred). On-disk bookkeeping schema goes to **v3** — a new `__localdb_snapshot_state` table, upgraded in place on open (the transition is a version stamp only: the shared DDL creates the table with `IF NOT EXISTS` on every open, and empty is the correct v3 state for a file that never baselined). **Wire-compatible with `0.1.x`/`0.2.x`, and `lib_schema_version` deliberately STAYS `1`** because `ValidateHandshake` compares it fail-closed for equality — bumping it would stop mixed pairs syncing outright, so the extension is negotiated by **capability**: `Handshake.supports_partial_snapshot` (new field 5) and `SnapshotBegin.tables` (new field 2, where **empty == all tables**, which is exactly what a pre-`0.3.0` sender emits and a pre-`0.3.0` receiver assumes). `RegisterReplicated(table, baselineExistingRows: true)` now flags **only that table**, so a snapshot streams only the flagged tables; the database-wide `__localdb_peer_state.needs_snapshot` survives unchanged for the cases that genuinely reach every table (oplog cap prune, impossible peer ack). **A peer that does not advertise the capability is served a FULL snapshot** — a scoped one would make it advance `last_applied_remote_seq` to the snapshot's as-of and silently discard every delta below it for the tables left out. For the same reason **a partial snapshot moves no watermark on either side**: the receiver keeps `last_applied_remote_seq` and the sender keeps `_sentThruSeq`, so the overlapping deltas are still pumped and merged by LWW. Crash safety: snapshot debt now clears on the receiver's new **`SnapshotAck`** (exchanged only between `0.3.0` peers) rather than on enqueue, so an ungraceful exit mid-snapshot re-streams the same scope on reconnect; against a pre-`0.3.0` peer the `0.2.x` clear-on-enqueue is kept, there being no ack to wait for. Composes with `0.2.1`'s bounded 64-message inbox and out-of-handshake snapshot (the ack rides the unbounded CONTROL lane, like `DeltaAck`). 199/199 tests, including a single-table-baseline scope test, a mixed-version interop test (old peer ⇒ full snapshot), an ack-gated-debt test and a v2→v3 upgrade test. **ScadaBridge pins `0.3.0`**; `Host/SiteLocalDbSetup.cs` needed no restructuring — it already calls `RegisterReplicated` once per table, which is what per-table flagging keys off, so adding an eleventh replicated table to an already-replicating site now streams that one table instead of all eleven. **`0.2.1` (2026-08-14) closes two adversarial-review findings against `0.2.0`** (same branch `localdb-perf-remediation`; wire-compatible, no proto/`lib_schema_version`/on-disk-schema change): **`DeregisterReplicated` now flushes the HLC's durable crash anchor inside its own transaction, before the prune** — clock recovery is `MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc)` and `hlc_high_water` was written only by `Dispose`, so on a file whose tables had all been deregistered an ungraceful exit left the clock nothing to recover from and it could re-issue stamps below HLCs peers already held (LWW then discards them as older); and **the gRPC sync inbox is bounded at 64 messages with wait-mode backpressure** instead of unbounded, capping inbound memory at ~`64 × MaxBatchBytes` rather than "whatever the peer sends" — the bad case being a **mutual full snapshot** (the snapshot flag is per-DATABASE and both sides of a late-enabled pair baseline unconditionally), previously buffered whole in memory on both nodes at once. Bounding required moving the snapshot a node owes **out of the handshake** to run alongside a live receive loop; the old ordering left neither side draining its inbox while sending and deadlocks under a bounded transport (pinned by a capacity-1 mutual-snapshot test). Per-table `needs_snapshot` was considered and **deferred** at `0.2.1` — it needs an on-disk schema change; the cost was instead documented (trim high-volume tables like event logs before enabling replication late). **That residual is closed by `0.3.0`, above.** 188/188 tests at `0.2.1`. **`0.2.0` (2026-08-14) is the performance + lifecycle release from the ScadaBridge architectural review (WP3.3, branch `localdb-perf-remediation`)** — wire-compatible with `0.1.x` peers (no proto change, no `lib_schema_version` bump); the on-disk bookkeeping schema goes to **v2**, upgraded in place on open. Six changes: **`DeregisterReplicated(table)`** drops a table's capture triggers and prunes its oplog + row-version rows (idempotent, and deliberately usable on triggers left by an EARLIER build — there was previously no way to stop capturing once a file had been registered); **`RegisterReplicated(table, baselineExistingRows: true)`** closes the late-opt-in hole, seeding the row-version ledger for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flagging a snapshot resync, because the snapshot streamer pages from that ledger and CDC never wrote pre-registration rows there — a node that started replicating LATE never sent them, silently and forever (the two APIs are a designed pair: the ledger prune is safe only because re-registration can re-baseline; deregistration must still be **symmetric**, which the fail-closed registered-table digest handshake enforces loudly); **byte-budget batching** via `MaxBatchBytes` (default 2 MB) with `MaxBatchSize` demoted to a secondary row cap, plus inbound delta/ack coalescing; **O(1) backlog depth** (`MAX(seq) − MAX(last_acked_seq, MIN(seq) − 1)`) replacing a `COUNT(*)` over the whole backlog that ran per maintenance tick AND per metrics scrape; `LwwApplier` prepare-once-per-statement; and the unused `__localdb_oplog_hlc` index dropped. **`0.1.2` (2026-07-21) fixes snapshot resync for a WIPED peer**: a converged pair prunes every oplog row on ack, and the handshake read an empty oplog as "no gap possible" — so the steady state of a healthy pair was the one state from which a rebuilt node (lost volume, re-imaged host) could never be back-filled; it came back empty and stayed empty until the next deploy. The gap is now measured against `last_acked_seq` when the oplog is empty. Found by the OtOpcUa Phase 1 live gate (check 4) as a documented limitation, then closed. **`0.1.3` (2026-07-21) closes the other half, which only became REACHABLE once back-fill worked**: the rebuilt peer's OWN writes were silently dropped. `last_applied_remote_seq` is a watermark in the *peer's* seq space and a rebuilt peer numbers from 1, so the healthy node's stale watermark made the sending side skip its whole oplog — the pair looked converged right up until the moment only the rebuilt node applied a deploy. Fixed by resetting the inbound watermark on a peer-node-id change (NOT last_acked_seq — that is what 0.1.2 keys off) plus rejecting a peer claim beyond anything we ever produced. Live-gated on the OtOpcUa docker-dev rig: `OtOpcUa/docs/plans/2026-07-21-localdb-followups-live-gate.md`. **`0.1.1` fixes a latent boot defect**: the library never created the parent directory of `LocalDb:Path` and `SqliteLocalDb` opens the file eagerly, so a missing directory was a hard boot failure (`SQLite Error 14`) — routine given the relative `./data/…` defaults, and masked on docker only by the volume mount. ScadaBridge had worked around it app-side; the fix moved into the library and the shim is deleted. **Phase 1** (merged in PR #23): `OperationTracking` + `site_events` consolidated into one `LocalDb:Path` database, fail-closed bearer interceptor on the sync endpoint, `ISyncStatus` health bridge, one-time legacy migrator. **Phase 2** (merged in the same PR): adds `sf_messages` + 7 site config tables (10 replicated in total) and **deletes** ScadaBridge's bespoke `SiteReplicationActor`, its messages, StoreAndForward's `ReplicationService`, and `ReplaceAllAsync` — including the whole **notify-and-fetch** exchange, so a standby now makes NO config fetch on deploy (`SiteReconciliationActor`'s startup fetch survives; different path). `notification_lists`/`smtp_configurations` are created but deliberately NEVER registered — permanently empty, and a channel for them would carry plaintext SMTP passwords. **Live gate PASS, all 10 checks** (`ScadaBridge/docs/plans/2026-07-19-localdb-phase2-live-gate.md`): byte-identical config rows with identical HLC + origin node id, standby zero-fetch, 3-table cascade delete converging with tombstones on both nodes, site failover with zero duplicate ids, clean both-nodes-together restart. **Replication remains default-OFF**, enabled on the docker rig's site-a pair only (site-b/site-c unreplicated as the default-OFF pin); **no production deployment has it enabled.** **Operational constraints:** a site pair must be stopped/started TOGETHER (rolling upgrade unsupported since the `SfBufferSnapshot` compat handler was deleted), a node offline beyond `TombstoneRetention` (7 d) can resurrect deleted rows, and — before `0.2.0` — `MaxBatchSize` batched by ROW COUNT against a 4 MB gRPC cap, which is why the rig pinned it to **16** (the 500 default would allow ~35 MB). **That pin is retired**: ScadaBridge consumed `0.2.0` on 2026-08-14 (branch `arch-review-remediation`), removed `MaxBatchSize` from both `docker/site-a` node configs and left `MaxBatchBytes` at its 2 MB default. The same consumption made `Host/SiteLocalDbSetup.cs` self-healing in both directions: an unreplicated node (site-b/site-c) now **deregisters** all ten tables at boot, dropping stale triggers an earlier build installed, and a replicating node registers with **`baselineExistingRows: true`**, so turning replication ON for a site that had been running without it finally converges on the rows already in the file rather than only on writes made after the restart. **Also adopted by OtOpcUa — Phase 1 COMPLETE and MERGED** to `master` 2026-07-21 (merge `c957db52`), and **Phase 2 (alarm store-and-forward on LocalDb) EXECUTED, live-gated and MERGED** to `master` (merge `06b5d81b`; the earlier "designed but NOT executed" claim was stale — corrected 2026-08-07; residual: live-gate checks 3+4 were never re-run on the post-mesh-Phase-6 topology): a pair-local consolidated SQLite cache of the deployed-configuration artifact (chunked, SHA-256-verified, newest-2 retention) that lets a driver node **boot from cache when central SQL Server is unreachable**, registered in the `hasDriver` branch only (admin-only nodes never register it), with the same default-OFF fail-closed replication. Its live gate passed 8/8 and caught four defects offline tests missed. **Consumers: ScadaBridge (Phases 1+2) and OtOpcUa (Phases 1+2)**, **ScadaBridge pins `0.3.0`** (2026-08-15, residual R1; it consumed `0.2.0` on 2026-08-14 under arch-review WP3.3) and **OtOpcUa pins `0.2.1`** (2026-08-15, residual R4, merged to its `master` @ `5a7aae14` — a pure pin bump, no code change; it also corrected its `MaxBatchSize = 16` rationale, kept at 16 as the DB read-page bound). The remaining 0.2.1-vs-0.3.0 skew is supported: every release is wire-compatible (`0.3.0` negotiates its scoped snapshots by capability and falls back to a full snapshot for an older peer) and the schema v2/v3 upgrades are local-file state. **OtOpcUa closed the ScadaBridge deferred-register row 39 gap on 2026-08-18** (branch `perf/followups-r3`, commit `3e62f161`): `LocalDbSetup.OnReady` no longer registers all four tables unconditionally — registration follows one predicate, `LocalDb:Replication:PeerAddress` **OR** `:ApiKey` non-blank (the OR because only the dialling half of a pair sets `PeerAddress`; `SyncListenPort` is deliberately excluded, a port with no key being fail-closed). A configured node registers with **`baselineExistingRows: true`**, so a late replication enable finally converges on the rows already in the file; an unconfigured node **deregisters** at boot, dropping capture triggers an earlier build installed and logging once only when it actually cleaned something. Same shape ScadaBridge landed under arch-review WP3.3, written against OtOpcUa's own seams. ⚠️ OtOpcUa stays on `0.2.1`, where the snapshot flag is per-DATABASE, so baselining there streams every registered table in full — trim a long `alarm_sf_events` backlog before enabling replication late. Flipping the predicate is a both-nodes-together change (fail-closed registered-table digest handshake). | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) + [`docs/plans/2026-07-19-scadabridge-localdb-design.md`](docs/plans/2026-07-19-scadabridge-localdb-design.md) + [`docs/plans/2026-07-20-otopcua-localdb-design.md`](docs/plans/2026-07-20-otopcua-localdb-design.md) (both adoption designs live here); ScadaBridge holds the Phase 1 plan + Phase 2 gate under its own `docs/plans/`, OtOpcUa its two phase plans + two live gates | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) |