scadaproj/ZB.MOM.WW.LocalDb/CLAUDE.md

ZB.MOM.WW.LocalDb

Shared embedded SQLite cache + 2-node LWW sync library for the ZB.MOM.WW SCADA family (3 packages; consumed by ScadaBridge and OtOpcUa). Build/test/pack from this directory: dotnet build ZB.MOM.WW.LocalDb.slnx, dotnet test ZB.MOM.WW.LocalDb.slnx (fully offline), dotnet pack ZB.MOM.WW.LocalDb.slnx -c Release -o artifacts. Design: ../docs/plans/2026-07-17-localdb-design.md; adoption designs ../docs/plans/2026-07-19-scadabridge-localdb-design.md + ../docs/plans/2026-07-20-otopcua-localdb-design.md.

Backlog / open work: lives in Gitea issues on dohertj2/scadaproj (tea issues list --repo dohertj2/scadaproj) — #16 (resurrection past TombstoneRetention) was fixed in 0.3.7. The history below is a record, not a status tracker; file new follow-ups as issues.

0.3.5 (unpublished, scadaproj#26): Microsoft.Data.Sqlite 10.0.7 -> 10.0.13 (efcore#39009 pool fault, which a LocalDb consumer is exposed to); SQLitePCLRaw.lib.e_sqlite3 transitive pin 2.1.12 -> 2.1.13 (10.0.13 floors 2.1.13). 216/216; no wire/schema change.

0.3.6 (unpublished, scadaproj#36–#39, #46, #50): code-review fixes, all wire-compatible (no proto, lib_schema_version or on-disk schema change).

• #36 LocalDb-001: the passive LocalDbSyncService now enforces LocalDb:Replication:ApiKey itself (Bearer, constant-time, checked before the handshake and before newest-wins takeover; Unauthenticated otherwise). Blank key = unchanged behaviour plus a startup warning. ScadaBridge's and OtOpcUa's own LocalDbSyncAuthInterceptors use the same key and header, so they keep working; they are now redundant except for their fail-closed-when-blank rule.
• #37 LocalDb-002: snapshots deliver FK-linked rows in any order. SnapshotStreamer sends referenced tables first (Kahn order, name tie-break); SnapshotApplier defers rows that fail only SQLITE_CONSTRAINT_FOREIGNKEY (LwwApplier.ApplyBatchAsync(..., deferForeignKeyViolations: true)), retries them as rows land, and dead-letters true orphans at SnapshotComplete. Before, a child that arrived first was dead-lettered and acked over permanently.
• #38 LocalDb-003: the applier upserts with INSERT … ON CONFLICT(<pk>) DO UPDATE (DO NOTHING when every column is PK) instead of INSERT OR REPLACE. A secondary-UNIQUE clash is now a dead-letter instead of a silent delete of another row; as a side effect a remote update of a parent with ON DELETE RESTRICT children applies instead of being dead-lettered.
• #39 LocalDb-004: SnapshotApplier now takes ReplicationOptions (wired by SyncSessionFactory) and runs the delta path's drift guard per snapshot row (shared HlcDriftGuard dead-letter); a dead-lettered row's HLC is kept out of last_seen_hlc. Before, FailClosedOnDrift was skipped on snapshots and one future-stamped row moved the receiver's HLC ahead across restarts.
• #46 LocalDb-005: RegisterReplicated refuses an AUTOINCREMENT primary key (LocalDbRegistrationException, detected from SQLite column metadata). Each node assigns its own, so two different rows shared a pk_json and LWW kept one on both nodes. Plain INTEGER PRIMARY KEY is still accepted (app-assigned ids); replicated keys must be unique across both nodes — documented. ScadaBridge's 12 and OtOpcUa's 5 replicated tables are all TEXT/composite keys (checked 2026-11-09), so neither boot is affected. This is the only 0.3.6 change that can refuse a schema an earlier version accepted.
• #46 LocalDb-006: apply/snapshot JSON paths double ' (shared JsonPathSql.For); a column named it's used to dead-letter every row of its table on the peer. LocalDb-007: README refreshed (version, OtOpcUa Phases 1+2, EXTRA, per-table baseline wording, test count).
• #50: snapshot-path dead-letters (poison, drift, FK orphan, cap spill) now count in localdb.sync.dead_lettered (SnapshotApplier takes LocalDbMetrics, wired by SyncSessionFactory). New option LocalDb:Replication:MaxSnapshotDeferredRows (default 100,000, validated > 0) caps snapshot rows held for a missing FK parent; past it the oldest are re-applied once and dead-lettered if still orphaned. Open note (cross-repo, #50): once ScadaBridge and OtOpcUa pin 0.3.6, their own LocalDbSyncAuthInterceptors duplicate the library check — they differ only in refusing every stream when no key is set. Each app decides whether to keep that stricter rule or delete its interceptor; nothing in this library changes for either choice. 249/249 tests.

0.3.7 (unpublished, scadaproj#16 — owner chose option A): a tombstone is pruned only when it is past TombstoneRetention and the peer holds it. OplogStore.PruneTombstonesAsync holds every tombstone while database-wide (__localdb_peer_state.needs_snapshot) or per-table (__localdb_snapshot_state) snapshot debt is outstanding, and a LOCAL tombstone while its delete is still unacked (hlc >= the first unacked oplog row's HLC — ledger and oplog share the HLC, local HLCs rise with seq). Delivery restamps tombstone_utc (in RecordPeerAckAsync for tombstones in the newly acked range; in ClearSnapshotDebtAsync for the snapshot's scope), so retention runs from delivery and the returning peer's stale queued writes get a full retention to drain. Remote-origin tombstones age on retention alone. New optional LocalDb:Replication:TombstoneHardCap (default null = off, validated ≥ TombstoneRetention) bounds an undelivered tombstone's life — past it the old risk returns. The snapshot path (issue's second note) is covered by the debt hold: the cap prune sets needs_snapshot in the same transaction that drops undelivered oplog rows, so the ledger tombstone the snapshot is built from survives until that snapshot is acknowledged. Cost: ledger growth for the length of an outage. Not covered: a peer restored from an old backup of its own file. No proto, lib_schema_version or on-disk schema change. Regression test TombstoneResurrectionTests.PeerOfflinePastRetention_DeleteStillWinsOnBothNodes (fails on 0.3.6 with the row resurrected, A=1) + positive control without the prune + pruned-after-delivery control; 5 OplogStoreTests with a controlled clock; every guard clause mutation-killed. 259/259.

Index history (migrated from the scadaproj root CLAUDE.md)

Moved verbatim out of the scadaproj root CLAUDE.md on 2026-09-03 so it loads only when needed. This file is now the authoritative place for this history — append new entries here and keep the root index row short.

Component-table row as it read in the index

Component	Status	Goal	Design	Implementation
LocalDb (embedded cache + 2-node sync)	Adopted by ScadaBridge — Phases 1 AND 2 COMPLETE and MERGED to main (2026-07-20, PR #23, merge 28ca04d7, pushed to origin; both feat/localdb-phase* branches fully contained and kept only as history). Libs 0.3.4 on the Gitea feed (3 pkgs; 0.1.0–0.3.3 also present). 0.3.4 (2026-08-20, scadaproj 75c1897 — ScadaBridge register row 106; sender-side, wire-compatible, no schema change): SnapshotStreamer stamps as_of_seq = max(MAX(seq), last_acked_seq) instead of bare MAX(seq), which read 0 once ack-pruning had emptied a converged sender's oplog — so a rebuilt peer (watermark 0) was healed by the snapshot but never lifted by it, and the 0.1.2 empty-oplog horizon rule re-streamed the whole database on EVERY reconnect until the sender wrote again (ScadaBridge's site-a-a: 127 k rows per restart, observed on two consecutive restarts). Pinned by WipedPeer_SnapshotLiftsItsWatermark_…; 216/216. ScadaBridge pins 0.3.4 (2026-08-20; live: first restart sent one last snapshot as_of 149, second restart sent none); OtOpcUa remained 0.2.1 (until its 2026-09-03 re-pin to 0.3.4) — note OtOpcUa's rebuilt-node path has the same loop until it re-pins. 0.3.3 (2026-08-20, scadaproj 122940c — ScadaBridge arch-review P4-7; wire-compatible, no proto/schema change) closes two latent LwwApplier hazards: (1) RegisterReplicated refuses a table that is the parent of a row-mutating ON DELETE FK (CASCADE/SET NULL/SET DEFAULT; RESTRICT/NO ACTION allowed) via LocalDbRegistrationException naming child.column ON DELETE <action> — the applier's INSERT OR REPLACE runs with foreign_keys=ON, so the implicit delete fired child actions while capture was suppressed, a silent divergence for any future FK'd table (none of ScadaBridge's ten or OtOpcUa's four has an FK today; registration-time check only, since SQLite can't add an FK to an existing table); (2) the per-entry apply (delete-or-upsert + ledger write) is atomic under a SAVEPOINT — a fault between the two statements left a deleted row with a live ledger that SnapshotStreamer warns-and-skips forever; poison-catch semantics preserved. 213/213 tests (+9, failure injected with a RAISE(ABORT) trigger on the ledger, no production seam). ScadaBridge pins 0.3.3 (2026-08-20); OtOpcUa remained 0.2.1 (until its 2026-09-03 re-pin to 0.3.4). 0.3.2 (2026-08-19, scadaproj f62b017 — ScadaBridge arch-review P4-1 + P4-3; wire-compatible, no proto/schema change): (1) the sync pump faults a live session when a cap-prune invalidates its position (oplog min-seq horizon check before each read batch + on the batch head; new public LocalDbOplogGapException rides the 0.3.1 fault teardown) — previously a slow-but-alive peer silently streamed past the pruned gap and the pair diverged until the next reconnect; (2) client h2 keepalive pings via new validated Replication:KeepAlivePingDelay/KeepAlivePingTimeout (60s/20s, Always policy); (3) the server single-stream guard is newest-wins with an awaited handover (TakeoverTimeout 10s) — a fresh authenticated Sync displaces a dead incumbent instead of being refused AlreadyExists forever by a half-open link whose zombie side reports connected. 204/204 tests. ScadaBridge pins 0.3.2 (2026-08-19, adding Kestrel h2 keepalive on its site gRPC listener as the server half); OtOpcUa remained 0.2.1 (until its 2026-09-03 re-pin to 0.3.4). 0.3.1 (2026-08-18, scadaproj 3e5a5fd) flushes a faulting session's control lane before its writer is cancelled — on a handshake-phase fault (i.e. a schema-mismatch rejection) RunAsync's teardown cancelled the writer while the session's OWN Handshake could still be sitting in the lane (the enqueue→receive→validate→fault path has no yields while the writer's wakeup waits in the thread-pool queue; a pinning test lost the race 39/40), and mismatch detection is symmetric — the peer computes its fail-closed verdict FROM that Handshake — so the healthy peer misread a permanent schema mismatch as a generic disconnect and reconnect-looped at the fast backoff instead of the classified FailedPrecondition→LogError+max-backoff path. Teardown now completes both lanes so the writer drains what is already enqueued (2 s deferred cancel as the wedged-transport backstop); caller-initiated shutdown is not delayed, no wire or config change — an older peer simply receives the Handshake it was always owed. Closes ScadaBridge deferred-register row 41 (its flaky Handshake_DigestMismatch_FailsClosed reproduced 2/6 pre-fix under load, 6/6 green post-fix; the other named flake, BatchBudgetTests.Pump_RowLargerThanTheBudget…, was a harness defect — session faults hidden as opaque 15 s timeouts — fixed with a run-observing NextAsync). ScadaBridge re-pinned through 0.3.1 to 0.3.2 (2026-08-19); OtOpcUa remained 0.2.1 (until its 2026-09-03 re-pin to 0.3.4). 0.3.0 (2026-08-15) makes the snapshot-resync flag PER-TABLE (branch localdb-per-table-snapshot; the residual 0.2.1 explicitly deferred). On-disk bookkeeping schema goes to v3 — a new __localdb_snapshot_state table, upgraded in place on open (the transition is a version stamp only: the shared DDL creates the table with IF NOT EXISTS on every open, and empty is the correct v3 state for a file that never baselined). Wire-compatible with 0.1.x/0.2.x, and lib_schema_version deliberately STAYS 1 because ValidateHandshake compares it fail-closed for equality — bumping it would stop mixed pairs syncing outright, so the extension is negotiated by capability: Handshake.supports_partial_snapshot (new field 5) and SnapshotBegin.tables (new field 2, where empty == all tables, which is exactly what a pre-0.3.0 sender emits and a pre-0.3.0 receiver assumes). RegisterReplicated(table, baselineExistingRows: true) now flags only that table, so a snapshot streams only the flagged tables; the database-wide __localdb_peer_state.needs_snapshot survives unchanged for the cases that genuinely reach every table (oplog cap prune, impossible peer ack). A peer that does not advertise the capability is served a FULL snapshot — a scoped one would make it advance last_applied_remote_seq to the snapshot's as-of and silently discard every delta below it for the tables left out. For the same reason a partial snapshot moves no watermark on either side: the receiver keeps last_applied_remote_seq and the sender keeps _sentThruSeq, so the overlapping deltas are still pumped and merged by LWW. Crash safety: snapshot debt now clears on the receiver's new SnapshotAck (exchanged only between 0.3.0 peers) rather than on enqueue, so an ungraceful exit mid-snapshot re-streams the same scope on reconnect; against a pre-0.3.0 peer the 0.2.x clear-on-enqueue is kept, there being no ack to wait for. Composes with 0.2.1's bounded 64-message inbox and out-of-handshake snapshot (the ack rides the unbounded CONTROL lane, like DeltaAck). 199/199 tests, including a single-table-baseline scope test, a mixed-version interop test (old peer ⇒ full snapshot), an ack-gated-debt test and a v2→v3 upgrade test. ScadaBridge pins 0.3.0; Host/SiteLocalDbSetup.cs needed no restructuring — it already calls RegisterReplicated once per table, which is what per-table flagging keys off, so adding an eleventh replicated table to an already-replicating site now streams that one table instead of all eleven. 0.2.1 (2026-08-14) closes two adversarial-review findings against 0.2.0 (same branch localdb-perf-remediation; wire-compatible, no proto/lib_schema_version/on-disk-schema change): DeregisterReplicated now flushes the HLC's durable crash anchor inside its own transaction, before the prune — clock recovery is MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc) and hlc_high_water was written only by Dispose, so on a file whose tables had all been deregistered an ungraceful exit left the clock nothing to recover from and it could re-issue stamps below HLCs peers already held (LWW then discards them as older); and the gRPC sync inbox is bounded at 64 messages with wait-mode backpressure instead of unbounded, capping inbound memory at ~64 × MaxBatchBytes rather than "whatever the peer sends" — the bad case being a mutual full snapshot (the snapshot flag is per-DATABASE and both sides of a late-enabled pair baseline unconditionally), previously buffered whole in memory on both nodes at once. Bounding required moving the snapshot a node owes out of the handshake to run alongside a live receive loop; the old ordering left neither side draining its inbox while sending and deadlocks under a bounded transport (pinned by a capacity-1 mutual-snapshot test). Per-table needs_snapshot was considered and deferred at 0.2.1 — it needs an on-disk schema change; the cost was instead documented (trim high-volume tables like event logs before enabling replication late). That residual is closed by 0.3.0, above. 188/188 tests at 0.2.1. 0.2.0 (2026-08-14) is the performance + lifecycle release from the ScadaBridge architectural review (WP3.3, branch localdb-perf-remediation) — wire-compatible with 0.1.x peers (no proto change, no lib_schema_version bump); the on-disk bookkeeping schema goes to v2, upgraded in place on open. Six changes: DeregisterReplicated(table) drops a table's capture triggers and prunes its oplog + row-version rows (idempotent, and deliberately usable on triggers left by an EARLIER build — there was previously no way to stop capturing once a file had been registered); RegisterReplicated(table, baselineExistingRows: true) closes the late-opt-in hole, seeding the row-version ledger for pre-existing rows at the LWW floor (HLC 0, this node's id) and flagging a snapshot resync, because the snapshot streamer pages from that ledger and CDC never wrote pre-registration rows there — a node that started replicating LATE never sent them, silently and forever (the two APIs are a designed pair: the ledger prune is safe only because re-registration can re-baseline; deregistration must still be symmetric, which the fail-closed registered-table digest handshake enforces loudly); byte-budget batching via MaxBatchBytes (default 2 MB) with MaxBatchSize demoted to a secondary row cap, plus inbound delta/ack coalescing; O(1) backlog depth (MAX(seq) − MAX(last_acked_seq, MIN(seq) − 1)) replacing a COUNT(*) over the whole backlog that ran per maintenance tick AND per metrics scrape; LwwApplier prepare-once-per-statement; and the unused __localdb_oplog_hlc index dropped. 0.1.2 (2026-07-21) fixes snapshot resync for a WIPED peer: a converged pair prunes every oplog row on ack, and the handshake read an empty oplog as "no gap possible" — so the steady state of a healthy pair was the one state from which a rebuilt node (lost volume, re-imaged host) could never be back-filled; it came back empty and stayed empty until the next deploy. The gap is now measured against last_acked_seq when the oplog is empty. Found by the OtOpcUa Phase 1 live gate (check 4) as a documented limitation, then closed. 0.1.3 (2026-07-21) closes the other half, which only became REACHABLE once back-fill worked: the rebuilt peer's OWN writes were silently dropped. last_applied_remote_seq is a watermark in the peer's seq space and a rebuilt peer numbers from 1, so the healthy node's stale watermark made the sending side skip its whole oplog — the pair looked converged right up until the moment only the rebuilt node applied a deploy. Fixed by resetting the inbound watermark on a peer-node-id change (NOT last_acked_seq — that is what 0.1.2 keys off) plus rejecting a peer claim beyond anything we ever produced. Live-gated on the OtOpcUa docker-dev rig: OtOpcUa/docs/plans/2026-07-21-localdb-followups-live-gate.md. 0.1.1 fixes a latent boot defect: the library never created the parent directory of LocalDb:Path and SqliteLocalDb opens the file eagerly, so a missing directory was a hard boot failure (SQLite Error 14) — routine given the relative ./data/… defaults, and masked on docker only by the volume mount. ScadaBridge had worked around it app-side; the fix moved into the library and the shim is deleted. Phase 1 (merged in PR #23): OperationTracking + site_events consolidated into one LocalDb:Path database, fail-closed bearer interceptor on the sync endpoint, ISyncStatus health bridge, one-time legacy migrator. Phase 2 (merged in the same PR): adds sf_messages + 7 site config tables (10 replicated in total) and deletes ScadaBridge's bespoke SiteReplicationActor, its messages, StoreAndForward's ReplicationService, and ReplaceAllAsync — including the whole notify-and-fetch exchange, so a standby now makes NO config fetch on deploy (SiteReconciliationActor's startup fetch survives; different path). notification_lists/smtp_configurations are created but deliberately NEVER registered — permanently empty, and a channel for them would carry plaintext SMTP passwords. Live gate PASS, all 10 checks (ScadaBridge/docs/plans/2026-07-19-localdb-phase2-live-gate.md): byte-identical config rows with identical HLC + origin node id, standby zero-fetch, 3-table cascade delete converging with tombstones on both nodes, site failover with zero duplicate ids, clean both-nodes-together restart. Replication remains default-OFF, enabled on the docker rig's site-a pair only (site-b/site-c unreplicated as the default-OFF pin); no production deployment has it enabled. Operational constraints: a site pair must be stopped/started TOGETHER (rolling upgrade unsupported since the SfBufferSnapshot compat handler was deleted), a node offline beyond TombstoneRetention (7 d) can resurrect deleted rows (→ #16), and — before 0.2.0 — MaxBatchSize batched by ROW COUNT against a 4 MB gRPC cap, which is why the rig pinned it to 16 (the 500 default would allow ~35 MB). That pin is retired: ScadaBridge consumed 0.2.0 on 2026-08-14 (branch arch-review-remediation), removed MaxBatchSize from both docker/site-a node configs and left MaxBatchBytes at its 2 MB default. The same consumption made Host/SiteLocalDbSetup.cs self-healing in both directions: an unreplicated node (site-b/site-c) now deregisters all ten tables at boot, dropping stale triggers an earlier build installed, and a replicating node registers with baselineExistingRows: true, so turning replication ON for a site that had been running without it finally converges on the rows already in the file rather than only on writes made after the restart. Also adopted by OtOpcUa — Phase 1 COMPLETE and MERGED to master 2026-07-21 (merge c957db52), and Phase 2 (alarm store-and-forward on LocalDb) EXECUTED, live-gated and MERGED to master (merge 06b5d81b; the earlier "designed but NOT executed" claim was stale — corrected 2026-08-07; residual: live-gate checks 3+4 were never re-run on the post-mesh-Phase-6 topology): a pair-local consolidated SQLite cache of the deployed-configuration artifact (chunked, SHA-256-verified, newest-2 retention) that lets a driver node boot from cache when central SQL Server is unreachable, registered in the hasDriver branch only (admin-only nodes never register it), with the same default-OFF fail-closed replication. Its live gate passed 8/8 and caught four defects offline tests missed. Consumers: ScadaBridge (Phases 1+2) and OtOpcUa (Phases 1+2) — both pin 0.3.4 as of 2026-09-03 (ScadaBridge since 2026-08-20; OtOpcUa b8b4c18f, a pure pin bump with tests unchanged versus its baseline). Earlier: ScadaBridge pinned 0.3.0 (2026-08-15, residual R1; it consumed 0.2.0 on 2026-08-14 under arch-review WP3.3) and OtOpcUa pinned 0.2.1 (2026-08-15, residual R4, merged to its master @ 5a7aae14 — a pure pin bump, no code change; it also corrected its MaxBatchSize = 16 rationale, kept at 16 as the DB read-page bound). The remaining 0.2.1-vs-0.3.0 skew is supported: every release is wire-compatible (0.3.0 negotiates its scoped snapshots by capability and falls back to a full snapshot for an older peer) and the schema v2/v3 upgrades are local-file state. OtOpcUa closed the ScadaBridge deferred-register row 39 gap on 2026-08-18 (branch perf/followups-r3, commit 3e62f161): LocalDbSetup.OnReady no longer registers all four tables unconditionally — registration follows one predicate, LocalDb:Replication:PeerAddress OR :ApiKey non-blank (the OR because only the dialling half of a pair sets PeerAddress; SyncListenPort is deliberately excluded, a port with no key being fail-closed). A configured node registers with baselineExistingRows: true, so a late replication enable finally converges on the rows already in the file; an unconfigured node deregisters at boot, dropping capture triggers an earlier build installed and logging once only when it actually cleaned something. Same shape ScadaBridge landed under arch-review WP3.3, written against OtOpcUa's own seams. OtOpcUa re-pinned to 0.3.4 on 2026-09-03 (b8b4c18f), so its snapshot flag is per-table too and the earlier per-DATABASE caveat (a late baseline streamed every registered table in full) no longer applies. Flipping the predicate is a both-nodes-together change (fail-closed registered-table digest handshake).	Shared ZB.MOM.WW.LocalDb lib	docs/plans/2026-07-17-localdb-design.md + docs/plans/2026-07-19-scadabridge-localdb-design.md + docs/plans/2026-07-20-otopcua-localdb-design.md (both adoption designs live here); ScadaBridge holds the Phase 1 plan + Phase 2 gate under its own docs/plans/, OtOpcUa its two phase plans + two live gates	ZB.MOM.WW.LocalDb/