deps(localdb)+docs: consume LocalDb 0.3.4 — snapshot as_of_seq floors at last_acked_seq, ending the full-snapshot-on-every-reconnect loop (register row 106 closed, live-proven on the rig: second pair restart exchanged zero snapshots)

CLAUDE.md

@@ -93,7 +93,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
     - **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all ten tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
     - **LocalDb 0.3.0 (2026-08-15) makes that snapshot flag PER-TABLE** — the residual 0.2.1 explicitly deferred. On-disk bookkeeping schema goes to **v3** (a new `__localdb_snapshot_state` table, upgraded in place on open); still wire-compatible, and `lib_schema_version` deliberately stays `1` because the handshake compares it fail-closed for *equality*. The extension is negotiated by **capability** (`Handshake.supports_partial_snapshot`, `SnapshotBegin.tables` where **empty == all tables**), so a peer that does not advertise it is served a full snapshot — a scoped one would make it advance `last_applied_remote_seq` past deltas it never received. For the same reason **a partial snapshot moves no watermark on either side**; the overlapping deltas are still pumped and absorbed by LWW. Snapshot debt now clears on the receiver's new `SnapshotAck` rather than on enqueue, so a crash mid-snapshot re-streams the same scope. **`SiteLocalDbSetup.cs` needed no restructuring** — it already calls `RegisterReplicated` once per table in a loop, which is exactly what per-table flagging keys off. The consequence to know: **adding an eleventh table to `ReplicatedTables` on an already-replicating site now snapshots that one table**, where through 0.2.x the same edit re-streamed all eleven in full in both directions. First-boot behaviour is unchanged (all ten seed at once ⇒ the flagged set is every registered table ⇒ an ordinary full snapshot), and so is upgrading the rig in place (the ten tables are already ledgered, nothing seeds, nothing is flagged). A future "register the whole list in one call" API would silently give the coarse behaviour back.
     - **LocalDb 0.3.2 (2026-08-19, ScadaBridge arch-review P4-1 + P4-3; wire-compatible, no schema change):** (1) the sync pump now **faults the live session when a cap-prune invalidates its position** (horizon check on the oplog's min seq before each read batch + on the batch head after it; throws the new public `LocalDbOplogGapException` through the 0.3.1 fault-teardown) — previously a slow-but-alive peer silently skipped the pruned gap and the pair diverged until the next reconnect; (2) **transport keepalives both sides** — client `SocketsHttpHandler` pings via new validated `LocalDb:Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (defaults 60s/20s, `Always` policy), and ScadaBridge's site gRPC listener sets Kestrel h2 keepalive (`Host/SiteKestrelSetup.cs` — server-wide, also governs the metrics port); (3) the server's single-stream guard is now **newest-wins with an awaited handover** — a fresh authenticated `Sync` cancels a dead/wedged incumbent and *waits* for it to finish (`TakeoverTimeout` 10s) before starting, so a half-open link can no longer wedge the pair behind `AlreadyExists` while the zombie side reports `LocalDbReplicationConnected=true`.
-    - **LocalDb 0.3.3 (2026-08-20, ScadaBridge arch-review P4-7; wire-compatible, no schema change):** two latent `LwwApplier` hazards closed before any table could hit them. (1) **`RegisterReplicated` now REFUSES a table that is the parent of a foreign key with a row-mutating `ON DELETE` action (`CASCADE`/`SET NULL`/`SET DEFAULT`) from any other table** (`LocalDbRegistrationException` naming `child.column ON DELETE <action>`; `RESTRICT`/`NO ACTION` still allowed) — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired those actions while CDC capture was suppressed, i.e. child rows changed on one node and never replicated. Registration-time only: SQLite can't add an FK to an existing table, so a later-created child is caught at the next boot's registration. None of ScadaBridge's ten tables has an FK (verified on the live rig file: zero `REFERENCES` across 29 `CREATE TABLE`s) — **keep it that way; a cascading-FK parent can never be one of `ReplicatedTables`** (register row 105). (2) The per-entry apply (delete-or-upsert + row-version ledger write) is now **atomic under a `SAVEPOINT`** — a fault between the two statements used to leave a deleted row with a live ledger entry, which `SnapshotStreamer` warns-and-skips forever; poison-catch semantics unchanged (narrow `SqliteException`, per-entry dead-letter, batch continues). Cost: two extra in-memory statements per applied entry on a backlog drain. **Both are pinned here** (2026-08-20): `SiteLocalDbReplicatedTableFkGuardTests` asserts at build time that no `ReplicatedTables` entry has an inbound row-mutating FK (register row 105), and **pre-2026-08-20 site files must be VACUUMed once offline** to get `auto_vacuum=INCREMENTAL` — `docker/vacuum-site-localdb.sh` (pair stopped together; done on all three rig pairs; topology guide → *one-off offline VACUUM*; row 104). **Known LocalDb defect, register row 106 (fix = 0.3.4):** a converged pair whose sender oplog is ack-pruned to empty stamps snapshots `as_of_seq 0`, so a peer with a zero watermark (rebuilt node) gets a FULL snapshot on every reconnect until the sender writes again — the rig's site-a-a does this today (127 k rows per restart).
+    - **LocalDb 0.3.3 (2026-08-20, ScadaBridge arch-review P4-7; wire-compatible, no schema change — superseded by 0.3.4 below in the same bullet):** two latent `LwwApplier` hazards closed before any table could hit them. (1) **`RegisterReplicated` now REFUSES a table that is the parent of a foreign key with a row-mutating `ON DELETE` action (`CASCADE`/`SET NULL`/`SET DEFAULT`) from any other table** (`LocalDbRegistrationException` naming `child.column ON DELETE <action>`; `RESTRICT`/`NO ACTION` still allowed) — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired those actions while CDC capture was suppressed, i.e. child rows changed on one node and never replicated. Registration-time only: SQLite can't add an FK to an existing table, so a later-created child is caught at the next boot's registration. None of ScadaBridge's ten tables has an FK (verified on the live rig file: zero `REFERENCES` across 29 `CREATE TABLE`s) — **keep it that way; a cascading-FK parent can never be one of `ReplicatedTables`** (register row 105). (2) The per-entry apply (delete-or-upsert + row-version ledger write) is now **atomic under a `SAVEPOINT`** — a fault between the two statements used to leave a deleted row with a live ledger entry, which `SnapshotStreamer` warns-and-skips forever; poison-catch semantics unchanged (narrow `SqliteException`, per-entry dead-letter, batch continues). Cost: two extra in-memory statements per applied entry on a backlog drain. **Both are pinned here** (2026-08-20): `SiteLocalDbReplicatedTableFkGuardTests` asserts at build time that no `ReplicatedTables` entry has an inbound row-mutating FK (register row 105), and **pre-2026-08-20 site files must be VACUUMed once offline** to get `auto_vacuum=INCREMENTAL` — `docker/vacuum-site-localdb.sh` (pair stopped together; done on all three rig pairs; topology guide → *one-off offline VACUUM*; row 104). **LocalDb 0.3.4 (2026-08-20, register row 106; wire-compatible, sender-side only):** a converged pair whose sender oplog was ack-pruned to empty used to stamp snapshots `as_of_seq 0`, so a peer with a zero watermark (a rebuilt node — the rig's site-a-a since 2026-08-15) received a FULL snapshot on every reconnect until the sender wrote again (127 k rows per restart, observed). The as-of now floors at `last_acked_seq`; proven live — second restart after the pin exchanged zero snapshots. Lesson for rebuilding a node from its peer: the first reconnect snapshots, the second must not.
     - **LocalDb 0.2.1** (two adversarial-review findings against 0.2.0, wire-compatible, no schema change) fixes: (1) `DeregisterReplicated` was deleting the HLC clock's only durable crash anchor along with the table rows — an ungraceful exit (crash, SIGKILL, power loss) on a fully-deregistered file left nothing to recover from, so the reopened clock could restart from wall-clock time and re-issue stamps below HLCs peers already hold (silently discarded by LWW); the current clock is now flushed inside the deregistration transaction, before the prune. (2) The sync inbox was an **unbounded** channel, so inbound memory was a function of what the peer sends — a mutual full snapshot (both sides of a pair baseline unconditionally when replication is enabled late) buffered whole, in memory, on both nodes at once; it is now **bounded at 64 messages** with wait-mode backpressure (unrelated to, and not to be confused with, the rig's separate `MaxBatchSize: 64` DB-read-page pin above), which required moving the snapshot to run alongside the receive loop (not inside the pre-loop handshake) so the inbox is drained for the whole session and can't deadlock itself.
 - All timestamps are UTC throughout the system.
 - Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.

Directory.Packages.props

@@ -109,9 +109,9 @@
     <PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.6.2" />
     <PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.6.2" />
     <PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.Grpc" Version="0.6.2" />
-    <PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.3.3" />
-    <PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.3.3" />
-    <PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.3.3" />
+    <PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.3.4" />
+    <PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.3.4" />
+    <PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.3.4" />
   </ItemGroup>
 
   <!--

docs/plans/2026-07-08-deferred-work-register.md

@@ -98,7 +98,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
 | 103 | ~~**Two lesser TTL-bounded circuit-teardown-vs-async-setup races remain in CentralUI**~~ **RESOLVED 2026-08-20 (Low-wave T12, agent commit `c8452136`).** Both instances closed with the Medium-wave T10 take-once discipline: `TransportImport.razor.cs` re-checks disposal after `TryLoadAsync`'s await and releases the import slot through exactly-once ownership (dispose-during-start frees the `BundleSession` slot immediately instead of stranding it for the 30-min TTL), and `Topology.razor` creates its timer under the same post-await re-check so a disposed page leaves no live timer (asserted event-style, not by sleep). Original note: (arch-review 2026-08-19 P6-6's optional tail, deliberately deferred by the fix). The permanent debug-stream leak was closed at `DebugView.razor` + `DebugStreamHub` (dispose-vs-connect take-once pattern, commit `11e623c2`), but two same-family instances with a *different shape* were left: (1) `TransportImport.razor.cs TryLoadAsync` (~:244) — a circuit torn down mid-load strands 1 of the 8 `MaxConcurrentImportSessions` slots until the 30-min `BundleSession` TTL reaps it (partially defeats row 43); (2) `Topology.razor` (~:201-214) — timer created before a disposal re-check. Both are TTL/lifetime-bounded (no permanent leak), which is why they missed the Medium bar. Fix is the same post-await `_disposed` re-check + take-once teardown, but routed through `BundleSession.ReleaseSession`/the wizard-step lifecycle rather than a session-stop call. | Arch-review 2026-08-19 P6-6 remediation (T10 explicitly scoped them out) | Open. | Next CentralUI hardening pass, or opportunistically when either file is next touched. |
 | 104 | ~~**Existing site LocalDb files stay `auto_vacuum=NONE`; `PRAGMA incremental_vacuum` after a cap trim is a no-op on them**~~ **RESOLVED 2026-08-20 — recipe shipped and applied to the rig.** New `docker/vacuum-site-localdb.sh` (stop both nodes of a pair together → `PRAGMA auto_vacuum=INCREMENTAL; VACUUM` each volume with integrity checks before/after → start both; `DRY_RUN=1` reports) plus the topology-guide section *Site LocalDb files created before 2026-08-20 need a one-off offline VACUUM*. Run on all three rig pairs: all six files `auto_vacuum 0 → 2`, `integrity_check=ok`, ~11 % smaller (site-a 20.3 → 18.1 MB), site-a replication back `Connected=true`/backlog 0 with no watermark or schema change — the VACUUM was replication-invisible as predicted (ledger keys by declared PK; oplog `seq` is an `INTEGER PRIMARY KEY AUTOINCREMENT`). The same maintenance surfaced the unrelated row 106. Original: | Arch-review 2026-08-19 P4-6 (Low-wave T4, `47f03fe6`) | `auto_vacuum` is only honoured on a table-less database, and `LocalDbSchema.EnsureCreated` had always created LocalDb's bookkeeping tables before `SiteEventLogSchema.Apply` ran — so the pragma had never once taken effect on any real file. The fix (`SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile`, run before `AddZbLocalDb`'s lazy singleton constructs `SqliteLocalDb`) only helps **fresh** files; an unconditional `VACUUM` against a live ~1 GB replicated file was explicitly rejected (long exclusive lock on the active node; the rewrite is CDC-invisible but blocks the sync pump). Every node on the docker rig today predates the fix, so freed `site_events` pages there are reused, not returned to the OS — the storage cap (row 81) still bounds the data volume, only the file size stops shrinking. | Operator-scheduled `VACUUM` in a maintenance window (both nodes of a pair stopped together), or when a site's file is next recreated. |
 | 105 | ~~**Design constraint (not a defect): a LocalDb-replicated table must never be the parent of a row-mutating `ON DELETE` foreign key (`CASCADE`/`SET NULL`/`SET DEFAULT`) from another table**~~ **PINNED 2026-08-20.** Enforcement moved from the first replicated boot to the build: `SiteLocalDbReplicatedTableFkGuardTests` (Host.Tests) builds the real site schema via `OnReady` on an unreplicated config (so 0.3.3's own guard can't mask which FK is at fault), walks `SiteLocalDbSetup.ReplicatedTables` (now `internal`), asserts each exists and has no inbound row-mutating `ON DELETE` FK with the same `sqlite_master JOIN pragma_foreign_key_list` query LocalDb uses, and carries a negative control proving the checker finds a cascading child. The constraint itself stands — the row is kept as its record. Original: | Arch-review 2026-08-19 P4-7 → LocalDb 0.3.3 (scadaproj `122940c` + review follow-up `9663279`, Low-wave T5) | `LwwApplier` upserts with `INSERT OR REPLACE` under `foreign_keys=ON`, so the implicit delete fires the child action while CDC capture is suppressed — the child rows change on the applying node only and the pair diverges silently. 0.3.3 makes `RegisterReplicated` refuse such a parent at registration (clear exception naming the child FK), which is a BOOT failure for the site node, so this is enforced, not advisory. None of the ten `SiteLocalDbSetup.ReplicatedTables` has any FK today (zero `REFERENCES` on the live rig file). Model cross-table cleanup as explicit application deletes on each table (as the 3-table deploy cascade already does), never as SQLite FK actions. | Whenever a schema change adds a foreign key between site tables, or an eleventh table joins `ReplicatedTables`. |
-| 106 | **LocalDb: a converged pair re-streams a FULL snapshot on EVERY reconnect once the sender's oplog has been ack-pruned to empty and the receiver's watermark is 0** | Found 2026-08-20 while verifying row 104 on the rig (site-a-a received the same 127 k-row full snapshot at both the 13:09 deploy restart and the 13:25 VACUUM restart) | Sender-side defect in `SnapshotStreamer.SendAsync`: `as_of_seq = MAX(seq)` of the oplog, which is **0** once the pair has converged and ack-pruning emptied it. The receiver's `SetLastAppliedRemoteSeqAsync` is MAX-guarded (no regression — good), but a peer whose watermark is genuinely 0 (site-a-a was rebuilt after the 2026-08-15 corruption; any wiped/re-imaged node is the same) can never be lifted by an `as_of_seq 0` snapshot, so the 0.1.2 empty-oplog horizon rule (`streamableFrom = last_acked_seq + 1`) sees `0 < 149` as a gap on every handshake and re-streams everything. Rig state: node-b `last_acked_seq=149`, oplog empty; node-a `last_applied_remote_seq=0`. Not a divergence (LWW absorbs the re-delivered rows) but a cost/availability bug — ~1 s and 127 k rows here, minutes and a full-file read on a 1 GB production pair, on every reconnect, until the sender happens to write a new row. Fix is sender-side and wire-compatible: stamp `as_of_seq = max(MAX(seq), last_acked_seq)` (a full snapshot taken now covers every delta ever pruned below `last_acked_seq`), so the receiver's MAX-guarded watermark lifts to 149 and the loop ends; add a pinning test (converged+pruned sender, zero-watermark receiver → exactly one snapshot across two handshakes). Ships as LocalDb 0.3.4. | Next LocalDb release (0.3.4) — small, well-understood, and worth doing before any production pair is rebuilt from its peer. |
+| 106 | ~~**LocalDb: a converged pair re-streams a FULL snapshot on EVERY reconnect once the sender's oplog has been ack-pruned to empty and the receiver's watermark is 0**~~ **RESOLVED 2026-08-20 — LocalDb 0.3.4 (scadaproj `75c1897`), published and pinned.** `SnapshotStreamer.SendAsync` now stamps `as_of_seq = max(MAX(seq), last_acked_seq)`; pinned by `WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot` (fails on 0.3.3 exactly as the rig did: watermark 0, expected 3). Live proof on the rig: the first restart on 0.3.4 sent one last full snapshot stamped `as_of_seq 149` (was 0) and node-a's `last_applied_remote_seq` went 0 → 149; a second restart of the pair exchanged **zero** snapshots, `Connected=true`, backlog 0. Original: | Found 2026-08-20 while verifying row 104 on the rig (site-a-a received the same 127 k-row full snapshot at both the 13:09 deploy restart and the 13:25 VACUUM restart) | Sender-side defect in `SnapshotStreamer.SendAsync`: `as_of_seq = MAX(seq)` of the oplog, which is **0** once the pair has converged and ack-pruning emptied it. The receiver's `SetLastAppliedRemoteSeqAsync` is MAX-guarded (no regression — good), but a peer whose watermark is genuinely 0 (site-a-a was rebuilt after the 2026-08-15 corruption; any wiped/re-imaged node is the same) can never be lifted by an `as_of_seq 0` snapshot, so the 0.1.2 empty-oplog horizon rule (`streamableFrom = last_acked_seq + 1`) sees `0 < 149` as a gap on every handshake and re-streams everything. Rig state: node-b `last_acked_seq=149`, oplog empty; node-a `last_applied_remote_seq=0`. Not a divergence (LWW absorbs the re-delivered rows) but a cost/availability bug — ~1 s and 127 k rows here, minutes and a full-file read on a 1 GB production pair, on every reconnect, until the sender happens to write a new row. Fix is sender-side and wire-compatible: stamp `as_of_seq = max(MAX(seq), last_acked_seq)` (a full snapshot taken now covers every delta ever pruned below `last_acked_seq`), so the receiver's MAX-guarded watermark lifts to 149 and the loop ends; add a pinning test (converged+pruned sender, zero-watermark receiver → exactly one snapshot across two handshakes). Ships as LocalDb 0.3.4. | Next LocalDb release (0.3.4) — small, well-understood, and worth doing before any production pair is rebuilt from its peer. |
 | 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — |
 
 ## Resolved (verified against the code 2026-07-10)