chore(deps): LocalDb 0.3.4 -> 0.3.7 (#130, scadaproj#53)

Pins ZB.MOM.WW.LocalDb, .Replication and .Contracts at 0.3.7 (published
2026-11-12). Wire-compatible; no config or code change needed.

- 0.3.6 (scadaproj#36): the library sync service enforces
  LocalDb:Replication:ApiKey itself. LocalDbSyncAuthInterceptor is kept for
  its refuse-every-stream-when-blank rule; its doc comment is updated. The
  rig's site-a pair already carries one identical key on both nodes.
- 0.3.7 (scadaproj#16): a tombstone is held until the peer has it, so a
  node offline past TombstoneRetention no longer resurrects deleted rows.
  The survivor's ledger (sf_messages / site_events deletes) grows for the
  length of the outage; optional TombstoneHardCap bounds it.

Deploy notes added to the topology + installation guides, the S&F
requirement and CLAUDE.md: stop/start the pair together, same ApiKey on
both nodes, ledger growth while a peer is down.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

CLAUDE.md

@@ -99,7 +99,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
 
     - CDC replication does all three jobs now: config deploys reach the standby as ordinary row changes — **the standby makes no fetch at all** during a deploy (`SiteReconciliationActor`'s node-STARTUP fetch when central reports gaps is a different, surviving path) — and buffer mutations replicate via triggers on `sf_messages`. `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync and is **unsafe to reintroduce**: a mass DELETE on a replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
     - **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
-    - **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
+    - **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. **LocalDb 0.3.7 (2026-11-12, scadaproj#53):** a tombstone is pruned only once past `TombstoneRetention` **and** held by the peer, so a node offline past retention no longer resurrects deleted rows — the cost is that the survivor's tombstone ledger (mostly `sf_messages` and `site_events` deletes) grows for the length of the outage; optional `TombstoneHardCap` (default off) bounds it at the price of the old risk. The library's sync endpoint also enforces `LocalDb:Replication:ApiKey` itself since 0.3.6 (scadaproj#36) — set the same key on both nodes; `LocalDbSyncAuthInterceptor` is kept for its stricter refuse-everything-when-blank rule. See `docs/deployment/topology-guide.md`.
     - **Batching is by BYTE BUDGET as of LocalDb 0.2.0** — `LocalDb:Replication:MaxBatchBytes` (default **2 MB**, sized under the 4 MB gRPC cap) bounds a delta/snapshot message by summed serialized size via a per-message split in `SyncSession.PumpLoopAsync`, with `MaxBatchSize` demoted to a secondary row cap; a single row over budget is sent alone rather than stalling the stream. **The rig's old `MaxBatchSize = 16` pin (a hand-computed byte-budget proxy: ~70 KB worst-case `config_json` x the 500 default is ~35 MB) is retired, but `MaxBatchSize` is NOT fully redundant with `MaxBatchBytes`** — it also bounds the separate DB READ page in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which materializes the whole page into memory *before* the byte-budget split runs, so an unset (500-default) `MaxBatchSize` still lets a reconnect drain transiently allocate ~35 MB per read even though every wire message stays under `MaxBatchBytes` (arch-review adversarial finding F2). Both site-a nodes on `docker/` therefore pin an explicit `"MaxBatchSize": 64` to bound that transient allocation, while `MaxBatchBytes` stays unset (its 2 MB default) to bound the wire message; site-b/site-c stay unreplicated so the key doesn't apply there.
     - **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all twelve tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
     - **LocalDb 0.3.0 (2026-08-15) makes that snapshot flag PER-TABLE** — the residual 0.2.1 explicitly deferred. On-disk bookkeeping schema goes to **v3** (a new `__localdb_snapshot_state` table, upgraded in place on open); still wire-compatible, and `lib_schema_version` deliberately stays `1` because the handshake compares it fail-closed for *equality*. The extension is negotiated by **capability** (`Handshake.supports_partial_snapshot`, `SnapshotBegin.tables` where **empty == all tables**), so a peer that does not advertise it is served a full snapshot — a scoped one would make it advance `last_applied_remote_seq` past deltas it never received. For the same reason **a partial snapshot moves no watermark on either side**; the overlapping deltas are still pumped and absorbed by LWW. Snapshot debt now clears on the receiver's new `SnapshotAck` rather than on enqueue, so a crash mid-snapshot re-streams the same scope. **`SiteLocalDbSetup.cs` needed no restructuring** — it already calls `RegisterReplicated` once per table in a loop, which is exactly what per-table flagging keys off. The consequence to know: **adding an eleventh table to `ReplicatedTables` on an already-replicating site now snapshots that one table**, where through 0.2.x the same edit re-streamed all eleven in full in both directions. First-boot behaviour is unchanged (all ten seed at once ⇒ the flagged set is every registered table ⇒ an ordinary full snapshot), and so is upgrading the rig in place (the ten tables are already ledgered, nothing seeds, nothing is flagged). A future "register the whole list in one call" API would silently give the coarse behaviour back.

Directory.Packages.props

@@ -119,9 +119,9 @@
     <PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.6.2" />
     <PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.6.2" />
     <PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.6.2" />
-    <PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.3.4" />
-    <PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.3.4" />
-    <PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.3.4" />
+    <PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.3.7" />
+    <PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.3.7" />
+    <PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.3.7" />
   </ItemGroup>
 
   <!--

docs/deployment/installation-guide.md

@@ -191,8 +191,16 @@ Ensure bidirectional TCP connectivity between all Akka.NET cluster peers. The re
 **Stop both nodes of a site pair, upgrade both, then start both.** Rolling one node at a time is
 not supported as of LocalDb Phase 2 — the legacy snapshot-compatibility handler that made a
 mixed-version pair converge was deleted with the bespoke replicator, and a mixed pair now diverges
-silently. See `docs/deployment/topology-guide.md` for the reasoning and for the related
-`TombstoneRetention` bound on how long one node may stay offline.
+silently. See `docs/deployment/topology-guide.md` for the reasoning.
+
+For a replicated pair (LocalDb 0.3.7):
+
+- `LocalDb:Replication:ApiKey` must be set, byte-identical, on **both** nodes (a `${secret:...}`
+  reference or an environment variable — never a literal in a shipped config). Both the library
+  and ScadaBridge's `LocalDbSyncAuthInterceptor` refuse the sync stream without it.
+- While one node is down, the survivor's delete tombstones are held until the partner returns and
+  acks them, so its LocalDb file grows for the length of the outage. Deleted rows no longer
+  resurrect on rejoin, however long the outage (scadaproj#16).
 
 ## Post-Installation Verification
 

docs/deployment/topology-guide.md

@@ -365,13 +365,28 @@ same boot, so a node picks up the new columns and the matching triggers in one s
 (Most recent instance: `external_systems` gained `max_retries` / `retry_delay_ms` for the
 per-system store-and-forward retry policy.)
 
-**Related bound — do not leave one node of a pair offline for long.** A node absent for longer than
-`LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when
-it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is
-nothing left to suppress the stale row the returning node still holds. Within the retention window
-a rejoin is safe and self-correcting (verified live: a node stopped and restarted mid-load rejoined
-with both nodes byte-identical and zero duplicates). Beyond it, rebuild the returning node's
-database from its peer rather than letting it rejoin.
+**Related bound — a node offline for long (changed in LocalDb 0.3.7, scadaproj#16).** Through
+0.3.6 a node absent for longer than `LocalDb:Replication:TombstoneRetention` (default **7 days**)
+could **resurrect deleted rows** when it rejoined, because its partner pruned the tombstones that
+would have suppressed them. **As of 0.3.7 a tombstone is pruned only once it is past retention AND
+the peer holds it** — an unacked local delete and any outstanding snapshot debt hold it, and
+retention restarts from the moment of delivery. A rejoin after any length of outage is therefore
+safe again. **The cost is ledger growth while the peer is down:** every delete made on the
+surviving node (on a site, mostly `sf_messages` delivery deletes and the daily `site_events` purge)
+keeps its tombstone in `__localdb_row_version` until the partner returns and acks it, so the site
+LocalDb file grows for the length of the outage and shrinks only after the drain. The optional
+`LocalDb:Replication:TombstoneHardCap` (default unset = off, must be ≥ `TombstoneRetention`) bounds
+that growth — but past the cap the old resurrection risk returns, so leave it unset unless disk is
+the binding constraint, and in that case rebuild the returning node from its peer instead of
+letting it rejoin. A peer restored from an old backup of its own file is still not covered.
+
+**The sync endpoint key is enforced by the library as of LocalDb 0.3.6 (scadaproj#36).**
+`LocalDb:Replication:ApiKey` must be **byte-identical on both nodes** of a replicated pair.
+ScadaBridge's own `LocalDbSyncAuthInterceptor` already refused a missing or mismatched key, so the
+pin bump changes nothing for a correctly configured pair; it is the second, library-side check of
+the same key. Supply it as a `${secret:...}` reference (resolved pre-host) or an environment
+variable, never a literal in a shipped config. A mismatch shows as `LocalDbReplicationConnected =
+false` with an `Unauthenticated` sync failure in the dialling node's log.
 
 ### Site LocalDb files created before 2026-08-20 need a one-off offline VACUUM
 

docs/requirements/Component-StoreAndForward.md

@@ -93,7 +93,7 @@ There is **no maximum buffer size**. Messages accumulate in the buffer until del
 - The standby node receives those row changes into its own copy of the table but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby's copy is kept warm purely for a future failover.
 - On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent changes may not have replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its status change not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit.
 - On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy.
-- **Peer-join anti-entropy resync (LocalDb CDC).** *(Rewritten for LocalDb Phase 2, 2026-07-20. The previous specification of a chunked, ack-confirmed `SfBufferSnapshotChunk` protocol described the bespoke `ReplicationService`, which Phase 2 deleted. The discussion is rewritten rather than removed, because the failure modes it reasoned about still exist — they are simply bounded differently now.)* The buffer lives in the consolidated LocalDb database as the replicated `sf_messages` table, and both nodes exchange changes over a gRPC sync stream rather than Akka remoting. A node that was down for an extended period no longer requests a full-buffer snapshot and **replaces** its local buffer; LocalDb's snapshot resync merges **per row under last-writer-wins and never deletes**. Several of the old hazards are therefore structurally gone rather than guarded against: **(a) the 128 000-byte Akka frame limit no longer applies** — the transport is gRPC, whose successor ceiling is the 4 MB default receive limit, managed since LocalDb 0.2.0 by the **byte-budget** bound `LocalDb:Replication:MaxBatchBytes` (default 2 MB, enforced per wire message). `MaxBatchSize` is demoted to a secondary row cap whose remaining job is bounding the DB **read page** that `OplogStore`/`SnapshotStreamer` materialize in memory before the byte split runs (arch-review adversarial finding F2) — the rig pins it to 64 for that reason; the old rig pin of 16, hand-sized against a ~70 KB worst-case `config_json` as a byte-budget proxy (Phase 2 plan, D6), is retired. Chunking, `ResyncId` assembly, assembly timeouts, and the truncation flag are all retired with it. **(b) The N1 directional-authority hazard is gone.** That guard existed because the bespoke resync applied a destructive delete-all-then-insert-all, so running it in the wrong direction wiped a live buffer. With a non-destructive merge there is no wipe to gate, and replication is symmetric — either node may write. `ActiveNodeEvaluator` survives, but only for the **delivery** gate and the heartbeat, which still genuinely need a single active node. **(c) The N5 orphan-row race is gone.** A `Remove` ordered before a re-add can no longer resurrect a row: deletes are tombstoned with an HLC, and a tombstone beats any older write for the same key. **The duplicate-delivery bound, stated explicitly.** Delivery remains single-node: only the primary runs the sweep (`IClusterNodeProvider.SelfIsPrimary`). A message can therefore be delivered twice only when the OLD primary delivered it and the resulting status change had not yet replicated at the instant the gate flipped. The window is one replication flush interval plus the in-flight ack, not an unbounded divergence — and unlike the old model it does not grow with backlog depth or with how long a node was absent. **One new bound replaces the old ones:** a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) may resurrect deleted rows on rejoin, because the tombstones that would have suppressed them have been pruned. Stop-and-start a site pair together, and do not leave one node of a pair offline across that horizon.
+- **Peer-join anti-entropy resync (LocalDb CDC).** *(Rewritten for LocalDb Phase 2, 2026-07-20. The previous specification of a chunked, ack-confirmed `SfBufferSnapshotChunk` protocol described the bespoke `ReplicationService`, which Phase 2 deleted. The discussion is rewritten rather than removed, because the failure modes it reasoned about still exist — they are simply bounded differently now.)* The buffer lives in the consolidated LocalDb database as the replicated `sf_messages` table, and both nodes exchange changes over a gRPC sync stream rather than Akka remoting. A node that was down for an extended period no longer requests a full-buffer snapshot and **replaces** its local buffer; LocalDb's snapshot resync merges **per row under last-writer-wins and never deletes**. Several of the old hazards are therefore structurally gone rather than guarded against: **(a) the 128 000-byte Akka frame limit no longer applies** — the transport is gRPC, whose successor ceiling is the 4 MB default receive limit, managed since LocalDb 0.2.0 by the **byte-budget** bound `LocalDb:Replication:MaxBatchBytes` (default 2 MB, enforced per wire message). `MaxBatchSize` is demoted to a secondary row cap whose remaining job is bounding the DB **read page** that `OplogStore`/`SnapshotStreamer` materialize in memory before the byte split runs (arch-review adversarial finding F2) — the rig pins it to 64 for that reason; the old rig pin of 16, hand-sized against a ~70 KB worst-case `config_json` as a byte-budget proxy (Phase 2 plan, D6), is retired. Chunking, `ResyncId` assembly, assembly timeouts, and the truncation flag are all retired with it. **(b) The N1 directional-authority hazard is gone.** That guard existed because the bespoke resync applied a destructive delete-all-then-insert-all, so running it in the wrong direction wiped a live buffer. With a non-destructive merge there is no wipe to gate, and replication is symmetric — either node may write. `ActiveNodeEvaluator` survives, but only for the **delivery** gate and the heartbeat, which still genuinely need a single active node. **(c) The N5 orphan-row race is gone.** A `Remove` ordered before a re-add can no longer resurrect a row: deletes are tombstoned with an HLC, and a tombstone beats any older write for the same key. **The duplicate-delivery bound, stated explicitly.** Delivery remains single-node: only the primary runs the sweep (`IClusterNodeProvider.SelfIsPrimary`). A message can therefore be delivered twice only when the OLD primary delivered it and the resulting status change had not yet replicated at the instant the gate flipped. The window is one replication flush interval plus the in-flight ack, not an unbounded divergence — and unlike the old model it does not grow with backlog depth or with how long a node was absent. **The old offline-horizon bound is gone as of LocalDb 0.3.7 (scadaproj#16):** through 0.3.6 a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) could resurrect deleted rows on rejoin, because the tombstones that would have suppressed them had been pruned. A tombstone is now pruned only once it is past retention **and** the peer holds it, so the trade is the other way round: while a peer is down, the survivor's `sf_messages` delete tombstones accumulate in the ledger until the peer returns and acks them (optional `TombstoneHardCap`, default off, bounds the growth at the price of the old risk). Stop-and-start a site pair together.
 
 ### Operation Tracking Table (lives in Site Runtime, not here)
 

src/ZB.MOM.WW.ScadaBridge.Host/LocalDbSyncAuthInterceptor.cs

@@ -9,10 +9,12 @@ using ZB.MOM.WW.LocalDb.Replication;
 namespace ZB.MOM.WW.ScadaBridge.Host;
 
 /// <summary>
-/// Gates the LocalDb passive sync endpoint. The replication library deliberately leaves
-/// inbound authentication to the host — its <c>LocalDbSyncService</c> verifies nothing —
-/// so without this interceptor anything that can reach the site node's gRPC port could
-/// stream arbitrary rows into the consolidated site database.
+/// Gates the LocalDb passive sync endpoint. Through LocalDb 0.3.5 the replication library
+/// left inbound authentication to the host — its <c>LocalDbSyncService</c> verified nothing —
+/// so without this interceptor anything that could reach the site node's gRPC port could
+/// stream arbitrary rows into the consolidated site database. Since 0.3.6 (scadaproj#36) the
+/// library checks the same key and header itself; this interceptor is kept for the one rule
+/// the library does not apply — refusing every stream when no key is configured.
 /// </summary>
 /// <remarks>
 /// <para>