chore(deps): LocalDb 0.3.4 -> 0.3.7 (#130, scadaproj#53)

Pins ZB.MOM.WW.LocalDb, .Replication and .Contracts at 0.3.7 (published
2026-11-12). Wire-compatible; no config or code change needed.

- 0.3.6 (scadaproj#36): the library sync service enforces
  LocalDb:Replication:ApiKey itself. LocalDbSyncAuthInterceptor is kept for
  its refuse-every-stream-when-blank rule; its doc comment is updated. The
  rig's site-a pair already carries one identical key on both nodes.
- 0.3.7 (scadaproj#16): a tombstone is held until the peer has it, so a
  node offline past TombstoneRetention no longer resurrects deleted rows.
  The survivor's ledger (sf_messages / site_events deletes) grows for the
  length of the outage; optional TombstoneHardCap bounds it.

Deploy notes added to the topology + installation guides, the S&F
requirement and CLAUDE.md: stop/start the pair together, same ApiKey on
both nodes, ledger growth while a peer is down.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

docs/deployment/installation-guide.md

@@ -191,8 +191,16 @@ Ensure bidirectional TCP connectivity between all Akka.NET cluster peers. The re
 **Stop both nodes of a site pair, upgrade both, then start both.** Rolling one node at a time is
 not supported as of LocalDb Phase 2 — the legacy snapshot-compatibility handler that made a
 mixed-version pair converge was deleted with the bespoke replicator, and a mixed pair now diverges
-silently. See `docs/deployment/topology-guide.md` for the reasoning and for the related
-`TombstoneRetention` bound on how long one node may stay offline.
+silently. See `docs/deployment/topology-guide.md` for the reasoning.
+
+For a replicated pair (LocalDb 0.3.7):
+
+- `LocalDb:Replication:ApiKey` must be set, byte-identical, on **both** nodes (a `${secret:...}`
+  reference or an environment variable — never a literal in a shipped config). Both the library
+  and ScadaBridge's `LocalDbSyncAuthInterceptor` refuse the sync stream without it.
+- While one node is down, the survivor's delete tombstones are held until the partner returns and
+  acks them, so its LocalDb file grows for the length of the outage. Deleted rows no longer
+  resurrect on rejoin, however long the outage (scadaproj#16).
 
 ## Post-Installation Verification
 

docs/deployment/topology-guide.md

@@ -365,13 +365,28 @@ same boot, so a node picks up the new columns and the matching triggers in one s
 (Most recent instance: `external_systems` gained `max_retries` / `retry_delay_ms` for the
 per-system store-and-forward retry policy.)
 
-**Related bound — do not leave one node of a pair offline for long.** A node absent for longer than
-`LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when
-it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is
-nothing left to suppress the stale row the returning node still holds. Within the retention window
-a rejoin is safe and self-correcting (verified live: a node stopped and restarted mid-load rejoined
-with both nodes byte-identical and zero duplicates). Beyond it, rebuild the returning node's
-database from its peer rather than letting it rejoin.
+**Related bound — a node offline for long (changed in LocalDb 0.3.7, scadaproj#16).** Through
+0.3.6 a node absent for longer than `LocalDb:Replication:TombstoneRetention` (default **7 days**)
+could **resurrect deleted rows** when it rejoined, because its partner pruned the tombstones that
+would have suppressed them. **As of 0.3.7 a tombstone is pruned only once it is past retention AND
+the peer holds it** — an unacked local delete and any outstanding snapshot debt hold it, and
+retention restarts from the moment of delivery. A rejoin after any length of outage is therefore
+safe again. **The cost is ledger growth while the peer is down:** every delete made on the
+surviving node (on a site, mostly `sf_messages` delivery deletes and the daily `site_events` purge)
+keeps its tombstone in `__localdb_row_version` until the partner returns and acks it, so the site
+LocalDb file grows for the length of the outage and shrinks only after the drain. The optional
+`LocalDb:Replication:TombstoneHardCap` (default unset = off, must be ≥ `TombstoneRetention`) bounds
+that growth — but past the cap the old resurrection risk returns, so leave it unset unless disk is
+the binding constraint, and in that case rebuild the returning node from its peer instead of
+letting it rejoin. A peer restored from an old backup of its own file is still not covered.
+
+**The sync endpoint key is enforced by the library as of LocalDb 0.3.6 (scadaproj#36).**
+`LocalDb:Replication:ApiKey` must be **byte-identical on both nodes** of a replicated pair.
+ScadaBridge's own `LocalDbSyncAuthInterceptor` already refused a missing or mismatched key, so the
+pin bump changes nothing for a correctly configured pair; it is the second, library-side check of
+the same key. Supply it as a `${secret:...}` reference (resolved pre-host) or an environment
+variable, never a literal in a shipped config. A mismatch shows as `LocalDbReplicationConnected =
+false` with an `Unauthenticated` sync failure in the dialling node's log.
 
 ### Site LocalDb files created before 2026-08-20 need a one-off offline VACUUM
 

docs/requirements/Component-StoreAndForward.md

@@ -93,7 +93,7 @@ There is **no maximum buffer size**. Messages accumulate in the buffer until del
 - The standby node receives those row changes into its own copy of the table but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby's copy is kept warm purely for a future failover.
 - On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent changes may not have replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its status change not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit.
 - On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy.
-- **Peer-join anti-entropy resync (LocalDb CDC).** *(Rewritten for LocalDb Phase 2, 2026-07-20. The previous specification of a chunked, ack-confirmed `SfBufferSnapshotChunk` protocol described the bespoke `ReplicationService`, which Phase 2 deleted. The discussion is rewritten rather than removed, because the failure modes it reasoned about still exist — they are simply bounded differently now.)* The buffer lives in the consolidated LocalDb database as the replicated `sf_messages` table, and both nodes exchange changes over a gRPC sync stream rather than Akka remoting. A node that was down for an extended period no longer requests a full-buffer snapshot and **replaces** its local buffer; LocalDb's snapshot resync merges **per row under last-writer-wins and never deletes**. Several of the old hazards are therefore structurally gone rather than guarded against: **(a) the 128 000-byte Akka frame limit no longer applies** — the transport is gRPC, whose successor ceiling is the 4 MB default receive limit, managed since LocalDb 0.2.0 by the **byte-budget** bound `LocalDb:Replication:MaxBatchBytes` (default 2 MB, enforced per wire message). `MaxBatchSize` is demoted to a secondary row cap whose remaining job is bounding the DB **read page** that `OplogStore`/`SnapshotStreamer` materialize in memory before the byte split runs (arch-review adversarial finding F2) — the rig pins it to 64 for that reason; the old rig pin of 16, hand-sized against a ~70 KB worst-case `config_json` as a byte-budget proxy (Phase 2 plan, D6), is retired. Chunking, `ResyncId` assembly, assembly timeouts, and the truncation flag are all retired with it. **(b) The N1 directional-authority hazard is gone.** That guard existed because the bespoke resync applied a destructive delete-all-then-insert-all, so running it in the wrong direction wiped a live buffer. With a non-destructive merge there is no wipe to gate, and replication is symmetric — either node may write. `ActiveNodeEvaluator` survives, but only for the **delivery** gate and the heartbeat, which still genuinely need a single active node. **(c) The N5 orphan-row race is gone.** A `Remove` ordered before a re-add can no longer resurrect a row: deletes are tombstoned with an HLC, and a tombstone beats any older write for the same key. **The duplicate-delivery bound, stated explicitly.** Delivery remains single-node: only the primary runs the sweep (`IClusterNodeProvider.SelfIsPrimary`). A message can therefore be delivered twice only when the OLD primary delivered it and the resulting status change had not yet replicated at the instant the gate flipped. The window is one replication flush interval plus the in-flight ack, not an unbounded divergence — and unlike the old model it does not grow with backlog depth or with how long a node was absent. **One new bound replaces the old ones:** a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) may resurrect deleted rows on rejoin, because the tombstones that would have suppressed them have been pruned. Stop-and-start a site pair together, and do not leave one node of a pair offline across that horizon.
+- **Peer-join anti-entropy resync (LocalDb CDC).** *(Rewritten for LocalDb Phase 2, 2026-07-20. The previous specification of a chunked, ack-confirmed `SfBufferSnapshotChunk` protocol described the bespoke `ReplicationService`, which Phase 2 deleted. The discussion is rewritten rather than removed, because the failure modes it reasoned about still exist — they are simply bounded differently now.)* The buffer lives in the consolidated LocalDb database as the replicated `sf_messages` table, and both nodes exchange changes over a gRPC sync stream rather than Akka remoting. A node that was down for an extended period no longer requests a full-buffer snapshot and **replaces** its local buffer; LocalDb's snapshot resync merges **per row under last-writer-wins and never deletes**. Several of the old hazards are therefore structurally gone rather than guarded against: **(a) the 128 000-byte Akka frame limit no longer applies** — the transport is gRPC, whose successor ceiling is the 4 MB default receive limit, managed since LocalDb 0.2.0 by the **byte-budget** bound `LocalDb:Replication:MaxBatchBytes` (default 2 MB, enforced per wire message). `MaxBatchSize` is demoted to a secondary row cap whose remaining job is bounding the DB **read page** that `OplogStore`/`SnapshotStreamer` materialize in memory before the byte split runs (arch-review adversarial finding F2) — the rig pins it to 64 for that reason; the old rig pin of 16, hand-sized against a ~70 KB worst-case `config_json` as a byte-budget proxy (Phase 2 plan, D6), is retired. Chunking, `ResyncId` assembly, assembly timeouts, and the truncation flag are all retired with it. **(b) The N1 directional-authority hazard is gone.** That guard existed because the bespoke resync applied a destructive delete-all-then-insert-all, so running it in the wrong direction wiped a live buffer. With a non-destructive merge there is no wipe to gate, and replication is symmetric — either node may write. `ActiveNodeEvaluator` survives, but only for the **delivery** gate and the heartbeat, which still genuinely need a single active node. **(c) The N5 orphan-row race is gone.** A `Remove` ordered before a re-add can no longer resurrect a row: deletes are tombstoned with an HLC, and a tombstone beats any older write for the same key. **The duplicate-delivery bound, stated explicitly.** Delivery remains single-node: only the primary runs the sweep (`IClusterNodeProvider.SelfIsPrimary`). A message can therefore be delivered twice only when the OLD primary delivered it and the resulting status change had not yet replicated at the instant the gate flipped. The window is one replication flush interval plus the in-flight ack, not an unbounded divergence — and unlike the old model it does not grow with backlog depth or with how long a node was absent. **The old offline-horizon bound is gone as of LocalDb 0.3.7 (scadaproj#16):** through 0.3.6 a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) could resurrect deleted rows on rejoin, because the tombstones that would have suppressed them had been pruned. A tombstone is now pruned only once it is past retention **and** the peer holds it, so the trade is the other way round: while a peer is down, the survivor's `sf_messages` delete tombstones accumulate in the ledger until the peer returns and acks them (optional `TombstoneHardCap`, default off, bounds the growth at the price of the old risk). Stop-and-start a site pair together.
 
 ### Operation Tracking Table (lives in Site Runtime, not here)