docs(localdb): phase 2 truth pass across both repos

Normative first: Component-StoreAndForward.md:83 specified the whole chunked,
ack-confirmed SfBufferSnapshotChunk resync protocol, which Phase 2 deleted. It
is rewritten rather than removed — the failure modes it reasoned about still
exist, they are just bounded differently — and it now states the
duplicate-delivery bound explicitly: a message can be delivered twice only when
the OLD primary delivered it and the status change had not yet replicated when
the gate flipped. One flush interval plus the in-flight ack, and unlike the old
model it does NOT grow with backlog depth or with how long a node was absent.
The N1 directional-authority and N5 orphan-row hazards are recorded as
structurally gone, not merely unguarded.

Frame-size known-issue amended: its 2026-06-26 resolution replaced the
intra-site hop with notify-and-fetch; Phase 2 then deleted notify-and-fetch
itself, so the 128 KB Akka frame constraint no longer applies to that hop in
any form. Successor ceiling recorded (4 MB gRPC cap via MaxBatchSize, which
batches by ROW COUNT), including that the failure mode differs — an oversized
gRPC message is rejected, not silently dropped.

Deployment docs gain the two operational constraints that have no home in code:
a site pair must be stopped and started TOGETHER (the SfBufferSnapshot compat
handler that made a mixed-version pair converge went with the replicator, and a
mixed pair now diverges silently), and a node offline beyond TombstoneRetention
can resurrect deleted rows on rejoin.

Both CLAUDE.md files corrected — each still said Phase 2 was NOT started.

Definition of done closed: build 0 warnings, all 10 suites green (3509 tests,
0 failures, 0 skips). Two DoD items needed amending rather than ticking: the
stale-symbol grep still matches 4 lines, all deliberate comment prose recording
what was deleted (a literal zero would delete the explanations that stop the
old design coming back), and the live gate has 10 evidence items, not the 9 the
checklist claimed.

Deletes the phase2 resume-state scratch doc, which said to delete it on landing.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-20 05:37:48 -04:00
parent 158e79bb50
commit 7b5a5a6f34
11 changed files with 89 additions and 219 deletions
+8
View File
@@ -187,6 +187,14 @@ ALTER ROLE db_owner ADD MEMBER scadabridge_svc;
Ensure bidirectional TCP connectivity between all Akka.NET cluster peers. The remoting port (default 8081) must be open in both directions.
## Upgrading a Site Pair
**Stop both nodes of a site pair, upgrade both, then start both.** Rolling one node at a time is
not supported as of LocalDb Phase 2 — the legacy snapshot-compatibility handler that made a
mixed-version pair converge was deleted with the bespoke replicator, and a mixed pair now diverges
silently. See `docs/deployment/topology-guide.md` for the reasoning and for the related
`TombstoneRetention` bound on how long one node may stay offline.
## Post-Installation Verification
1. Start the service: `sc.exe start ScadaBridge-Central`
+21 -1
View File
@@ -150,7 +150,27 @@ Each site has its own two-node cluster:
- Same split-brain resolver as central (keep-oldest).
- Singleton actors: Site Deployment Manager migrates on failover.
- Staggered instance startup: 50ms delay between Instance Actor creation to prevent reconnection storms.
- SQLite persistence: Both nodes access the same SQLite files (or each has its own copy with async replication).
- SQLite persistence: each node owns its own consolidated LocalDb database, kept in step by
asynchronous CDC replication over a gRPC sync stream (LocalDb Phase 1 + 2). The nodes do NOT
share a SQLite file.
### Site Pair Upgrades — stop and start BOTH nodes together
**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while
the bespoke replicator kept a legacy `SfBufferSnapshot` compatibility handler so a new standby
could still apply an old active node's monolithic snapshot. LocalDb Phase 2 deleted that handler
along with the replicator, so a mixed-version pair has no common replication path: the two nodes
will run, but they will not converge, and the divergence is silent.
Stop both nodes of a site pair, upgrade both, then start both.
**Related bound — do not leave one node of a pair offline for long.** A node absent for longer than
`LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when
it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is
nothing left to suppress the stale row the returning node still holds. Within the retention window
a rejoin is safe and self-correcting (verified live: a node stopped and restarted mid-load rejoined
with both nodes byte-identical and zero duplicates). Beyond it, rebuild the returning node's
database from its peer rather than letting it rejoin.
### Central-Site Communication