fix(localdb): registration follows replication config — baseline on enable, deregister self-heal on disable (pair-symmetric)

LocalDbSetup.OnReady registered all four replicated tables unconditionally on
every driver node, replicating or not. That was not merely wasteful (an
unreplicated node paid two INSERTs plus a json_object of the whole row inside
every write transaction, for an oplog nothing ever drained) — it made
baselineExistingRows unusable, so enabling replication on a long-running
default-OFF node converged on SUBSEQUENT WRITES ONLY and the rows already in the
file stayed invisible to the peer, silently and forever.

Registration now follows one predicate, LocalDbSetup.ReplicationIsConfigured:
LocalDb:Replication:PeerAddress OR :ApiKey non-blank. The OR is load-bearing —
only the dialling half of a pair sets PeerAddress, so keying on it alone would
strip capture from every passive node and the pair would converge in one
direction only. SyncListenPort deliberately does not join it: the passive
endpoint is fail-closed on the key, so a port with no ApiKey syncs with nobody.

  configured   => RegisterReplicated(table, baselineExistingRows: true), per
                  table, in the unchanged load-bearing DDL->register order
  unconfigured => DeregisterReplicated(table), an idempotent boot self-heal that
                  drops the capture triggers an earlier build left behind and
                  prunes that table's oplog/row-version rows. One Information
                  line, only when it actually cleaned something.

PAIR SYMMETRY: the sync handshake compares registered-table digests fail-closed,
so flipping either key is a both-nodes-together change — set it on both and
restart both. Documented in CLAUDE.md, docs/Configuration.md and the replication
runbook (which claimed "this repo never calls DeregisterReplicated").

Verified against the pinned LocalDb 0.2.1: RegisterReplicated(string, bool) and
DeregisterReplicated(string) -> bool both exist; baselining seeds
__localdb_row_version at HLC 0 and the deregister return value distinguishes
"cleaned" from "nothing to clean". No pin bump needed.

Tests: LocalDbSetupTests gains pins for both branches (configured via
PeerAddress alone and via ApiKey alone; unconfigured => no tables, zero
triggers; late-enable => an HLC-0 baseline ledger entry; the log line fires once
and only when something was cleaned) — four went red against the old
unconditional code, and all four go red again under a constant-true predicate.
LocalDbWiringTests splits its exact-set pin into a replicating graph and a
default-OFF one. LocalDbPairHarness and AlarmSfLegacyMigratorTests built their
database from LocalDb:Path alone while configuring the replication engine from a
second config — production reads one configuration, so the two are now aligned.
Host.IntegrationTests: 224 passed, 0 failed, 5 skipped.
This commit is contained in:
Joseph Doherty
2026-08-18 07:25:01 -04:00
parent e29fab19f6
commit 3e62f1617b
8 changed files with 542 additions and 93 deletions
+43 -16
View File
@@ -650,7 +650,9 @@ deliberately unresolvable gateway endpoint, so the buffer is always a live, non-
`.Replication`), bumped from `0.1.3` on 2026-08-15. It was a **pure pin bump — no code change**:
`RegisterReplicated(string)` is unchanged, everything 0.2.x added is additive, the wire protocol is
untouched (0.1.x peers still sync), and the on-disk bookkeeping schema v1 → v2 upgrades in place on
open, so no rig or volume action was needed. Three things about it are worth knowing:
open, so no rig or volume action was needed. **Its two new APIs — `DeregisterReplicated` and
`RegisterReplicated(table, baselineExistingRows:)` — are now both used** (2026-08-18); the bullets
below are the shape that landed, not the bump itself. Two things are worth knowing:
- **`MaxBatchSize` no longer means what the rig's `16` was chosen to mean.** 0.2.0 made batching
**byte-budgeted** — `LocalDb:Replication:MaxBatchBytes` (2 MB default, left unset here, sized
@@ -660,21 +662,46 @@ open, so no rig or volume action was needed. Three things about it are worth kno
`16` — it now bounds that transient allocation (≈2.7 MB at ≈171 KB artifact chunk rows), not the
message size. Every comment and doc row that called it "row-count-only against gRPC's 4 MB cap"
was corrected with the bump; do not restore that rationale.
- **`DeregisterReplicated` exists and this repo deliberately does not call it.** `LocalDbSetup.OnReady`
registers all four tables **unconditionally**, on every driver node, replicating or not — a
replication-configured node and a default-OFF one (central, site-b) carry identical CDC triggers.
ScadaBridge went the other way (conditional registration + a boot-time deregister self-heal), so
do not read that pattern across as settled here: it would be a behaviour change, and
`LocalDbSetupTests` pins the registered-table set and the DDL→register ordering exactly. The
cost of the current shape is three unused triggers per table on an unreplicated node; the
benefit is that turning replication on is a config change that needs no re-registration pass.
- **Consequently, `baselineExistingRows` is not used either.** Rows written on a node *before*
replication is enabled are not in `__localdb_row_version`, so enabling replication on a
long-running default-OFF node (central, site-b) converges on subsequent **writes**, not on the
rows already in the file. Passing `RegisterReplicated(table, baselineExistingRows: true)` is the
library's answer, but it flags a snapshot resync and is only correct when gated on replication
actually being configured — which the unconditional registration above cannot express. Enable
replication on a fresh pair, or accept the gap and let the next deploy re-cache.
- **Registration FOLLOWS the replication config — it is no longer unconditional (2026-08-18).**
`LocalDbSetup.OnReady` reads one predicate, `LocalDbSetup.ReplicationIsConfigured`:
**`LocalDb:Replication:PeerAddress` OR `LocalDb:Replication:ApiKey` is non-blank.** The **OR is
not belt-and-braces** — only the dialling half of a pair sets `PeerAddress`, so keying on it
alone would strip capture from every *passive* node (site-a-2 carries `ApiKey` +
`LocalDb:SyncListenPort` and nothing else) and the pair would converge in one direction only,
silently. ⚠️ **`SyncListenPort` deliberately does NOT join the OR:** the passive endpoint is
fail-closed on the key, so a port without an `ApiKey` rejects every inbound stream and has
nothing to capture for.
- **Configured ⇒ `RegisterReplicated(table, baselineExistingRows: true)`**, per table, in the
same load-bearing DDL→register order. Baselining is what makes enabling replication on a node
that has been running *without* it actually converge: capture is CDC and the snapshot streamer
pages from `__localdb_row_version`, so pre-registration rows were in neither and a late opt-in
used to replicate only writes made after the restart. It seeds the ledger at the LWW floor
(HLC `0`, this node's id) and flags a resync — a baselined row loses to any genuine remote
write of the same key and wins only where the peer has no version at all. It is idempotent, so
it is free on every boot after the first. ⚠️ On the pinned **0.2.1** the snapshot flag is
per-**database**, not per-table, so baselining anything streams *every* registered table in
full, in both directions at once — trim `alarm_sf_events` before enabling replication on a node
that has been buffering alarm history for months, and expect one slow first sync.
- **Not configured ⇒ `DeregisterReplicated(table)` per table**, an idempotent boot-time
self-heal: it drops the capture triggers an earlier (unconditional) build left in the file and
prunes that table's oplog + row-version rows. Merely declining to register would leave every
pre-2026-08-18 file capturing forever — two INSERTs plus a `json_object` of the whole row
inside every write transaction — on precisely the node that has no peer to capture for. The
call returns `true` when it removed something, so the Information line fires **once, and only
when it actually cleaned**; a default-OFF node's ordinary boot stays silent. The logger is
Serilog's static `Log.Logger` (this runs inside the `AddZbLocalDb` singleton factory, where
resolving one from DI would nest a resolution inside a singleton construction; `Program.cs`
assigns it before anything resolves `ILocalDb`), with an optional parameter so tests observe
the line rather than infer it.
- ⚠️ **PAIR SYMMETRY: flipping registration is a BOTH-NODES-TOGETHER change.** The sync handshake
compares registered-table digests **fail-closed**, so a node that deregisters a table its peer
still replicates stops syncing with a schema mismatch. Enabling replication on an established
pair means setting the config and **restarting both nodes** — do it on both, or on neither.
- **Rollout note:** the only replicating pair in the fleet is docker-dev **site-a**, and it was
enabled from a fresh pair, so the impact of the old gap was low — central and site-b are
default-OFF and simply shed their unused triggers on their next boot. `LocalDbSetupTests`
pins both branches (registered-table set on a configured node, empty set + zero triggers +
the HLC-0 baseline ledger entry on the other) and the DDL→register ordering, unchanged.
See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
cache), `docs/AlarmHistorian.md`, and the runbook