LocalDbSetup.OnReady registered all four replicated tables unconditionally on
every driver node, replicating or not. That was not merely wasteful (an
unreplicated node paid two INSERTs plus a json_object of the whole row inside
every write transaction, for an oplog nothing ever drained) — it made
baselineExistingRows unusable, so enabling replication on a long-running
default-OFF node converged on SUBSEQUENT WRITES ONLY and the rows already in the
file stayed invisible to the peer, silently and forever.
Registration now follows one predicate, LocalDbSetup.ReplicationIsConfigured:
LocalDb:Replication:PeerAddress OR :ApiKey non-blank. The OR is load-bearing —
only the dialling half of a pair sets PeerAddress, so keying on it alone would
strip capture from every passive node and the pair would converge in one
direction only. SyncListenPort deliberately does not join it: the passive
endpoint is fail-closed on the key, so a port with no ApiKey syncs with nobody.
configured => RegisterReplicated(table, baselineExistingRows: true), per
table, in the unchanged load-bearing DDL->register order
unconfigured => DeregisterReplicated(table), an idempotent boot self-heal that
drops the capture triggers an earlier build left behind and
prunes that table's oplog/row-version rows. One Information
line, only when it actually cleaned something.
PAIR SYMMETRY: the sync handshake compares registered-table digests fail-closed,
so flipping either key is a both-nodes-together change — set it on both and
restart both. Documented in CLAUDE.md, docs/Configuration.md and the replication
runbook (which claimed "this repo never calls DeregisterReplicated").
Verified against the pinned LocalDb 0.2.1: RegisterReplicated(string, bool) and
DeregisterReplicated(string) -> bool both exist; baselining seeds
__localdb_row_version at HLC 0 and the deregister return value distinguishes
"cleaned" from "nothing to clean". No pin bump needed.
Tests: LocalDbSetupTests gains pins for both branches (configured via
PeerAddress alone and via ApiKey alone; unconfigured => no tables, zero
triggers; late-enable => an HLC-0 baseline ledger entry; the log line fires once
and only when something was cleaned) — four went red against the old
unconditional code, and all four go red again under a constant-true predicate.
LocalDbWiringTests splits its exact-set pin into a replicating graph and a
default-OFF one. LocalDbPairHarness and AlarmSfLegacyMigratorTests built their
database from LocalDb:Path alone while configuring the replication engine from a
second config — production reads one configuration, so the two are now aligned.
Host.IntegrationTests: 224 passed, 0 failed, 5 skipped.
Copies the pre-consolidation store-and-forward queue into the consolidated
database on first boot, then renames the legacy file aside. Rows are in that
file precisely because the historian could not be reached, so dropping them
on upgrade would discard exactly the alarm audit trail the queue exists to
protect.
Runs last in OnReady, after every RegisterReplicated call. It is the only
thing in OnReady that writes rows, and capture is trigger-based: a migration
that ran before registration would recover the backlog locally and never
replicate a line of it, silently and permanently.
Ids are derived from the payload rather than the plan's mig-{node}-{legacyId}
scheme. Node-prefixing solves the collision the legacy AUTOINCREMENT key
would cause -- node A's row 7 and node B's row 7 are different alarms -- but
it preserves a duplication that should be collapsed instead. A warm pair's
two legacy files OVERLAP: HistorianAdapterActor default-writes while its
redundancy role is unknown, so both nodes accepted the same transitions
during every boot window. Prefixed ids would carry those duplicates into the
merged buffer forever; equal-payload ids converge them. The same property
makes a crash between commit and rename harmless under INSERT OR IGNORE.
OnReady now takes IConfiguration rather than offering an overload that skips
the migration. A wiring mistake that silently discarded a node's undelivered
alarm history is not a mistake worth making possible.
The copy is restricted to the columns the legacy table actually has. Naming
a column an older build never wrote throws "no such column", which would
discard every row in the table rather than the one field.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
1. Wiped-node back-fill (live gate check 4). Fixed upstream in
ZB.MOM.WW.LocalDb 0.1.2 and pinned here. The gate's diagnosis was slightly
off: the oplog cap was not the gate. Snapshot detection measured the peer's
gap against the oldest surviving oplog row and read an empty oplog as "no
gap possible" — and an empty oplog is the steady state of a converged pair,
since ack-pruning deletes everything the peer confirmed. The healthy state
was the one state that could not heal a wiped peer. Now measured against
last_acked_seq when the oplog is empty.
Pinned at this level too, not just the library's: the pair harness grows a
WipePassiveAsync (a NEW database — a rebuilt node comes back with a new node
id and a zero watermark), and the new scenario asserts the emptied oplog as
its precondition before wiping, then requires the deployment artifact to
come back byte-identical with no new deploy. Verified RED against the pinned
0.1.1 (times out waiting for back-fill) and green on 0.1.2.
2. Oplog growth on default-OFF nodes (live gate check 8), fixed at the source:
StoreAsync now skips entirely when the pointer already names this
deployment/revision, the SHA matches the bytes, and the expected chunk count
is present. Re-caching an artifact the node already holds — every restart's
boot-from-cache, every RestoreApplied — writes nothing and mints no oplog
rows, where before it cost a delete plus an insert per chunk plus a pointer
update, all identical to what was already there.
Identity is over the bytes and the chunks are counted, both deliberately: a
re-composed artifact can carry the same ids with different bytes, and a
pointer can name a deployment whose chunks are missing, where skipping would
make an unreadable cache permanent. Both guards verified RED against a naive
pointer-only skip.
The gate's stated bound was also wrong and is corrected in the doc: growth
was never headed for the 1M row cap. AddZbLocalDbReplication is registered
unconditionally, so MaintenanceBackgroundService runs on default-OFF nodes
and the 7-day MaxOplogAge cap prunes.
Runtime.Tests 412/0/31, Host.IntegrationTests LocalDb 45/45. Pre-existing and
unrelated: AbCip_Green_AgainstSim (needs the AB CIP docker fixture, red on a
clean master too) and a flaky Roslyn race test (green 2/2 in isolation).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two driver nodes replicate the deployment-artifact cache over a real loopback
h2c transport through the real fail-closed LocalDbSyncAuthInterceptor; both DBs
init via the production LocalDbSetup.OnReady. Scenarios: byte-identical
convergence with matching pointer HLC+origin, retention prune reaching B as
tombstones, writes-while-transport-down surviving rejoin, and wrong-key
non-convergence with a matching-key positive control.
DoD positive control: commenting out both RegisterReplicated calls turns all
four scenarios red (verified locally, restored before commit).
Joseph Doherty