Two driver nodes replicating the alarm buffer over the real loopback h2c
transport, through the real fail-closed interceptor. This is the test the
phase exists to pass: schema, registration, sink rewire and drain gate can
all be individually green while a pair still fails to share the undelivered
alarm history that is the whole point of moving the queue.
Rows are written through the production sink rather than hand-rolled
INSERTs, so the id derivation, column set and delete-on-ack semantics under
test are the ones production uses. The scenarios cover a buffered burst
converging with the oplog draining to zero, delivered rows being removed from
BOTH nodes as tombstones (the no-redeliver-after-failover property), writes
during a transport outage surviving the rejoin, and the same event accepted
on both nodes collapsing to one row.
The positive control found a weak assertion. With
RegisterReplicated("alarm_sf_events") commented out, three of the four went
red immediately -- but the same-event-on-both-nodes scenario still PASSED,
because with replication off each node trivially holds its own single copy
and "one row on each" is satisfied by two databases that never spoke. It now
also enqueues a distinct event on B and requires both nodes to hold two rows,
which cannot be satisfied without convergence; it goes red under the control
like the others. Control restored, all four green.
Also updates the second exact-set replicated-tables pin, in LocalDbWiringTests.
The plan predicted these pins would go red here, and that is them working:
one of them is asserted through the full driver DI graph rather than the
OnReady callback, so it catches a registration that exists in the callback
but never reaches a running host.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Copies the pre-consolidation store-and-forward queue into the consolidated
database on first boot, then renames the legacy file aside. Rows are in that
file precisely because the historian could not be reached, so dropping them
on upgrade would discard exactly the alarm audit trail the queue exists to
protect.
Runs last in OnReady, after every RegisterReplicated call. It is the only
thing in OnReady that writes rows, and capture is trigger-based: a migration
that ran before registration would recover the backlog locally and never
replicate a line of it, silently and permanently.
Ids are derived from the payload rather than the plan's mig-{node}-{legacyId}
scheme. Node-prefixing solves the collision the legacy AUTOINCREMENT key
would cause -- node A's row 7 and node B's row 7 are different alarms -- but
it preserves a duplication that should be collapsed instead. A warm pair's
two legacy files OVERLAP: HistorianAdapterActor default-writes while its
redundancy role is unknown, so both nodes accepted the same transitions
during every boot window. Prefixed ids would carry those duplicates into the
merged buffer forever; equal-payload ids converge them. The same property
makes a crash between commit and rename harmless under INSERT OR IGNORE.
OnReady now takes IConfiguration rather than offering an overload that skips
the migration. A wiring mistake that silently discarded a node's undelivered
alarm history is not a mistake worth making possible.
The copy is restricted to the columns the legacy table actually has. Naming
a column an older build never wrote throws "no such column", which would
discard every row in the table rather than the one field.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
1. Wiped-node back-fill (live gate check 4). Fixed upstream in
ZB.MOM.WW.LocalDb 0.1.2 and pinned here. The gate's diagnosis was slightly
off: the oplog cap was not the gate. Snapshot detection measured the peer's
gap against the oldest surviving oplog row and read an empty oplog as "no
gap possible" — and an empty oplog is the steady state of a converged pair,
since ack-pruning deletes everything the peer confirmed. The healthy state
was the one state that could not heal a wiped peer. Now measured against
last_acked_seq when the oplog is empty.
Pinned at this level too, not just the library's: the pair harness grows a
WipePassiveAsync (a NEW database — a rebuilt node comes back with a new node
id and a zero watermark), and the new scenario asserts the emptied oplog as
its precondition before wiping, then requires the deployment artifact to
come back byte-identical with no new deploy. Verified RED against the pinned
0.1.1 (times out waiting for back-fill) and green on 0.1.2.
2. Oplog growth on default-OFF nodes (live gate check 8), fixed at the source:
StoreAsync now skips entirely when the pointer already names this
deployment/revision, the SHA matches the bytes, and the expected chunk count
is present. Re-caching an artifact the node already holds — every restart's
boot-from-cache, every RestoreApplied — writes nothing and mints no oplog
rows, where before it cost a delete plus an insert per chunk plus a pointer
update, all identical to what was already there.
Identity is over the bytes and the chunks are counted, both deliberately: a
re-composed artifact can carry the same ids with different bytes, and a
pointer can name a deployment whose chunks are missing, where skipping would
make an unreadable cache permanent. Both guards verified RED against a naive
pointer-only skip.
The gate's stated bound was also wrong and is corrected in the doc: growth
was never headed for the 1M row cap. AddZbLocalDbReplication is registered
unconditionally, so MaintenanceBackgroundService runs on default-OFF nodes
and the 7-day MaxOplogAge cap prunes.
Runtime.Tests 412/0/31, Host.IntegrationTests LocalDb 45/45. Pre-existing and
unrelated: AbCip_Green_AgainstSim (needs the AB CIP docker fixture, red on a
clean master too) and a flaky Roslyn race test (green 2/2 in isolation).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two driver nodes replicate the deployment-artifact cache over a real loopback
h2c transport through the real fail-closed LocalDbSyncAuthInterceptor; both DBs
init via the production LocalDbSetup.OnReady. Scenarios: byte-identical
convergence with matching pointer HLC+origin, retention prune reaching B as
tombstones, writes-while-transport-down surviving rejoin, and wrong-key
non-convergence with a matching-key positive control.
DoD positive control: commenting out both RegisterReplicated calls turns all
four scenarios red (verified locally, restored before commit).
Joseph Doherty