Two driver nodes replicating the alarm buffer over the real loopback h2c
transport, through the real fail-closed interceptor. This is the test the
phase exists to pass: schema, registration, sink rewire and drain gate can
all be individually green while a pair still fails to share the undelivered
alarm history that is the whole point of moving the queue.
Rows are written through the production sink rather than hand-rolled
INSERTs, so the id derivation, column set and delete-on-ack semantics under
test are the ones production uses. The scenarios cover a buffered burst
converging with the oplog draining to zero, delivered rows being removed from
BOTH nodes as tombstones (the no-redeliver-after-failover property), writes
during a transport outage surviving the rejoin, and the same event accepted
on both nodes collapsing to one row.
The positive control found a weak assertion. With
RegisterReplicated("alarm_sf_events") commented out, three of the four went
red immediately -- but the same-event-on-both-nodes scenario still PASSED,
because with replication off each node trivially holds its own single copy
and "one row on each" is satisfied by two databases that never spoke. It now
also enqueues a distinct event on B and requires both nodes to hold two rows,
which cannot be satisfied without convergence; it goes red under the control
like the others. Control restored, all four green.
Also updates the second exact-set replicated-tables pin, in LocalDbWiringTests.
The plan predicted these pins would go red here, and that is them working:
one of them is asserted through the full driver DI graph rather than the
OnReady callback, so it catches a registration that exists in the callback
but never reaches a running host.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Copies the pre-consolidation store-and-forward queue into the consolidated
database on first boot, then renames the legacy file aside. Rows are in that
file precisely because the historian could not be reached, so dropping them
on upgrade would discard exactly the alarm audit trail the queue exists to
protect.
Runs last in OnReady, after every RegisterReplicated call. It is the only
thing in OnReady that writes rows, and capture is trigger-based: a migration
that ran before registration would recover the backlog locally and never
replicate a line of it, silently and permanently.
Ids are derived from the payload rather than the plan's mig-{node}-{legacyId}
scheme. Node-prefixing solves the collision the legacy AUTOINCREMENT key
would cause -- node A's row 7 and node B's row 7 are different alarms -- but
it preserves a duplication that should be collapsed instead. A warm pair's
two legacy files OVERLAP: HistorianAdapterActor default-writes while its
redundancy role is unknown, so both nodes accepted the same transitions
during every boot window. Prefixed ids would carry those duplicates into the
merged buffer forever; equal-payload ids converge them. The same property
makes a crash between commit and rename harmless under INSERT OR IGNORE.
OnReady now takes IConfiguration rather than offering an overload that skips
the migration. A wiring mistake that silently discarded a node's undelivered
alarm history is not a mistake worth making possible.
The copy is restricted to the columns the legacy table actually has. Naming
a column an older build never wrote throws "no such column", which would
discard every row in the table rather than the one field.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Moves the alarm store-and-forward buffer out of its own alarm-historian.db
and into the node's consolidated LocalDb, where it replicates to the
redundant pair peer. A node that dies holding undelivered alarm history no
longer takes it to the grave.
Tasks 2 and 3 land together, as the plan anticipated. They are not separable:
the gate is a constructor argument of the rewritten sink, and a commit that
replicated the queue without gating the drain would be a commit in which both
nodes of a pair deliver every alarm event, continuously.
That drain gate is the load-bearing part of the change, and the recon
explains why it is new work rather than a refinement. Exactly-once delivery
across a pair is enforced today on the ENQUEUE side, by
HistorianAdapterActor: only the Primary enqueues, so the Secondary's queue is
empty and its ungated drain has nothing to send. Replicating the table
destroys that invariant.
The gate is a Func<bool> the caller supplies, because the drain runs on a
timer the sink owns rather than on a mailbox, and Core.AlarmHistorian cannot
reference PrimaryGatePolicy in Runtime. Runtime supplies it via a new
IRedundancyRoleView singleton that DriverHostActor publishes its Primary-gate
verdict to -- the same verdict the inbound-write and native-ack gates use, so
there is no second notion of am-I-the-Primary to drift.
Two failure modes are deliberately closed:
- The view is seeded OPEN, matching the policy's own answer for an unknown
role with no driver peer. A deployment that runs no redundancy never
publishes to it, and defaulting closed would silently stop its alarm
history forever.
- A gate that throws is read as not-now, never as permission, and a closed
gate reports the new HistorianDrainState.NotPrimary rather than Idle. A
Secondary's rising queue is supposed to look different from a stalled
drain, and if BOTH nodes report NotPrimary the pair is misconfigured and
says so instead of quietly filling toward the capacity ceiling.
Row ids are a hash of the payload rather than fresh GUIDs. Both adapters
accept the same fanned transition in the window before the first redundancy
snapshot arrives, and under last-writer-wins an equal key converges those two
accepts into one row instead of duplicating them.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Adds the alarm store-and-forward buffer to the consolidated LocalDb file and
registers it for replication, so a node that dies with undelivered alarm
history no longer takes it to the grave.
The table keeps the legacy queue's shape rather than the status column the
plan sketched. An acknowledged row is deleted, not marked delivered: that
needs no second sweeper to keep the table bounded, leaves the capacity
semantics untouched, and the replication engine carries the delete as a
tombstone so the peer drops its copy anyway -- which is what the status
column was for. last_error is retained because it is the only operator-facing
record of why a row was dead-lettered.
The primary key is app-minted TEXT. The legacy AUTOINCREMENT RowId cannot
replicate under last-writer-wins: two nodes would independently allocate
rowid 7 to different alarms and silently overwrite each other. The drain
index therefore orders by enqueued_at_utc rather than insertion order, with
id as a tiebreak so the ordering is total.
Tables are created unconditionally, independent of AlarmHistorian:Enabled.
An empty registered table costs three triggers; creating it lazily would mean
a node that enables the historian later writes rows before its capture
triggers exist, which is exactly the silent-loss shape OnReady's ordering
comment warns about.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Answers Task 0 with file:line citations against master 2e46d054.
The STOP condition does not fire: PayloadJson is TEXT, no BLOB column, so
alarm_sf_events can be registered for replication as-is.
Two findings reshape the plan's assumptions:
- The drain worker is not a hosted service or an actor. It is a
self-rescheduling Timer owned by the sink and started from the DI
factory, so the Primary gate has to live inside DrainOnceAsync as an
injected delegate -- Core.AlarmHistorian cannot reference
PrimaryGatePolicy, which lives in Runtime.
- A Primary gate already exists on the enqueue side
(HistorianAdapterActor.ShouldHistorize), using a different policy than
the one the plan specifies for the drain. That gate is why today's
ungated drain is safe: the Secondary never enqueues, so its queue is
empty. Replicating the table breaks that invariant, which makes the
drain gate mandatory in the same commit rather than a refinement.
Records deviations D-1 (deterministic content-hash ids so a boot-window
double-enqueue converges instead of duplicating), D-2 (gate denial must be
observable or a snapshot identity mismatch silently stops alarm history),
D-3 (Tasks 2+3 land as one commit) and D-4 (the rig will dead-letter in
~5 minutes at MaxAttempts 10 with no gateway).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty