Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs
FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when
no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs
JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting
central-b never came Up (the "registered outage gap"), and self-first ordering
closes it using Akka's own protocol.
- 6 node appsettings swapped (the *-node-b configs; the -a nodes were already
self-first). All 14 shipped node configs now satisfy the invariant.
- StartupValidator enforces it at boot, comparing host AND port -- the invariant
fails silently when broken, so it is enforced loudly. NOTE: the gitignored
deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or
that node will refuse to boot.
- SelfFirstSeedBootstrapTests: real in-process clusters at production
failure-detection timings, incl. a falsifiability control proving the OLD
peer-first ordering never forms.
Rejected alternative (implemented, measured, discarded): an external self-form
timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's
join handshake and so cannot tell "no seed answered" from "a seed answered and
the join is in flight". On a routine standby restart the peer is alive but the
join stalls behind removal of the node's own stale incarnation; a Join(self)
during TryingToJoin abandons the in-flight join and forms a second cluster at
the same address -- still split after 90s. Docs that claimed self-first ordering
was unsafe for simultaneous cold start are corrected: while mutually reachable
the InitJoin handshake converges them to one cluster (measured).
CLAUDE.md was corrected in 34227991 but the component docs were not, so the
same understatements — plus several outright wrong statements — survived where
a reader is most likely to meet them.
ClusterInfrastructure.md carried the worst of it. It described active/standby as
cluster leadership and showed an IsActiveNode snippet doing
`cluster.State.Leader == self.Address`. No such code exists: ActiveNodeGate
returns ClusterActivityEvaluator.SelfIsOldest, and ActiveNodeEvaluator's own doc
comment says "never cluster.State.Leader". A second snippet (siteCallAuditShutdown
.AddTask) described a hand-rolled drain that has since been folded into
SingletonRegistrar.Start. Both snippets are replaced with what the code does. The
central singleton table listed 3 of the 7 registered singletons. The downing
section still described keep-oldest as the strategy and omitted
downing-provider-class entirely; it now shows the branching block cf3bd52f
introduced, with the accepted dual-active trade stated rather than the old
"impossible to boot" framing. Note the requirements-side spec,
docs/requirements/Component-ClusterInfrastructure.md, was already rewritten by
cf3bd52f — this is the components-side doc, which was untouched.
Communication.md described two transports; there are three — the deployment
config fetch over HTTP was missing. Each row now names which side dials, because
the gRPC entry gave a data direction (site to central) without saying who hosts
the server, which reads as the opposite of the truth: the only MapGrpcService in
the tree is in Program.cs's Site branch, and central dials in. The SiteEnvelope
snippet called DeployInstanceAsync, which is not a member of CommunicationService;
the heartbeat bullet claimed a cluster-leadership check; the proto summary listed
4 of 6 RPCs.
DeploymentManager.md still said the pipeline sends DeployInstanceCommand carrying
FlattenedConfigurationJson. It stages a PendingDeployment and sends a
RefreshDeploymentCommand; the config travels over the HTTP fetch. That is the
change the 128 KB frame-size issue forced, and the doc predated it.
Two of the five briefed drifts turned out not to be errors: nothing claimed the
transports were authenticated, and nothing asserted per-site ActorSystem names —
both were simply unstated. They are now stated, since silence about an
unauthenticated boundary is its own problem.
Docs only; no code changed.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
Same pattern as the SQLitePCLRaw pin: direct PackageReference at the chain's entry
project (ConfigurationDatabase). Bumping the DataProtection parent instead was tried
and rejected — 10.0.10 floors Microsoft.Extensions.*/EF at 10.0.10 (NU1605 cascade).
Found while OtOpcUa researched this repo as the model for its own per-cluster
mesh work. Each was verified against the code, not inferred from the doc.
Three transports cross the boundary, not two: ClusterClient, gRPC, and plain
token-gated HTTP for the deploy config fetch (DeploymentConfigEndpoints,
X-Deployment-Token, AllowAnonymous with the per-deployment token as the entire
security boundary).
The gRPC direction is inverted from the data flow. Data moves site to central,
but each SITE hosts the gRPC server and central dials in — MapGrpcService appears
only in the Site branch of Program.cs. There is no gRPC server on central, which
is why the two Ingest* RPCs are dead in practice. Also records the 6-RPC surface
(the doc implied 2), the (site, endpoint) factory key that fixed an arch-review
High, and the vendored-generated-code caveat.
Active/standby is ActiveNodeEvaluator.SelfIsOldestUp — the OLDEST Up member, and
explicitly never cluster.State.Leader, because leadership diverges from singleton
placement permanently after a restart-and-rejoin and both sides claim it during a
partition. The equivalence oldest-Up == singleton placement is the design, and it
was not stated anywhere in this file.
All clusters share one ActorSystem name ("scadabridge", hardcoded at
AkkaHostedService.cs:191); they are separate clusters only by seed partitioning.
Required, not incidental — Akka.Remote address matching means ClusterClient could
not reach a differently-named system. Site nodes also carry two roles, base plus
site-{SiteId}, with singletons scoped to the site-specific one.
Neither inter-cluster transport is authenticated or encrypted: no Akka TLS or
secure cookie, gRPC is h2c, and LocalDbSyncAuthInterceptor gates only the LocalDb
sync path — so the whole SiteStreamService surface, including the two Pull RPCs
returning audit rows, is reachable by anyone who can hit :8083. The boundary
assumes a trusted network; that assumption deserves to be explicit.
Also promotes two things from docs/ into CLAUDE.md because they change decisions:
the default 128 KB Akka frame size with log-frame-size-exceeding off and no custom
serializer (a silent single-message drop that leaves the association healthy), and
the registered two-node keep-oldest total-outage gap, whose own drill has a mode
existing "to make the registered gap observable — not to pretend it is covered."
Committed to a branch rather than main; merge at your discretion.
Picks up two rebuilt-peer replication fixes found by the OtOpcUa LocalDb
Phase 1 live gate. They matter more here than in OtOpcUa, which replicates 2
tables to this repo's 10:
- 0.1.2 — a converged pair prunes its oplog to empty on ack, and snapshot
detection read that as 'no gap possible', so a node whose database was lost
rejoined empty and stayed empty until the next deploy.
- 0.1.3 — with back-fill working, the rebuilt node's own writes were silently
dropped until its restarted seq counter climbed past the peer's stale
watermark.
Build clean; SiteRuntime 512 green.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two fixes, both about a node whose LocalDb file is lost. They matter more here
than in OtOpcUa, which replicates 2 tables to this repo's 10.
0.1.2 — a converged pair prunes every oplog row on ack, and snapshot detection
read an empty oplog as "no gap possible". The steady state of a healthy pair
was the one state from which a rebuilt node could never be healed: it rejoined
empty and stayed empty until the next deploy.
0.1.3 — with back-fill working, the rebuilt node's OWN writes turned out to be
silently dropped: last_applied_remote_seq is a watermark in the peer's seq
space, and a rebuilt peer numbers from 1, so the healthy node's stale watermark
made the sender skip its whole oplog.
Both found by the OtOpcUa LocalDb Phase 1 live gate on the docker-dev rig; the
second only became reachable once the first was fixed.
Build clean; SiteRuntime 512, SiteEventLogging 70 green (Host 330,
HealthMonitoring 97 green on 0.1.2, unchanged by the pin).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
An audit row whose tracking snapshot cannot be resolved is skipped and left
Pending, on the reasoning that central reconciliation will pick it up. Nothing
removes it from the local drain queue, so the next tick re-reads it, fails
identically, and logs again — forever. Measured on the rig at ~2,800
warnings/minute, sustained across both a process restart and a container
restart, until the audit database itself was discarded.
The code comment already names the cause ("the tracking store was reset"), so
this is an anticipated input, not an exotic one. Two production triggers need
no operator error: the tracking retention window elapsing before the drain
catches up (long central outage, large backlog), or restoring one store
independently of the other. The two stores are easy to desync because they live
in different places — audit in auditlog.db, which on the docker rig is INSIDE
the container, and tracking in the bind-mounted LocalDb database.
Skipping the row is right; leaving it Pending with no other state change is
not. There is no attempt counter, no backoff, no terminal state, and one
Warning per row per pass. Suggested fixes ranked by effort, the cheapest being
to rate-limit the warning the way MaintenanceBackgroundService already does for
oplog caps.
No data loss — the audit rows are intact and still reach central by the
reconciliation path. Found while cleaning the rig after the Phase 2 live gate.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Normative first: Component-StoreAndForward.md:83 specified the whole chunked,
ack-confirmed SfBufferSnapshotChunk resync protocol, which Phase 2 deleted. It
is rewritten rather than removed — the failure modes it reasoned about still
exist, they are just bounded differently — and it now states the
duplicate-delivery bound explicitly: a message can be delivered twice only when
the OLD primary delivered it and the status change had not yet replicated when
the gate flipped. One flush interval plus the in-flight ack, and unlike the old
model it does NOT grow with backlog depth or with how long a node was absent.
The N1 directional-authority and N5 orphan-row hazards are recorded as
structurally gone, not merely unguarded.
Frame-size known-issue amended: its 2026-06-26 resolution replaced the
intra-site hop with notify-and-fetch; Phase 2 then deleted notify-and-fetch
itself, so the 128 KB Akka frame constraint no longer applies to that hop in
any form. Successor ceiling recorded (4 MB gRPC cap via MaxBatchSize, which
batches by ROW COUNT), including that the failure mode differs — an oversized
gRPC message is rejected, not silently dropped.
Deployment docs gain the two operational constraints that have no home in code:
a site pair must be stopped and started TOGETHER (the SfBufferSnapshot compat
handler that made a mixed-version pair converge went with the replicator, and a
mixed pair now diverges silently), and a node offline beyond TombstoneRetention
can resurrect deleted rows on rejoin.
Both CLAUDE.md files corrected — each still said Phase 2 was NOT started.
Definition of done closed: build 0 warnings, all 10 suites green (3509 tests,
0 failures, 0 skips). Two DoD items needed amending rather than ticking: the
stale-symbol grep still matches 4 lines, all deliberate comment prose recording
what was deleted (a literal zero would delete the explanations that stop the
old design coming back), and the live gate has 10 evidence items, not the 9 the
checklist claimed.
Deletes the phase2 resume-state scratch doc, which said to delete it on landing.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
The blocker in the previous (incomplete) run was diagnosed: external systems
reach a site only through ArtifactDeploymentService, which `instance deploy`
never invokes. `deploy artifacts` delivered the probe harness AND propagated
the owed ExternalSystemDefinitions restore to both site nodes, unblocking
checks 5 and 10.
Highlights:
- 3+4: the deployed row is byte-identical on both nodes with the SAME
__localdb_row_version HLC and origin node id, and the standby logged zero
config fetches in the deploy window. B holds the row A wrote, by CDC alone.
- 6: stopped the active node; the standby took over in 10s and kept buffering,
oplog rose to 4 unacked while partitioned, drained to 0 on rejoin, and both
nodes ended byte-identical with ZERO duplicate ids (exactly-once).
- 7: the 3-table cascade converged with explicit TOMBSTONES on both nodes —
the rows are not merely absent on B, B applied the deletes.
- 9: both nodes stopped and started together; clean rejoin, zero SQLite I/O
errors.
Two caveats recorded rather than glossed: check 2's zero-count is vacuous on
its own (the legacy source was also empty) and rests instead on the ABSENCE of
CDC triggers for smtp_configurations/notification_lists; check 7's
native_alarm_state leg was empty live and is covered only offline.
Also records three method corrections — the plan's host-side sqlite3
instruction is unsafe, `docker exec curl` cannot scrape metrics from an image
with no curl (a silent failure that nearly became a false finding), and the
artifact-vs-instance deploy distinction above.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Checks 1, 2, 3, 4 and the oplog/dead-letter/metrics half of 8 captured and
passing. Check 5 blocked on rig state, checks 6, 7, 9, 10 not run. Task 21 must
NOT proceed on this evidence.
Strongest result is check 3+4 together: the deployed config row is byte-
identical on both site-a nodes with the SAME __localdb_row_version HLC and
originating node id, and the standby logged zero config fetches in the deploy
window. The standby holds the row node A wrote, obtained purely by CDC — which
is the whole point of deleting notify-and-fetch.
Check 5 is blocked by a rig-shaping problem, not a product fault: removing the
owed soakgen instances orphaned their 11,804 buffered sf_messages, and a
replacement probe's external system never reached the site nodes (site config
tables come from deployment artifacts, and an external system referenced only
by name inside script code is not carried). Same blocker stops check 10.
Two method corrections recorded in the doc:
- The plan says to run DB checks host-side against the bind mounts. That is
WRONG — a host sqlite3 open poisons the container's WAL. All reads here copy
the db/-wal/-shm triplet and query the copy.
- An apparent "localdb_* metrics missing" finding was an artifact of the
aspnet:10.0 image having no curl, with the error swallowed by 2>/dev/null.
Re-scraped via a network-sharing sidecar: the metrics are present.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
LocalDb 0.1.1 creates the parent directory of LocalDb:Path itself, so the
SiteLocalDbDirectory shim this repo carried through Phase 2 is deleted along
with its call site. The gap was found here but was never ScadaBridge's alone —
every LocalDb consumer had it — so the fix moved to the library.
SiteLocalDbDirectoryTests is RETAINED and retargeted rather than deleted with
the shim. It was already written against the site registration path, not the
mechanism, so it needed only its Ensure() call removed: what a site node
requires is that resolving ILocalDb not fail on a fresh machine, regardless of
who provides that. Verified it still earns its place — pinned back to LocalDb
0.1.0 it fails inside SqliteLocalDb..ctor -> SqliteConnection.Open(), so it
genuinely depends on the library behaviour and not on a coincidence.
Also corrects the coverage-split note in StoreAndForwardStorageTests, which
asserted that directory creation is "NOT LocalDb's" — true when written, wrong
as of 0.1.1.
Build 0 warnings; Host 330, StoreAndForward 130, LocalDb integration 20 pass.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
MaxBatchSize 500 -> 16. The default is a ROW count, not a byte budget, so the
batch size in bytes is set by the widest replicated column — config_json, which
Task 1 measured at up to ~60-70 KB in production. 70 KB x 500 is ~35 MB against
gRPC's 4 MB default receive limit; 16 keeps a worst-case batch near 1.1 MB.
MaxOplogRows 1,000,000 -> 250,000 and MaxOplogAge 7d -> 2d, sized from the
soak's 0.80 sf_messages rows/sec (~69k rows/day). Tighter than default is
correct here because exceeding a cap is not data loss: the oplog prunes to the
ceiling and sets needs_snapshot, so the peer catches up by snapshot resync
instead of incrementally. That trades a rare full resync for a bounded file.
site-b and site-c stay unreplicated, so the default-OFF posture is still proven
side by side on one rig.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Four scenarios over the real site pair, driven through the REAL
SiteStorageService rather than hand-written SQL. The sibling suites had to
hand-write SQL because they were specifications written BEFORE the cutover;
this suite runs after it, so it can drive the shipped writers — which is what
makes the cascade scenario meaningful.
- DeployedConfigRow_ConvergesToB_ColumnForColumn: the existing suite asserts
config_json only, so a capture that dropped or defaulted any other column
would still pass. deployed_at matters most — SiteReconciliationActor's
guarded write compares it.
- RemovingAnInstance_ConvergesAllThreeCascadeTables: the plan's flagged
highest-risk case. RemoveDeployedConfigAsync deletes from three tables in one
transaction, the schema has no foreign keys, and CDC ships three independent
per-table streams. A dropped delete leaves a permanently stale override or
alarm row on the standby, invisible until the instance name is redeployed.
Carries a never-removed control instance, without which "the cascade
converged" is indistinguishable from "node B lost these tables entirely".
- ANativeAlarmBurst_Converges_AndTheOplogDrains: convergence alone would pass
if entries replicated but were never acked, and an oplog that only grows
trips the caps into a snapshot resync.
- RowsWrittenOnBWhileItsListenerIsDown_SurviveTheRejoin: the union-survives
property is per-table, and an unregistered table is silently local-only
rather than an error, so it is re-proved on the tables the N1 scenario does
not touch.
Non-vacuity verified as the plan requires: with the eight Phase 2
RegisterReplicated calls commented out in SiteLocalDbSetup, all four go red
(4 failed / 0 passed); restored, 20/20 pass across the three LocalDb suites.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
LocalDb Phase 2 deleted the bespoke replicators, so three config keys changed
meaning or died outright:
- ScadaBridge:StoreAndForward:ReplicationEnabled is fully dead. Deleted the
property, its 10 config entries, and the 5 test references.
- SqliteDbPath / SiteDbPath are now migration-only: they name the legacy files
SiteLocalDbLegacyMigrator drains at boot, not live databases. Both mandatory
rules are relaxed accordingly (StartupValidator's Site-only Require, and the
S&F validator's non-empty rule) — an absent value now means "nothing to
migrate", so an already-migrated node can drop the key. DatabaseOptions-
Validator still rejects a present-but-blank value.
- SiteRuntime:ConfigFetchRetryCount's only reader was SiteReplicationActor.
Deleted with its validator rule.
The two path keys stay present in every config, now with a comment explaining
why: removing them would strand un-migrated data on a node that has not yet
started once.
Both relaxations are pinned by the inverse of the test they replace
(Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each
verified to fail with the old rule restored.
Note: deploy/wonder-app-vd03/appsettings.Site.json is under a gitignored
deploy/ tree, so its edit is local-only and must be repeated on the box.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Tasks 14, 15 and 16, landed as ONE commit.
PLAN DEFECT: these three tasks cannot compile separately. SiteReplicationActor
takes a ReplicationService and calls ReplaceAllAsync (Task 14 deletes both);
DeploymentManagerActor Tells message types declared in ReplicationMessages.cs
(Task 15 deletes it); AkkaHostedService constructs the actor (Task 16). Any
ordering leaves a broken intermediate. Combining them also strengthens the
invariant Task 14 already stated for itself — the two mechanisms never both
run, and never neither.
Registered 8 tables in SiteLocalDbSetup.OnReady: sf_messages plus the 7 site
config tables. notification_lists and smtp_configurations are deliberately NOT
registered — permanently empty by design, so registering them would open a
standing replication channel whose only historical payload was plaintext SMTP
passwords. Migrate stays the LAST call in OnReady, after all registrations, so
migrated rows enter the oplog through live capture triggers.
Deleted: SiteReplicationActor, ReplicationMessages.cs, ReplicationService,
StoreAndForwardStorage.ReplaceAllAsync, and 6 test files. ReplaceAllAsync is
not merely unused but unsafe to keep: a mass DELETE on a now-replicated table
would be captured and shipped to the peer.
Kept ActiveNodeEvaluator (delivery gate + heartbeat still need it) with its doc
corrected, and activeNodeCheck in AkkaHostedService (SiteCommunicationActor).
The positional-argument hazard the plan flagged was real: removing
DeploymentManagerActor's optional IActorRef? replicationActor shifted 6
trailing optionals, and 4 test call sites bound the wrong arguments with no
compile error at some positions. Converted them to named arguments where
possible — Props.Create builds an expression tree, which rejects out-of-position
named args, so the rest are padded positionally with a comment saying why.
The Task 7 'not yet registered' test was INVERTED rather than deleted, and is
exact in both directions: too few means a table silently stops replicating, too
many means the SMTP tables leak. Added a separate security-named test for those
two, and a composite-PK test (LWW keys on the full PK, so a truncated key set
would collapse distinct rows). The convergence suites now get their
registrations from the real OnReady — their temporary harness registration is
deleted, so they prove the cutover rather than agreeing with themselves.
Verified: build 0 warnings; SiteRuntime 512, StoreAndForward 130, Host 330,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, LocalDb
integration 16 — all pass, 0 failures.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 12 + Task 13. No production behaviour change in either.
Task 12: DeploymentManagerActor.HandleDeployArtifacts already purges
notification_lists and smtp_configurations on every artifact apply, but nothing
pinned the actor's CALL to it — ArtifactStorageTests covers the storage method
only. Task 15 deletes SiteReplicationActor's copy, making this the sole
remaining call site, and Task 16 edits this actor's wiring; dropping the call
would leave plaintext SMTP passwords on disk with every suite still green.
Verified red-first by commenting the call out: the pin fails with that message.
Task 13: StoreDeployedConfigIfNewerAsync STAYS. Re-verified both callers —
SiteReplicationActor:375 (dies at Task 15) and SiteReconciliationActor:166
(survives). Reconciliation is a per-node startup self-heal against central
whose fetch races real deploys, so the deployed_at guard still does real work
there. Doc comment rewritten to say so, and to warn against porting the guard
onto the replication path where it would fight the HLC rather than help it.
Also corrected a stale 'guarded standby write' section header in the tests.
Re-ran the Task 13 step 2 scope check: ConfigFetchRetryCount's only production
reader remains SiteReplicationActor:157, so its option + validator rule stay
until Task 17, after Task 15 deletes the actor.
Verified: build 0 warnings; SiteRuntime 533, Host 329, StoreAndForward 153,
LocalDb integration 16 — all pass.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 11. Companion to Task 10, covering the intents held by
SiteReplicationActorTests and SfBufferResyncPredicateTests.
N1 Critical is re-expressed, not ported literally (D2). SfBufferResyncPredicate
existed because ReplaceAllAsync was a destructive DELETE-then-INSERT, so a
wrong-direction resync wiped a live buffer and the code needed an oldest-Up
predicate to decide authority. LocalDb's snapshot resync merges per row under
LWW and never deletes, so this asserts the property the guard protected — no
node loses rows to a peer's snapshot — with no directional-authority assertion,
because there is no active/standby asymmetry left to enforce. The setup is the
wipe scenario made concrete: each node writes rows the other never sees while
partitioned, plus a contended row, then they resync.
DEVIATION on the zero-fetch assertion. The plan asked for a fetcher test double
recording zero invocations; this harness has no actor system, no central and no
IDeploymentConfigFetcher in the graph, so the double would record zero calls
whether or not the fetch path still existed. An assertion that cannot fail is
worse than none, so the test proves the positive half — config reaches B over
replication alone — and the file records that the negative half is proved by
Task 15 deleting the code and the build still passing. It also records the D1
scope note: SiteReconciliationActor survives and legitimately fetches at
startup, so 'the standby never fetches, ever' would be false.
Non-vacuity verified by unregistering deployed_configurations: all 4 fail.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 10. Specifications first, deletion second: the bespoke ReplicationService
(explicit Add/Remove/Park/Requeue over Akka) dies at Task 14, so its behaviour
is restated here as outcomes the CDC replacement must still deliver. Written
in terms of ROWS, not operations — under CDC there is no Add or Park message to
observe, only a row that must end up right on both nodes.
Not ported: ReplicationOperations_AreDispatchedInIssueOrder. It asserts the
mechanism (inline fire-and-forget dispatch), and CDC capture is asynchronous
and batched by construction. Its portable content is the ordering OUTCOME —
add-then-remove must never converge to present — which is a test here, with
that reasoning recorded in the file so it does not read as an accidental drop.
DEVIATION: extracted the Phase 1 fixture into LocalDbSitePairHarness rather
than duplicating ~150 lines. Phase 1's tests now derive from it and still pass
unchanged. The harness registers the Phase 2 tables itself, since production
OnReady does not until Task 14; that method is marked for deletion at the
cutover, and the 8-table list is written literally so a cutover registering the
wrong set fails these tests instead of agreeing with itself.
Non-vacuity verified by unregistering sf_messages: 6 of 7 failed. The 7th —
the ordering test — PASSED, because an absent row is also what a pair that
replicates nothing looks like. Fixed with a control row that must converge in
the same window, so the absence is evidence rather than silence.
Also corrected two comments from Task 9 that claimed Task 14 makes
notification_lists/smtp_configurations replicated. It explicitly does not
register them, for the same reason the migrator skips them.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 9. Seven of the nine site configuration tables are copied out of the
legacy scadabridge.db in a single transaction, with one rename at the end: a
partial config migration would leave a site node running against half its old
configuration, which is worse than failing startup outright.
notification_lists and smtp_configurations are deliberately NOT migrated.
Both are purged on every deploy and permanently empty by design since the
site write paths were removed (2026-07-10), but a pre-fix legacy file can
still hold rows — and smtp_configurations.password is plaintext. Task 14
makes these tables replicated, so migrating them would push plaintext SMTP
passwords across a channel whose only historical payload was exactly that.
The tables are still created; only their historical contents stay behind.
Generalizes Task 8's MigrateTable into MigrateFile(many tables, one
transaction, one rename), keeping the legacy/current column intersection.
Five tests, including a column-parity test asserting each declared column
list equals the live SiteStorageSchema's. That mismatch is otherwise
invisible: a typo'd column is silently dropped by the intersection, and a
missing one silently leaves data behind. Non-vacuity verified twice — once by
removing the Migrate call (3 fail), once by wrongly adding notification_lists
and smtp_configurations to the table map (the skip test fails).
Verified: solution build 0 warnings; Host 329, SiteRuntime 532,
StoreAndForward 153 — all pass.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 8. Adds ResolveStoreAndForwardPath + MigrateStoreAndForward, wired as a
third call in Migrate alongside the two Phase 1 files.
Unlike the Phase 1 paths, the store-and-forward default sits INSIDE the data
volume, so a real deployment has a real file here holding undelivered
messages. This migration genuinely moves data; losing it would discard exactly
the buffered calls store-and-forward exists to protect.
No id synthesis: sf_messages.id is already a caller-assigned TEXT primary key,
so INSERT OR IGNORE is idempotent across a crash-then-rerun.
The copy intersects the legacy column set with the current one rather than
naming all 16 columns outright. A file from an older build predates
execution_id / parent_execution_id / last_attempt_at_ms, and naming a missing
column throws 'no such column' — which the existing reader treats as an
unrecognised shape and silently discards every row. A required-column (PK)
guard keeps that tolerance from degrading into copying NULL-keyed rows.
Four tests: copy+rename, crash-before-rename idempotence, the __localdb_oplog
assertion that catches migrate-before-register, and the older-column-set case.
All four verified to fail without the Migrate call.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 7. SiteLocalDbSetup.OnReady now applies SiteStorageSchema and
StoreAndForwardSchema alongside the Phase 1 schemas, so the nine site
configuration tables and the store-and-forward buffer live in the
consolidated LocalDb file.
Deliberately NOT registered for replication. The bespoke SiteReplicationActor
and the StoreAndForward ReplicationService still own these tables until the
Task 14 cutover deletes both and registers them in one commit; registering
early would run two replicators over the same rows and let either one's
defects hide behind the other's writes.
Pinned by two tests through the real composition root: one asserting all
twelve tables exist, one asserting ReplicatedTables is EXACTLY the Phase 1
pair. Non-vacuity verified by removing the DDL and observing the failure.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Records the latent Phase 1 directory defect and its open library-vs-app decision,
the CreateConnection contract change, the new TestSupport library, and the
verification numbers at the wave-2 boundary.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Tasks 5 and 6 of the Phase 2 plan, committed together because their test
fallout is entangled — several fixtures construct both stores.
StoreAndForwardStorage and SiteStorageService now take ILocalDb. Connections
come from ILocalDb.CreateConnection(), which hands out an already-open,
pragma-configured connection carrying the zb_hlc_next() UDF the capture triggers
call; a raw connection would lack the UDF and every write to a replicated table
would fail closed. Deleted with the connection strings: S&F's
EnsureDatabaseDirectoryExists and its per-open busy_timeout pragma, and the site
service's BusyTimeoutFloorSeconds normalization — LocalDb owns all of it now.
DI: AddSiteRuntime's string overload is gone (nothing left to supply), so the
Host calls the no-arg form. ScadaBridge:Database:SiteDbPath and
StoreAndForwardOptions.SqliteDbPath survive only as the migrator's source
locations in Tasks 8/9.
Two things the plan did not anticipate, both worth reading:
1. FOUND A REAL LATENT DEFECT, from Phase 1, now fixed. The plan assumed
directory creation simply moved to LocalDb along with file ownership. It did
not: the LocalDb library never creates the parent directory, and
SqliteLocalDb opens the file eagerly in its constructor — so a missing
directory is a hard boot failure ("SQLite Error 14: unable to open database
file"), not a degraded start. The default site config points at the RELATIVE
path ./data/site-localdb.db, so any site node without a pre-existing data/
directory fails to boot. The docker rig escapes only because its volume mount
happens to create /app/data — a coincidence that would have hidden this until
a bare-metal or fresh deployment. This has been latent since Phase 1 made
LocalDb:Path required; deleting S&F's EnsureDatabaseDirectoryExists here
would have widened it. Re-established the guarantee at the layer that now
owns the path (SiteLocalDbDirectory.Ensure, called before AddZbLocalDb) and
pinned it with SiteLocalDbDirectoryTests. Non-vacuity is not assumed: two
tests written against the wrong assumption failed with exactly this
SQLite Error 14 before the fix existed.
2. Test fallout was ~7x the plan's estimate. The plan named "fixtures" in one
project; the constructor change actually reaches 40 files across 7 test
projects, and most used Mode=Memory;Cache=Shared — which LocalDb has no
equivalent for, so every one had to move to a real temp file. Rather than
copy the Phase 1 TestLocalDb fixture into 7 projects, added a shared
tests/ZB.MOM.WW.ScadaBridge.TestSupport library (not a test project) so the
WAL-sidecar cleanup and the "real, not stubbed" rationale live in one place.
Retargeted rather than deleted, in both directions: the S&F WAL test now asserts
against the LocalDb-backed store (WAL genuinely is LocalDb's job), while the
directory-creation test moved to Host.Tests (that guarantee is NOT LocalDb's).
SiteStorageServiceTests.Initialize_EnablesWalJournalMode got the same treatment.
DeploymentManagerMediumFindingsTests induced a persistence failure via an
unopenable path, which no longer reaches the assertion since the fixture now
throws first; it induces the same failure shape via an uninitialized store.
Verified: full solution build 0 warnings; SiteRuntime 532, Host 318,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97,
StoreAndForward 153 — 1597 passed, 0 failed.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 4 of the Phase 2 plan. All nine table definitions plus MigrateSchemaAsync
and TryAddColumnAsync move into a static sync Apply(SqliteConnection) depending
only on Microsoft.Data.Sqlite, so the Host can apply the DDL to a LocalDb-managed
connection before RegisterReplicated installs the capture triggers.
Per the plan, PRAGMA journal_mode=WAL stays in InitializeAsync — LocalDb owns
the connection's pragmas, and the service still opens its own connections until
Task 6.
One deviation: TryAddColumnAsync's `catch (SqliteException) when
(ex.Message.Contains("duplicate column"))` becomes a PRAGMA table_info probe,
matching OperationTrackingSchema. The message-matching form depends on an error
string that is not part of SQLite's contract, and it swallowed every
SqliteException whose message happened to contain that substring. The probe also
drops the ILogger dependency, which is what let the class become static. Cost:
the per-column "Migrated: added column" info log is gone — it fired once per
column per legacy database and nothing consumes it.
Also added a test beyond the plan's specified one, for the same reason as Task 3:
the specified test asserts against a freshly-created database, where CREATE TABLE
already lists the migration columns, so it would pass with every ALTER deleted.
The added test starts from the pre-migration shapes with a row present and proves
the migration path runs and preserves data across a NOT NULL DEFAULT add.
SiteRuntime suite 532 passed / 0 failed; full solution build 0 warnings.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 3 of the Phase 2 plan. Mirrors OperationTrackingSchema: the sf_messages
DDL, the four additive ALTERs, the last_attempt_at_ms backfill and the due-index
move verbatim into a static sync Apply(SqliteConnection), depending only on
Microsoft.Data.Sqlite. The Host needs to apply this to a LocalDb-managed
connection before RegisterReplicated installs the capture triggers; the store
still calls it so a directly-constructed store stays self-sufficient.
Two deviations from the plan as written, both deliberate:
1. The PRAGMA journal_mode=WAL in InitializeAsync STAYS. The plan's Step 4
snippet drops it, but Task 4 explicitly says not to move the equivalent
pragma ("LocalDb owns the connection's pragmas") — the two tasks contradict
each other. Keeping it preserves behaviour for the intermediate commits and
for directly-constructed stores; it becomes moot in Task 5 when the store
stops opening its own connections. Dropping it now would quietly regress the
concurrent-writer support the pragma's own comment documents.
2. Added a second test beyond the plan's. The specified test asserts only
against a freshly-created table, where CREATE TABLE already lists all 16
columns — it would pass with every ALTER deleted. The added test starts from
the pre-upgrade 12-column shape with a row in it and proves the upgrade path
runs and preserves data.
That second test also surfaced that the backfill lands 1 ms low: julianday()'s
double day-fraction cannot represent every millisecond. Pre-existing behaviour,
carried over verbatim, asserted with a 1 ms tolerance rather than pinning a
precision the implementation never had.
Suite: 154 passed, 0 failed.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 2 of the Phase 2 plan, plus the clean re-run of Task 1's cut-short
sampling that Task 2 was waiting on.
Gate doc: status NOT STARTED -> CLOSED. All five §5 open questions answered
inline. The questions are kept, not deleted — several of the answers overturn
a premise the gate itself stated, and that reasoning is the value:
- Oplog sizing is not the binding constraint. Cap overrun prunes and flags
needs_snapshot (graceful resync), so the caps need no defensive sizing; the
real ceiling is D6's 4 MB gRPC cap, binding on MaxBatchSize x row-bytes.
- LWW-vs-in-flight-send cannot arise on a correct pair (only the active node
sweeps). The honest cost is D2's semantic change: the standby is convergent,
no longer byte-identical.
- Migration-under-load is designed out, not managed — both mechanisms are
deleted in the same commit and D5 forecloses rolling upgrades.
- Rollback is genuinely "revert the commit": the migration is additive and
leaves the legacy files intact. Bounded cost is the post-cutover delta.
- One DB per process stays; every write axis is bounded.
§2's requirement to state CDC's duplicate-delivery bound is routed explicitly
to Task 21 rather than left implicit.
Task 1 re-run (safe copy-based snap(), both nodes restarted first, six 60s
intervals): sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0,
alarms 0, max payload_json 76 B, zero SQLite errors across 30 min, both site-a
nodes converged at 3564 rows. Independently confirms the disk-I/O storm was
observer-induced. Recorded with its limits stated: 0.80/s is ~1.6% of the 50/s
ceiling and the retry path was never exercised — acceptable only because that
ceiling is structural rather than empirical, and the rig's 721 B config rows
are explicitly NOT representative.
GATE VERDICT: PROCEED to Task 3. Binding output is MaxBatchSize 500 -> 16.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Scratch handoff so the plan can be picked back up cleanly: Task 1 done and its
STOP verdict superseded, what must happen before Task 3, the four plan-premise
corrections, the rig gotchas that are not discoverable from the repo, and the
rig cleanup the Task 20 live gate needs.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
The Phase 2 soak's "LocalDb fails under load" blocker is NOT a product defect.
Root cause: host-side (macOS) sqlite3 reads of the live, bind-mounted WAL
databases. POSIX advisory locks do not propagate across the virtiofs boundary,
so the host reader believes it is the only connection, checkpoints on close and
resets the WAL to 0 bytes under the container. The container's still-mapped
WAL index then references frames that no longer exist and every subsequent
statement fails SQLITE_IOERR_SHORT_READ (522) -> primary code 10, permanently
until the process reopens the database.
Reproduced on demand both on the rig (one sqlite3 SELECT reset a 4.6 MiB WAL and
produced the first error one second later) and in a minimal python:3.12-alpine
repro with no LocalDb or .NET involved. Refuted the converse: a freshly-reopened
node sustained the full soak load 10+ minutes with zero errors.
The original brief's isolation was confounded - both nodes had been poisoned by
the same sampling pass, and a poisoned standby looks healthy only because it
issues almost no statements. Corrections annotated in place.
Hardening shipped alongside:
- SqliteErrorCodes.Describe: log SQLite primary AND extended codes at the
LocalDb-adjacent catch sites (the missing extended code is what made the
original diagnosis so slow).
- SiteAuditTelemetryActor: stop touching ActorContext across an await
(NotSupportedException), with regression coverage.
- infra/reseed.sh: apply the MSSQL init scripts explicitly. The official
mssql/server image does not implement /docker-entrypoint-initdb.d, so a fresh
volume hung the reseed forever; compose mounts annotated as informational.
Deliberately NOT done: detect-and-reopen self-heal in SqliteLocalDb. It defends
only against external interference, which is now prevented at the source, and
same-kernel production readers see the locks correctly.
Build 0 warnings; SiteAuditTelemetryActorTests 9/9, SiteEventLogging 70/70.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Standalone handoff for a follow-up investigation. Records the symptom, a full
reproduction (including the two rig-tooling blockers and the CachedCall-vs-Call
detail needed to generate load at all), the isolation evidence, a code map of
the LocalDb connection model, four ranked hypotheses, and acceptance criteria.
Not root-caused. Highest-value next step identified: capture the SQLite EXTENDED
result code (logs only carry the generic primary code 10), and run the load with
LocalDb:Path off the bind mount to partition environmental vs library causes.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 1 of the Phase 2 plan. The gate stops the plan, but not for the reason it
anticipated: oplog sizing is fine and D6 is resolved.
BLOCKER: the Phase 1 consolidated LocalDb (site-localdb.db) throws SQLite
Error 10 'disk I/O error' on essentially every write on the ACTIVE node under
sustained load — site_events, OperationTracking and the audit telemetry paths
all fail, and site event logging silently drops events. Isolated to the load
(not the node, not host-side observation) by failing over between nodes, and to
LocalDb specifically (legacy store-and-forward.db / scadabridge.db in the same
bind-mounted directory take zero errors under identical load).
Phase 2 would register 8 more tables into that database — including the two
highest-volume ones — while deleting the bespoke mechanisms that currently
carry them. Must be root-caused first.
Also recorded: D6's premise corrected (largest known production config_json is
~60-70 KB, not >128 KB — but MaxBatchSize 500 is still unsafe, use 16); D4's
premise corrected (alarm writes are bounded by per-SourceReference coalescing at
a 100 ms flush); sf_messages has a hard 50 rows/sec structural ceiling; and
exceeding the oplog caps is a graceful snapshot-resync, not a failure.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
docker/seed-sites.sh inserted the pre-rename 'Design' / 'Deployment' role
strings, but the canonical vocabulary in Roles.cs is 'Designer' / 'Deployer'.
The mismatch authorized nothing: on a freshly reseeded rig every
Designer/Deployer-gated management command failed UNAUTHORIZED, including
seed-sites.sh's own trailing `deploy artifacts` and reseed.sh's stage 6d
encrypted-secret restore.
Found while standing up the Phase 2 soak rig.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Three verification sweeps over SiteRuntime, StoreAndForward, and Host/rig/docs
plus the LocalDb library source. Load-bearing corrections: D1 (the guarded write
has a second surviving caller, SiteReconciliationActor), D3 (the active node
already purges — Task 12 becomes a pin), D6 (new: the 4 MB gRPC cap vs
row-count-only batching), Task 1 (rewritten method — the Phase 2 tables are not
in the Phase 1 oplog), and Task 14 (do not register the notification/SMTP tables).
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Moves scadabridge.db's 9 config tables + sf_messages into the consolidated
LocalDb file and deletes SiteReplicationActor + StoreAndForward's
ReplicationService.
Resolves the phase 2 gate's five open questions from code recon (D1-D5):
D1 notify-and-fetch is DELETED, not preserved. It exists only because the
config blob exceeds Akka's 128KB frame; LocalDb sync is gRPC. The
deployed_at version guard protected against a stale fetch racing, so it
dies with the fetch. Do not reproduce it on LWW - different clocks.
D2 ReplaceAllAsync is deleted and the N1 directional guard becomes
unnecessary: LocalDb's snapshot resync merges per-row LWW and never
deletes (SnapshotApplier has no DELETE; LwwApplier.cs:69-78 discards a
lower-HLC incoming row). Semantic change - the standby is convergent,
no longer byte-identical.
D3 The SMTP purge (plaintext passwords) rides the replication path and is
re-homed to the active node BEFORE any deletion.
D4 native_alarm_state volume is measured by a rig soak, not assumed. Task 1
gates the plan and stops it if the oplog cannot absorb the churn.
D5 No dual-mechanism period forecloses rolling site upgrades - both nodes
must stop and start together.
Recon also found the gate doc's "two test files are the spec" undercounts:
the real specification is five files, including the N1 Critical regression
test and the only Requeue coverage.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 14. A GATE document, not an implementation plan - the plan explicitly wants
Phase 2 planned against post-Phase-1 reality, which means after this record
exists rather than from the original design alone.
Records what Phase 1 actually established (ordering constraints, where
registration lives and why, the fail-closed interceptor, the silent meter
allowlist), what must be ported before anything is deleted (both bespoke
replicator test files, treated as specifications, plus the accepted N5
bounded-duplicate race), and why Phase 2 is a harder risk class than Phase 1:
it REPLACES a working mechanism rather than adding one where none existed,
sf_messages has external side effects, config tables drive deployment, there is
no dual-mechanism period, and the migration is real this time rather than the
usual no-op.
The most important line is an open question, not an answer: choosing
MaxOplogRows / MaxOplogAge / snapshot-resync thresholds for sf_messages write
rates needs a Phase-1 rig SOAK that has not been run. The live gate proved
correctness, not steady-state behaviour over time. Phase 2 should not be planned
without it.
tasks.json finalized: all 15 tasks completed, every deviation from the plan
recorded with its reason.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 13.
SiteEventLogger's XML doc said 'Not replicated to standby. On failover, the new
active node starts a fresh log.' That is now false and was the exact behaviour
Phase 1 set out to fix.
OperationTrackingOptions.ConnectionString and SiteEventLogOptions.DatabasePath
marked MIGRATION-ONLY: nothing reads them but SiteLocalDbLegacyMigrator, and
they are safe to delete from a config once a node has migrated.
ScadaBridge CLAUDE.md gains a consolidated-site-database entry covering the
GUID/opaque-token contract change, the migration-only keys, the incidental
CWD-outside-the-volume data-loss fix, the fail-closed default-OFF replication
posture, and the fact that ZbTelemetryOptions.Meters is a silent allowlist.
scadaproj CLAUDE.md's LocalDb row goes from 'no app adoption yet' to the honest
current state, per the component-status-honesty convention: ScadaBridge Phase 1
only, on an UNMERGED branch, replication live-proven on the rig's site-a pair
ONLY, default-OFF everywhere else, Phase 2 not started, no other app adopted, no
production deployment replicating.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 12 (live gate on the docker rig) found one real defect, which this fixes.
THE DEFECT. The plan asserted LocalDb metrics "need no work - the
ZB.MOM.WW.LocalDb.Replication meter flows through the existing AddZbTelemetry
Prometheus export". It does not. ZbTelemetryOptions.Meters is an ALLOWLIST: an
unlisted meter's instruments are never observed, with no error, no warning, and
no missing-metric signal anywhere. The rig's /metrics scrape returned 301 lines
and zero localdb_* series. Only looking for them found it.
Fixed by hoisting the allowlist to SiteServiceRegistration.ObservedMeters and
adding LocalDbMetrics.MeterName. The pin test asserts on that constant rather
than on OTel's observed set, because AddZbTelemetry applies its options to a
local instance and never registers IOptions - there is nothing resolvable to
interrogate. The end-to-end proof is the live scrape below; the test exists to
stop the entry being dropped again. Called out as such in the test.
LIVE GATE EVIDENCE (docker rig, 8 nodes, rebuilt via docker/deploy.sh):
1. Consolidated DB created on every site node at /app/data/site-localdb.db -
inside the mounted volume, unlike the legacy CWD-relative databases.
2. Replication proven REAL, not two independent local writes. Both site-a nodes
hold the identical row and, decisively, the identical originating node_id and
HLC in __localdb_row_version:
site_events|{"id":"8b06b37c..."}|116946936661147648|65b00319-...|0
__localdb_peer_state shows a completed handshake in both directions.
3. Convergence: 5/5 events on both nodes, __localdb_row_version byte-identical
across the pair, oplog drained to 0, dead_letter 0 on both.
4. SITE failover drill. Note the repo's failover-drill.sh targets CENTRAL nodes
and does not exercise this claim, so the site-pair flip was run directly:
stopped site-a-a (the node holding the history); site-a-b survived with the
complete pre-flip history (3/3 events) plus its own takeover event. Restarted
site-a-a; it caught up on what it missed and both nodes reconverged to the
same 5 ids. This is the failure Phase 1 exists to prevent.
5. localdb_* now exported after the fix:
localdb_sync_reconnects_total{...} 1
localdb_oplog_depth{...} 0
6. DEFAULT-OFF pin, live and side-by-side: site-b (no Replication section) boots
identically, creates its consolidated DB, registers the engine - and stays
inert. Zero replication log lines, no reconnect counter at all, so it never
dialled anything.
One transient WRN at startup on site-a-a ("Connection refused", reconnecting) is
node A dialling before node B's listener was up; it reconnected within a second.
Expected for a pair that starts together.
Verified: build 0 warnings; Host wiring tests 11/11.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 11. site-a-a is the initiator (dials scadabridge-site-a-b:8083, the
existing site gRPC h2c listener - no new port); site-a-b is passive. Both carry
the SAME dev ApiKey, which is load-bearing: LocalDbSyncAuthInterceptor is
fail-closed, so a mismatch does not degrade to unauthenticated replication, it
rejects every stream and the pair silently stops converging.
site-b and site-c stay unreplicated on purpose, so the default-OFF posture is
proven side-by-side on one rig rather than asserted.
Plain dev key matches the rig's existing posture; production uses a
${secret:...} reference through the pre-host expander. Noted in both files.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 10 of the LocalDb Phase 1 adoption plan. Two ScadaBridge site nodes
replicating the consolidated site database over loopback Kestrel h2c, through
the REAL fail-closed auth interceptor - not a stand-in.
This is the test that answers the question Phase 1 exists to answer. Every piece
upstream of it - schema helpers, DI wiring, the interceptor - can be
individually green while the pair still fails to converge.
It initializes both databases through SiteLocalDbSetup.OnReady rather than a
hand-written schema, so the tables, primary keys and registration ORDER under
test are the ones the host actually runs. A local schema would prove only that
the test agrees with itself.
Scenarios: A->B, B->A (bidirectional even though only A dials, which is what
makes failover safe in either direction), LWW convergence on a contended
operation, the event union across both nodes, and a peer-offline-then-rejoin
catch-up.
The event-union scenario is the one that pins the GUID id change: under the old
autoincrement scheme both nodes independently mint id=1,2,3..., and
last-writer-wins on the primary key would silently DESTROY one node's events
rather than merge them. Asserting the union count is what catches that.
The whole suite passes in ~0.6s, which is fast enough to be suspicious of a
vacuous pass, so it was verified red-first the hard way: deliberately mismatching
the two nodes' ApiKey turns all 5 red with 2m30s of timeouts. The tests really do
run through the interceptor and the real transport.
Node B's database survives its host teardown (registered as a pre-constructed
instance, which MS.DI does not dispose), so the rejoin scenario is a genuine
rejoin rather than a fresh node.
Offline - no docker, no external services.
Verified: build 0 warnings; IntegrationTests 80/80.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 6 of the LocalDb Phase 1 adoption plan. Copies site-tracking.db and
site_events.db into the consolidated database, then renames each source to
<name>.migrated.
Runs AFTER RegisterReplicated, deliberately: capture is trigger-based, so rows
inserted before registration would never enter the oplog and would never reach
the peer - silently. There is a test asserting migrated rows land in
__localdb_oplog, because every other test in the file would still pass with that
ordering reversed.
Idempotent twice over. Tracking rows keep their existing TEXT primary key, so
INSERT OR IGNORE dedupes naturally. Legacy events had autoincrement integer ids,
which the consolidated schema replaced with GUIDs; they get DETERMINISTIC ids of
the form mig-{NodeName}-{legacyId} rather than fresh GUIDs, so a migration that
crashes between commit and rename cannot duplicate history on the next boot. The
NodeName prefix keeps the two nodes of a pair from colliding on their independent
legacy id sequences. Both properties are covered, including an explicit
crash-window test that undoes the rename and re-runs.
All-or-nothing: the copy commits in one transaction and the rename happens only
afterwards, so a failure throws out of OnReady, fails host startup, and leaves
the legacy files untouched. Deviation from the plan: it specified
BeginTransactionAsync, but OnReady is a synchronous Action<ILocalDb>. A raw
transaction on a CreateConnection() connection gives identical atomicity and
identical trigger capture without blocking on async in a startup callback.
Legacy paths come from the OLD config keys and fall back to the CWD-relative code
defaults, because that is where the old code actually wrote them. Note this means
both legacy databases have always lived outside the mounted data volume and were
already discarded on every container recreate - so on the rig this migrator will
usually find nothing, which is a legitimate no-op rather than a failure. Phase 1
incidentally fixes that data-loss bug.
Non-failure cases are covered too: missing files, a legacy file with no such
table, and in-memory legacy connection strings from test/dev configs.
Verified: build 0 warnings; Host 316/316 (9 migrator tests new).
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Tasks 8 and 9 of the LocalDb Phase 1 adoption plan.
Task 8 - LocalDbSyncAuthInterceptor. The replication library's LocalDbSyncService
verifies nothing; inbound auth is explicitly the host's job. Without this,
anything able to reach a site node's gRPC port could stream arbitrary rows into
the consolidated site database - including OperationTracking, which central
reconciles from.
Scoped strictly to /localdb_sync.v1.LocalDbSync/; SiteStream shares the same
AddGrpc pipeline and passes through untouched. Fail-closed: with no
LocalDb:Replication:ApiKey configured NO sync stream is accepted, authenticated
or not. That is deliberate - "no key" is the default every site node ships with,
so treating it as "no auth required" would expose the endpoint on precisely the
most common configuration. Comparison is FixedTimeEquals over UTF-8 bytes.
All four server handler shapes are gated, not just unary: the sync RPC is a
bidirectional stream, so gating only the unary path would leave the real endpoint
open while every unary test still passed. There is a test for that.
Deviation from the plan: it specified Grpc.Core.Testing for the fake
ServerCallContext. That type ships in the retired native Grpc.Core package and
does not exist on the grpc-dotnet stack this solution uses; a minimal
FakeServerCallContext in the test file was the better trade than adding a dead
dependency.
Task 9 - ISyncStatus onto the site health report as LocalDbReplicationConnected
and LocalDbOplogBacklog, via a delegate-seam hosted service following the
AddSiteEventLogHealthMetricsBridge precedent (HealthMonitoring takes no reference
on the replication library). Both are additive init properties, so the
Akka-remoted SiteHealthReport constructor signature is untouched.
Both fields are nullable and the distinction is load-bearing:
- null = the reporter has not run / replication is not wired ("no data");
- false/0 = a real reading. On a node with no peer that IS the healthy
default-OFF state, not an outage.
OplogBacklog is passed through nullable end-to-end because ISyncStatus returns
null when the poll fails - flattening it to 0 would report a replication pair
that cannot read its own oplog as perfectly healthy. The collector stores both
values as one tuple and CollectReport reads it once, so a torn read cannot pair a
fresh Connected with a stale backlog.
Verified: build 0 warnings; Host 307/307 (8 interceptor + 3 health tests new),
HealthMonitoring 97/97, Commons 684/684.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Joseph Doherty
dohertj2