Rebuilt scadabridge:latest from main @ 8524a7f7 and recreated only the
env2 containers. All three gate checks pass on site-x:
1. both site nodes boot with the key (StartupValidator fail-closed →
reaching 'Application started' proves the key present);
2. control-plane PSK auth: no-header / wrong-key ⇒ PermissionDenied,
correct key ⇒ success, on both nodes (:9123, :9124);
3. LocalDb unaffected (local-only; 0 errors, healthy boot).
Bonus: central registers site-x online via gRPC heartbeat; no real
ClusterClient/receptionist (only the benign ClusterClientSiteAuditClient
label, same as the primary rig). Noted a seed-data gap (ScadaBridgeConfig2
dbo.Sites is empty) — orthogonal to the transport.
ClusterClient→gRPC migration Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md).
Phases 2/3 proved both directions on gRPC; this removes the Akka transport underneath.
Deleted:
- AkkaCentralTransport, AkkaSiteTransport (+ their dedicated tests)
- ISiteClientFactory + DefaultSiteClientFactory; CentralCommunicationActor legacy
ctor + SelectTransport (Host now builds GrpcSiteTransport and injects it)
- ClusterClient creation + both ClusterClientReceptionist.RegisterService calls in
AkkaHostedService; the RegisterCentralClient message + receive block
- CommunicationOptions.CentralContactPoints; the CentralTransport/SiteTransport
coexistence flags; the CentralTransportMode/SiteTransportKind enums
gRPC is now the only site↔central transport (site→central CentralControlService via
GrpcCentralTransport; central→site SiteCommandService via GrpcSiteTransport), both
built unconditionally by the Host. NoOpCentralTransport is the fail-loud null-default
so TestKit command-dispatch suites still construct the site actor without a wired
transport; production always injects GrpcCentralTransport.
Config: CentralGrpcEndpoints is now unconditional — CommunicationOptionsValidator
rejects blank entries (role-agnostic), and StartupValidator requires a Site node to
list >=1 endpoint (fail-fast, mirrors GrpcPsk). Rig configs moved
CentralContactPoints -> CentralGrpcEndpoints (docker x6, docker-env2 x2, Host default,
deploy/wonder-app-vd03). Kept Akka.Cluster.Tools (ClusterSingleton still used).
Tests: build 0/0; Communication.Tests 640, Host.Tests 421 green. Removed the
ClusterClient.Send per-site-routing tests (covered by the transport suites), swapped
the ISiteClientFactory-based ctors to a substitute ISiteCommandTransport, converted
the audit-push integration relay to an in-process bridge transport.
Docs: Component-Communication/Host/StoreAndForward, components/Communication,
topology-guide, grpc_streams (SUPERSEDED note), the frame-size known-issue (retired
amendment), and CLAUDE.md transport decisions.
Not included: the dead IntegrationCallRequest path (#32) is a separate user-owned
behavioral decision — SiteEnvelope routing is transport-agnostic so it still compiles.
Central flipped to SiteTransport=Grpc (both nodes, central-wide flag), sites
kept on gRPC from Phase 2 → both directions on gRPC at once. Central logs
'central→site command transport: gRPC (SiteCommandService)'; 0 ClusterClient-to-
site on either central.
Command matrix over SiteCommandService (all 200):
- ExecuteQuery (event-log) + ExecuteParked (parked) — all 3 sites
- ExecuteLifecycle disable/enable #95 on site-a — NEW live coverage vs 1B/P2
(SoakNotify instances survived the recreate as Enabled)
Resilience:
- hard-kill the ACTIVE site-a node mid-query-loop → in-flight call returned
TIMEOUT at exactly the 30s QueryTimeout deadline (bounded, no hang; deadline≠
retry — an in-flight call can't be safely re-sent)
- next + all subsequent queries recovered automatically via site-a-b
(SitePairChannelProvider NodeA→NodeB); site→central S&F never stopped
(notif 619→715, still no dupes)
- 0 PermissionDenied across all 8 nodes
ExecuteOpcUa/ExecuteRoute/TriggerFailover deferred (no OPC-bound instance / no
CLI verb; unit-proven). Rig left both-gRPC; git reverted to Akka default.
All 6 site nodes flipped to CentralTransport=Grpc; whole control plane
(heartbeat/health/notification S&F/audit) rides gRPC CentralControlService,
196 RPCs/90s 0 non-200, 0 PermissionDenied, 0 health-sequence regressions.
Soak driven by a live SoakNotify workload (3 instances, 3 notifs/5s):
- single-node active-kill (central-a): sticky failover central-a:8083→b:8083
logged instantly, central-b active in 29s, buffer drained 72→101, every 5s
bucket = exactly 3 through the gap, 101==101 distinct
- failback: central-a rejoined ready ~5s as standby, central-b kept active
(oldest-Up, no flap), traffic uninterrupted
- full outage (both central down ~59s): count frozen, cold re-form central-b
active ~14s, ~42 buffered drained, every 5s bucket = exactly 3 across the
whole dead window, 216==216 distinct — zero loss, zero dupes
Also previews Phase 5 checks 4 (failover/failback) and 5 (mid-drain kill).
Rig config reverted to Akka default (defaults stay Akka until Phase 4).
Refactor SiteCommunicationActor's central→site routing table into one
SiteCommandDispatcher — the single routing truth for the 28 migrated commands
(IntegrationCallRequest, the dead 29th, stays on the actor and out of the
dispatcher). The Akka actor and the new SiteCommandGrpcService both route through
one dispatcher instance so the two transports can never drift on where a command
goes. Server-side only: nothing central flips to gRPC yet (that is T1B.3);
ClusterClient remains the live path.
Decisions worth recording:
- Targets preserved byte-for-byte. Lifecycle/OPC UA/query/route → the Deployment
Manager singleton proxy; DeployArtifacts/EventLog/parked → their null-guarded
handlers with the exact same "handler not available" replies; the parked
handler stays NODE-LOCAL (per-node replicated-store owner), never the singleton
proxy — pinned by a dispatcher test that asserts the target is the parked probe
and NOT the dm proxy.
- Sender preservation intact. The actor's command handlers became thin
DispatchCommand delegations that still Forward (central Ask → reply routes
straight back); the existing SiteCommunicationActorTests pass unchanged, which
is the regression guard for that plumbing. UnsubscribeDebugView keeps its
fire-and-forget shape: the actor Forwards, the gRPC service Tells + returns the
synthetic UnsubscribeDebugViewAck so a unary RPC still answers.
- Ack-before-Leave on failover. The dispatcher's PrepareFailover resolves the
standby with a DRY-RUN (no leave) to build the ack, and hands back a deferred
CommitLeave; the gRPC service returns the ack, then schedules the real
Cluster.Leave — so a caller reaching the very node about to leave still gets its
ack instead of a broken stream. The actor path keeps today's coupled
resolve-and-leave (over ClusterClient the ack Tell only enqueues, so order is
immaterial). Proven at both levels: a dispatcher test asserts the ack is built
before CommitLeave runs, and a TestServer test asserts the recorded seam order
is resolve-then-leave.
- ControlPlaneAuthInterceptor gates SiteCommandService by EXTENDING
DefaultGatedPrefixes (descriptor-derived), not by adding a constructor — the
one-public-ctor invariant and its test stay green.
Tests: SiteCommandDispatcherTests (28-command routing incl. parked node-locality
and both failover paths) and SiteCommandGrpcService TestServer tests (auth,
readiness→Unavailable, one command per oneof group, failover ordering). Full
solution build 0/0; Communication.Tests 574 and Host.Tests 377 green. No active
<Protobuf> item.
Phase 1B's contract slice: the wire shape and the canonical translation for the
28 central→site commands that leave ClusterClient. No behaviour changes yet —
SiteCommunicationActor and CentralCommunicationActor are untouched; the
dispatcher refactor (T1B.2) and the central transport seam (T1B.3) consume this.
Protos/site_command.proto (package scadabridge.sitecommand.v1, service
SiteCommandService): six domain RPCs, each with a `oneof` request/reply
envelope. The grouping is what carries deadline policy — every command inside a
group shares a CommunicationOptions timeout class today, so one RPC per group
keeps the deadline choice in one place on the client and one dispatch switch on
the server, while the oneof keeps each command individually typed:
ExecuteLifecycle(6) · ExecuteOpcUa(8) · ExecuteQuery(4) · ExecuteParked(5) ·
ExecuteRoute(4) · TriggerFailover(1). IntegrationCallRequest — the 29th entry on
SiteCommunicationActor's receive table — is deliberately excluded as dead code
(2026-07-22-integration-call-routing-is-dead-code.md).
Contract decisions worth knowing:
- Nullable COLLECTIONS ride in per-collection wrapper messages
(DeployArtifactsCommand's six artifact lists, CertTrustResult.Certs,
RouteToCallRequest.Parameters). proto3 repeated/map collapses null into empty,
and that distinction is live at the site — the same silent-data-loss class the
transport round-trip guard exposed in PLAN-05 T8. Goldens cover null, empty
and populated for each.
- Nullable strings use the empty-string-means-null convention already set by
AuditEventDtoMapper, with ONE exception: RouteToWaitForAttributeRequest's
TargetValueEncoded, where "wait for the empty string" is a real target, so it
carries a StringValue wrapper. Both behaviours are asserted, not assumed.
- Nullable enums ride in one-field messages (proto3 enums have no presence and
no stock wrapper). Enum translation is an explicit switch in both directions —
never by ordinal — so reordering a C# enum cannot re-map the wire; every wire
enum reserves 0 for _UNSPECIFIED and decodes to a documented safe default
rather than faulting a command from a version-skewed peer.
- New LooseValueCodec carries the surviving `object?` members (script params and
return values, attribute values, tag read/write values) as a type-tagged union
so a boxed value keeps its runtime CLR type, as it does today under Akka's
type-preserving JSON serializer. Dates ride as invariant round-trip strings,
not Timestamp, which would silently normalise away DateTime.Kind and
DateTimeOffset.Offset. Lists/maps recurse; anything outside the tagged set
falls back to JSON and is documented as CLR-type-lossy.
- DebugViewSnapshot gets its own full-fidelity alarm/attribute messages rather
than reusing sitestream's AlarmStateUpdate, which flattens values to display
strings — right for a live stream, lossy for a snapshot the UI treats as
authoritative. The encoder omits an AlarmStateChanged.Condition that already
equals the record's derived default, so computed alarms round-trip exactly
(record equality compares the nullable backing field, not the property).
Tests are reflection-driven so the coverage cannot drift: the round-trip theory
enumerates the mapper's own ToProto overloads, the envelope guards enumerate the
generated oneof descriptors, and a missing golden fails the build. 216 new tests
green (Communication 532 total, Commons 684 total, solution build 0/0).
Codegen is checked in under SiteCommandGrpc/ per the sitestream recipe; the
<Protobuf> ItemGroup stays commented out (an active one segfaults protoc in the
linux_arm64 Docker image). docker/regen-proto.sh now handles every proto in that
ItemGroup instead of just sitestream, and re-comments idempotently.
Phase 0's gate doc now carries the full suite picture, not just the rig checks.
Playwright: 170 pass / 2 fail / 1 skip of 173. Both failures were run down to
root cause and both are pre-existing on main, unrelated to this branch (which
touches no EF, CentralUI, Transport or ManagementService file):
- TransportImportTests is a REAL production bug: BundleImporter.cs:1298 opens a
user-initiated transaction while the central context has EnableRetryOnFailure,
so SqlServerRetryingExecutionStrategy refuses the split query inside it and
bundle import fails against real MS SQL. The unit/integration suite cannot see
it -- the in-memory EF provider has no retrying strategy and BeginTransaction
is a no-op there.
- SmsNotificationE2ETests is a stale fixture: SID 'ACtest123' (2026-06-19) vs the
^AC[0-9a-fA-F]{32}$ guard added 2026-07-10 (40088a21). Failing since then, which
has also silenced everything after the toast assertion -- including the
secret-non-leak check on the Auth Token.
Also records that the earlier 44-failure run is void: a concurrent deploy.sh was
recreating the cluster underneath it.
Neither is fixed here; both are out of scope for a PSK-auth branch.
The gate's first run failed on a defect the green suite could not see: two public
constructors on ControlPlaneAuthInterceptor made Grpc.AspNetCore's activation
throw per call, so correct key, wrong key and no key all produced identical
errors. Recorded in full because the symptom (Unknown / "Exception was thrown by
handler") points at the handler, not at auth, and because phases 1A/1B both add
services to this same interceptor — they must extend DefaultGatedPrefixes rather
than add a second public constructor.
Also records what the gate does NOT cover: live streaming under load, key
rotation on a running pair, and docker-env2 (keyed but neither redeployed nor
gated).
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
CentralFailoverControl lives in Components/Health/ (matching AuditKpiTiles /
SiteCallKpiTiles) rather than inline in Health.razor, so it is testable without
standing up the whole dashboard's DI graph.
- Admin-gated via AuthorizeView + RequireAdmin. The Health page itself is
intentionally all-roles, so the gate belongs on the control, not the page.
- Disabled with an explanatory title when the pair has no online standby;
the authoritative guard remains server-side against live cluster membership.
- Confirmation dialog (IDialogService, the page idiom) warns that singletons
hand over, in-flight work on the active node is interrupted, and THIS PAGE
will disconnect and reconnect against the new active node -- Traefik routes
the UI to the node being restarted, so a working failover otherwise reads as
a crash the admin caused.
- A refused failover (service returns null) surfaces the refusal; the UI never
reports a failover that did not happen.
7 bUnit tests. Two harness requirements that bit first: AuthorizeView needs a
cascading AuthenticationState (the app supplies it from the layout), and
BunitContext pre-registers a placeholder IAuthorizationService that throws on
policy evaluation -- both handled the same way NavMenuTests documents.
Runbook paragraphs added to Component-ClusterInfrastructure.md (new Manual
Failover section) and docker/README.md.
Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs
FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when
no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs
JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting
central-b never came Up (the "registered outage gap"), and self-first ordering
closes it using Akka's own protocol.
- 6 node appsettings swapped (the *-node-b configs; the -a nodes were already
self-first). All 14 shipped node configs now satisfy the invariant.
- StartupValidator enforces it at boot, comparing host AND port -- the invariant
fails silently when broken, so it is enforced loudly. NOTE: the gitignored
deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or
that node will refuse to boot.
- SelfFirstSeedBootstrapTests: real in-process clusters at production
failure-detection timings, incl. a falsifiability control proving the OLD
peer-first ordering never forms.
Rejected alternative (implemented, measured, discarded): an external self-form
timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's
join handshake and so cannot tell "no seed answered" from "a seed answered and
the join is in flight". On a routine standby restart the peer is alive but the
join stalls behind removal of the node's own stale incarnation; a Join(self)
during TryingToJoin abandons the in-flight join and forms a second cluster at
the same address -- still split after 90s. Docs that claimed self-first ordering
was unsafe for simultaneous cold start are corrected: while mutually reachable
the InitJoin handshake converges them to one cluster (measured).
CLAUDE.md was corrected in 34227991 but the component docs were not, so the
same understatements — plus several outright wrong statements — survived where
a reader is most likely to meet them.
ClusterInfrastructure.md carried the worst of it. It described active/standby as
cluster leadership and showed an IsActiveNode snippet doing
`cluster.State.Leader == self.Address`. No such code exists: ActiveNodeGate
returns ClusterActivityEvaluator.SelfIsOldest, and ActiveNodeEvaluator's own doc
comment says "never cluster.State.Leader". A second snippet (siteCallAuditShutdown
.AddTask) described a hand-rolled drain that has since been folded into
SingletonRegistrar.Start. Both snippets are replaced with what the code does. The
central singleton table listed 3 of the 7 registered singletons. The downing
section still described keep-oldest as the strategy and omitted
downing-provider-class entirely; it now shows the branching block cf3bd52f
introduced, with the accepted dual-active trade stated rather than the old
"impossible to boot" framing. Note the requirements-side spec,
docs/requirements/Component-ClusterInfrastructure.md, was already rewritten by
cf3bd52f — this is the components-side doc, which was untouched.
Communication.md described two transports; there are three — the deployment
config fetch over HTTP was missing. Each row now names which side dials, because
the gRPC entry gave a data direction (site to central) without saying who hosts
the server, which reads as the opposite of the truth: the only MapGrpcService in
the tree is in Program.cs's Site branch, and central dials in. The SiteEnvelope
snippet called DeployInstanceAsync, which is not a member of CommunicationService;
the heartbeat bullet claimed a cluster-leadership check; the proto summary listed
4 of 6 RPCs.
DeploymentManager.md still said the pipeline sends DeployInstanceCommand carrying
FlattenedConfigurationJson. It stages a PendingDeployment and sends a
RefreshDeploymentCommand; the config travels over the HTTP fetch. That is the
change the 128 KB frame-size issue forced, and the doc predated it.
Two of the five briefed drifts turned out not to be errors: nothing claimed the
transports were authenticated, and nothing asserted per-site ActorSystem names —
both were simply unstated. They are now stated, since silence about an
unauthenticated boundary is its own problem.
Docs only; no code changed.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
An audit row whose tracking snapshot cannot be resolved is skipped and left
Pending, on the reasoning that central reconciliation will pick it up. Nothing
removes it from the local drain queue, so the next tick re-reads it, fails
identically, and logs again — forever. Measured on the rig at ~2,800
warnings/minute, sustained across both a process restart and a container
restart, until the audit database itself was discarded.
The code comment already names the cause ("the tracking store was reset"), so
this is an anticipated input, not an exotic one. Two production triggers need
no operator error: the tracking retention window elapsing before the drain
catches up (long central outage, large backlog), or restoring one store
independently of the other. The two stores are easy to desync because they live
in different places — audit in auditlog.db, which on the docker rig is INSIDE
the container, and tracking in the bind-mounted LocalDb database.
Skipping the row is right; leaving it Pending with no other state change is
not. There is no attempt counter, no backoff, no terminal state, and one
Warning per row per pass. Suggested fixes ranked by effort, the cheapest being
to rate-limit the warning the way MaintenanceBackgroundService already does for
oplog caps.
No data loss — the audit rows are intact and still reach central by the
reconciliation path. Found while cleaning the rig after the Phase 2 live gate.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Normative first: Component-StoreAndForward.md:83 specified the whole chunked,
ack-confirmed SfBufferSnapshotChunk resync protocol, which Phase 2 deleted. It
is rewritten rather than removed — the failure modes it reasoned about still
exist, they are just bounded differently — and it now states the
duplicate-delivery bound explicitly: a message can be delivered twice only when
the OLD primary delivered it and the status change had not yet replicated when
the gate flipped. One flush interval plus the in-flight ack, and unlike the old
model it does NOT grow with backlog depth or with how long a node was absent.
The N1 directional-authority and N5 orphan-row hazards are recorded as
structurally gone, not merely unguarded.
Frame-size known-issue amended: its 2026-06-26 resolution replaced the
intra-site hop with notify-and-fetch; Phase 2 then deleted notify-and-fetch
itself, so the 128 KB Akka frame constraint no longer applies to that hop in
any form. Successor ceiling recorded (4 MB gRPC cap via MaxBatchSize, which
batches by ROW COUNT), including that the failure mode differs — an oversized
gRPC message is rejected, not silently dropped.
Deployment docs gain the two operational constraints that have no home in code:
a site pair must be stopped and started TOGETHER (the SfBufferSnapshot compat
handler that made a mixed-version pair converge went with the replicator, and a
mixed pair now diverges silently), and a node offline beyond TombstoneRetention
can resurrect deleted rows on rejoin.
Both CLAUDE.md files corrected — each still said Phase 2 was NOT started.
Definition of done closed: build 0 warnings, all 10 suites green (3509 tests,
0 failures, 0 skips). Two DoD items needed amending rather than ticking: the
stale-symbol grep still matches 4 lines, all deliberate comment prose recording
what was deleted (a literal zero would delete the explanations that stop the
old design coming back), and the live gate has 10 evidence items, not the 9 the
checklist claimed.
Deletes the phase2 resume-state scratch doc, which said to delete it on landing.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
The blocker in the previous (incomplete) run was diagnosed: external systems
reach a site only through ArtifactDeploymentService, which `instance deploy`
never invokes. `deploy artifacts` delivered the probe harness AND propagated
the owed ExternalSystemDefinitions restore to both site nodes, unblocking
checks 5 and 10.
Highlights:
- 3+4: the deployed row is byte-identical on both nodes with the SAME
__localdb_row_version HLC and origin node id, and the standby logged zero
config fetches in the deploy window. B holds the row A wrote, by CDC alone.
- 6: stopped the active node; the standby took over in 10s and kept buffering,
oplog rose to 4 unacked while partitioned, drained to 0 on rejoin, and both
nodes ended byte-identical with ZERO duplicate ids (exactly-once).
- 7: the 3-table cascade converged with explicit TOMBSTONES on both nodes —
the rows are not merely absent on B, B applied the deletes.
- 9: both nodes stopped and started together; clean rejoin, zero SQLite I/O
errors.
Two caveats recorded rather than glossed: check 2's zero-count is vacuous on
its own (the legacy source was also empty) and rests instead on the ABSENCE of
CDC triggers for smtp_configurations/notification_lists; check 7's
native_alarm_state leg was empty live and is covered only offline.
Also records three method corrections — the plan's host-side sqlite3
instruction is unsafe, `docker exec curl` cannot scrape metrics from an image
with no curl (a silent failure that nearly became a false finding), and the
artifact-vs-instance deploy distinction above.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Checks 1, 2, 3, 4 and the oplog/dead-letter/metrics half of 8 captured and
passing. Check 5 blocked on rig state, checks 6, 7, 9, 10 not run. Task 21 must
NOT proceed on this evidence.
Strongest result is check 3+4 together: the deployed config row is byte-
identical on both site-a nodes with the SAME __localdb_row_version HLC and
originating node id, and the standby logged zero config fetches in the deploy
window. The standby holds the row node A wrote, obtained purely by CDC — which
is the whole point of deleting notify-and-fetch.
Check 5 is blocked by a rig-shaping problem, not a product fault: removing the
owed soakgen instances orphaned their 11,804 buffered sf_messages, and a
replacement probe's external system never reached the site nodes (site config
tables come from deployment artifacts, and an external system referenced only
by name inside script code is not carried). Same blocker stops check 10.
Two method corrections recorded in the doc:
- The plan says to run DB checks host-side against the bind mounts. That is
WRONG — a host sqlite3 open poisons the container's WAL. All reads here copy
the db/-wal/-shm triplet and query the copy.
- An apparent "localdb_* metrics missing" finding was an artifact of the
aspnet:10.0 image having no curl, with the error swallowed by 2>/dev/null.
Re-scraped via a network-sharing sidecar: the metrics are present.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Records the latent Phase 1 directory defect and its open library-vs-app decision,
the CreateConnection contract change, the new TestSupport library, and the
verification numbers at the wave-2 boundary.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 2 of the Phase 2 plan, plus the clean re-run of Task 1's cut-short
sampling that Task 2 was waiting on.
Gate doc: status NOT STARTED -> CLOSED. All five §5 open questions answered
inline. The questions are kept, not deleted — several of the answers overturn
a premise the gate itself stated, and that reasoning is the value:
- Oplog sizing is not the binding constraint. Cap overrun prunes and flags
needs_snapshot (graceful resync), so the caps need no defensive sizing; the
real ceiling is D6's 4 MB gRPC cap, binding on MaxBatchSize x row-bytes.
- LWW-vs-in-flight-send cannot arise on a correct pair (only the active node
sweeps). The honest cost is D2's semantic change: the standby is convergent,
no longer byte-identical.
- Migration-under-load is designed out, not managed — both mechanisms are
deleted in the same commit and D5 forecloses rolling upgrades.
- Rollback is genuinely "revert the commit": the migration is additive and
leaves the legacy files intact. Bounded cost is the post-cutover delta.
- One DB per process stays; every write axis is bounded.
§2's requirement to state CDC's duplicate-delivery bound is routed explicitly
to Task 21 rather than left implicit.
Task 1 re-run (safe copy-based snap(), both nodes restarted first, six 60s
intervals): sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0,
alarms 0, max payload_json 76 B, zero SQLite errors across 30 min, both site-a
nodes converged at 3564 rows. Independently confirms the disk-I/O storm was
observer-induced. Recorded with its limits stated: 0.80/s is ~1.6% of the 50/s
ceiling and the retry path was never exercised — acceptable only because that
ceiling is structural rather than empirical, and the rig's 721 B config rows
are explicitly NOT representative.
GATE VERDICT: PROCEED to Task 3. Binding output is MaxBatchSize 500 -> 16.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Scratch handoff so the plan can be picked back up cleanly: Task 1 done and its
STOP verdict superseded, what must happen before Task 3, the four plan-premise
corrections, the rig gotchas that are not discoverable from the repo, and the
rig cleanup the Task 20 live gate needs.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
The Phase 2 soak's "LocalDb fails under load" blocker is NOT a product defect.
Root cause: host-side (macOS) sqlite3 reads of the live, bind-mounted WAL
databases. POSIX advisory locks do not propagate across the virtiofs boundary,
so the host reader believes it is the only connection, checkpoints on close and
resets the WAL to 0 bytes under the container. The container's still-mapped
WAL index then references frames that no longer exist and every subsequent
statement fails SQLITE_IOERR_SHORT_READ (522) -> primary code 10, permanently
until the process reopens the database.
Reproduced on demand both on the rig (one sqlite3 SELECT reset a 4.6 MiB WAL and
produced the first error one second later) and in a minimal python:3.12-alpine
repro with no LocalDb or .NET involved. Refuted the converse: a freshly-reopened
node sustained the full soak load 10+ minutes with zero errors.
The original brief's isolation was confounded - both nodes had been poisoned by
the same sampling pass, and a poisoned standby looks healthy only because it
issues almost no statements. Corrections annotated in place.
Hardening shipped alongside:
- SqliteErrorCodes.Describe: log SQLite primary AND extended codes at the
LocalDb-adjacent catch sites (the missing extended code is what made the
original diagnosis so slow).
- SiteAuditTelemetryActor: stop touching ActorContext across an await
(NotSupportedException), with regression coverage.
- infra/reseed.sh: apply the MSSQL init scripts explicitly. The official
mssql/server image does not implement /docker-entrypoint-initdb.d, so a fresh
volume hung the reseed forever; compose mounts annotated as informational.
Deliberately NOT done: detect-and-reopen self-heal in SqliteLocalDb. It defends
only against external interference, which is now prevented at the source, and
same-kernel production readers see the locks correctly.
Build 0 warnings; SiteAuditTelemetryActorTests 9/9, SiteEventLogging 70/70.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Standalone handoff for a follow-up investigation. Records the symptom, a full
reproduction (including the two rig-tooling blockers and the CachedCall-vs-Call
detail needed to generate load at all), the isolation evidence, a code map of
the LocalDb connection model, four ranked hypotheses, and acceptance criteria.
Not root-caused. Highest-value next step identified: capture the SQLite EXTENDED
result code (logs only carry the generic primary code 10), and run the load with
LocalDb:Path off the bind mount to partition environmental vs library causes.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 1 of the Phase 2 plan. The gate stops the plan, but not for the reason it
anticipated: oplog sizing is fine and D6 is resolved.
BLOCKER: the Phase 1 consolidated LocalDb (site-localdb.db) throws SQLite
Error 10 'disk I/O error' on essentially every write on the ACTIVE node under
sustained load — site_events, OperationTracking and the audit telemetry paths
all fail, and site event logging silently drops events. Isolated to the load
(not the node, not host-side observation) by failing over between nodes, and to
LocalDb specifically (legacy store-and-forward.db / scadabridge.db in the same
bind-mounted directory take zero errors under identical load).
Phase 2 would register 8 more tables into that database — including the two
highest-volume ones — while deleting the bespoke mechanisms that currently
carry them. Must be root-caused first.
Also recorded: D6's premise corrected (largest known production config_json is
~60-70 KB, not >128 KB — but MaxBatchSize 500 is still unsafe, use 16); D4's
premise corrected (alarm writes are bounded by per-SourceReference coalescing at
a 100 ms flush); sf_messages has a hard 50 rows/sec structural ceiling; and
exceeding the oplog caps is a graceful snapshot-resync, not a failure.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Three verification sweeps over SiteRuntime, StoreAndForward, and Host/rig/docs
plus the LocalDb library source. Load-bearing corrections: D1 (the guarded write
has a second surviving caller, SiteReconciliationActor), D3 (the active node
already purges — Task 12 becomes a pin), D6 (new: the 4 MB gRPC cap vs
row-count-only batching), Task 1 (rewritten method — the Phase 2 tables are not
in the Phase 1 oplog), and Task 14 (do not register the notification/SMTP tables).
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Moves scadabridge.db's 9 config tables + sf_messages into the consolidated
LocalDb file and deletes SiteReplicationActor + StoreAndForward's
ReplicationService.
Resolves the phase 2 gate's five open questions from code recon (D1-D5):
D1 notify-and-fetch is DELETED, not preserved. It exists only because the
config blob exceeds Akka's 128KB frame; LocalDb sync is gRPC. The
deployed_at version guard protected against a stale fetch racing, so it
dies with the fetch. Do not reproduce it on LWW - different clocks.
D2 ReplaceAllAsync is deleted and the N1 directional guard becomes
unnecessary: LocalDb's snapshot resync merges per-row LWW and never
deletes (SnapshotApplier has no DELETE; LwwApplier.cs:69-78 discards a
lower-HLC incoming row). Semantic change - the standby is convergent,
no longer byte-identical.
D3 The SMTP purge (plaintext passwords) rides the replication path and is
re-homed to the active node BEFORE any deletion.
D4 native_alarm_state volume is measured by a rig soak, not assumed. Task 1
gates the plan and stops it if the oplog cannot absorb the churn.
D5 No dual-mechanism period forecloses rolling site upgrades - both nodes
must stop and start together.
Recon also found the gate doc's "two test files are the spec" undercounts:
the real specification is five files, including the N1 Critical regression
test and the only Requeue coverage.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Joseph Doherty