Design doc and code travel together (CLAUDE.md editing rules), so this records
what the two preceding commits shipped and, more usefully, WHY the non-obvious
choices were made -- the parts a future reader would otherwise re-litigate:
- Component-SiteRuntime.md: the Alarms.CurrentAsync() runtime API entry (why
it is not scope-prefixed, why it is read-only, why placeholder rows are
included), the full ScriptAlarm shape, AckTime on the enriched
AlarmStateChanged, proto field 24, and the metadata_json-vs-new-column
persistence rationale (native_alarm_state is RegisterReplicated; LocalDb
builds its CDC triggers from the column list at registration time).
- Component-DataConnectionLayer.md already carried the AckTime section in the
first commit; this adds the SiteRuntime/ScriptAnalysis/InboundAPI halves.
- Component-ScriptAnalysis.md: accessors returning domain types return the
SAME type on both surfaces, and the trust-model note that a deny-list needs
no entry for a new globals member -- only that its return type resolves in
a permitted namespace.
- Component-InboundAPI.md records the NEGATIVE decision: there is
deliberately no Route.To(...).GetAlarms(...) verb. Alarm state is
per-instance and lives on the site's Instance Actor, so the read goes
through a routed site script and the filtering happens where the data is;
central stays a thin router.
- CLAUDE.md native-alarm bullet gains the enrichment + accessor summary.
- The plan's §7 Phase 1 rows are ticked with 2026-08-01 and annotated with
what was actually built (incl. the two choices that differ from the plan's
"or" options: a dedicated snapshot message rather than DebugSnapshotRequest,
and the extra SandboxScriptHost mirror the plan did not list). Phases 2-4
stay open -- they are deployed config and need a live rig.
T34c only fixed the bounded modal-surface offenders. Bootstrap 5.3's bg-light
and bg-white are fixed light values that do NOT flip under [data-bs-theme=dark],
so every remaining use was a dark-mode contrast break. 35 swaps across 19 files:
surface / <pre> / <code> bg-light -> bg-body-secondary
panel bg-white -> bg-body
neutral badge bg-light text-dark -> bg-secondary-subtle text-secondary-emphasis
muted badge / input group bg-light text-muted -> bg-body-secondary text-body-secondary
DELIBERATELY LEFT (7 sites): the neutral member of a status-badge switch or
ternary whose siblings are all solid, non-theme-aware colours (bg-success,
bg-danger, bg-warning) — swapping only the neutral one to a subtle token breaks
the visual weight of the set, so these stay until the whole family is retoned:
Topology.razor:513 (Current badge) and :588 (InstanceState.NotDeployed)
InstanceConfigure.razor:1520 (same InstanceState switch)
NotificationReport.razor StatusBadgeClass fallback
TransportImport.razor ConflictKind badge fallback
SecuredWrites.razor text-bg-light fallback (a text-bg-* family with no
theme-aware member at all)
Health.razor:377 depth ternary
Also untouched by design: SchemaBuilder.razor:88 (already bg-light-subtle,
theme-aware), bg-dark text-light console panels, and site.css / #reconnect-modal.
InstanceConfigure.razor is edited concurrently elsewhere; its change here is 5
pure class-string swaps on existing lines, no reflow. No test asserted any of
these classes. Also marks all four M10 residuals done in the plan doc.
Two verified-absent parity gaps between the service layer and the CLI /
ManagementActor command surface, both left as follow-ups by the 2026-05-11
design plans.
(1) area move. AreaService.MoveAreaAsync had existed since the deployment
topology page shipped but was reachable only from the Blazor UI. Adds
MoveAreaCommand(AreaId, NewParentAreaId?) to Commons, a ManagementActor
dispatch arm delegating straight to AreaService.MoveAreaAsync (not-found /
self-parent / descendant-cycle / cross-site / name-collision all surface as
the standard curated ManagementCommandException failure response; the service
writes its own "Move" audit row), and the CLI verb `site area move --id
[--parent-id]`. Omitting --parent-id moves the area to the site root, matching
the command's nullable NewParentAreaId. The command carries the SAME any-of
[Designer, Deployer] gate as CreateArea/UpdateArea/DeleteArea (arch-review C6):
re-parenting is the same structural authoring act, exposed on the same two
surfaces. Placed under the existing `site area` group rather than a new
top-level `area` group, alongside its create/update/delete siblings.
(2) template folder verbs. The five folder management commands have been
handled by ManagementActor since the folder-hierarchy plan, but the promised
CLI surface was never written. Adds `template folder
list|create|rename|move|reorder|delete` mapping 1:1 onto ListTemplateFolders /
CreateTemplateFolder / RenameTemplateFolder / MoveTemplateFolder /
ReorderTemplateFolder / DeleteTemplateFolder. --parent-id is omitted to target
the tree root; --direction takes the lowercase literals up/down, validated at
parse time by AcceptOnlyFromAmong (same case-sensitive contract as the audit
--channel/--kind/--status options).
Follow-on updates: the frozen authorization matrix gains its MoveArea entry
(reflection-driven, so a missing entry would have failed CI); CommandTreeTests
pins both new verb sets plus the omit-parent-id-means-root parse behaviour and
registry round-trips; ManagementActorTests covers the MoveArea role gate and
the delegate-to-service success/root/cycle/not-found paths; the CLI README and
Component-ManagementService.md document the new surface (the latter also gained
the previously-undocumented ReorderTemplateFolder); both plan docs' follow-up
lines are marked done.
Adds rows 23-26 to the deferred-work register: live LDAP group-membership
re-query (blocked on ZB.MOM.WW.Auth.Ldap gaining a passwordless group
search), M8 large-bundle perf hardening (logged in the M9 completion design,
never given a plan or perf test), the Phase-8 WP-4 target-scale load test
(claimed complete by a 107-byte checklist stub with no evidence), and the
Ipsen MES MoveIn tail (-LT routing, PLC-output flags, Z28062 data).
Splits the review-08 "Failover-timing + broader perf envelope" row: the
failover-timing half is resolved by FailoverTimingTests, now a live [Fact]
on TwoNodeClusterFixture at production timings (PLAN-R2-01 T4); the S&F
drain-rate + per-subscriber backpressure half stays open as its own row.
Moves the closed folder drag-drop row (18, [PERM]) out of Deferred into the
Resolved table per the register's own rule.
Design review 2026-08-01: (1) MES relevance = dedicated severity band 900-999
(script constants, real IsFlaggedForMES predicate); (2) MESReceiver version
DROPPED — CvdReactor is the only implementation, router returns a clean
not-supported error elsewhere; (3) Description = Message else AlarmTypeName;
(4) enrich the native mirror NOW with an additive AckTime (proto +
native_alarm_state + DCL stamping); (5) script names match the endpoints;
(6) shared ReactorAlarms on both sides, suffix behavior as planned;
(7) existing MES API key, no new roles. Plan is now ready to execute.
Rebuilt scadabridge:latest from main @ 8524a7f7 and recreated only the
env2 containers. All three gate checks pass on site-x:
1. both site nodes boot with the key (StartupValidator fail-closed →
reaching 'Application started' proves the key present);
2. control-plane PSK auth: no-header / wrong-key ⇒ PermissionDenied,
correct key ⇒ success, on both nodes (:9123, :9124);
3. LocalDb unaffected (local-only; 0 errors, healthy boot).
Bonus: central registers site-x online via gRPC heartbeat; no real
ClusterClient/receptionist (only the benign ClusterClientSiteAuditClient
label, same as the primary rig). Noted a seed-data gap (ScadaBridgeConfig2
dbo.Sites is empty) — orthogonal to the transport.
ClusterClient→gRPC migration Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md).
Phases 2/3 proved both directions on gRPC; this removes the Akka transport underneath.
Deleted:
- AkkaCentralTransport, AkkaSiteTransport (+ their dedicated tests)
- ISiteClientFactory + DefaultSiteClientFactory; CentralCommunicationActor legacy
ctor + SelectTransport (Host now builds GrpcSiteTransport and injects it)
- ClusterClient creation + both ClusterClientReceptionist.RegisterService calls in
AkkaHostedService; the RegisterCentralClient message + receive block
- CommunicationOptions.CentralContactPoints; the CentralTransport/SiteTransport
coexistence flags; the CentralTransportMode/SiteTransportKind enums
gRPC is now the only site↔central transport (site→central CentralControlService via
GrpcCentralTransport; central→site SiteCommandService via GrpcSiteTransport), both
built unconditionally by the Host. NoOpCentralTransport is the fail-loud null-default
so TestKit command-dispatch suites still construct the site actor without a wired
transport; production always injects GrpcCentralTransport.
Config: CentralGrpcEndpoints is now unconditional — CommunicationOptionsValidator
rejects blank entries (role-agnostic), and StartupValidator requires a Site node to
list >=1 endpoint (fail-fast, mirrors GrpcPsk). Rig configs moved
CentralContactPoints -> CentralGrpcEndpoints (docker x6, docker-env2 x2, Host default,
deploy/wonder-app-vd03). Kept Akka.Cluster.Tools (ClusterSingleton still used).
Tests: build 0/0; Communication.Tests 640, Host.Tests 421 green. Removed the
ClusterClient.Send per-site-routing tests (covered by the transport suites), swapped
the ISiteClientFactory-based ctors to a substitute ISiteCommandTransport, converted
the audit-push integration relay to an in-process bridge transport.
Docs: Component-Communication/Host/StoreAndForward, components/Communication,
topology-guide, grpc_streams (SUPERSEDED note), the frame-size known-issue (retired
amendment), and CLAUDE.md transport decisions.
Not included: the dead IntegrationCallRequest path (#32) is a separate user-owned
behavioral decision — SiteEnvelope routing is transport-agnostic so it still compiles.
Central flipped to SiteTransport=Grpc (both nodes, central-wide flag), sites
kept on gRPC from Phase 2 → both directions on gRPC at once. Central logs
'central→site command transport: gRPC (SiteCommandService)'; 0 ClusterClient-to-
site on either central.
Command matrix over SiteCommandService (all 200):
- ExecuteQuery (event-log) + ExecuteParked (parked) — all 3 sites
- ExecuteLifecycle disable/enable #95 on site-a — NEW live coverage vs 1B/P2
(SoakNotify instances survived the recreate as Enabled)
Resilience:
- hard-kill the ACTIVE site-a node mid-query-loop → in-flight call returned
TIMEOUT at exactly the 30s QueryTimeout deadline (bounded, no hang; deadline≠
retry — an in-flight call can't be safely re-sent)
- next + all subsequent queries recovered automatically via site-a-b
(SitePairChannelProvider NodeA→NodeB); site→central S&F never stopped
(notif 619→715, still no dupes)
- 0 PermissionDenied across all 8 nodes
ExecuteOpcUa/ExecuteRoute/TriggerFailover deferred (no OPC-bound instance / no
CLI verb; unit-proven). Rig left both-gRPC; git reverted to Akka default.
All 6 site nodes flipped to CentralTransport=Grpc; whole control plane
(heartbeat/health/notification S&F/audit) rides gRPC CentralControlService,
196 RPCs/90s 0 non-200, 0 PermissionDenied, 0 health-sequence regressions.
Soak driven by a live SoakNotify workload (3 instances, 3 notifs/5s):
- single-node active-kill (central-a): sticky failover central-a:8083→b:8083
logged instantly, central-b active in 29s, buffer drained 72→101, every 5s
bucket = exactly 3 through the gap, 101==101 distinct
- failback: central-a rejoined ready ~5s as standby, central-b kept active
(oldest-Up, no flap), traffic uninterrupted
- full outage (both central down ~59s): count frozen, cold re-form central-b
active ~14s, ~42 buffered drained, every 5s bucket = exactly 3 across the
whole dead window, 216==216 distinct — zero loss, zero dupes
Also previews Phase 5 checks 4 (failover/failback) and 5 (mid-drain kill).
Rig config reverted to Akka default (defaults stay Akka until Phase 4).
Refactor SiteCommunicationActor's central→site routing table into one
SiteCommandDispatcher — the single routing truth for the 28 migrated commands
(IntegrationCallRequest, the dead 29th, stays on the actor and out of the
dispatcher). The Akka actor and the new SiteCommandGrpcService both route through
one dispatcher instance so the two transports can never drift on where a command
goes. Server-side only: nothing central flips to gRPC yet (that is T1B.3);
ClusterClient remains the live path.
Decisions worth recording:
- Targets preserved byte-for-byte. Lifecycle/OPC UA/query/route → the Deployment
Manager singleton proxy; DeployArtifacts/EventLog/parked → their null-guarded
handlers with the exact same "handler not available" replies; the parked
handler stays NODE-LOCAL (per-node replicated-store owner), never the singleton
proxy — pinned by a dispatcher test that asserts the target is the parked probe
and NOT the dm proxy.
- Sender preservation intact. The actor's command handlers became thin
DispatchCommand delegations that still Forward (central Ask → reply routes
straight back); the existing SiteCommunicationActorTests pass unchanged, which
is the regression guard for that plumbing. UnsubscribeDebugView keeps its
fire-and-forget shape: the actor Forwards, the gRPC service Tells + returns the
synthetic UnsubscribeDebugViewAck so a unary RPC still answers.
- Ack-before-Leave on failover. The dispatcher's PrepareFailover resolves the
standby with a DRY-RUN (no leave) to build the ack, and hands back a deferred
CommitLeave; the gRPC service returns the ack, then schedules the real
Cluster.Leave — so a caller reaching the very node about to leave still gets its
ack instead of a broken stream. The actor path keeps today's coupled
resolve-and-leave (over ClusterClient the ack Tell only enqueues, so order is
immaterial). Proven at both levels: a dispatcher test asserts the ack is built
before CommitLeave runs, and a TestServer test asserts the recorded seam order
is resolve-then-leave.
- ControlPlaneAuthInterceptor gates SiteCommandService by EXTENDING
DefaultGatedPrefixes (descriptor-derived), not by adding a constructor — the
one-public-ctor invariant and its test stay green.
Tests: SiteCommandDispatcherTests (28-command routing incl. parked node-locality
and both failover paths) and SiteCommandGrpcService TestServer tests (auth,
readiness→Unavailable, one command per oneof group, failover ordering). Full
solution build 0/0; Communication.Tests 574 and Host.Tests 377 green. No active
<Protobuf> item.
Phase 1B's contract slice: the wire shape and the canonical translation for the
28 central→site commands that leave ClusterClient. No behaviour changes yet —
SiteCommunicationActor and CentralCommunicationActor are untouched; the
dispatcher refactor (T1B.2) and the central transport seam (T1B.3) consume this.
Protos/site_command.proto (package scadabridge.sitecommand.v1, service
SiteCommandService): six domain RPCs, each with a `oneof` request/reply
envelope. The grouping is what carries deadline policy — every command inside a
group shares a CommunicationOptions timeout class today, so one RPC per group
keeps the deadline choice in one place on the client and one dispatch switch on
the server, while the oneof keeps each command individually typed:
ExecuteLifecycle(6) · ExecuteOpcUa(8) · ExecuteQuery(4) · ExecuteParked(5) ·
ExecuteRoute(4) · TriggerFailover(1). IntegrationCallRequest — the 29th entry on
SiteCommunicationActor's receive table — is deliberately excluded as dead code
(2026-07-22-integration-call-routing-is-dead-code.md).
Contract decisions worth knowing:
- Nullable COLLECTIONS ride in per-collection wrapper messages
(DeployArtifactsCommand's six artifact lists, CertTrustResult.Certs,
RouteToCallRequest.Parameters). proto3 repeated/map collapses null into empty,
and that distinction is live at the site — the same silent-data-loss class the
transport round-trip guard exposed in PLAN-05 T8. Goldens cover null, empty
and populated for each.
- Nullable strings use the empty-string-means-null convention already set by
AuditEventDtoMapper, with ONE exception: RouteToWaitForAttributeRequest's
TargetValueEncoded, where "wait for the empty string" is a real target, so it
carries a StringValue wrapper. Both behaviours are asserted, not assumed.
- Nullable enums ride in one-field messages (proto3 enums have no presence and
no stock wrapper). Enum translation is an explicit switch in both directions —
never by ordinal — so reordering a C# enum cannot re-map the wire; every wire
enum reserves 0 for _UNSPECIFIED and decodes to a documented safe default
rather than faulting a command from a version-skewed peer.
- New LooseValueCodec carries the surviving `object?` members (script params and
return values, attribute values, tag read/write values) as a type-tagged union
so a boxed value keeps its runtime CLR type, as it does today under Akka's
type-preserving JSON serializer. Dates ride as invariant round-trip strings,
not Timestamp, which would silently normalise away DateTime.Kind and
DateTimeOffset.Offset. Lists/maps recurse; anything outside the tagged set
falls back to JSON and is documented as CLR-type-lossy.
- DebugViewSnapshot gets its own full-fidelity alarm/attribute messages rather
than reusing sitestream's AlarmStateUpdate, which flattens values to display
strings — right for a live stream, lossy for a snapshot the UI treats as
authoritative. The encoder omits an AlarmStateChanged.Condition that already
equals the record's derived default, so computed alarms round-trip exactly
(record equality compares the nullable backing field, not the property).
Tests are reflection-driven so the coverage cannot drift: the round-trip theory
enumerates the mapper's own ToProto overloads, the envelope guards enumerate the
generated oneof descriptors, and a missing golden fails the build. 216 new tests
green (Communication 532 total, Commons 684 total, solution build 0/0).
Codegen is checked in under SiteCommandGrpc/ per the sitestream recipe; the
<Protobuf> ItemGroup stays commented out (an active one segfaults protoc in the
linux_arm64 Docker image). docker/regen-proto.sh now handles every proto in that
ItemGroup instead of just sitestream, and re-comments idempotently.
Phase 0's gate doc now carries the full suite picture, not just the rig checks.
Playwright: 170 pass / 2 fail / 1 skip of 173. Both failures were run down to
root cause and both are pre-existing on main, unrelated to this branch (which
touches no EF, CentralUI, Transport or ManagementService file):
- TransportImportTests is a REAL production bug: BundleImporter.cs:1298 opens a
user-initiated transaction while the central context has EnableRetryOnFailure,
so SqlServerRetryingExecutionStrategy refuses the split query inside it and
bundle import fails against real MS SQL. The unit/integration suite cannot see
it -- the in-memory EF provider has no retrying strategy and BeginTransaction
is a no-op there.
- SmsNotificationE2ETests is a stale fixture: SID 'ACtest123' (2026-06-19) vs the
^AC[0-9a-fA-F]{32}$ guard added 2026-07-10 (40088a21). Failing since then, which
has also silenced everything after the toast assertion -- including the
secret-non-leak check on the Auth Token.
Also records that the earlier 44-failure run is void: a concurrent deploy.sh was
recreating the cluster underneath it.
Neither is fixed here; both are out of scope for a PSK-auth branch.
The gate's first run failed on a defect the green suite could not see: two public
constructors on ControlPlaneAuthInterceptor made Grpc.AspNetCore's activation
throw per call, so correct key, wrong key and no key all produced identical
errors. Recorded in full because the symptom (Unknown / "Exception was thrown by
handler") points at the handler, not at auth, and because phases 1A/1B both add
services to this same interceptor — they must extend DefaultGatedPrefixes rather
than add a second public constructor.
Also records what the gate does NOT cover: live streaming under load, key
rotation on a running pair, and docker-env2 (keyed but neither redeployed nor
gated).
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
Normative first: Component-StoreAndForward.md:83 specified the whole chunked,
ack-confirmed SfBufferSnapshotChunk resync protocol, which Phase 2 deleted. It
is rewritten rather than removed — the failure modes it reasoned about still
exist, they are just bounded differently — and it now states the
duplicate-delivery bound explicitly: a message can be delivered twice only when
the OLD primary delivered it and the status change had not yet replicated when
the gate flipped. One flush interval plus the in-flight ack, and unlike the old
model it does NOT grow with backlog depth or with how long a node was absent.
The N1 directional-authority and N5 orphan-row hazards are recorded as
structurally gone, not merely unguarded.
Frame-size known-issue amended: its 2026-06-26 resolution replaced the
intra-site hop with notify-and-fetch; Phase 2 then deleted notify-and-fetch
itself, so the 128 KB Akka frame constraint no longer applies to that hop in
any form. Successor ceiling recorded (4 MB gRPC cap via MaxBatchSize, which
batches by ROW COUNT), including that the failure mode differs — an oversized
gRPC message is rejected, not silently dropped.
Deployment docs gain the two operational constraints that have no home in code:
a site pair must be stopped and started TOGETHER (the SfBufferSnapshot compat
handler that made a mixed-version pair converge went with the replicator, and a
mixed pair now diverges silently), and a node offline beyond TombstoneRetention
can resurrect deleted rows on rejoin.
Both CLAUDE.md files corrected — each still said Phase 2 was NOT started.
Definition of done closed: build 0 warnings, all 10 suites green (3509 tests,
0 failures, 0 skips). Two DoD items needed amending rather than ticking: the
stale-symbol grep still matches 4 lines, all deliberate comment prose recording
what was deleted (a literal zero would delete the explanations that stop the
old design coming back), and the live gate has 10 evidence items, not the 9 the
checklist claimed.
Deletes the phase2 resume-state scratch doc, which said to delete it on landing.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
The blocker in the previous (incomplete) run was diagnosed: external systems
reach a site only through ArtifactDeploymentService, which `instance deploy`
never invokes. `deploy artifacts` delivered the probe harness AND propagated
the owed ExternalSystemDefinitions restore to both site nodes, unblocking
checks 5 and 10.
Highlights:
- 3+4: the deployed row is byte-identical on both nodes with the SAME
__localdb_row_version HLC and origin node id, and the standby logged zero
config fetches in the deploy window. B holds the row A wrote, by CDC alone.
- 6: stopped the active node; the standby took over in 10s and kept buffering,
oplog rose to 4 unacked while partitioned, drained to 0 on rejoin, and both
nodes ended byte-identical with ZERO duplicate ids (exactly-once).
- 7: the 3-table cascade converged with explicit TOMBSTONES on both nodes —
the rows are not merely absent on B, B applied the deletes.
- 9: both nodes stopped and started together; clean rejoin, zero SQLite I/O
errors.
Two caveats recorded rather than glossed: check 2's zero-count is vacuous on
its own (the legacy source was also empty) and rests instead on the ABSENCE of
CDC triggers for smtp_configurations/notification_lists; check 7's
native_alarm_state leg was empty live and is covered only offline.
Also records three method corrections — the plan's host-side sqlite3
instruction is unsafe, `docker exec curl` cannot scrape metrics from an image
with no curl (a silent failure that nearly became a false finding), and the
artifact-vs-instance deploy distinction above.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Checks 1, 2, 3, 4 and the oplog/dead-letter/metrics half of 8 captured and
passing. Check 5 blocked on rig state, checks 6, 7, 9, 10 not run. Task 21 must
NOT proceed on this evidence.
Strongest result is check 3+4 together: the deployed config row is byte-
identical on both site-a nodes with the SAME __localdb_row_version HLC and
originating node id, and the standby logged zero config fetches in the deploy
window. The standby holds the row node A wrote, obtained purely by CDC — which
is the whole point of deleting notify-and-fetch.
Check 5 is blocked by a rig-shaping problem, not a product fault: removing the
owed soakgen instances orphaned their 11,804 buffered sf_messages, and a
replacement probe's external system never reached the site nodes (site config
tables come from deployment artifacts, and an external system referenced only
by name inside script code is not carried). Same blocker stops check 10.
Two method corrections recorded in the doc:
- The plan says to run DB checks host-side against the bind mounts. That is
WRONG — a host sqlite3 open poisons the container's WAL. All reads here copy
the db/-wal/-shm triplet and query the copy.
- An apparent "localdb_* metrics missing" finding was an artifact of the
aspnet:10.0 image having no curl, with the error swallowed by 2>/dev/null.
Re-scraped via a network-sharing sidecar: the metrics are present.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Joseph Doherty