Nullable ConfigDb factory; UpsertNodeDeploymentState no-ops when absent
(central persists acks from the ApplyAck); scripted-alarm condition state
served from the LocalDb store instead of the ConfigDb-backed Ef store.
Combines Phase-4 Tasks 4 + 6. Removes the interim dbFactory! cast.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 2 code review surfaced two unguarded ConfigDb sites reachable on a
driver-only node once Tasks 0+1 landed (SpawnScriptedAlarmHost's Ef store,
unconditional UpsertNodeDeploymentState). Task 4 brief now names them + the
dbFactory! removal; Task 6 re-homes the alarm store to LocalDb.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 4 of the per-cluster mesh program. Removes the four remaining driver-side
ConfigDb consumers + the registration itself so a driver-only node boots with no
ConfigDb connection string:
- ConfigDb registered iff hasAdmin (Program.cs); driver-only requires none
- driver-only ⇒ FetchAndCache mandatory (validator)
- DbHealthProbeActor not spawned on driver-only; DbReachable=true constant
(client-visible ServiceLevel change — a healthy DB-less node publishes 240/250
with central down, per user decision 2026-07-23)
- DriverHostActor: nullable factory, drop redundant SQL ack-writes (central's
PersistNodeAck is the ack system of record)
- EfAlarmConditionStateStore → LocalDbAlarmConditionStateStore (new replicated
alarm_condition_state table)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Driver nodes can obtain config via a gRPC fetch from central + LocalDb cache instead of
reading central SQL, behind the ConfigSource:Mode=FetchAndCache dark switch (Direct
default). Central serves the artifact over a dedicated h2c listener gated by a shared node
key; the node SHA-verifies against the dispatched RevisionHash, applies from the bytes in
hand, and boots from the LocalDb pointer with no SQL read. Live gate PASSED on docker-dev
(caught + fixed a Status==Sealed serve-gate deadlock). Phase boundary: config READS only.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The Phase 3 live gate deadlocked every FetchAndCache deploy: DeploymentArtifactGrpcService
gated on Status==Sealed, but central seals a deployment only AFTER every node acks Applied,
and a FetchAndCache node cannot ack until it has fetched the artifact — seal-needs-ack ->
ack-needs-fetch -> fetch-needs-sealed. The node's fetch reached central and passed the
shared-key interceptor, then got a clean NotFound; the #485 apply-failure path correctly
kept last-known-good, but the deploy could never seal.
Direct mode reads the same ArtifactBlob from SQL while the row is still AwaitingApplyAcks,
so the serve path must too. Drop the Sealed gate: serve any deployment whose blob is
non-empty (unknown id / empty blob still collapse to NotFound — existence-hiding + #485).
Access is already gated by the interceptor, so there is no reason to hide a non-sealed
deployment a node is legitimately applying. The Task 2 'non-sealed -> NotFound' test flips
to 'non-sealed with a blob is served'.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 added an in-repo .proto compiled by Grpc.Tools; its bundled linux_arm64 protoc
segfaults (exit 139) on Apple-Silicon Docker, so the native-arm64 build stage could not
generate the gRPC stubs. dotnet publish is framework-dependent (portable IL + every RID's
native assets), so pinning ONLY the build stage to linux/amd64 leaves the runtime image
native arm64 (fast) while the one-time build is emulated. Matches the Dockerfile's stated
'targeting linux-x64' intent.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 9. docker-dev rig: central-1/2 carry ConfigServe (:4055 h2c + committed
dev key) and stay Direct; the four site nodes carry ConfigSource (both central
endpoints + matching key) defaulting to Direct. Flip only the site nodes with
OTOPCUA_CONFIG_MODE=FetchAndCache at 'docker compose up' — central keeps SQL and models
the target topology. Binding :4055 activates the dedicated-h2c Kestrel takeover on
central (re-binds :9000 too). Docs: new docs/Configuration.md ConfigSource/ConfigServe
section; docs/Redundancy.md 'config bytes travel out-of-band' note; the program plan
Phase 3 section + tracking row flipped to CODE-COMPLETE; a CLAUDE.md 'Config source'
section.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 8. Stands a real h2c-only Kestrel listener serving the real
DeploymentArtifactGrpcService behind the real ConfigServeAuthInterceptor (in-memory
config DB seeded with one sealed deployment), and drives the real
GrpcDeploymentArtifactFetcher across it — proving a stream crosses the real boundary,
not just that the fetcher's logic is right (Task 4's fakes). Asserts: (1) right key +
>256 KB blob reassembles byte-equal and verifies against RevisionHash; (2) FALSIFIABILITY
CONTROL — wrong key returns null while the right key succeeds on the SAME server (so the
null is the auth gate, not a dead server); (3) unknown id -> null (NotFound-hidden);
(4) [dead-port, real-port] failover still returns the bytes. h2c GrpcChannel over http://
works out of the box in .NET 10. Tagged [Trait Category=ArtifactBoundary].
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 7. Regression net pinning the #485 negative on every FetchAndCache seam —
nothing rebuilds, nothing is torn down on a failure: (1) a null fetch mid-steady-state
(all endpoints down / NotFound / SHA mismatch — indistinguishable to the actor) keeps
the served address space and revision (no RebuildAddressSpace for the failed revision);
(2) a zero-length blob handed to the actor fails via ReconcileDriversFromBlob's own #485
guard, independent of the fetcher's null contract; (3) a corrupt cache at boot (surfaced
as GetCurrentUnkeyedAsync == null) lands Steady-no-revision and applies NOTHING — never
a partial address space. No new mechanism; all green as a net. Removing the #485 empty
guard reddens the zero-length test.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 6. In FetchAndCache mode Bootstrap() branches to BootstrapFromCache(),
which restores served state from the LocalDb pointer (GetCurrentUnkeyedAsync → set
_currentRevision → ApplyCachedArtifact from the cached bytes) with NO central-SQL read.
Distinct from the Direct-mode TryBootFromCache fallback: reading the cache is NORMAL
operation here so _isRunningFromCache stays false (the node can still fetch new
deploys). An empty OR unreadable cache lands Steady-with-no-revision (the first dispatch
fetches), never Stale — Stale means 'central SQL down, retry it', and FetchAndCache has
no config SQL read to recover. TryRecoverFromStale gains a defensive FetchAndCache guard
(it is unreachable in that mode, since the retry-db timer only starts in Stale).
Proven with a ThrowingDbFactory in every test: reaching Steady + applying a dispatch
proves the boot never touched central SQL. Sabotaging the mode branch (fall through to
the SQL read) reddens all three — test A flips RunningFromCache; B/C enter Stale, which
ignores the dispatch.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 5. Under ConfigSource:Mode=FetchAndCache, a DispatchDeployment whose
revision differs from _currentRevision kicks off a gRPC artifact fetch that PipeTo's
its result back to Self as FetchedForApply (never a blocking await in a receive — that
would freeze the mailbox for the fetch deadline). HandleFetchedForApply then applies
synchronously on the actor thread, mirroring ApplyAndAck: null bytes = apply FAILURE
(keep last-known-good, do not advance the revision, ack Failed — #485); non-null bytes
reconcile drivers, rebuild the address space and push subscriptions FROM the bytes in
hand (OpcUaPublishActor never reads central SQL), then cache them. A faulted fetch task
maps to null bytes so it can't escape as an unhandled message.
Extracted ReconcileDriversFromBlob from ReconcileDrivers so Direct-read, FetchAndCache
and boot-from-cache share one reconcile body (the #485 empty guard moves into it);
ApplyCachedArtifact now reuses it instead of duplicating the spawn-plan logic.
DI: DriverHostActor.Props gains fetchAndCacheMode + artifactFetcher (LAST, per the
positional-forwarding warning); Runtime resolves them from IOptions<ConfigSourceOptions>
(fetcher only in FetchAndCache mode); Host registers GrpcDeploymentArtifactFetcher under
hasDriver.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 4. GrpcDeploymentArtifactFetcher streams the artifact from the first
configured central endpoint that answers, reassembles the chunks, and verifies
SHA-256(bytes) lowercase-hex == the dispatched RevisionHash (byte-identical to
ConfigComposer's Convert.ToHexStringLower(SHA256.HashData(blob))). Every per-endpoint
problem — RpcException, zero-length stream, or hash mismatch — is logged and the next
endpoint tried; if none yield verified bytes it returns null (apply failure, keep
last-known-good — the #485 contract), never throwing into the actor loop. A
Func<endpoint, client> seam keeps the gRPC boundary out of the unit tests (that is
Task 8's job); the real path caches one h2c GrpcChannel per endpoint.
Files live under the .DeploymentCache namespace (folder Deployment/) to avoid shadowing
the Configuration.Entities.Deployment type in the test assembly — same convention as
LocalDbDeploymentArtifactCache.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 3. Central serves the deployment artifact over a dedicated h2c
Kestrel listener (ConfigServe:GrpcListenPort), gated by the ConfigServeAuthInterceptor
(shared bearer key, FixedTimeEquals, fail-closed, path-scoped to
/deployment_artifact.v1.DeploymentArtifactService/). The listener block is merged
with the LocalDb-sync listener so a fused admin+driver node re-applies its existing
HTTP surface exactly ONCE and adds both dedicated h2c ports on top — double-binding
would throw 'address already in use'; zero re-binding would silently unbind the
AdminUI behind Traefik. AddGrpc now registers under hasDriver || hasAdmin with both
path-scoped interceptors sharing one pipeline.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Streams a sealed deployment's ArtifactBlob to a fetching driver node in
128-KiB chunks (matching the LocalDb cache chunk size). Unknown id, a
non-sealed deployment, and a zero-length blob all collapse to one
indistinguishable NotFound — existence-hiding plus the #485 serve-side guard
(never stream empty bytes as a valid empty config). The impl is named
DeploymentArtifactGrpcService to avoid colliding with the generated
DeploymentArtifactService container type.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The repo has consumed packaged gRPC clients (LocalDb.Replication,
HistorianGateway.Client) but never compiled a proto locally. This adds the
Grpc.Tools toolchain to Commons and a deployment_artifact.proto with a single
server-streaming Fetch(deployment_id) -> stream ArtifactChunk RPC, generating
both the server base and the client stub (GrpcServices="Both") so central
(Host/AdminUI) and the node (Runtime) share one reference. A compile-touch test
pins the codegen: it stops compiling if generation regresses.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
ConfigSource:Mode = Direct (default, read central SQL) | FetchAndCache (fetch
the artifact from central over gRPC, read LocalDb). The validator fails host
start on a FetchAndCache shape that cannot fetch — no endpoints, a non-http(s)
endpoint, no shared key, or a non-positive timeout — because each otherwise
surfaces as a silent absence (a deploy that never applies). ConfigServe (the
central serve surface) takes a plain Configure: a 0 port is a valid "disabled".
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Eleven tasks. User decisions baked in: gRPC fetch RPC (not token-gated HTTP),
shared node bearer key (not a per-deployment token, so no migration and
DispatchDeployment is unchanged), a ConfigSource:Mode dark switch (Direct
default), both pair nodes fetch (no Primary gating). Header carries the five
recon facts and the Phase 3/4 boundary (config READS only; the NodeDeploymentState
write, DbHealthProbe and EfAlarmConditionStateStore stay for Phase 4).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Central↔node command transport behind the MeshTransport:Mode dark switch
(Dps default | ClusterClient). Both comm actors registered with the
receptionist in both modes; SendToAll fan-out, DB-sourced contacts, node-local
EventStream re-emit, ApplyAck relay. Live-gated on docker-dev end-to-end.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Flipped the docker-dev rig end-to-end to MeshTransport:Mode=ClusterClient
(OTOPCUA_MESH_MODE, no rebuild) and ran the gate. PASSED for the transport:
deploy seals on ClusterClient, a stopped node fails the deploy naming it, the
MaintenanceMode hatch shrinks the DB-sourced contact set live (6->5->6 on both
centrals independently), and the frame-size canary stayed silent.
Four findings, recorded in 2026-07-22-mesh-phase2-live-gate.md:
A. A 10-30s post-restart delivery window exists, but the DPS control fails
identically -- it is a cluster-convergence property, not a Phase 2
regression. Running the control is the only reason this is a note and not
a false attribution.
B. buffer-size=0's stated rationale was OVERSTATED. A node applies a TimedOut
deployment anyway -- live-observed at +44s -- via DriverHostActor.Bootstrap
replaying the orphan Applying row on restart. Zero buffering removes the
silent, timer-driven replay, not the boot-time DB one. Corrected in the
plan decisions table, Configuration.md, and CLAUDE.md; the decision stands,
the justification narrowed.
C. Stopping BOTH centrals strands the non-seed nodes at the Akka membership
layer (frozen mid-Exiting, not Unreachable, so auto-down does not help).
Transport-independent; a Phase 7 downing/rejoin drill item.
D. The AdminUI driver Restart/Reconnect buttons (DriverStatusPanel) are
rendered by no page -- orphaned since v3 Batch 2 retired /clusters/{id}/
drivers. So gate steps 6/7 could not run; the driver-control node-side leg
is proven live only because DispatchDeployment shares its EventStream path.
Step 8's planned log-signal does not exist (HandleApplyAck is silent on
success); proven instead by both centrals independently logging per-instance
ClusterClient creation, which a singleton cannot.
Also adds a repo-root .dockerignore: docker-dev/Dockerfile does COPY . ., which
was pulling .claude/worktrees (19GB of agent checkouts with their own bin/obj)
into the build context and exhausting the Docker VM disk mid-COPY.
Rig restored to Mode=Dps, six members, no leftover maintenance flags.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Rig: all six docker-dev nodes carry MeshTransport__Mode plus both central
contact points. Mode is "${OTOPCUA_MESH_MODE:-Dps}", so the rig still comes up
on the transport it has always used and the live gate can flip the whole fleet
at `docker compose up` without a compose edit or a rebuild.
Docs: a MeshTransport section in Configuration.md (including why contact points
are addresses only, why the contact set does not scope delivery, and why
buffer-size=0 is a behaviour decision rather than a tuning knob), a "Command
transport" section in Redundancy.md contrasting the two modes, and a summary
beside the Redundancy notes in CLAUDE.md.
Corrects the design doc's claim that ScadaBridge replies with a typed failure to
every unhandled message. Neither of their comm actors has a ReceiveAny or an
Unhandled override; the idiom is per message type and fires on a missing
registration, and an unknown type dead-letters there as it does anywhere else.
Records the exit-gate deviation in the program plan: "an Ask timing out cleanly
against a stopped node" cannot be run, because no Ask crosses this boundary. The
honest equivalents are a stopped node failing the deploy at the apply deadline
and an unreachable contact dropping the command with a Warning.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two genuinely separate single-member clusters sharing the ActorSystem name and
joined only by ClusterClient, so the mesh boundary is exercised over the wire
rather than through an in-process relay: a relay cannot fail for a missing
receptionist extension, an unregistered service, a malformed contact path or a
serialization break, which are the Phase 2 failure modes.
Covers both legs plus the control:
* central SendToAll -> node comm actor -> node EventStream
* node ApplyAck -> central comm actor, through the node's own client
* a node without the receptionist extension receives nothing
Sabotage-verified. Suppressing the node-side EventStream publish and the
outbound ack Tell reddens one leg each. Restoring the receptionist and the
service registration on the control node reddens the control, which is what
proves the control's send is live rather than silently misaddressed.
The control removes the extension AND the RegisterService call, since resolving
the receptionist to register would materialise the extension whose absence is
the point; it falsifies "delivery happens without a receptionist boundary", not
the extension line alone. Recorded in the test.
Also corrects MeshCommActorPathTests: with one node per side, per-node and
singleton registration are indistinguishable here, so the property the dropped
Task 6 assertion covered belongs to live-gate step 8, not to this class.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 7. The five central->node publish sites -- ConfigPublishCoordinator's
deploy dispatch, and AdminOperationsActor's restart, reconnect, acknowledge and
shelve -- now hand a MeshCommand to the transport router instead of telling the
DistributedPubSub mediator directly. Neither actor knows which transport is in
force any more.
_meshRouter is NULLABLE with a mediator fallback, and that is deliberate: about
a dozen existing coordinator and admin-operations tests construct these actors
without a router, and they must keep exercising the real DPS path rather than a
silently-disabled one. The fallback is not a permanent seam -- Phase 6 deletes
it with the DPS branch and the single mesh.
Wiring note: the comm actor's WithActors block moved ABOVE the singleton
registrations so both singletons can resolve it with registry.Get, which is the
ordering-by-registration idiom AdminOperationsActor already uses for the
coordinator. An earlier version used ActorSelection.ResolveOne().GetAwaiter()
.GetResult() inside the props factory -- that blocks inside actor construction
and races the very spawn it waits for.
Sabotage-verified by inverting the branch so the mediator always wins: all six
new tests go red, including the fallback test (which proves it is asserting the
mediator path rather than passing by accident).
Suites after: ControlPlane 118/118, Runtime 450/450, Host.IntegrationTests
196/202 -- sole failure AbCip_Green_AgainstSim, the fixture baseline that fails
on master too.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 6. Both comm actors are now spawned and receptionist-registered:
CentralCommunicationActor from WithOtOpcUaControlPlaneSingletons, and
NodeCommunicationActor from WithOtOpcUaRuntimeActors (before DriverHostActor,
because it is the host's ackRouter under ClusterClient mode; it has no
dependency of its own on the host, since inbound commands travel via the node's
EventStream).
Both are plain WithActors registrations, NOT singletons -- a driver node's
ClusterClient rotates across contact points and must find a live comm actor at
whichever node answers. Both register with the receptionist in BOTH transport
modes: an idle registration costs nothing, whereas registering only under
ClusterClient would mean flipping the flag on a running fleet needs a restart
before anything can be reached, turning a config change back into a deployment.
The coordinator ref is resolved lazily per message (Func<IActorRef?> +
registry.TryGet) so the comm actor never depends on the order in which
Akka.Hosting materialises the singleton proxy relative to this block.
Adds MeshCommActorPathTests, which boots the real two-node host and Identify-
probes both paths. These strings are the wire contract and nothing else asserts
they agree: a rename would compile, pass every unit test, deploy, and deliver
nothing, because a ClusterClient send to an unregistered path is dropped
silently. Sabotage-verified with an actual rename.
DELIBERATELY NOT TESTED, with the reason recorded in the file: that the actors
are per-node rather than singletons. Two discriminators were tried and both were
invalid -- "the path resolves on both nodes" passes under either shape because a
ClusterSingletonManager is also created on every node in the role, and "no
/singleton child" is null even for the KNOWN singleton /user/config-publish, so
Akka.Hosting does not lay them out that way. A sabotage re-registering the actor
via WithSingleton left both green, which is what exposed them. Rather than ship
an assertion that cannot fail, the property is left to the Task 8 boundary test,
where a send only arrives if the target really is registered.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 5. DriverHostActor and ScriptedAlarmHostActor now subscribe to the
node-local EventStream alongside their existing DPS topic subscriptions, so the
transport flag lives ENTIRELY on the publishing side -- exactly one of the two
ever publishes, and the switch can be flipped or reverted without touching a
single subscriber.
Note the alarm case is not symmetric: the DPS subscribe there still carries the
node-LOCAL publisher (OtOpcUaServerHostedService's OPC UA Part 9 method calls),
which never crosses the boundary and stays on DPS in both modes. Only the
central AdminUI leg moves.
Renamed DriverHostActor._coordinatorOverride to _ackRouter. Under ClusterClient
mode that ref is the NodeCommunicationActor, emphatically not a coordinator,
and the old name would send the next reader looking for one.
Adds two wiring tests, because the unit tests either side of this seam BOTH pass
with the subscription deleted -- the comm actor's test asserts a probe receives
the message, and the host tests drive the actor by direct Tell. Neither notices
a missing PreStart subscribe. Sabotage-verified: deleting either subscription
reddens exactly its own wiring test and nothing else.
Three things went wrong while writing those tests, all worth keeping:
- The first version raced. ActorOf returns before PreStart runs, and an
EventStream.Publish with no subscriber is dropped silently -- no buffering, no
dead letter. Fixed with a warm-up whose ack proves PreStart completed; the
comment says why it is not ceremony.
- The alarm version was VACUOUS: it told the host directly INSIDE the assertion
window alongside the relay, so the direct Tell alone produced the log the
assertion waited for. Split into warm-up (outside) and relay (inside).
- The alarm test needs its own ActorSystem: the shared harness pins
loglevel = WARNING and the only signal an unowned alarm gives is a Debug line.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 4. Node-side end of the boundary, registered with the receptionist
per node rather than as a singleton so central's contact rotation reaches
whichever node answers.
Inbound commands are re-emitted on the node's EventStream, NOT back onto their
DistributedPubSub topic. DPS is mesh-wide and central SendToAll-s to every
node's comm actor, so a DPS republish would deliver N copies of every command
to every subscriber. The EventStream is node-local by construction. It also
avoids a registration handshake with actors spawned later: ScriptedAlarmHostActor
is a CHILD of DriverHostActor and has no registry key, so there is no ref to
hand in at wiring time.
Each inbound type is listed explicitly rather than caught by a ReceiveAny, so an
unknown type dead-letters loudly instead of being republished blind onto a
stream where every subscriber ignores it.
Outbound is ApplyAck only, and it uses Send rather than SendToAll -- the deploy
coordinator is one singleton behind central's proxy, so SendToAll would deliver
one ack per central node and the coordinator would count this node twice.
Notably there is NO Ask across this boundary in Phase 2: every migrated command
is fire-and-forget, which is why this actor is far smaller than the sister
project's equivalent and needs no sender preservation at all.
Sabotage-verified twice: deleting the AlarmCommand handler and flipping Send to
SendToAll each redden exactly their own test.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 3. Central-side end of the boundary: routes every central->node
command out over DPS or ClusterClient per MeshTransport:Mode, and forwards
inbound ApplyAcks to the deploy coordinator singleton (Forward, not Tell, so
the coordinator sees the node rather than this relay).
Three things here are load-bearing and easy to get wrong later:
SendToAll, never Send. Today's DPS publish reaches EVERY DriverHostActor and
the node side has no ClusterId or node filter to compensate -- scoping happens
later, inside the artifact. ClusterClient.Send delivers to exactly ONE
registered actor, so it would deploy to a single node while every other node
silently kept its old config, and the deployment could still seal green.
Sabotage-verified: swapping to Send reddens exactly that one test.
Exactly ONE ClusterClient, fleet-wide. A receptionist serves its whole cluster,
so SendToAll reaches every registered node-comm actor in the mesh regardless of
which contact point was dialled. One client per application Cluster -- the
shape Phase 6 wants -- would fan each command out once per cluster while the
fleet is still one mesh: N x duplicate DispatchDeployment and ApplyAck. Marked
TODO(Phase 6).
The contact set does NOT scope delivery. Excluding a maintenance-mode node from
the contacts does not stop it receiving commands; it still receives them, as it
does under today's broadcast. The filter is about which receptionists are worth
dialling, not about who gets the message. Stated in the code because the
opposite is the natural assumption.
Also carries the sister project's shipped fixes: per-row address parsing so one
malformed ClusterNode cannot abort the refresh and leave the contact set
half-built (regression test orders the bad row FIRST), the cache entry cleared
before recreate so a failed create cannot leave commands routing into a
stopping actor, and a Status.Failure handler so a DB outage is a Warning rather
than a silent debug-level unhandled message.
Two test-harness bugs found and fixed while writing this, both mine: SubscribeAck
goes to the SENDER (so the mediator subscribe needed an explicit sender), and
EventFilter matches case-INSENSITIVELY, so a "was NOT" filter also caught this
actor's own "was not created" warning.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 2. Two small pieces the comm actors are built on.
MeshCommand carries a command plus the DPS topic it would have been published
on, so the admin singletons stop knowing which transport is in force. Without
it the dark switch would have to be threaded through five publish sites across
two actors, each free to drift. Central-side only -- it never crosses the wire,
so it carries no serialization contract.
DefaultMeshClusterClientFactory appends a generation counter to the actor name.
Context.Stop is asynchronous and the name stays reserved until termination
completes, so recreating the same name inside one message handler throws
InvalidActorNameException -- which is exactly what a contact-set change does
(stop the old client, create the replacement, one handler). Ported from the
sister project, where it was a shipped bug fix rather than foresight.
Sabotage-verified: freezing the name to a constant reddens both name tests with
InvalidActorNameException.
Dropped a third test I had written -- it asserted ClusterClientSettings
directly and never touched the factory, so it was coverage theatre. Replaced
with a comment saying where contact propagation IS covered (the Task 8 boundary
test, the only place it can actually fail).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 1. Three changes to the embedded base config, plus a test that
reads every one of them back off a RUNNING ActorSystem rather than off the
HOCON text -- the settled idiom here (SplitBrainResolverActivationTests),
because several fragments compete and the losing one still reads correctly in
the file.
- Registers the ClusterClientReceptionist extension. It is not auto-started, so
without this the boundary only listens if some code path happens to call
ClusterClientReceptionist.Get(system) first: "is the boundary up?" would
depend on startup ordering rather than on configuration.
- buffer-size = 0. This is the one genuine behaviour change and it is NOT what
the sister project runs -- they never wrote this section and inherit 1000.
Buffering would replay a DispatchDeployment on reconnect and apply a
deployment the coordinator already sealed as TimedOut, leaving the node on a
configuration the database says failed.
- log-frame-size-exceeding = 32000b. An oversized frame is dropped WITHOUT
tearing down the association: heartbeats keep flowing, the node reports
healthy, and the only symptom is a command that never arrived. Our deploy
notify is payload-free, so this canary should stay quiet.
reconnect-timeout and receptionist.role restate Akka defaults; they are stated
explicitly and pinned because the behaviour is load-bearing, and the comments
say which is which.
One test was VACUOUS on first write: Config.GetInt returns 0 for a missing key,
so the buffering assertion passed before the feature existed -- it could not
distinguish "explicitly 0" from "absent, default 1000 in force". Rewritten to
assert through ClusterClientSettings/ClusterReceptionistSettings, the objects
the client is actually built from. Sabotage-verified after the fix: setting
buffer-size back to 1000 reddens exactly that one test.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 0. Binds a validated `MeshTransport` section selecting which
transport carries central->node commands, defaulting to Dps so nothing changes
until the rig gate passes.
The validator exists because every fault it catches otherwise surfaces as an
ABSENCE -- a deployment that never arrives, an alarm ack that does nothing --
which has no stack trace and no failing node to point at. Two of the shapes it
rejects are ported from the sister project's shipped mistakes: a contact point
carrying an actor-path suffix, and (documented in the options XML) a template
listing the node's OWN remoting port as a central contact, which is a permanent
failure in the initial-contact rotation.
Contact points are required only under ClusterClient mode; requiring them under
the default would make the section mandatory on admin-only nodes that have no
reason to carry them.
Sabotage-verified: relaxing the empty-contacts guard and the actor-path-suffix
guard turns exactly those two tests red, and nothing else.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 of the per-cluster mesh program: move the three central->node command
channels (deployments, driver-control, alarm-commands) and the deployment-acks
reply channel off DistributedPubSub onto Akka ClusterClient, behind a config
flag.
Recon of both repos turned up three corrections to the design doc's account of
the sister project, all folded into the plan:
- Design doc S2 claims "every unhandled message replies with a typed failure".
ScadaBridge has no ReceiveAny/Unhandled override in either comm actor; the
typed-failure idiom fires only for a MISSING REGISTRATION, per message type.
- "No central buffering" was never a setting they wrote. They have zero
akka.cluster.client HOCON and run the defaults (buffer-size 1000,
reconnect-timeout off). We choose buffer-size = 0 deliberately.
- Publish -> Send is the wrong substitution. Today's deploy notify is a
broadcast every DriverHostActor receives (no ClusterId filter exists on the
node side), so it must become SendToAll. Send would deploy to exactly one
node of the fleet and seal green on partial acks.
Also records the single-mesh duplicate-delivery trap (one ClusterClient in
Phase 2, not one per Cluster -- a receptionist serves its whole mesh) and one
deviation from the program plan's exit gate: there is no cross-boundary Ask in
Phase 2, because every migrated command is fire-and-forget with a local reply,
so "an Ask timing out cleanly" cannot be run as written.
Marks the program tracking tables current: prereq + Phase 1 are DONE (they
still read "not executed"), Phase 2 in progress.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Closes the last branch-only failures. Measured across five full-suite runs on
this branch versus two on master in a clean worktree: master failed only AbCip
in this assembly (2/2), while the branch additionally failed one or two E2E
deploy tests, rotating between DeployHappyPath, DriverReconnect and
EquipmentNamespaceMaterialization (4/4). Each passed in isolation, and the
assembly on its own was 195/201 throughout — the failures only appeared under
full-suite CPU contention.
Cause is contention, not correctness. Every test here builds a real two-node
Akka cluster — two Kestrel hosts, two ActorSystems with remoting, six admin
singletons, an EF context, a deploy pipeline — and xUnit ran them concurrently
with each other and with every other assembly. That was survivable while the
coordinator sealed instantly on an empty expected-ack set; now that it waits for
a real ack from every configured node, these tests measure an end-to-end
round-trip and starve.
Serialising this one assembly makes them deterministic: 195/201 with only
AbCip_Green_AgainstSim, and the full-solution failure set is now IDENTICAL to
master's — 7 shared, zero branch-only. Cost is wall-clock for this assembly,
40s -> 3m35s; it is a handful of heavyweight E2E tests, not a broad unit suite,
and a widened timeout alone did not fix it (tried first, at 45s).
Also drops createProxyToo for the address reconciler. Nothing resolves
ClusterNodeAddressReconcilerKey from the registry — the actor only reads cluster
state and logs — so the proxy was a second actor per node hunting for a
singleton nobody addresses. Kept because it removes dead machinery, not because
it fixed the flakiness; measured on its own, it did not.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Comparing full-suite runs on this branch against master in a clean worktree:
both fail 11 tests, 9 identical. Master's two extras are load-flaky unit tests
(AbCip Probe_loops, Galaxy EventPumpBoundedChannel); this branch's two extras
were both Host.IntegrationTests deploy-path tests — the area this change
re-times — so they are attributable here rather than to background flakiness.
Cause is margin, not correctness. These waits used to observe a coordinator
that sealed instantly on an empty expected-ack set; they now observe a real
ApplyAck round-trip from every configured node. 15s was enormous margin against
"instant" and thin against the real thing, so they failed only under full-suite
CPU contention.
Hoisted to TwoNodeClusterHarness.DeploySealTimeout (45s) so the reason is
recorded once rather than as four unexplained numbers.
DriverReconnectE2eTests is deliberately left alone: it seeds both ClusterNode
rows itself and unconditionally, so its expected-ack set is identical before and
after this change. Widening its timeout would be papering over a flake this
change did not cause.
Host.IntegrationTests 195/201; sole failure AbCip_Green_AgainstSim, verified
failing on master.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The full-suite run surfaced two load-dependent failures that were green in
isolation. Root cause: SeedDefaultClusterAsync no-opped unless
OTOPCUA_HARNESS_USE_SQL=1, on the reasoning that "the in-memory provider ignores
FK constraints, so deploy E2E tests pass without it". Phase 1 invalidated that.
With no ClusterNode rows the coordinator's expected-ack set is empty, so it
seals IMMEDIATELY. "Wait for Sealed" therefore stopped implying "every node has
applied" — and because DriverHostActor.UpsertNodeDeploymentState writes each
node's row on its own schedule, every test counting those rows after a seal
became a race. Reproducibly green alone, red under suite load, and it surfaced
in a different test each run, which is what made it look like flakiness rather
than a contract change.
Seeding in both modes restores the invariant those tests were written against
and is closer to production either way: a real fleet always has these rows,
because the FK requires them.
Deployment_started_with_node_b_down_seals_with_one_node_state asserted the
behaviour Phase 1 deliberately removed — its own comment documented membership
snapshotting, i.e. sealing green while a configured node never received the
deployment. Replaced by two tests covering the new contract: a stopped node is
still expected (both state rows exist, B still Applying, deployment does not
seal), and MaintenanceMode is what makes it seal with one.
The new "does not seal" test asserts both rows exist rather than only the
absence of a seal, so it cannot pass against a coordinator that simply died.
Host.IntegrationTests 195/201, sole remaining failure AbCip_Green_AgainstSim —
verified failing on master in a clean worktree, so pre-existing fixture
baseline, not a regression.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty