Phase 3 Task 8. Stands a real h2c-only Kestrel listener serving the real
DeploymentArtifactGrpcService behind the real ConfigServeAuthInterceptor (in-memory
config DB seeded with one sealed deployment), and drives the real
GrpcDeploymentArtifactFetcher across it — proving a stream crosses the real boundary,
not just that the fetcher's logic is right (Task 4's fakes). Asserts: (1) right key +
>256 KB blob reassembles byte-equal and verifies against RevisionHash; (2) FALSIFIABILITY
CONTROL — wrong key returns null while the right key succeeds on the SAME server (so the
null is the auth gate, not a dead server); (3) unknown id -> null (NotFound-hidden);
(4) [dead-port, real-port] failover still returns the bytes. h2c GrpcChannel over http://
works out of the box in .NET 10. Tagged [Trait Category=ArtifactBoundary].
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 7. Regression net pinning the #485 negative on every FetchAndCache seam —
nothing rebuilds, nothing is torn down on a failure: (1) a null fetch mid-steady-state
(all endpoints down / NotFound / SHA mismatch — indistinguishable to the actor) keeps
the served address space and revision (no RebuildAddressSpace for the failed revision);
(2) a zero-length blob handed to the actor fails via ReconcileDriversFromBlob's own #485
guard, independent of the fetcher's null contract; (3) a corrupt cache at boot (surfaced
as GetCurrentUnkeyedAsync == null) lands Steady-no-revision and applies NOTHING — never
a partial address space. No new mechanism; all green as a net. Removing the #485 empty
guard reddens the zero-length test.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 6. In FetchAndCache mode Bootstrap() branches to BootstrapFromCache(),
which restores served state from the LocalDb pointer (GetCurrentUnkeyedAsync → set
_currentRevision → ApplyCachedArtifact from the cached bytes) with NO central-SQL read.
Distinct from the Direct-mode TryBootFromCache fallback: reading the cache is NORMAL
operation here so _isRunningFromCache stays false (the node can still fetch new
deploys). An empty OR unreadable cache lands Steady-with-no-revision (the first dispatch
fetches), never Stale — Stale means 'central SQL down, retry it', and FetchAndCache has
no config SQL read to recover. TryRecoverFromStale gains a defensive FetchAndCache guard
(it is unreachable in that mode, since the retry-db timer only starts in Stale).
Proven with a ThrowingDbFactory in every test: reaching Steady + applying a dispatch
proves the boot never touched central SQL. Sabotaging the mode branch (fall through to
the SQL read) reddens all three — test A flips RunningFromCache; B/C enter Stale, which
ignores the dispatch.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 5. Under ConfigSource:Mode=FetchAndCache, a DispatchDeployment whose
revision differs from _currentRevision kicks off a gRPC artifact fetch that PipeTo's
its result back to Self as FetchedForApply (never a blocking await in a receive — that
would freeze the mailbox for the fetch deadline). HandleFetchedForApply then applies
synchronously on the actor thread, mirroring ApplyAndAck: null bytes = apply FAILURE
(keep last-known-good, do not advance the revision, ack Failed — #485); non-null bytes
reconcile drivers, rebuild the address space and push subscriptions FROM the bytes in
hand (OpcUaPublishActor never reads central SQL), then cache them. A faulted fetch task
maps to null bytes so it can't escape as an unhandled message.
Extracted ReconcileDriversFromBlob from ReconcileDrivers so Direct-read, FetchAndCache
and boot-from-cache share one reconcile body (the #485 empty guard moves into it);
ApplyCachedArtifact now reuses it instead of duplicating the spawn-plan logic.
DI: DriverHostActor.Props gains fetchAndCacheMode + artifactFetcher (LAST, per the
positional-forwarding warning); Runtime resolves them from IOptions<ConfigSourceOptions>
(fetcher only in FetchAndCache mode); Host registers GrpcDeploymentArtifactFetcher under
hasDriver.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 4. GrpcDeploymentArtifactFetcher streams the artifact from the first
configured central endpoint that answers, reassembles the chunks, and verifies
SHA-256(bytes) lowercase-hex == the dispatched RevisionHash (byte-identical to
ConfigComposer's Convert.ToHexStringLower(SHA256.HashData(blob))). Every per-endpoint
problem — RpcException, zero-length stream, or hash mismatch — is logged and the next
endpoint tried; if none yield verified bytes it returns null (apply failure, keep
last-known-good — the #485 contract), never throwing into the actor loop. A
Func<endpoint, client> seam keeps the gRPC boundary out of the unit tests (that is
Task 8's job); the real path caches one h2c GrpcChannel per endpoint.
Files live under the .DeploymentCache namespace (folder Deployment/) to avoid shadowing
the Configuration.Entities.Deployment type in the test assembly — same convention as
LocalDbDeploymentArtifactCache.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 3. Central serves the deployment artifact over a dedicated h2c
Kestrel listener (ConfigServe:GrpcListenPort), gated by the ConfigServeAuthInterceptor
(shared bearer key, FixedTimeEquals, fail-closed, path-scoped to
/deployment_artifact.v1.DeploymentArtifactService/). The listener block is merged
with the LocalDb-sync listener so a fused admin+driver node re-applies its existing
HTTP surface exactly ONCE and adds both dedicated h2c ports on top — double-binding
would throw 'address already in use'; zero re-binding would silently unbind the
AdminUI behind Traefik. AddGrpc now registers under hasDriver || hasAdmin with both
path-scoped interceptors sharing one pipeline.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Streams a sealed deployment's ArtifactBlob to a fetching driver node in
128-KiB chunks (matching the LocalDb cache chunk size). Unknown id, a
non-sealed deployment, and a zero-length blob all collapse to one
indistinguishable NotFound — existence-hiding plus the #485 serve-side guard
(never stream empty bytes as a valid empty config). The impl is named
DeploymentArtifactGrpcService to avoid colliding with the generated
DeploymentArtifactService container type.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two genuinely separate single-member clusters sharing the ActorSystem name and
joined only by ClusterClient, so the mesh boundary is exercised over the wire
rather than through an in-process relay: a relay cannot fail for a missing
receptionist extension, an unregistered service, a malformed contact path or a
serialization break, which are the Phase 2 failure modes.
Covers both legs plus the control:
* central SendToAll -> node comm actor -> node EventStream
* node ApplyAck -> central comm actor, through the node's own client
* a node without the receptionist extension receives nothing
Sabotage-verified. Suppressing the node-side EventStream publish and the
outbound ack Tell reddens one leg each. Restoring the receptionist and the
service registration on the control node reddens the control, which is what
proves the control's send is live rather than silently misaddressed.
The control removes the extension AND the RegisterService call, since resolving
the receptionist to register would materialise the extension whose absence is
the point; it falsifies "delivery happens without a receptionist boundary", not
the extension line alone. Recorded in the test.
Also corrects MeshCommActorPathTests: with one node per side, per-node and
singleton registration are indistinguishable here, so the property the dropped
Task 6 assertion covered belongs to live-gate step 8, not to this class.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 7. The five central->node publish sites -- ConfigPublishCoordinator's
deploy dispatch, and AdminOperationsActor's restart, reconnect, acknowledge and
shelve -- now hand a MeshCommand to the transport router instead of telling the
DistributedPubSub mediator directly. Neither actor knows which transport is in
force any more.
_meshRouter is NULLABLE with a mediator fallback, and that is deliberate: about
a dozen existing coordinator and admin-operations tests construct these actors
without a router, and they must keep exercising the real DPS path rather than a
silently-disabled one. The fallback is not a permanent seam -- Phase 6 deletes
it with the DPS branch and the single mesh.
Wiring note: the comm actor's WithActors block moved ABOVE the singleton
registrations so both singletons can resolve it with registry.Get, which is the
ordering-by-registration idiom AdminOperationsActor already uses for the
coordinator. An earlier version used ActorSelection.ResolveOne().GetAwaiter()
.GetResult() inside the props factory -- that blocks inside actor construction
and races the very spawn it waits for.
Sabotage-verified by inverting the branch so the mediator always wins: all six
new tests go red, including the fallback test (which proves it is asserting the
mediator path rather than passing by accident).
Suites after: ControlPlane 118/118, Runtime 450/450, Host.IntegrationTests
196/202 -- sole failure AbCip_Green_AgainstSim, the fixture baseline that fails
on master too.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 6. Both comm actors are now spawned and receptionist-registered:
CentralCommunicationActor from WithOtOpcUaControlPlaneSingletons, and
NodeCommunicationActor from WithOtOpcUaRuntimeActors (before DriverHostActor,
because it is the host's ackRouter under ClusterClient mode; it has no
dependency of its own on the host, since inbound commands travel via the node's
EventStream).
Both are plain WithActors registrations, NOT singletons -- a driver node's
ClusterClient rotates across contact points and must find a live comm actor at
whichever node answers. Both register with the receptionist in BOTH transport
modes: an idle registration costs nothing, whereas registering only under
ClusterClient would mean flipping the flag on a running fleet needs a restart
before anything can be reached, turning a config change back into a deployment.
The coordinator ref is resolved lazily per message (Func<IActorRef?> +
registry.TryGet) so the comm actor never depends on the order in which
Akka.Hosting materialises the singleton proxy relative to this block.
Adds MeshCommActorPathTests, which boots the real two-node host and Identify-
probes both paths. These strings are the wire contract and nothing else asserts
they agree: a rename would compile, pass every unit test, deploy, and deliver
nothing, because a ClusterClient send to an unregistered path is dropped
silently. Sabotage-verified with an actual rename.
DELIBERATELY NOT TESTED, with the reason recorded in the file: that the actors
are per-node rather than singletons. Two discriminators were tried and both were
invalid -- "the path resolves on both nodes" passes under either shape because a
ClusterSingletonManager is also created on every node in the role, and "no
/singleton child" is null even for the KNOWN singleton /user/config-publish, so
Akka.Hosting does not lay them out that way. A sabotage re-registering the actor
via WithSingleton left both green, which is what exposed them. Rather than ship
an assertion that cannot fail, the property is left to the Task 8 boundary test,
where a send only arrives if the target really is registered.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 5. DriverHostActor and ScriptedAlarmHostActor now subscribe to the
node-local EventStream alongside their existing DPS topic subscriptions, so the
transport flag lives ENTIRELY on the publishing side -- exactly one of the two
ever publishes, and the switch can be flipped or reverted without touching a
single subscriber.
Note the alarm case is not symmetric: the DPS subscribe there still carries the
node-LOCAL publisher (OtOpcUaServerHostedService's OPC UA Part 9 method calls),
which never crosses the boundary and stays on DPS in both modes. Only the
central AdminUI leg moves.
Renamed DriverHostActor._coordinatorOverride to _ackRouter. Under ClusterClient
mode that ref is the NodeCommunicationActor, emphatically not a coordinator,
and the old name would send the next reader looking for one.
Adds two wiring tests, because the unit tests either side of this seam BOTH pass
with the subscription deleted -- the comm actor's test asserts a probe receives
the message, and the host tests drive the actor by direct Tell. Neither notices
a missing PreStart subscribe. Sabotage-verified: deleting either subscription
reddens exactly its own wiring test and nothing else.
Three things went wrong while writing those tests, all worth keeping:
- The first version raced. ActorOf returns before PreStart runs, and an
EventStream.Publish with no subscriber is dropped silently -- no buffering, no
dead letter. Fixed with a warm-up whose ack proves PreStart completed; the
comment says why it is not ceremony.
- The alarm version was VACUOUS: it told the host directly INSIDE the assertion
window alongside the relay, so the direct Tell alone produced the log the
assertion waited for. Split into warm-up (outside) and relay (inside).
- The alarm test needs its own ActorSystem: the shared harness pins
loglevel = WARNING and the only signal an unowned alarm gives is a Debug line.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 4. Node-side end of the boundary, registered with the receptionist
per node rather than as a singleton so central's contact rotation reaches
whichever node answers.
Inbound commands are re-emitted on the node's EventStream, NOT back onto their
DistributedPubSub topic. DPS is mesh-wide and central SendToAll-s to every
node's comm actor, so a DPS republish would deliver N copies of every command
to every subscriber. The EventStream is node-local by construction. It also
avoids a registration handshake with actors spawned later: ScriptedAlarmHostActor
is a CHILD of DriverHostActor and has no registry key, so there is no ref to
hand in at wiring time.
Each inbound type is listed explicitly rather than caught by a ReceiveAny, so an
unknown type dead-letters loudly instead of being republished blind onto a
stream where every subscriber ignores it.
Outbound is ApplyAck only, and it uses Send rather than SendToAll -- the deploy
coordinator is one singleton behind central's proxy, so SendToAll would deliver
one ack per central node and the coordinator would count this node twice.
Notably there is NO Ask across this boundary in Phase 2: every migrated command
is fire-and-forget, which is why this actor is far smaller than the sister
project's equivalent and needs no sender preservation at all.
Sabotage-verified twice: deleting the AlarmCommand handler and flipping Send to
SendToAll each redden exactly their own test.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 3. Central-side end of the boundary: routes every central->node
command out over DPS or ClusterClient per MeshTransport:Mode, and forwards
inbound ApplyAcks to the deploy coordinator singleton (Forward, not Tell, so
the coordinator sees the node rather than this relay).
Three things here are load-bearing and easy to get wrong later:
SendToAll, never Send. Today's DPS publish reaches EVERY DriverHostActor and
the node side has no ClusterId or node filter to compensate -- scoping happens
later, inside the artifact. ClusterClient.Send delivers to exactly ONE
registered actor, so it would deploy to a single node while every other node
silently kept its old config, and the deployment could still seal green.
Sabotage-verified: swapping to Send reddens exactly that one test.
Exactly ONE ClusterClient, fleet-wide. A receptionist serves its whole cluster,
so SendToAll reaches every registered node-comm actor in the mesh regardless of
which contact point was dialled. One client per application Cluster -- the
shape Phase 6 wants -- would fan each command out once per cluster while the
fleet is still one mesh: N x duplicate DispatchDeployment and ApplyAck. Marked
TODO(Phase 6).
The contact set does NOT scope delivery. Excluding a maintenance-mode node from
the contacts does not stop it receiving commands; it still receives them, as it
does under today's broadcast. The filter is about which receptionists are worth
dialling, not about who gets the message. Stated in the code because the
opposite is the natural assumption.
Also carries the sister project's shipped fixes: per-row address parsing so one
malformed ClusterNode cannot abort the refresh and leave the contact set
half-built (regression test orders the bad row FIRST), the cache entry cleared
before recreate so a failed create cannot leave commands routing into a
stopping actor, and a Status.Failure handler so a DB outage is a Warning rather
than a silent debug-level unhandled message.
Two test-harness bugs found and fixed while writing this, both mine: SubscribeAck
goes to the SENDER (so the mediator subscribe needed an explicit sender), and
EventFilter matches case-INSENSITIVELY, so a "was NOT" filter also caught this
actor's own "was not created" warning.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 2. Two small pieces the comm actors are built on.
MeshCommand carries a command plus the DPS topic it would have been published
on, so the admin singletons stop knowing which transport is in force. Without
it the dark switch would have to be threaded through five publish sites across
two actors, each free to drift. Central-side only -- it never crosses the wire,
so it carries no serialization contract.
DefaultMeshClusterClientFactory appends a generation counter to the actor name.
Context.Stop is asynchronous and the name stays reserved until termination
completes, so recreating the same name inside one message handler throws
InvalidActorNameException -- which is exactly what a contact-set change does
(stop the old client, create the replacement, one handler). Ported from the
sister project, where it was a shipped bug fix rather than foresight.
Sabotage-verified: freezing the name to a constant reddens both name tests with
InvalidActorNameException.
Dropped a third test I had written -- it asserted ClusterClientSettings
directly and never touched the factory, so it was coverage theatre. Replaced
with a comment saying where contact propagation IS covered (the Task 8 boundary
test, the only place it can actually fail).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Closes the last branch-only failures. Measured across five full-suite runs on
this branch versus two on master in a clean worktree: master failed only AbCip
in this assembly (2/2), while the branch additionally failed one or two E2E
deploy tests, rotating between DeployHappyPath, DriverReconnect and
EquipmentNamespaceMaterialization (4/4). Each passed in isolation, and the
assembly on its own was 195/201 throughout — the failures only appeared under
full-suite CPU contention.
Cause is contention, not correctness. Every test here builds a real two-node
Akka cluster — two Kestrel hosts, two ActorSystems with remoting, six admin
singletons, an EF context, a deploy pipeline — and xUnit ran them concurrently
with each other and with every other assembly. That was survivable while the
coordinator sealed instantly on an empty expected-ack set; now that it waits for
a real ack from every configured node, these tests measure an end-to-end
round-trip and starve.
Serialising this one assembly makes them deterministic: 195/201 with only
AbCip_Green_AgainstSim, and the full-solution failure set is now IDENTICAL to
master's — 7 shared, zero branch-only. Cost is wall-clock for this assembly,
40s -> 3m35s; it is a handful of heavyweight E2E tests, not a broad unit suite,
and a widened timeout alone did not fix it (tried first, at 45s).
Also drops createProxyToo for the address reconciler. Nothing resolves
ClusterNodeAddressReconcilerKey from the registry — the actor only reads cluster
state and logs — so the proxy was a second actor per node hunting for a
singleton nobody addresses. Kept because it removes dead machinery, not because
it fixed the flakiness; measured on its own, it did not.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Comparing full-suite runs on this branch against master in a clean worktree:
both fail 11 tests, 9 identical. Master's two extras are load-flaky unit tests
(AbCip Probe_loops, Galaxy EventPumpBoundedChannel); this branch's two extras
were both Host.IntegrationTests deploy-path tests — the area this change
re-times — so they are attributable here rather than to background flakiness.
Cause is margin, not correctness. These waits used to observe a coordinator
that sealed instantly on an empty expected-ack set; they now observe a real
ApplyAck round-trip from every configured node. 15s was enormous margin against
"instant" and thin against the real thing, so they failed only under full-suite
CPU contention.
Hoisted to TwoNodeClusterHarness.DeploySealTimeout (45s) so the reason is
recorded once rather than as four unexplained numbers.
DriverReconnectE2eTests is deliberately left alone: it seeds both ClusterNode
rows itself and unconditionally, so its expected-ack set is identical before and
after this change. Widening its timeout would be papering over a flake this
change did not cause.
Host.IntegrationTests 195/201; sole failure AbCip_Green_AgainstSim, verified
failing on master.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The full-suite run surfaced two load-dependent failures that were green in
isolation. Root cause: SeedDefaultClusterAsync no-opped unless
OTOPCUA_HARNESS_USE_SQL=1, on the reasoning that "the in-memory provider ignores
FK constraints, so deploy E2E tests pass without it". Phase 1 invalidated that.
With no ClusterNode rows the coordinator's expected-ack set is empty, so it
seals IMMEDIATELY. "Wait for Sealed" therefore stopped implying "every node has
applied" — and because DriverHostActor.UpsertNodeDeploymentState writes each
node's row on its own schedule, every test counting those rows after a seal
became a race. Reproducibly green alone, red under suite load, and it surfaced
in a different test each run, which is what made it look like flakiness rather
than a contract change.
Seeding in both modes restores the invariant those tests were written against
and is closer to production either way: a real fleet always has these rows,
because the FK requires them.
Deployment_started_with_node_b_down_seals_with_one_node_state asserted the
behaviour Phase 1 deliberately removed — its own comment documented membership
snapshotting, i.e. sealing green while a configured node never received the
deployment. Replaced by two tests covering the new contract: a stopped node is
still expected (both state rows exist, B still Applying, deployment does not
seal), and MaintenanceMode is what makes it seal with one.
The new "does not seal" test asserts both rows exist rather than only the
absence of a seal, so it cannot pass against a coordinator that simply died.
Host.IntegrationTests 195/201, sole remaining failure AbCip_Green_AgainstSim —
verified failing on master in a clean worktree, so pre-existing fixture
baseline, not a regression.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service
returns 422 ClusterEnabledNodeCountMismatch, because
DraftValidator.ValidateClusterTopology requires the enabled-node count to equal
ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and
every cluster in the target topology — Enabled can therefore never be the
maintenance hatch. The plan called step 4 "the check that makes step 3
acceptable to ship", so Task 3's behaviour change was not shippable as it stood:
a node down for maintenance would block every deployment to its cluster.
The two rules were each reasonable and contradictory together — the validator
reads Enabled as "part of the declared topology", Phase 1 additionally read it
as "expect an ack". Split the meanings rather than weaken either rule:
Enabled part of the declared topology (validator, untouched)
MaintenanceMode expected to participate now (coordinator + reconciler)
Rejected alternatives: counting configured rather than enabled nodes (drops the
guard against booting a pair into InvalidTopology); downgrading the rule to a
warning (weakens a deploy gate for everyone); shipping with no hatch.
Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new
test red. It also asserts both nodes remain Enabled, so the fix cannot quietly
regress to disabling the row after all.
Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
ClusterNode.AkkaPort and the node's own Cluster:Port are the same fact in two
places and nothing made them agree. Phase 2 dials the row instead of gossiping,
so a node binding 4054 while its row says 4053 becomes unreachable from central
— and the symptom is a silent absence of acks rather than an error. That is the
shape of the Modbus/ModbusTcp and TwinCat/Focas drifts already in this repo.
Since Phase 1 also made the rows the deploy path's expected-ack set, drift
already costs a failed deployment today.
Implemented as the plan's preferred option: an admin-role singleton comparing
rows against the membership an admin node can already see, rather than each
driver node asserting its own row — Phase 4 removes the driver nodes' ConfigDb
connection, so a self-assertion written there would have to be deleted again.
Three shapes, split by severity: a row whose dial target disagrees with its own
NodeId and a running node with no row are Errors; an enabled row with no
matching member is a Warning, because a node down for maintenance is a
legitimate state. Findings are logged only when the set changes — a check that
reprints the same warning every sweep trains operators to filter it out.
Documented limitation: Phase 2 must revisit this. Once the fleet splits into
one mesh per cluster an admin node cannot see site members, and every site row
would report EnabledRowNotInCluster forever.
Sabotage: removing the change-detection guard turns the repeat test red.
ControlPlane.Tests 100/100.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Cuts ConfigPublishCoordinator's last genuinely mesh-bound dependency: central
must be able to name the nodes a deployment is for without sharing a gossip
ring with them, because Phase 2 splits the fleet into one mesh per Cluster.
Behaviour change, confirmed before implementing: a configured node that is
switched off is now expected, so deploying while it is down fails at the apply
deadline instead of sealing green without it. Under the membership rule the
operator was told the fleet was deployed when it was not. ClusterNode.Enabled
= 0 is the maintenance hatch, and the deadline log now names the silent nodes
and points at that hatch — "4/5 acks landed" would leave an operator reading
logs on five machines.
Two assumptions are now documented on the class rather than left implicit:
every ClusterNode row is a driver node (the DB has no role column and
deliberately does not gain one — it would drift from Cluster:Roles), and
ServerCluster.Enabled is not consulted because nothing else consults it.
Dropped from the plan: cluster-scope filtering of the expected set. There is
no cluster-scoped deployment to filter on — Deployment has no ClusterId,
ConfigComposer always snapshots the whole DB, and ResolveClusterScope is
node-side self-scoping of a fleet-wide artifact.
Positive control: reverting DiscoverDriverNodes to the membership scan turns
three of the four new tests red. The fourth pins the seal-empty branch and is
documented as derivation-insensitive rather than left to look like coverage.
ControlPlane.Tests 90/90.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
RedundancyStateActor derived the driver Primary from ClusterState.RoleLeader("driver").
Akka offers two different notions of a "first" member and they are not the same
one: role leader is the lowest-ADDRESSED Up member (host, then port), while
ClusterSingletonManager places singletons on the OLDEST — lowest up-number.
They agree on a freshly-formed cluster, which is why every existing test passed.
They diverge after any restart: the restarted node re-joins as the youngest while
keeping its address, so if it holds the lower address it becomes role leader while
the singletons stay put. The snapshot would then name a Primary that is not
hosting the work, and every Primary-gated surface follows it — inbound device
writes, native-alarm acks, the fleet-wide alerts emit, and the alarm-history
drain would all enable on the wrong node while the node actually running the
singletons stayed gated off.
BuildSnapshot now selects the oldest Up member carrying the driver role, matching
singleton placement. Leaving members are excluded: a node handing its singletons
over must not be named Primary.
NodeRedundancyState.IsRoleLeaderForDriver is renamed IsDriverPrimary, and
NodeHealthInputs.IsDriverRoleLeader likewise. Keeping the old names would have
left the wire contract asserting a derivation the code no longer uses — the same
drift that made this defect invisible.
Proven by a real two-node cluster rather than a mock. RedundancyPrimaryElectionTests
binds the first-joining node to the HIGHER port, so oldest and lowest-address name
different nodes, and includes a fixture assertion that the divergence actually
occurred — without it the real assertion could pass for the wrong reason. Positive
control: restoring the RoleLeader derivation turns exactly the two election tests
red while the fixture check stays green.
Runtime.Tests 440 passed, ControlPlane.Tests 82, Cluster.Tests 36.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Completes 835fc08c, which fixed the harness's DisposeAsync but left StopNodeBAsync
calling NodeB.DisposeAsync() directly — despite that commit's own message naming
it as carrying the same bug. Caught by re-reading the code rather than trusting
the commit message.
This one is not just about memory. IHost.DisposeAsync never invokes
IHostedService.StopAsync, so CoordinatedShutdown never runs and node B never
performs a graceful Cluster.Leave. Node A instead notices only when the failure
detector times the member out — a slower and materially different path from the
graceful leave the failover tests are written to exercise. The doc comment
asserted the opposite ("shuts down node B via DisposeAsync, which runs
CoordinatedShutdown"); it now says what actually happens and why.
Verified: the three failover suites that call it pass (5/5), and the project is
unchanged at 194 passed with only the AbCip fixture failure, which needs a docker
fixture that is not running.
Also checked the sibling harness while here: LocalDbPairHarness already stops each
host before disposing it, so it needed nothing.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
User-reported: the integration tests leave instances behind and take 18+ GB.
Measured before changing anything — a SINGLE Host.IntegrationTests process peaked
at 30,153 MB RSS across its ~200 tests.
The cause was not a leak and not parallelism. Referencing the Host drags in the
ASP.NET Core framework reference, which turns Server GC on by default: one heap
per core, tuned for throughput and deliberately reluctant to hand memory back.
Correct for a server, wrong for a test host. Workstation GC takes the same suite
from 30,153 MB to 6,717 MB — 78% less — with no change in runtime (45s).
Two hypotheses were measured and rejected on the way, recorded here so nobody
re-runs the experiment: capping xunit maxParallelThreads to 4 cost 29% wall-clock
and saved nothing (30,153 -> 31,115 MB, i.e. noise), and it turns out a single
process was doing all of it. Reverted.
The harness fix is real but was worth only 4% on its own: TwoNodeClusterHarness
disposed each node without stopping it first, and IHost.DisposeAsync only disposes
the service provider — it never invokes IHostedService.StopAsync, which is where
Akka.Hosting terminates the ActorSystem. So every test left two live ActorSystems,
remoting transports and dispatcher pools included, rooted for the process
lifetime. The class doc-comment asserted the opposite ("DisposeAsync, which runs
CoordinatedShutdown"); it was wrong, and StopNodeBAsync inherited the same bug on
a path failover tests depend on.
The payoff is larger than the memory number. Two failures I had classified as
known-flaky baseline noise now pass consistently:
RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed
and ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks.
They were never flaky — they were starved. Full solution: peak summed test-process
RSS 15.2 GB, zero new failures, two fewer.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Gate record: docs/plans/2026-07-20-localdb-phase2-live-gate.md.
Checks 1, 2, 5, 6 pass. Checks 3 and 4 are NOT satisfied — the defect they
were meant to confirm turned out to be the opposite of what the plan assumed.
Three defects crash-looped every driver node before check 1 could even run:
1. An empty ServerHistorian:ApiKey kills the host. ServerHistorianOptions-
Validator exists to turn exactly that class of failure into a named
OptionsValidationException, but its documented fail tier explicitly excluded
ApiKey on the reasoning that a keyless client "degrades — the gateway rejects
calls". It does not: the client validates its own options at construction, so
the process dies during Akka startup and never makes a call.
2. UseTls disagreeing with the endpoint scheme kills the host too, in both
directions (both messages confirmed in the shipped client assembly). Moving an
endpoint from https to http without clearing UseTls is an ordinary migration
slip.
3. Plaintext h2c was UNREACHABLE. HistorianGatewayClientAdapter forwarded the
TLS-only options unconditionally, and AllowUntrustedServerCertificate defaults
to false, so it always sent RequireCertificateValidation=true — which the
client rejects outright when UseTls=false. Every http:// deployment crashed,
though the scheme is documented as the supported way to select h2c, and the
only workaround was to assert a certificate posture for a connection that has
no certificate.
The fourth was the blocker, and it is Phase 2's own:
4. The drain gate deferred to a Primary that cannot deliver. Redundancy roles are
elected CLUSTER-WIDE; the alarm queue is PAIR-LOCAL. On the rig the elected
driver Primary is central-1 — it carries the driver Akka role, replicates
nobody's LocalDb and does not even run the alarm historian — so every driver
node logged "Historian drain suspended", including the two site-b nodes that
have no peer at all. Nothing drained anywhere, where before Phase 2 it drained
fine. The cost is not a duplicate; it is the buffer growing to the capacity
wall and evicting the audit trail it exists to protect.
Fixed in three layers: a separate ShouldDrainAlarmHistory policy (unknown role
drains; the two gates now deliberately disagree, and a test pins that); peer-
host matching in DriverHostActor so a node stands down only for a Primary
holding its rows; and AddAlarmHistorian short-circuiting the gate when
replication is unconfigured — testing BOTH Replication:PeerAddress and
SyncListenPort, since only the dialing half sets the former while both halves
share the queue.
Every one of these follows from the asymmetry: a false allow costs a duplicate
row, which at-least-once delivery already accepts and payload-hash ids
collapse; a false deny loses data silently.
A third vacuous test, caught by the same delete-the-guard discipline: the
role-view tests stayed green with the guard removed, because AwaitAssert polls
until an assertion passes and the assertion was "reads open" — which is the
SEEDED value, satisfied at the first poll before the actor processed anything.
They now assert the sequence of published values through a recording view; the
control then goes red for exactly the cases that matter.
Migration evidence: 11 legacy rows across two deliberately overlapping files
converged to exactly 9 identical rows on both nodes, proving D-6's payload-hash
identity on real nodes rather than in a fixture.
Open design fork, recorded in the gate doc rather than decided here: a pair
cannot currently identify its own Primary, so both halves drain. Safe in every
topology — nothing loses data — but the gate's de-duplication benefit is
unrealised until roles are scoped per pair.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Two driver nodes replicating the alarm buffer over the real loopback h2c
transport, through the real fail-closed interceptor. This is the test the
phase exists to pass: schema, registration, sink rewire and drain gate can
all be individually green while a pair still fails to share the undelivered
alarm history that is the whole point of moving the queue.
Rows are written through the production sink rather than hand-rolled
INSERTs, so the id derivation, column set and delete-on-ack semantics under
test are the ones production uses. The scenarios cover a buffered burst
converging with the oplog draining to zero, delivered rows being removed from
BOTH nodes as tombstones (the no-redeliver-after-failover property), writes
during a transport outage surviving the rejoin, and the same event accepted
on both nodes collapsing to one row.
The positive control found a weak assertion. With
RegisterReplicated("alarm_sf_events") commented out, three of the four went
red immediately -- but the same-event-on-both-nodes scenario still PASSED,
because with replication off each node trivially holds its own single copy
and "one row on each" is satisfied by two databases that never spoke. It now
also enqueues a distinct event on B and requires both nodes to hold two rows,
which cannot be satisfied without convergence; it goes red under the control
like the others. Control restored, all four green.
Also updates the second exact-set replicated-tables pin, in LocalDbWiringTests.
The plan predicted these pins would go red here, and that is them working:
one of them is asserted through the full driver DI graph rather than the
OnReady callback, so it catches a registration that exists in the callback
but never reaches a running host.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Copies the pre-consolidation store-and-forward queue into the consolidated
database on first boot, then renames the legacy file aside. Rows are in that
file precisely because the historian could not be reached, so dropping them
on upgrade would discard exactly the alarm audit trail the queue exists to
protect.
Runs last in OnReady, after every RegisterReplicated call. It is the only
thing in OnReady that writes rows, and capture is trigger-based: a migration
that ran before registration would recover the backlog locally and never
replicate a line of it, silently and permanently.
Ids are derived from the payload rather than the plan's mig-{node}-{legacyId}
scheme. Node-prefixing solves the collision the legacy AUTOINCREMENT key
would cause -- node A's row 7 and node B's row 7 are different alarms -- but
it preserves a duplication that should be collapsed instead. A warm pair's
two legacy files OVERLAP: HistorianAdapterActor default-writes while its
redundancy role is unknown, so both nodes accepted the same transitions
during every boot window. Prefixed ids would carry those duplicates into the
merged buffer forever; equal-payload ids converge them. The same property
makes a crash between commit and rename harmless under INSERT OR IGNORE.
OnReady now takes IConfiguration rather than offering an overload that skips
the migration. A wiring mistake that silently discarded a node's undelivered
alarm history is not a mistake worth making possible.
The copy is restricted to the columns the legacy table actually has. Naming
a column an older build never wrote throws "no such column", which would
discard every row in the table rather than the one field.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Moves the alarm store-and-forward buffer out of its own alarm-historian.db
and into the node's consolidated LocalDb, where it replicates to the
redundant pair peer. A node that dies holding undelivered alarm history no
longer takes it to the grave.
Tasks 2 and 3 land together, as the plan anticipated. They are not separable:
the gate is a constructor argument of the rewritten sink, and a commit that
replicated the queue without gating the drain would be a commit in which both
nodes of a pair deliver every alarm event, continuously.
That drain gate is the load-bearing part of the change, and the recon
explains why it is new work rather than a refinement. Exactly-once delivery
across a pair is enforced today on the ENQUEUE side, by
HistorianAdapterActor: only the Primary enqueues, so the Secondary's queue is
empty and its ungated drain has nothing to send. Replicating the table
destroys that invariant.
The gate is a Func<bool> the caller supplies, because the drain runs on a
timer the sink owns rather than on a mailbox, and Core.AlarmHistorian cannot
reference PrimaryGatePolicy in Runtime. Runtime supplies it via a new
IRedundancyRoleView singleton that DriverHostActor publishes its Primary-gate
verdict to -- the same verdict the inbound-write and native-ack gates use, so
there is no second notion of am-I-the-Primary to drift.
Two failure modes are deliberately closed:
- The view is seeded OPEN, matching the policy's own answer for an unknown
role with no driver peer. A deployment that runs no redundancy never
publishes to it, and defaulting closed would silently stop its alarm
history forever.
- A gate that throws is read as not-now, never as permission, and a closed
gate reports the new HistorianDrainState.NotPrimary rather than Idle. A
Secondary's rising queue is supposed to look different from a stalled
drain, and if BOTH nodes report NotPrimary the pair is misconfigured and
says so instead of quietly filling toward the capacity ceiling.
Row ids are a hash of the payload rather than fresh GUIDs. Both adapters
accept the same fanned transition in the window before the first redundancy
snapshot arrives, and under last-writer-wins an equal key converges those two
accepts into one row instead of duplicating them.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Adds the alarm store-and-forward buffer to the consolidated LocalDb file and
registers it for replication, so a node that dies with undelivered alarm
history no longer takes it to the grave.
The table keeps the legacy queue's shape rather than the status column the
plan sketched. An acknowledged row is deleted, not marked delivered: that
needs no second sweeper to keep the table bounded, leaves the capacity
semantics untouched, and the replication engine carries the delete as a
tombstone so the peer drops its copy anyway -- which is what the status
column was for. last_error is retained because it is the only operator-facing
record of why a row was dead-lettered.
The primary key is app-minted TEXT. The legacy AUTOINCREMENT RowId cannot
replicate under last-writer-wins: two nodes would independently allocate
rowid 7 to different alarms and silently overwrite each other. The drain
index therefore orders by enqueued_at_utc rather than insertion order, with
id as a tiebreak so the ordering is total.
Tables are created unconditionally, independent of AlarmHistorian:Enabled.
An empty registered table costs three triggers; creating it lazily would mean
a node that enables the historian later writes rows before its capture
triggers exist, which is exactly the silent-loss shape OnReady's ordering
comment warns about.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
ApplyAndAck advanced _currentRevision and recorded NodeDeploymentState=Applied
immediately after ReconcileDrivers, which returns null when the artifact could
not be read. So a node that applied NOTHING reported success, claimed a
revision whose configuration it never applied, and — because
HandleDispatchFromSteady short-circuits on a revision match — could never be
healed by re-dispatching that revision. Only a later, different revision
recovered it.
Now a null blob fails the apply: the revision is left where it was,
NodeDeploymentState records Failed with the reason, and the coordinator gets a
Failed ACK. Leaving the revision alone is what makes the retry actually land
instead of being waved through.
This is about telling the truth, not about tearing anything down — the node
keeps serving its last-known-good address space, drivers and subscriptions
throughout (#485).
Tests, RED-first (both verified failing with "should be Failed but was
Applied"): the ACK/revision assertion, plus the one that matters — after a
failed apply, re-dispatching the SAME revision now genuinely applies. Its
recovered artifact adds a second driver precisely so a short-circuited ACK
(which has no side effects) cannot satisfy it.
Four existing tests changed, and both changes are deliberate:
- DriverHostActorTests.SeedDeployment never set ArtifactBlob at all. Its three
consumers are about the apply/ACK/state machine, not the artifact, and now
need a genuine apply — so the helper seeds a well-formed artifact declaring
no drivers. That is what the composer emits for an empty configuration, and
is precisely what "bytes we could not read" is NOT.
- EmptyArtifact_IsNotCached asserted "the apply still succeeds — an empty
artifact is a legitimate no-op deployment". That premise is the bug. Flipped
to Failed; the test's actual subject (nothing is cached) is untouched and now
holds for two independent reasons.
Closes#486
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The driver-side half of the same mistake the address-space fix corrected.
ReconcileDrivers and PushDesiredSubscriptionsFromArtifact both read the
artifact as `...FirstOrDefault() ?? Array.Empty<byte>()`. Their THROW paths
already returned early, but empty bytes flowed onwards as a real answer: zero
driver specs, so DriverSpawnPlanner planned every running child for StopChild,
and then an empty desired set dropped each surviving driver's live
subscription handle. A node lost its entire field I/O and still ACKed Applied.
Same rule as the address space: no bytes is no answer, not "a configuration
with no drivers". Nothing legitimate produces a zero-length blob — deleting
the last driver still deploys a JSON document with empty arrays. Both sites
now skip and keep what is running. The two guards are independent because
PushDesiredSubscriptions does its OWN ConfigDb read, so the row can go missing
between them.
Also corrects the ReconcileDrivers doc comment, which claimed an empty blob
made it "effectively a no-op" — with children running it was the opposite.
Tests: DriverHostActorUnreadableArtifactTests, RED-first (verified failing —
after the empty dispatch the driver list was empty). A zero-length ArtifactBlob
reproduces byte-for-byte what the missing-row case delivers, so the race does
not have to be constructed.
Two controls, both load-bearing:
- a READABLE driverless artifact still stops the driver, so the guard keys on
"the read gave us nothing", not on "fewer drivers than before";
- dropping a driver's LAST TAG still clears its subscription. This one was
added after the absence assertion was caught passing on a race: the
unsubscribe is an async self-tell, so with the second guard deleted the test
still went green. The control observes that same unsubscribe arriving well
inside the settle window, which is what makes the absence meaningful — with
the guard deleted the suite now correctly goes RED.
SubscribableStubDriver gains an UnsubscribeCount for that observation.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
A transient ConfigDb error while loading a deployment's artifact emptied the
served address space. LoadArtifact caught the exception, logged "rebuild
becomes no-op" and returned zero bytes; those parsed to an EMPTY composition,
which the planner diffed against the live one as a PureRemove and the applier
faithfully executed — 16 nodes gone, while the log claimed nothing happened.
An artifact we could not obtain is not an empty configuration. Nothing
legitimate produces a zero-length blob (an operator who really deletes
everything still deploys a JSON document with empty arrays), so "no bytes" can
only mean "no answer". HandleRebuild now abandons the rebuild on one, leaving
both the materialised nodes and _lastApplied intact so the retry diffs against
what is actually being served. The loader's own log line is now true.
The two cases are logged differently: warn when there is a live address space
being protected, debug when nothing has been deployed to this node yet.
Covered by a RED-first actor test (verified failing: the load error tore down
eq-2's subtree), plus a positive control proving a READABLE artifact that
genuinely drops an equipment still removes it — the guard suppresses teardown
only when the read failed.
Found on the LocalDb Phase 1 follow-up live gate, which induced it by flapping
SQL; that gate's log is the production evidence for the seam.
Closes#485
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
1. Wiped-node back-fill (live gate check 4). Fixed upstream in
ZB.MOM.WW.LocalDb 0.1.2 and pinned here. The gate's diagnosis was slightly
off: the oplog cap was not the gate. Snapshot detection measured the peer's
gap against the oldest surviving oplog row and read an empty oplog as "no
gap possible" — and an empty oplog is the steady state of a converged pair,
since ack-pruning deletes everything the peer confirmed. The healthy state
was the one state that could not heal a wiped peer. Now measured against
last_acked_seq when the oplog is empty.
Pinned at this level too, not just the library's: the pair harness grows a
WipePassiveAsync (a NEW database — a rebuilt node comes back with a new node
id and a zero watermark), and the new scenario asserts the emptied oplog as
its precondition before wiping, then requires the deployment artifact to
come back byte-identical with no new deploy. Verified RED against the pinned
0.1.1 (times out waiting for back-fill) and green on 0.1.2.
2. Oplog growth on default-OFF nodes (live gate check 8), fixed at the source:
StoreAsync now skips entirely when the pointer already names this
deployment/revision, the SHA matches the bytes, and the expected chunk count
is present. Re-caching an artifact the node already holds — every restart's
boot-from-cache, every RestoreApplied — writes nothing and mints no oplog
rows, where before it cost a delete plus an insert per chunk plus a pointer
update, all identical to what was already there.
Identity is over the bytes and the chunks are counted, both deliberately: a
re-composed artifact can carry the same ids with different bytes, and a
pointer can name a deployment whose chunks are missing, where skipping would
make an unreadable cache permanent. Both guards verified RED against a naive
pointer-only skip.
The gate's stated bound was also wrong and is corrected in the doc: growth
was never headed for the 1M row cap. AddZbLocalDbReplication is registered
unconditionally, so MaintenanceBackgroundService runs on default-OFF nodes
and the 7-day MaxOplogAge cap prunes.
Runtime.Tests 412/0/31, Host.IntegrationTests LocalDb 45/45. Pre-existing and
unrelated: AbCip_Green_AgainstSim (needs the AB CIP docker fixture, red on a
clean master too) and a flaky Roslyn race test (green 2/2 in isolation).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
CacheAppliedArtifact ran only on a fresh apply (ApplyAndAck), never on the
bootstrap RestoreApplied path. So a node whose cache was lost — fresh/wiped
volume, disk failure — recovered its served state from central yet stayed
cache-less, unable to boot-from-cache on the NEXT central outage until some
future new deploy happened to land. Replication does not heal this: a peer's
already-acked cache rows are pruned from its oplog, so a fully-wiped, converged
node is never back-filled. RestoreApplied now writes the served artifact back to
the cache (invariant: the cache holds what the node serves). Found on the
docker-dev live gate wiping a node's LocalDb volume.
Boot-from-cache told OpcUaPublishActor to RebuildAddressSpace(deploymentId), which
re-loads the artifact from the ConfigDb by id — the very database that is
unreachable during the outage this path exists to survive. The rebuild no-op'd, so
a cache-booted node ran its drivers but served OPC UA clients an EMPTY address
space. RebuildAddressSpace now carries an optional in-hand Artifact blob;
ApplyCachedArtifact passes the cached bytes so materialisation happens from them
directly. Found on the docker-dev live gate: healthy peer served 16 nodes, the
cache-booted node served 1.
The Kestrel re-bind read only urls/ASPNETCORE_URLS. The aspnet:8.0+ base images
set ASPNETCORE_HTTP_PORTS=8080 (all interfaces) as the CONTAINER default in place
of ASPNETCORE_URLS, so a driver node that never sets URLS fell straight through to
Kestrel's localhost:5000 default — silently moving its health/metrics surface to
loopback when the sync listener took over Kestrel. Now falls through URLS →
HTTP_PORTS/HTTPS_PORTS (wildcard, all-interfaces) → localhost:5000, mirroring
Kestrel's own source precedence. Found on the docker-dev live gate (central nodes
were fine — they set ASPNETCORE_URLS=9000 explicitly).
Two driver nodes replicate the deployment-artifact cache over a real loopback
h2c transport through the real fail-closed LocalDbSyncAuthInterceptor; both DBs
init via the production LocalDbSetup.OnReady. Scenarios: byte-identical
convergence with matching pointer HLC+origin, retention prune reaching B as
tombstones, writes-while-transport-down surviving rejoin, and wrong-key
non-convergence with a matching-key positive control.
DoD positive control: commenting out both RegisterReplicated calls turns all
four scenarios red (verified locally, restored before commit).
Builds a container from the same AddOtOpcUa* methods Program.cs's driver branch
calls (AddOtOpcUaLocalDb / AddGrpc / AddOtOpcUaHealth / AddOtOpcUaObservability) and
resolves through it - DI extensions have shipped inert three times in this family.
Does not boot the real host: WebApplicationFactory<Program> would start hosted
services (Akka, OPC UA server, ValidateOnStart) needing SQL + a mesh, per D-5. The
seam under test is DI resolution, so the container is built and resolved without
starting.
Five pins: ILocalDb singleton + exact replicated-table set; IDeploymentArtifactCache
-> concrete; default-OFF ISyncStatus; localdb-replication health check registered and
constructible; admin-only graph has NO ILocalDb and does not demand LocalDb:Path.
LocalDbReplicationHealthCheck reports Healthy when replication is default-OFF or
LocalDb is absent (admin-only graphs) - so a plain node is never degraded by it -
Degraded when configured-but-disconnected, connected-with-unknown-backlog (a failed
oplog poll must not read as zero), or connected-with-backlog past the threshold.
Never Unhealthy: a replication fault does not stop the node serving its address
space. Pure decision core is table-tested; registered unconditionally via a factory
so AddOtOpcUaHealth keeps its no-arg signature and resolves ISyncStatus optionally.
Adds LocalDbMetrics.MeterName to the observability allowlist. o.Meters is a strict
allowlist with no wildcard, so the localdb.sync.* / localdb.oplog.depth series were
otherwise silently absent from /metrics - the omission a ScadaBridge live gate
caught.
Hooks the Bootstrap catch that previously went straight to Stale. On a cache hit the
node serves its last-known-good configuration through the outage instead of coming
up with an empty address space; on a miss, behaviour is byte-for-byte what it was.
The read is unkeyed because ClusterId is only derivable from an artifact you already
hold or from the unreachable central DB (recon D-1). Newest pointer wins, which is
correct in every real topology - a node belongs to one cluster and its peer
replicates that same row.
Splits PushDesiredSubscriptionsFromArtifact out of PushDesiredSubscriptions: the
cache path runs precisely when the ConfigDb read cannot succeed, so re-reading the
artifact there would fail by definition.
RunningFromCache is surfaced on NodeDiagnosticsSnapshot (optional param, existing
call sites unaffected). A node running from cache looks entirely healthy from
outside - full address space, live values - but its config is frozen and no deploy
can reach it. It clears only on a real apply from central.
New Props/ctor parameter appended LAST at all three positions - the forwarding
list inside Props.Create is positional and compiles into an expression tree, so a
mis-ordered argument is usually type-compatible and binds the wrong dependency at
runtime. All 26 existing test construction sites use named arguments and needed no
change.
ReconcileDrivers now returns the blob it loaded. It catches its own DB failures and
returns without rethrowing, so ApplyAndAck can reach its success path having spawned
zero drivers; caching on 'the apply succeeded' would persist an empty artifact as
last-known-good and boot the node into an empty address space during the next
outage. Verified: weakening the guard to null-only turns EmptyArtifact_IsNotCached
red.
The store runs in its own try/catch because an Applied ACK has already been sent by
then - an escaping exception would land in ApplyAndAck's catch and emit a second,
contradictory Failed ACK for a deployment the fleet believes is live.
128 KiB raw chunks base64-encoded into deployment_artifacts, with a per-cluster
pointer carrying a SHA-256 over the raw bytes. Reassembly verifies chunk count
AND the SHA before returning, so a partially replicated artifact reads back as a
clean miss rather than a silently truncated address space.
Adds GetCurrentUnkeyedAsync beyond the plan's interface: at the boot-failure seam
the actor cannot know its ClusterId (it is derivable only from an artifact you
already have, or from the central DB that is unreachable). Recon D-1.
The prune's 'deployment_id <> @DeploymentId' clause makes 'the pointer's target is
always present' structural rather than a side effect of clock resolution. Three
stores inside one timestamp tick fall through to a deployment_id tiebreak, and
since real ids are GUIDs that ordering is random - the row just written could lose,
leaving the pointer naming chunks that no longer exist. Verified red without the
clause.
The Secrets.Ui RCL's routable page was routed directly, but this host's
router is deliberately static (cookie SignInAsync needs SSR) with per-page
@rendermode opt-in - a directive the RCL page cannot carry, because the
other three family hosts render it under a globally interactive router
where a nested render mode throws. Routing straight to the RCL page
therefore rendered /admin/secrets with no circuit: it displayed, but every
@onclick was silently dead.
Fix: a host-side wrapper page (Pages/SecretsAdmin.razor) now owns
/admin/secrets, carries the standard per-page InteractiveServer render
mode, re-states the RCL page's own authorization policy (the router only
enforces [Authorize] on the routed component, which is now the wrapper),
and renders the RCL page as an ordinary child. The RCL assembly is
de-registered from both AdditionalAssemblies sites (router + endpoint) so
the route is unambiguous.
Guards: SecretsPageWiringTests pins the render mode (the #483 regression),
route parity with the RCL page, and policy parity; the page census in
PageAuthorizationGuardTests classifies the new page and admits the
secrets:manage policy. AdminUI.Tests 665/665.
Live-verified on the rebuilt docker-dev rig with Playwright: /_blazor
circuit negotiated on /admin/secrets and Add secret opens the editor
(before the fix: zero interactive markers, no circuit, dead click).
Closes#483.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Bumps the four ZB.MOM.WW.Secrets pins 0.2.1 -> 0.2.2 (closes the OtOpcUa side
of scadaproj#1, tracked here as #482). 0.2.1's Akka replicator deadlocked any
hosted process at startup when Secrets:Replication:Enabled was true: the
package's DI graph closed a circular singleton dependency through factory
lambdas (store decorator -> replicator -> actor provider -> cache invalidator
-> resolver -> store), which MS.DI's StackGuard turns into a silent
cross-thread call-site-lock deadlock. 0.2.2 defers the invalidator edge to
first eviction. The flag stays default-false; enabling remains a
per-environment decision.
The_startup_hook_actually_creates_the_replication_actor is now a real test:
SecretReplicationStarter's docs had promised it since the adoption, and the
upstream fix finally makes a provider-based resolve runnable - container built
exactly as the host does, hook started under a watchdog, replication actor
proven to exist by ActorSelection on a self-joined single-node cluster (no
TestKit needed, which matters because Akka.TestKit.Xunit2 is xunit-v2-only and
this project is on xunit.v3). Also corrects the stale rationale that blamed
the old hang on DistributedPubSub needing a joined cluster - the actor
constructor was never reached; it was the DI cycle.
Verified: SecretsReplicationRegistrationTests 8/8 on the 0.2.2 feed packages;
full slnx build 0 errors; the 2-node Akka live convergence gate re-run against
the published 0.2.2 packages passes 6/6 (write->peer, tombstone propagation
without resurrection, delete visibility through the resolver cache, reverse
direction, wrong-KEK fail-closed).
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Akka.TestKit is the family convention for non-end-to-end Akka testing and would
be the right tool here, but Akka.TestKit.Xunit2 is xunit-v2-only and this project
is on xunit.v3 - adding it produces CS0433 type conflicts. Directory.Packages.props
already documents the same constraint for the three projects deliberately held on
xunit v2 for TestKit. Revisit when Akka ships an xunit.v3 TestKit.
0.2.0's AddZbSecretsAkkaReplication never bound its own ISecretReplicator: it
called AddZbSecrets first, which TryAdds NoOpSecretReplicator, so the package's
own TryAdd was silently discarded. Replication published into a no-op sink and no
actor was ever spawned - no exception, no log line. Found by the Task 6 wiring
work, which is the first code that ever built a container around that extension.
Fixed upstream in 0.2.1.
Test consequence worth recording: the replication-enabled registration tests
previously passed against a plain ActorSystem only BECAUSE of that bug - the
no-op never touched Akka. With 0.2.1 they resolve a real replicator, which spawns
the actor, whose PreStart needs DistributedPubSub and therefore a joined cluster,
so provider-based assertions hang. Standing a real cluster up inside this test
assembly was attempted and did not work, so these assertions are now made against
the ServiceCollection, which is precisely where the defect lives. Actor creation
and convergence remain covered upstream by the library's TwoNodeClusterReplication
tests against a genuine 2-node cluster.
7 registration tests pass in 23ms. Full solution builds 0 errors; no warnings
originate from these files.
Joseph Doherty