d01b0695c27608833346212d15577254cd09c00f
1336 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
d9de4e3019 |
fix(mesh): serve the artifact regardless of deployment status (Phase 3 live-gate deadlock)
The Phase 3 live gate deadlocked every FetchAndCache deploy: DeploymentArtifactGrpcService gated on Status==Sealed, but central seals a deployment only AFTER every node acks Applied, and a FetchAndCache node cannot ack until it has fetched the artifact — seal-needs-ack -> ack-needs-fetch -> fetch-needs-sealed. The node's fetch reached central and passed the shared-key interceptor, then got a clean NotFound; the #485 apply-failure path correctly kept last-known-good, but the deploy could never seal. Direct mode reads the same ArtifactBlob from SQL while the row is still AwaitingApplyAcks, so the serve path must too. Drop the Sealed gate: serve any deployment whose blob is non-empty (unknown id / empty blob still collapse to NotFound — existence-hiding + #485). Access is already gated by the interceptor, so there is no reason to hide a non-sealed deployment a node is legitimately applying. The Task 2 'non-sealed -> NotFound' test flips to 'non-sealed with a blob is served'. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
ffbcaa93f2 |
test(mesh): real two-Host gRPC artifact-fetch boundary test with wrong-key control
Phase 3 Task 8. Stands a real h2c-only Kestrel listener serving the real DeploymentArtifactGrpcService behind the real ConfigServeAuthInterceptor (in-memory config DB seeded with one sealed deployment), and drives the real GrpcDeploymentArtifactFetcher across it — proving a stream crosses the real boundary, not just that the fetcher's logic is right (Task 4's fakes). Asserts: (1) right key + >256 KB blob reassembles byte-equal and verifies against RevisionHash; (2) FALSIFIABILITY CONTROL — wrong key returns null while the right key succeeds on the SAME server (so the null is the auth gate, not a dead server); (3) unknown id -> null (NotFound-hidden); (4) [dead-port, real-port] failover still returns the bytes. h2c GrpcChannel over http:// works out of the box in .NET 10. Tagged [Trait Category=ArtifactBoundary]. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
fc568ae0fa |
test(mesh): #485 empty/corrupt/mismatch coverage on the FetchAndCache paths
Phase 3 Task 7. Regression net pinning the #485 negative on every FetchAndCache seam — nothing rebuilds, nothing is torn down on a failure: (1) a null fetch mid-steady-state (all endpoints down / NotFound / SHA mismatch — indistinguishable to the actor) keeps the served address space and revision (no RebuildAddressSpace for the failed revision); (2) a zero-length blob handed to the actor fails via ReconcileDriversFromBlob's own #485 guard, independent of the fetcher's null contract; (3) a corrupt cache at boot (surfaced as GetCurrentUnkeyedAsync == null) lands Steady-no-revision and applies NOTHING — never a partial address space. No new mechanism; all green as a net. Removing the #485 empty guard reddens the zero-length test. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
6d00473866 |
feat(mesh): FetchAndCache bootstrap from the LocalDb pointer; no driver-side config SQL read
Phase 3 Task 6. In FetchAndCache mode Bootstrap() branches to BootstrapFromCache(), which restores served state from the LocalDb pointer (GetCurrentUnkeyedAsync → set _currentRevision → ApplyCachedArtifact from the cached bytes) with NO central-SQL read. Distinct from the Direct-mode TryBootFromCache fallback: reading the cache is NORMAL operation here so _isRunningFromCache stays false (the node can still fetch new deploys). An empty OR unreadable cache lands Steady-with-no-revision (the first dispatch fetches), never Stale — Stale means 'central SQL down, retry it', and FetchAndCache has no config SQL read to recover. TryRecoverFromStale gains a defensive FetchAndCache guard (it is unreachable in that mode, since the retry-db timer only starts in Stale). Proven with a ThrowingDbFactory in every test: reaching Steady + applying a dispatch proves the boot never touched central SQL. Sabotaging the mode branch (fall through to the SQL read) reddens all three — test A flips RunningFromCache; B/C enter Stale, which ignores the dispatch. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
648f173b18 |
feat(mesh): FetchAndCache apply path — fetch->cache->apply-from-bytes, null=apply-failure
Phase 3 Task 5. Under ConfigSource:Mode=FetchAndCache, a DispatchDeployment whose revision differs from _currentRevision kicks off a gRPC artifact fetch that PipeTo's its result back to Self as FetchedForApply (never a blocking await in a receive — that would freeze the mailbox for the fetch deadline). HandleFetchedForApply then applies synchronously on the actor thread, mirroring ApplyAndAck: null bytes = apply FAILURE (keep last-known-good, do not advance the revision, ack Failed — #485); non-null bytes reconcile drivers, rebuild the address space and push subscriptions FROM the bytes in hand (OpcUaPublishActor never reads central SQL), then cache them. A faulted fetch task maps to null bytes so it can't escape as an unhandled message. Extracted ReconcileDriversFromBlob from ReconcileDrivers so Direct-read, FetchAndCache and boot-from-cache share one reconcile body (the #485 empty guard moves into it); ApplyCachedArtifact now reuses it instead of duplicating the spawn-plan logic. DI: DriverHostActor.Props gains fetchAndCacheMode + artifactFetcher (LAST, per the positional-forwarding warning); Runtime resolves them from IOptions<ConfigSourceOptions> (fetcher only in FetchAndCache mode); Host registers GrpcDeploymentArtifactFetcher under hasDriver. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
7a8fb5f129 |
feat(mesh): node gRPC artifact fetcher — SHA-verified reassembly, endpoint failover, null-on-any-failure
Phase 3 Task 4. GrpcDeploymentArtifactFetcher streams the artifact from the first configured central endpoint that answers, reassembles the chunks, and verifies SHA-256(bytes) lowercase-hex == the dispatched RevisionHash (byte-identical to ConfigComposer's Convert.ToHexStringLower(SHA256.HashData(blob))). Every per-endpoint problem — RpcException, zero-length stream, or hash mismatch — is logged and the next endpoint tried; if none yield verified bytes it returns null (apply failure, keep last-known-good — the #485 contract), never throwing into the actor loop. A Func<endpoint, client> seam keeps the gRPC boundary out of the unit tests (that is Task 8's job); the real path caches one h2c GrpcChannel per endpoint. Files live under the .DeploymentCache namespace (folder Deployment/) to avoid shadowing the Configuration.Entities.Deployment type in the test assembly — same convention as LocalDbDeploymentArtifactCache. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
4d88f67c0d |
feat(mesh): config-serve auth interceptor + dedicated h2c listener (fail-closed bearer)
Phase 3 Task 3. Central serves the deployment artifact over a dedicated h2c Kestrel listener (ConfigServe:GrpcListenPort), gated by the ConfigServeAuthInterceptor (shared bearer key, FixedTimeEquals, fail-closed, path-scoped to /deployment_artifact.v1.DeploymentArtifactService/). The listener block is merged with the LocalDb-sync listener so a fused admin+driver node re-applies its existing HTTP surface exactly ONCE and adds both dedicated h2c ports on top — double-binding would throw 'address already in use'; zero re-binding would silently unbind the AdminUI behind Traefik. AddGrpc now registers under hasDriver || hasAdmin with both path-scoped interceptors sharing one pipeline. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
7d7de482b7 |
feat(mesh): central DeploymentArtifactService — stream the sealed artifact, NotFound-hide the rest
Streams a sealed deployment's ArtifactBlob to a fetching driver node in 128-KiB chunks (matching the LocalDb cache chunk size). Unknown id, a non-sealed deployment, and a zero-length blob all collapse to one indistinguishable NotFound — existence-hiding plus the #485 serve-side guard (never stream empty bytes as a valid empty config). The impl is named DeploymentArtifactGrpcService to avoid colliding with the generated DeploymentArtifactService container type. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
62e8f54f71 |
feat(mesh): deployment_artifact.v1 proto + first in-repo Grpc.Tools codegen
The repo has consumed packaged gRPC clients (LocalDb.Replication, HistorianGateway.Client) but never compiled a proto locally. This adds the Grpc.Tools toolchain to Commons and a deployment_artifact.proto with a single server-streaming Fetch(deployment_id) -> stream ArtifactChunk RPC, generating both the server base and the client stub (GrpcServices="Both") so central (Host/AdminUI) and the node (Runtime) share one reference. A compile-touch test pins the codegen: it stops compiling if generation regresses. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
9d1e60c00c |
feat(mesh): ConfigSource/ConfigServe options + validator (Phase 3 dark switch)
ConfigSource:Mode = Direct (default, read central SQL) | FetchAndCache (fetch the artifact from central over gRPC, read LocalDb). The validator fails host start on a FetchAndCache shape that cannot fetch — no endpoints, a non-http(s) endpoint, no shared key, or a non-positive timeout — because each otherwise surfaces as a silent absence (a deploy that never applies). ConfigServe (the central serve surface) takes a plain Configure: a 0 port is a valid "disabled". Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
5468b79d19 |
test(mesh): real two-ActorSystem ClusterClient boundary test with falsifiability control
Two genuinely separate single-member clusters sharing the ActorSystem name and joined only by ClusterClient, so the mesh boundary is exercised over the wire rather than through an in-process relay: a relay cannot fail for a missing receptionist extension, an unregistered service, a malformed contact path or a serialization break, which are the Phase 2 failure modes. Covers both legs plus the control: * central SendToAll -> node comm actor -> node EventStream * node ApplyAck -> central comm actor, through the node's own client * a node without the receptionist extension receives nothing Sabotage-verified. Suppressing the node-side EventStream publish and the outbound ack Tell reddens one leg each. Restoring the receptionist and the service registration on the control node reddens the control, which is what proves the control's send is live rather than silently misaddressed. The control removes the extension AND the RegisterService call, since resolving the receptionist to register would materialise the extension whose absence is the point; it falsifies "delivery happens without a receptionist boundary", not the extension line alone. Recorded in the test. Also corrects MeshCommActorPathTests: with one node per side, per-node and singleton registration are indistinguishable here, so the property the dropped Task 6 assertion covered belongs to live-gate step 8, not to this class. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
2a2e54f083 |
feat(mesh): route deploy, driver-control and alarm commands through the mesh router
Phase 2 Task 7. The five central->node publish sites -- ConfigPublishCoordinator's deploy dispatch, and AdminOperationsActor's restart, reconnect, acknowledge and shelve -- now hand a MeshCommand to the transport router instead of telling the DistributedPubSub mediator directly. Neither actor knows which transport is in force any more. _meshRouter is NULLABLE with a mediator fallback, and that is deliberate: about a dozen existing coordinator and admin-operations tests construct these actors without a router, and they must keep exercising the real DPS path rather than a silently-disabled one. The fallback is not a permanent seam -- Phase 6 deletes it with the DPS branch and the single mesh. Wiring note: the comm actor's WithActors block moved ABOVE the singleton registrations so both singletons can resolve it with registry.Get, which is the ordering-by-registration idiom AdminOperationsActor already uses for the coordinator. An earlier version used ActorSelection.ResolveOne().GetAwaiter() .GetResult() inside the props factory -- that blocks inside actor construction and races the very spawn it waits for. Sabotage-verified by inverting the branch so the mediator always wins: all six new tests go red, including the fallback test (which proves it is asserting the mediator path rather than passing by accident). Suites after: ControlPlane 118/118, Runtime 450/450, Host.IntegrationTests 196/202 -- sole failure AbCip_Green_AgainstSim, the fixture baseline that fails on master too. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
16d598560a |
feat(mesh): register comm actors with the receptionist; pin the wire-contract paths
Phase 2 Task 6. Both comm actors are now spawned and receptionist-registered: CentralCommunicationActor from WithOtOpcUaControlPlaneSingletons, and NodeCommunicationActor from WithOtOpcUaRuntimeActors (before DriverHostActor, because it is the host's ackRouter under ClusterClient mode; it has no dependency of its own on the host, since inbound commands travel via the node's EventStream). Both are plain WithActors registrations, NOT singletons -- a driver node's ClusterClient rotates across contact points and must find a live comm actor at whichever node answers. Both register with the receptionist in BOTH transport modes: an idle registration costs nothing, whereas registering only under ClusterClient would mean flipping the flag on a running fleet needs a restart before anything can be reached, turning a config change back into a deployment. The coordinator ref is resolved lazily per message (Func<IActorRef?> + registry.TryGet) so the comm actor never depends on the order in which Akka.Hosting materialises the singleton proxy relative to this block. Adds MeshCommActorPathTests, which boots the real two-node host and Identify- probes both paths. These strings are the wire contract and nothing else asserts they agree: a rename would compile, pass every unit test, deploy, and deliver nothing, because a ClusterClient send to an unregistered path is dropped silently. Sabotage-verified with an actual rename. DELIBERATELY NOT TESTED, with the reason recorded in the file: that the actors are per-node rather than singletons. Two discriminators were tried and both were invalid -- "the path resolves on both nodes" passes under either shape because a ClusterSingletonManager is also created on every node in the role, and "no /singleton child" is null even for the KNOWN singleton /user/config-publish, so Akka.Hosting does not lay them out that way. A sabotage re-registering the actor via WithSingleton left both green, which is what exposed them. Rather than ship an assertion that cannot fail, the property is left to the Task 8 boundary test, where a send only arrives if the target really is registered. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
5cb0e72166 |
feat(mesh): node subscribers accept EventStream commands; _coordinatorOverride -> _ackRouter
Phase 2 Task 5. DriverHostActor and ScriptedAlarmHostActor now subscribe to the node-local EventStream alongside their existing DPS topic subscriptions, so the transport flag lives ENTIRELY on the publishing side -- exactly one of the two ever publishes, and the switch can be flipped or reverted without touching a single subscriber. Note the alarm case is not symmetric: the DPS subscribe there still carries the node-LOCAL publisher (OtOpcUaServerHostedService's OPC UA Part 9 method calls), which never crosses the boundary and stays on DPS in both modes. Only the central AdminUI leg moves. Renamed DriverHostActor._coordinatorOverride to _ackRouter. Under ClusterClient mode that ref is the NodeCommunicationActor, emphatically not a coordinator, and the old name would send the next reader looking for one. Adds two wiring tests, because the unit tests either side of this seam BOTH pass with the subscription deleted -- the comm actor's test asserts a probe receives the message, and the host tests drive the actor by direct Tell. Neither notices a missing PreStart subscribe. Sabotage-verified: deleting either subscription reddens exactly its own wiring test and nothing else. Three things went wrong while writing those tests, all worth keeping: - The first version raced. ActorOf returns before PreStart runs, and an EventStream.Publish with no subscriber is dropped silently -- no buffering, no dead letter. Fixed with a warm-up whose ack proves PreStart completed; the comment says why it is not ceremony. - The alarm version was VACUOUS: it told the host directly INSIDE the assertion window alongside the relay, so the direct Tell alone produced the log the assertion waited for. Split into warm-up (outside) and relay (inside). - The alarm test needs its own ActorSystem: the shared harness pins loglevel = WARNING and the only signal an unowned alarm gives is a Debug line. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
915beec11a |
feat(mesh): NodeCommunicationActor — inbound commands to the node-local EventStream
Phase 2 Task 4. Node-side end of the boundary, registered with the receptionist per node rather than as a singleton so central's contact rotation reaches whichever node answers. Inbound commands are re-emitted on the node's EventStream, NOT back onto their DistributedPubSub topic. DPS is mesh-wide and central SendToAll-s to every node's comm actor, so a DPS republish would deliver N copies of every command to every subscriber. The EventStream is node-local by construction. It also avoids a registration handshake with actors spawned later: ScriptedAlarmHostActor is a CHILD of DriverHostActor and has no registry key, so there is no ref to hand in at wiring time. Each inbound type is listed explicitly rather than caught by a ReceiveAny, so an unknown type dead-letters loudly instead of being republished blind onto a stream where every subscriber ignores it. Outbound is ApplyAck only, and it uses Send rather than SendToAll -- the deploy coordinator is one singleton behind central's proxy, so SendToAll would deliver one ack per central node and the coordinator would count this node twice. Notably there is NO Ask across this boundary in Phase 2: every migrated command is fire-and-forget, which is why this actor is far smaller than the sister project's equivalent and needs no sender preservation at all. Sabotage-verified twice: deleting the AlarmCommand handler and flipping Send to SendToAll each redden exactly their own test. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
d5b5cb6ede |
feat(mesh): CentralCommunicationActor — dark-switched fan-out, DB-sourced contacts
Phase 2 Task 3. Central-side end of the boundary: routes every central->node command out over DPS or ClusterClient per MeshTransport:Mode, and forwards inbound ApplyAcks to the deploy coordinator singleton (Forward, not Tell, so the coordinator sees the node rather than this relay). Three things here are load-bearing and easy to get wrong later: SendToAll, never Send. Today's DPS publish reaches EVERY DriverHostActor and the node side has no ClusterId or node filter to compensate -- scoping happens later, inside the artifact. ClusterClient.Send delivers to exactly ONE registered actor, so it would deploy to a single node while every other node silently kept its old config, and the deployment could still seal green. Sabotage-verified: swapping to Send reddens exactly that one test. Exactly ONE ClusterClient, fleet-wide. A receptionist serves its whole cluster, so SendToAll reaches every registered node-comm actor in the mesh regardless of which contact point was dialled. One client per application Cluster -- the shape Phase 6 wants -- would fan each command out once per cluster while the fleet is still one mesh: N x duplicate DispatchDeployment and ApplyAck. Marked TODO(Phase 6). The contact set does NOT scope delivery. Excluding a maintenance-mode node from the contacts does not stop it receiving commands; it still receives them, as it does under today's broadcast. The filter is about which receptionists are worth dialling, not about who gets the message. Stated in the code because the opposite is the natural assumption. Also carries the sister project's shipped fixes: per-row address parsing so one malformed ClusterNode cannot abort the refresh and leave the contact set half-built (regression test orders the bad row FIRST), the cache entry cleared before recreate so a failed create cannot leave commands routing into a stopping actor, and a Status.Failure handler so a DB outage is a Warning rather than a silent debug-level unhandled message. Two test-harness bugs found and fixed while writing this, both mine: SubscribeAck goes to the SENDER (so the mediator subscribe needed an explicit sender), and EventFilter matches case-INSENSITIVELY, so a "was NOT" filter also caught this actor's own "was not created" warning. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
0d6669d5ff |
feat(mesh): MeshCommand envelope + generation-suffixed ClusterClient factory
Phase 2 Task 2. Two small pieces the comm actors are built on. MeshCommand carries a command plus the DPS topic it would have been published on, so the admin singletons stop knowing which transport is in force. Without it the dark switch would have to be threaded through five publish sites across two actors, each free to drift. Central-side only -- it never crosses the wire, so it carries no serialization contract. DefaultMeshClusterClientFactory appends a generation counter to the actor name. Context.Stop is asynchronous and the name stays reserved until termination completes, so recreating the same name inside one message handler throws InvalidActorNameException -- which is exactly what a contact-set change does (stop the old client, create the replacement, one handler). Ported from the sister project, where it was a shipped bug fix rather than foresight. Sabotage-verified: freezing the name to a constant reddens both name tests with InvalidActorNameException. Dropped a third test I had written -- it asserted ClusterClientSettings directly and never touched the factory, so it was coverage theatre. Replaced with a comment saying where contact propagation IS covered (the Task 8 boundary test, the only place it can actually fail). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
7b71a6a35d |
feat(mesh): receptionist extension, zero ClusterClient buffering, frame-size logging
Phase 2 Task 1. Three changes to the embedded base config, plus a test that reads every one of them back off a RUNNING ActorSystem rather than off the HOCON text -- the settled idiom here (SplitBrainResolverActivationTests), because several fragments compete and the losing one still reads correctly in the file. - Registers the ClusterClientReceptionist extension. It is not auto-started, so without this the boundary only listens if some code path happens to call ClusterClientReceptionist.Get(system) first: "is the boundary up?" would depend on startup ordering rather than on configuration. - buffer-size = 0. This is the one genuine behaviour change and it is NOT what the sister project runs -- they never wrote this section and inherit 1000. Buffering would replay a DispatchDeployment on reconnect and apply a deployment the coordinator already sealed as TimedOut, leaving the node on a configuration the database says failed. - log-frame-size-exceeding = 32000b. An oversized frame is dropped WITHOUT tearing down the association: heartbeats keep flowing, the node reports healthy, and the only symptom is a command that never arrived. Our deploy notify is payload-free, so this canary should stay quiet. reconnect-timeout and receptionist.role restate Akka defaults; they are stated explicitly and pinned because the behaviour is load-bearing, and the comments say which is which. One test was VACUOUS on first write: Config.GetInt returns 0 for a missing key, so the buffering assertion passed before the feature existed -- it could not distinguish "explicitly 0" from "absent, default 1000 in force". Rewritten to assert through ClusterClientSettings/ClusterReceptionistSettings, the objects the client is actually built from. Sabotage-verified after the fix: setting buffer-size back to 1000 reddens exactly that one test. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
5439f14804 |
feat(mesh): MeshTransportOptions — dark switch between DPS and ClusterClient
Phase 2 Task 0. Binds a validated `MeshTransport` section selecting which transport carries central->node commands, defaulting to Dps so nothing changes until the rig gate passes. The validator exists because every fault it catches otherwise surfaces as an ABSENCE -- a deployment that never arrives, an alarm ack that does nothing -- which has no stack trace and no failing node to point at. Two of the shapes it rejects are ported from the sister project's shipped mistakes: a contact point carrying an actor-path suffix, and (documented in the options XML) a template listing the node's OWN remoting port as a central contact, which is a permanent failure in the initial-contact rotation. Contact points are required only under ClusterClient mode; requiring them under the default would make the section mandatory on admin-only nodes that have no reason to carry them. Sabotage-verified: relaxing the empty-contacts guard and the actor-path-suffix guard turns exactly those two tests red, and nothing else. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
7654f24dab |
test(harness): serialize Host.IntegrationTests; drop the reconciler's unused singleton proxy
Closes the last branch-only failures. Measured across five full-suite runs on this branch versus two on master in a clean worktree: master failed only AbCip in this assembly (2/2), while the branch additionally failed one or two E2E deploy tests, rotating between DeployHappyPath, DriverReconnect and EquipmentNamespaceMaterialization (4/4). Each passed in isolation, and the assembly on its own was 195/201 throughout — the failures only appeared under full-suite CPU contention. Cause is contention, not correctness. Every test here builds a real two-node Akka cluster — two Kestrel hosts, two ActorSystems with remoting, six admin singletons, an EF context, a deploy pipeline — and xUnit ran them concurrently with each other and with every other assembly. That was survivable while the coordinator sealed instantly on an empty expected-ack set; now that it waits for a real ack from every configured node, these tests measure an end-to-end round-trip and starve. Serialising this one assembly makes them deterministic: 195/201 with only AbCip_Green_AgainstSim, and the full-solution failure set is now IDENTICAL to master's — 7 shared, zero branch-only. Cost is wall-clock for this assembly, 40s -> 3m35s; it is a handful of heavyweight E2E tests, not a broad unit suite, and a widened timeout alone did not fix it (tried first, at 45s). Also drops createProxyToo for the address reconciler. Nothing resolves ClusterNodeAddressReconcilerKey from the registry — the actor only reads cluster state and logs — so the proxy was a second actor per node hunting for a singleton nobody addresses. Kept because it removes dead machinery, not because it fixed the flakiness; measured on its own, it did not. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
0b7c53f64f |
test(harness): widen the deploy-seal timeout to match a real two-node deploy
Comparing full-suite runs on this branch against master in a clean worktree: both fail 11 tests, 9 identical. Master's two extras are load-flaky unit tests (AbCip Probe_loops, Galaxy EventPumpBoundedChannel); this branch's two extras were both Host.IntegrationTests deploy-path tests — the area this change re-times — so they are attributable here rather than to background flakiness. Cause is margin, not correctness. These waits used to observe a coordinator that sealed instantly on an empty expected-ack set; they now observe a real ApplyAck round-trip from every configured node. 15s was enormous margin against "instant" and thin against the real thing, so they failed only under full-suite CPU contention. Hoisted to TwoNodeClusterHarness.DeploySealTimeout (45s) so the reason is recorded once rather than as four unexplained numbers. DriverReconnectE2eTests is deliberately left alone: it seeds both ClusterNode rows itself and unconditionally, so its expected-ack set is identical before and after this change. Widening its timeout would be papering over a flake this change did not cause. Host.IntegrationTests 195/201; sole failure AbCip_Green_AgainstSim, verified failing on master. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
14746f2995 |
test(harness): seed ClusterNode rows in in-memory mode; retarget the failover deploy test
The full-suite run surfaced two load-dependent failures that were green in isolation. Root cause: SeedDefaultClusterAsync no-opped unless OTOPCUA_HARNESS_USE_SQL=1, on the reasoning that "the in-memory provider ignores FK constraints, so deploy E2E tests pass without it". Phase 1 invalidated that. With no ClusterNode rows the coordinator's expected-ack set is empty, so it seals IMMEDIATELY. "Wait for Sealed" therefore stopped implying "every node has applied" — and because DriverHostActor.UpsertNodeDeploymentState writes each node's row on its own schedule, every test counting those rows after a seal became a race. Reproducibly green alone, red under suite load, and it surfaced in a different test each run, which is what made it look like flakiness rather than a contract change. Seeding in both modes restores the invariant those tests were written against and is closer to production either way: a real fleet always has these rows, because the FK requires them. Deployment_started_with_node_b_down_seals_with_one_node_state asserted the behaviour Phase 1 deliberately removed — its own comment documented membership snapshotting, i.e. sealing green while a configured node never received the deployment. Replaced by two tests covering the new contract: a stopped node is still expected (both state rows exist, B still Applying, deployment does not seal), and MaintenanceMode is what makes it seal with one. The new "does not seal" test asserts both rows exist rather than only the absence of a seal, so it cannot pass against a coordinator that simply died. Host.IntegrationTests 195/201, sole remaining failure AbCip_Green_AgainstSim — verified failing on master in a clean worktree, so pre-existing fixture baseline, not a regression. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
ee69caa270 |
feat(config): add ClusterNode.MaintenanceMode — the hatch the live gate proved missing
Phase 1 gate step 4 failed: setting Enabled = 0 to take a node out of service returns 422 ClusterEnabledNodeCountMismatch, because DraftValidator.ValidateClusterTopology requires the enabled-node count to equal ServerCluster.NodeCount. On a Warm/Hot pair — every cluster on the rig, and every cluster in the target topology — Enabled can therefore never be the maintenance hatch. The plan called step 4 "the check that makes step 3 acceptable to ship", so Task 3's behaviour change was not shippable as it stood: a node down for maintenance would block every deployment to its cluster. The two rules were each reasonable and contradictory together — the validator reads Enabled as "part of the declared topology", Phase 1 additionally read it as "expect an ack". Split the meanings rather than weaken either rule: Enabled part of the declared topology (validator, untouched) MaintenanceMode expected to participate now (coordinator + reconciler) Rejected alternatives: counting configured rather than enabled nodes (drops the guard against booting a pair into InvalidTopology); downgrading the rule to a warning (weakens a deploy gate for everyone); shipping with no hatch. Sabotage: dropping !n.MaintenanceMode from the coordinator query turns the new test red. It also asserts both nodes remain Enabled, so the fix cannot quietly regress to disabling the row after all. Configuration.Tests 95/95, ControlPlane.Tests 101/101, solution builds clean. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
90c1302f71 |
feat(fleet): reconcile ClusterNode dial targets against live membership
ClusterNode.AkkaPort and the node's own Cluster:Port are the same fact in two places and nothing made them agree. Phase 2 dials the row instead of gossiping, so a node binding 4054 while its row says 4053 becomes unreachable from central — and the symptom is a silent absence of acks rather than an error. That is the shape of the Modbus/ModbusTcp and TwinCat/Focas drifts already in this repo. Since Phase 1 also made the rows the deploy path's expected-ack set, drift already costs a failed deployment today. Implemented as the plan's preferred option: an admin-role singleton comparing rows against the membership an admin node can already see, rather than each driver node asserting its own row — Phase 4 removes the driver nodes' ConfigDb connection, so a self-assertion written there would have to be deleted again. Three shapes, split by severity: a row whose dial target disagrees with its own NodeId and a running node with no row are Errors; an enabled row with no matching member is a Warning, because a node down for maintenance is a legitimate state. Findings are logged only when the set changes — a check that reprints the same warning every sweep trains operators to filter it out. Documented limitation: Phase 2 must revisit this. Once the fleet splits into one mesh per cluster an admin node cannot see site members, and every site row would report EnabledRowNotInCluster forever. Sabotage: removing the change-detection guard turns the repeat test red. ControlPlane.Tests 100/100. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
d88e245503 |
feat(deploy): coordinator sources its expected-ack set from ClusterNode rows
Cuts ConfigPublishCoordinator's last genuinely mesh-bound dependency: central must be able to name the nodes a deployment is for without sharing a gossip ring with them, because Phase 2 splits the fleet into one mesh per Cluster. Behaviour change, confirmed before implementing: a configured node that is switched off is now expected, so deploying while it is down fails at the apply deadline instead of sealing green without it. Under the membership rule the operator was told the fleet was deployed when it was not. ClusterNode.Enabled = 0 is the maintenance hatch, and the deadline log now names the silent nodes and points at that hatch — "4/5 acks landed" would leave an operator reading logs on five machines. Two assumptions are now documented on the class rather than left implicit: every ClusterNode row is a driver node (the DB has no role column and deliberately does not gain one — it would drift from Cluster:Roles), and ServerCluster.Enabled is not consulted because nothing else consults it. Dropped from the plan: cluster-scope filtering of the expected set. There is no cluster-scoped deployment to filter on — Deployment has no ClusterId, ConfigComposer always snapshots the whole DB, and ResolveClusterScope is node-side self-scoping of a fleet-wide artifact. Positive control: reverting DiscoverDriverNodes to the membership scan turns three of the four new tests red. The fourth pins the seal-empty branch and is documented as derivation-insensitive rather than left to look like coverage. ControlPlane.Tests 90/90. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
da74ebd696 |
feat(config): migration AddClusterNodeTransportPorts
Additive only — two AddColumn ops, no AlterColumn, so no accumulated model drift is riding along with this change: ALTER TABLE [ClusterNode] ADD [AkkaPort] int NOT NULL DEFAULT 4053; ALTER TABLE [ClusterNode] ADD [GrpcPort] int NULL; Adds a third test covering the DB-side default specifically. The entity round-trip cannot see it: ClusterNode.AkkaPort's CLR initializer is also 4053, so that assertion passes with the mapping default deleted. The new test inserts through raw SQL with the column omitted, so only the schema can supply the value — verified by setting defaultValue: 0, which turns exactly that one test red and leaves the other two green. Configuration.Tests 95/95. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
2bbb02713c |
feat(config): add ClusterNode.AkkaPort/GrpcPort — central's dial targets
Per-cluster mesh Phase 1 groundwork. Once the meshes split (Phase 2) central can no longer see a site node's appsettings, so the transport ports it must dial have to live in a row central can read. AkkaPort is non-nullable with a 4053 default — every node listens on a remoting port, so 0 is never a truthful value and pre-existing rows must migrate to something real. GrpcPort is nullable with no default: nothing listens on it until Phase 5, and a non-null default would assert a port that does not exist. Both are documented as central's dial targets rather than the node's own binding config; the duplication against Cluster:Port is reconciled in Task 4. The new SchemaCompliance test is red until the migration lands (Task 2). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
3f24d4d6bf |
feat(cluster)!: self-first seed ordering replaces the InitJoin self-form watchdog
Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.
- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
self-first). The site nodes are untouched -- they seed off central-1 only and
are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
would refuse to boot every driver-only site node. Identity is compared on
PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
outside Akka's join handshake, so it could not distinguish "no seed answered"
from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
is not ignored mid-handshake, it wins. The live gate (
|
||
|
Joseph Doherty
|
ea45ace1c3 |
fix(cluster): don't self-form while a seed peer is reachable — the fallback islanded a restarting node
The live gate for the manual-failover control caught the self-form fallback forming a SECOND cluster. A node bounced by a failover restarted, received InitJoinAck from its live peer — the join was in flight and healthy — but did not get the Welcome inside the 10s window, because the peer's ring still held the node's previous incarnation (Exiting -> Down -> Removed). The fallback fired on the timer and the node islanded itself until an operator restarted it. The original design assumed Cluster.Join(SelfAddress) would be ignored mid-handshake. It is not — it wins. And since manual failover deliberately produces that restart, every failover could island the node it bounced. Before self-forming, TCP-probe the other seed addresses; a reachable peer means wait another window instead. That is also what the fallback claims to detect: 'no seed answered InitJoin (peer down at boot)'. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
d8a85c3d89 |
feat(adminui): manual failover control on the cluster redundancy page
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
69b697bc3e |
feat(redundancy): manual failover service — graceful Leave of the driver Primary
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
b8ac7e35e6 |
test(cluster): self-form fallback — lone-seed forms, disabled waits, site-node guard
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
b2d19a1730 |
fix(tests): repair the DriverTypeNames guard, and stop MSBuild hoarding worker nodes
Two independent housekeeping defects. DriverTypeNamesGuardTests (3 failures, pre-existing). The guard discovers each driver's *DriverFactoryExtensions.Register by reflection and invokes it, then asserts the registered type-name set matches the DriverTypeNames constants. It built the argument array positionally — registry first, null for everything after — on the assumption that the remaining parameters were optional. Galaxy's are not: its Register takes a REQUIRED ISecretResolver and guards it with ArgumentNullException.ThrowIfNull, so the reflective call threw and all three parity facts failed. OpcUaClient's equivalent parameter is optional, which is why only Galaxy tripped it. Worth stating plainly: the guard was not partially broken, it was completely inert. The throw happened while building the registered set, so NO driver's parity was ever checked — a genuine drift in any of the nine would have been invisible behind the same three red tests. Arguments are now supplied by parameter type, and a parameter the helper cannot satisfy fails with a message naming the factory and the parameter instead of an opaque ArgumentNullException. The stand-in resolver reports every secret absent, so if the "factory func is never invoked" assumption is ever broken, the driver fails closed. Core.Abstractions.Tests: 138 passed, 0 failed. MSBuild node reuse. MSBuild leaves worker nodes resident between builds so the next one starts warm; across this session's repeated solution builds and test runs that reached 55 idle processes holding ~6 GB. That is not just untidy — the integration suites assert on timeouts, so memory pressure produces failures indistinguishable from real regressions, the same failure mode the Workstation-GC fix addressed from a different direction. Directory.Build.rsp now passes -nodeReuse:false. Measured on the same incremental solution build: 15 residual nodes without it, 2 with, and elapsed time 6.12s vs 6.11s — no cost worth the memory on this machine. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
50b55ee4b9 |
fix(redundancy): elect the Primary by cluster age, not by address
RedundancyStateActor derived the driver Primary from ClusterState.RoleLeader("driver").
Akka offers two different notions of a "first" member and they are not the same
one: role leader is the lowest-ADDRESSED Up member (host, then port), while
ClusterSingletonManager places singletons on the OLDEST — lowest up-number.
They agree on a freshly-formed cluster, which is why every existing test passed.
They diverge after any restart: the restarted node re-joins as the youngest while
keeping its address, so if it holds the lower address it becomes role leader while
the singletons stay put. The snapshot would then name a Primary that is not
hosting the work, and every Primary-gated surface follows it — inbound device
writes, native-alarm acks, the fleet-wide alerts emit, and the alarm-history
drain would all enable on the wrong node while the node actually running the
singletons stayed gated off.
BuildSnapshot now selects the oldest Up member carrying the driver role, matching
singleton placement. Leaving members are excluded: a node handing its singletons
over must not be named Primary.
NodeRedundancyState.IsRoleLeaderForDriver is renamed IsDriverPrimary, and
NodeHealthInputs.IsDriverRoleLeader likewise. Keeping the old names would have
left the wire contract asserting a derivation the code no longer uses — the same
drift that made this defect invisible.
Proven by a real two-node cluster rather than a mock. RedundancyPrimaryElectionTests
binds the first-joining node to the HIGHER port, so oldest and lowest-address name
different nodes, and includes a fixture assertion that the divergence actually
occurred — without it the real assertion could pass for the wrong reason. Positive
control: restoring the RoleLeader derivation turns exactly the two election tests
red while the fixture check stays green.
Runtime.Tests 440 passed, ControlPlane.Tests 82, Cluster.Tests 36.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
2964361a6f |
fix(cluster): auto-down downing strategy — a two-node pair could not survive an oldest-node crash
Ports the sister project's live-proven fix (ScadaBridge cf3bd52f). OtOpcUa ran the identical configuration it indicts: keep-oldest with down-if-alone = on. Akka.NET 1.5.62's KeepOldest.OldestDecision only lets the down-if-alone branch rescue a side holding >= 2 members. A two-node cluster losing a peer is always a 1-vs-1 split, so the branch never fires, the lone survivor falls through to DownReachable and downs ITSELF, and run-coordinated-shutdown-when-down then terminates it. down-if-alone is a 3+-node feature and does not do what its name suggests for a pair: a crash of the oldest node is a total outage, which is the exact failure the redundancy pair exists to absorb. Cluster:SplitBrainResolverStrategy now selects the provider, defaulting to auto-down: Akka's AutoDowning with auto-down-unreachable-after = 15s, so the leader among the reachable members downs the unreachable peer and a crash of either node fails over in place. keep-oldest remains available for deployments that would rather take an outage than ever run dual-active during a real partition. An unrecognised value fails the host at startup rather than falling through to the fatal default. The tests assert the EFFECTIVE configuration — they start a real host through WithOtOpcUaClusterBootstrap and read akka.cluster.downing-provider-class back off the running ActorSystem. This is not incidental. The first draft inlined the three calls the bootstrap makes instead of calling it, which pinned the test's own wiring: sabotaging production's HOCON precedence left all 36 green. The prior file had the same shape at a smaller scale, asserting only that BuildClusterOptions returned a KeepOldestOption — true, and true of a configuration that cannot fail over. Positive control: making BuildDowningHocon emit nothing for auto-down turns exactly the two effective-config guards red. Measured while verifying rather than assumed: HoconAddMode.Append also wins here, purely because it is added last, so the mode name is not the guarantee. The comment now says so instead of asserting a precedence rule that does not hold. Not yet live-drilled on OtOpcUa. The docker-dev rig is a single six-node mesh where the 1-vs-1 pathology cannot occur; the kill-the-oldest drill belongs with the per-cluster mesh work that makes every mesh exactly two nodes (design doc 6.2 / Phase 0a). Recorded as an outstanding gate in docs/Redundancy.md. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
8838e92d0a |
fix(tests): StopNodeBAsync also has to stop the node, not just dispose it
Completes
|
||
|
Joseph Doherty
|
835fc08c3f |
fix(tests): stop the integration host from ballooning to 30 GB
User-reported: the integration tests leave instances behind and take 18+ GB.
Measured before changing anything — a SINGLE Host.IntegrationTests process peaked
at 30,153 MB RSS across its ~200 tests.
The cause was not a leak and not parallelism. Referencing the Host drags in the
ASP.NET Core framework reference, which turns Server GC on by default: one heap
per core, tuned for throughput and deliberately reluctant to hand memory back.
Correct for a server, wrong for a test host. Workstation GC takes the same suite
from 30,153 MB to 6,717 MB — 78% less — with no change in runtime (45s).
Two hypotheses were measured and rejected on the way, recorded here so nobody
re-runs the experiment: capping xunit maxParallelThreads to 4 cost 29% wall-clock
and saved nothing (30,153 -> 31,115 MB, i.e. noise), and it turns out a single
process was doing all of it. Reverted.
The harness fix is real but was worth only 4% on its own: TwoNodeClusterHarness
disposed each node without stopping it first, and IHost.DisposeAsync only disposes
the service provider — it never invokes IHostedService.StopAsync, which is where
Akka.Hosting terminates the ActorSystem. So every test left two live ActorSystems,
remoting transports and dispatcher pools included, rooted for the process
lifetime. The class doc-comment asserted the opposite ("DisposeAsync, which runs
CoordinatedShutdown"); it was wrong, and StopNodeBAsync inherited the same bug on
a path failover tests depend on.
The payoff is larger than the memory number. Two failures I had classified as
known-flaky baseline noise now pass consistently:
RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed
and ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks.
They were never flaky — they were starved. Full solution: peak summed test-process
RSS 15.2 GB, zero new failures, two fewer.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
f9f1b8fcee |
fix(localdb): phase-2 live gate — 4 production defects found and fixed
Gate record: docs/plans/2026-07-20-localdb-phase2-live-gate.md. Checks 1, 2, 5, 6 pass. Checks 3 and 4 are NOT satisfied — the defect they were meant to confirm turned out to be the opposite of what the plan assumed. Three defects crash-looped every driver node before check 1 could even run: 1. An empty ServerHistorian:ApiKey kills the host. ServerHistorianOptions- Validator exists to turn exactly that class of failure into a named OptionsValidationException, but its documented fail tier explicitly excluded ApiKey on the reasoning that a keyless client "degrades — the gateway rejects calls". It does not: the client validates its own options at construction, so the process dies during Akka startup and never makes a call. 2. UseTls disagreeing with the endpoint scheme kills the host too, in both directions (both messages confirmed in the shipped client assembly). Moving an endpoint from https to http without clearing UseTls is an ordinary migration slip. 3. Plaintext h2c was UNREACHABLE. HistorianGatewayClientAdapter forwarded the TLS-only options unconditionally, and AllowUntrustedServerCertificate defaults to false, so it always sent RequireCertificateValidation=true — which the client rejects outright when UseTls=false. Every http:// deployment crashed, though the scheme is documented as the supported way to select h2c, and the only workaround was to assert a certificate posture for a connection that has no certificate. The fourth was the blocker, and it is Phase 2's own: 4. The drain gate deferred to a Primary that cannot deliver. Redundancy roles are elected CLUSTER-WIDE; the alarm queue is PAIR-LOCAL. On the rig the elected driver Primary is central-1 — it carries the driver Akka role, replicates nobody's LocalDb and does not even run the alarm historian — so every driver node logged "Historian drain suspended", including the two site-b nodes that have no peer at all. Nothing drained anywhere, where before Phase 2 it drained fine. The cost is not a duplicate; it is the buffer growing to the capacity wall and evicting the audit trail it exists to protect. Fixed in three layers: a separate ShouldDrainAlarmHistory policy (unknown role drains; the two gates now deliberately disagree, and a test pins that); peer- host matching in DriverHostActor so a node stands down only for a Primary holding its rows; and AddAlarmHistorian short-circuiting the gate when replication is unconfigured — testing BOTH Replication:PeerAddress and SyncListenPort, since only the dialing half sets the former while both halves share the queue. Every one of these follows from the asymmetry: a false allow costs a duplicate row, which at-least-once delivery already accepts and payload-hash ids collapse; a false deny loses data silently. A third vacuous test, caught by the same delete-the-guard discipline: the role-view tests stayed green with the guard removed, because AwaitAssert polls until an assertion passes and the assertion was "reads open" — which is the SEEDED value, satisfied at the first poll before the actor processed anything. They now assert the sequence of published values through a recording view; the control then goes red for exactly the cases that matter. Migration evidence: 11 legacy rows across two deliberately overlapping files converged to exactly 9 identical rows on both nodes, proving D-6's payload-hash identity on real nodes rather than in a fixture. Open design fork, recorded in the gate doc rather than decided here: a pair cannot currently identify its own Primary, so both halves drain. Safe in every topology — nothing loses data — but the gate's de-duplication benefit is unrealised until roles are scoped per pair. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
2e4ccf7fe9 |
chore(localdb): phase-2 DoD sweep
Build: 0 errors solution-wide, and 0 warnings from every project this branch
touches. The ~816 solution-wide warnings are pre-existing xUnit1051 /
OTOPCUA0001 / CS86xx in untouched driver + client test projects.
Tests: full solution run compared against a full run on a detached worktree at
the pre-branch baseline
|
||
|
Joseph Doherty
|
271fcc4e15 |
test(localdb): alarm S&F convergence scenarios
Two driver nodes replicating the alarm buffer over the real loopback h2c
transport, through the real fail-closed interceptor. This is the test the
phase exists to pass: schema, registration, sink rewire and drain gate can
all be individually green while a pair still fails to share the undelivered
alarm history that is the whole point of moving the queue.
Rows are written through the production sink rather than hand-rolled
INSERTs, so the id derivation, column set and delete-on-ack semantics under
test are the ones production uses. The scenarios cover a buffered burst
converging with the oplog draining to zero, delivered rows being removed from
BOTH nodes as tombstones (the no-redeliver-after-failover property), writes
during a transport outage surviving the rejoin, and the same event accepted
on both nodes collapsing to one row.
The positive control found a weak assertion. With
RegisterReplicated("alarm_sf_events") commented out, three of the four went
red immediately -- but the same-event-on-both-nodes scenario still PASSED,
because with replication off each node trivially holds its own single copy
and "one row on each" is satisfied by two databases that never spoke. It now
also enqueues a distinct event on B and requires both nodes to hold two rows,
which cannot be satisfied without convergence; it goes red under the control
like the others. Control restored, all four green.
Also updates the second exact-set replicated-tables pin, in LocalDbWiringTests.
The plan predicted these pins would go red here, and that is them working:
one of them is asserted through the full driver DI graph rather than the
OnReady callback, so it catches a registration that exists in the callback
but never reaches a running host.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
af545efdf5 |
feat(localdb): one-time alarm-historian.db migrator
Copies the pre-consolidation store-and-forward queue into the consolidated
database on first boot, then renames the legacy file aside. Rows are in that
file precisely because the historian could not be reached, so dropping them
on upgrade would discard exactly the alarm audit trail the queue exists to
protect.
Runs last in OnReady, after every RegisterReplicated call. It is the only
thing in OnReady that writes rows, and capture is trigger-based: a migration
that ran before registration would recover the backlog locally and never
replicate a line of it, silently and permanently.
Ids are derived from the payload rather than the plan's mig-{node}-{legacyId}
scheme. Node-prefixing solves the collision the legacy AUTOINCREMENT key
would cause -- node A's row 7 and node B's row 7 are different alarms -- but
it preserves a duplication that should be collapsed instead. A warm pair's
two legacy files OVERLAP: HistorianAdapterActor default-writes while its
redundancy role is unknown, so both nodes accepted the same transitions
during every boot window. Prefixed ids would carry those duplicates into the
merged buffer forever; equal-payload ids converge them. The same property
makes a crash between commit and rename harmless under INSERT OR IGNORE.
OnReady now takes IConfiguration rather than offering an overload that skips
the migration. A wiring mistake that silently discarded a node's undelivered
alarm history is not a mistake worth making possible.
The copy is restricted to the columns the legacy table actually has. Naming
a column an older build never wrote throws "no such column", which would
discard every row in the table rather than the one field.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
71379816e7 |
feat(localdb): alarm S&F on LocalDb with primary-gated drain (cutover)
Moves the alarm store-and-forward buffer out of its own alarm-historian.db and into the node's consolidated LocalDb, where it replicates to the redundant pair peer. A node that dies holding undelivered alarm history no longer takes it to the grave. Tasks 2 and 3 land together, as the plan anticipated. They are not separable: the gate is a constructor argument of the rewritten sink, and a commit that replicated the queue without gating the drain would be a commit in which both nodes of a pair deliver every alarm event, continuously. That drain gate is the load-bearing part of the change, and the recon explains why it is new work rather than a refinement. Exactly-once delivery across a pair is enforced today on the ENQUEUE side, by HistorianAdapterActor: only the Primary enqueues, so the Secondary's queue is empty and its ungated drain has nothing to send. Replicating the table destroys that invariant. The gate is a Func<bool> the caller supplies, because the drain runs on a timer the sink owns rather than on a mailbox, and Core.AlarmHistorian cannot reference PrimaryGatePolicy in Runtime. Runtime supplies it via a new IRedundancyRoleView singleton that DriverHostActor publishes its Primary-gate verdict to -- the same verdict the inbound-write and native-ack gates use, so there is no second notion of am-I-the-Primary to drift. Two failure modes are deliberately closed: - The view is seeded OPEN, matching the policy's own answer for an unknown role with no driver peer. A deployment that runs no redundancy never publishes to it, and defaulting closed would silently stop its alarm history forever. - A gate that throws is read as not-now, never as permission, and a closed gate reports the new HistorianDrainState.NotPrimary rather than Idle. A Secondary's rising queue is supposed to look different from a stalled drain, and if BOTH nodes report NotPrimary the pair is misconfigured and says so instead of quietly filling toward the capacity ceiling. Row ids are a hash of the payload rather than fresh GUIDs. Both adapters accept the same fanned transition in the window before the first redundancy snapshot arrives, and under last-writer-wins an equal key converges those two accepts into one row instead of duplicating them. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
8a9cb40a72 |
feat(localdb): alarm_sf_events replicated table
Adds the alarm store-and-forward buffer to the consolidated LocalDb file and registers it for replication, so a node that dies with undelivered alarm history no longer takes it to the grave. The table keeps the legacy queue's shape rather than the status column the plan sketched. An acknowledged row is deleted, not marked delivered: that needs no second sweeper to keep the table bounded, leaves the capacity semantics untouched, and the replication engine carries the delete as a tombstone so the peer drops its copy anyway -- which is what the status column was for. last_error is retained because it is the only operator-facing record of why a row was dead-lettered. The primary key is app-minted TEXT. The legacy AUTOINCREMENT RowId cannot replicate under last-writer-wins: two nodes would independently allocate rowid 7 to different alarms and silently overwrite each other. The drain index therefore orders by enqueued_at_utc rather than insertion order, with id as a tiebreak so the ordering is total. Tables are created unconditionally, independent of AlarmHistorian:Enabled. An empty registered table costs three triggers; creating it lazily would mean a node that enables the historian later writes rows before its capture triggers exist, which is exactly the silent-loss shape OnReady's ordering comment warns about. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
ef41a43102 |
fix(drivers): fail the apply when the artifact could not be read (#486)
ApplyAndAck advanced _currentRevision and recorded NodeDeploymentState=Applied immediately after ReconcileDrivers, which returns null when the artifact could not be read. So a node that applied NOTHING reported success, claimed a revision whose configuration it never applied, and — because HandleDispatchFromSteady short-circuits on a revision match — could never be healed by re-dispatching that revision. Only a later, different revision recovered it. Now a null blob fails the apply: the revision is left where it was, NodeDeploymentState records Failed with the reason, and the coordinator gets a Failed ACK. Leaving the revision alone is what makes the retry actually land instead of being waved through. This is about telling the truth, not about tearing anything down — the node keeps serving its last-known-good address space, drivers and subscriptions throughout (#485). Tests, RED-first (both verified failing with "should be Failed but was Applied"): the ACK/revision assertion, plus the one that matters — after a failed apply, re-dispatching the SAME revision now genuinely applies. Its recovered artifact adds a second driver precisely so a short-circuited ACK (which has no side effects) cannot satisfy it. Four existing tests changed, and both changes are deliberate: - DriverHostActorTests.SeedDeployment never set ArtifactBlob at all. Its three consumers are about the apply/ACK/state machine, not the artifact, and now need a genuine apply — so the helper seeds a well-formed artifact declaring no drivers. That is what the composer emits for an empty configuration, and is precisely what "bytes we could not read" is NOT. - EmptyArtifact_IsNotCached asserted "the apply still succeeds — an empty artifact is a legitimate no-op deployment". That premise is the bug. Flipped to Failed; the test's actual subject (nothing is cached) is untouched and now holds for two independent reasons. Closes #486 Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
b0c031f09f |
fix(drivers): don't tear down field I/O when a deploy's artifact reads back empty (#485)
The driver-side half of the same mistake the address-space fix corrected. ReconcileDrivers and PushDesiredSubscriptionsFromArtifact both read the artifact as `...FirstOrDefault() ?? Array.Empty<byte>()`. Their THROW paths already returned early, but empty bytes flowed onwards as a real answer: zero driver specs, so DriverSpawnPlanner planned every running child for StopChild, and then an empty desired set dropped each surviving driver's live subscription handle. A node lost its entire field I/O and still ACKed Applied. Same rule as the address space: no bytes is no answer, not "a configuration with no drivers". Nothing legitimate produces a zero-length blob — deleting the last driver still deploys a JSON document with empty arrays. Both sites now skip and keep what is running. The two guards are independent because PushDesiredSubscriptions does its OWN ConfigDb read, so the row can go missing between them. Also corrects the ReconcileDrivers doc comment, which claimed an empty blob made it "effectively a no-op" — with children running it was the opposite. Tests: DriverHostActorUnreadableArtifactTests, RED-first (verified failing — after the empty dispatch the driver list was empty). A zero-length ArtifactBlob reproduces byte-for-byte what the missing-row case delivers, so the race does not have to be constructed. Two controls, both load-bearing: - a READABLE driverless artifact still stops the driver, so the guard keys on "the read gave us nothing", not on "fewer drivers than before"; - dropping a driver's LAST TAG still clears its subscription. This one was added after the absence assertion was caught passing on a race: the unsubscribe is an async self-tell, so with the second guard deleted the test still went green. The control observes that same unsubscribe arriving well inside the settle window, which is what makes the absence meaningful — with the guard deleted the suite now correctly goes RED. SubscribableStubDriver gains an UnsubscribeCount for that observation. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
4b0be1335e |
fix(opcua): hold last-known-good when a deploy's artifact cannot be read (#485)
A transient ConfigDb error while loading a deployment's artifact emptied the served address space. LoadArtifact caught the exception, logged "rebuild becomes no-op" and returned zero bytes; those parsed to an EMPTY composition, which the planner diffed against the live one as a PureRemove and the applier faithfully executed — 16 nodes gone, while the log claimed nothing happened. An artifact we could not obtain is not an empty configuration. Nothing legitimate produces a zero-length blob (an operator who really deletes everything still deploys a JSON document with empty arrays), so "no bytes" can only mean "no answer". HandleRebuild now abandons the rebuild on one, leaving both the materialised nodes and _lastApplied intact so the retry diffs against what is actually being served. The loader's own log line is now true. The two cases are logged differently: warn when there is a live address space being protected, debug when nothing has been deployed to this node yet. Covered by a RED-first actor test (verified failing: the load error tore down eq-2's subtree), plus a positive control proving a READABLE artifact that genuinely drops an equipment still removes it — the guard suppresses teardown only when the read failed. Found on the LocalDb Phase 1 follow-up live gate, which induced it by flapping SQL; that gate's log is the production evidence for the seam. Closes #485 Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
232bff5df7 |
fix(localdb): close both Phase 1 live-gate follow-ups
1. Wiped-node back-fill (live gate check 4). Fixed upstream in ZB.MOM.WW.LocalDb 0.1.2 and pinned here. The gate's diagnosis was slightly off: the oplog cap was not the gate. Snapshot detection measured the peer's gap against the oldest surviving oplog row and read an empty oplog as "no gap possible" — and an empty oplog is the steady state of a converged pair, since ack-pruning deletes everything the peer confirmed. The healthy state was the one state that could not heal a wiped peer. Now measured against last_acked_seq when the oplog is empty. Pinned at this level too, not just the library's: the pair harness grows a WipePassiveAsync (a NEW database — a rebuilt node comes back with a new node id and a zero watermark), and the new scenario asserts the emptied oplog as its precondition before wiping, then requires the deployment artifact to come back byte-identical with no new deploy. Verified RED against the pinned 0.1.1 (times out waiting for back-fill) and green on 0.1.2. 2. Oplog growth on default-OFF nodes (live gate check 8), fixed at the source: StoreAsync now skips entirely when the pointer already names this deployment/revision, the SHA matches the bytes, and the expected chunk count is present. Re-caching an artifact the node already holds — every restart's boot-from-cache, every RestoreApplied — writes nothing and mints no oplog rows, where before it cost a delete plus an insert per chunk plus a pointer update, all identical to what was already there. Identity is over the bytes and the chunks are counted, both deliberately: a re-composed artifact can carry the same ids with different bytes, and a pointer can name a deployment whose chunks are missing, where skipping would make an unreadable cache permanent. Both guards verified RED against a naive pointer-only skip. The gate's stated bound was also wrong and is corrected in the doc: growth was never headed for the 1M row cap. AddZbLocalDbReplication is registered unconditionally, so MaintenanceBackgroundService runs on default-OFF nodes and the 7-day MaxOplogAge cap prunes. Runtime.Tests 412/0/31, Host.IntegrationTests LocalDb 45/45. Pre-existing and unrelated: AbCip_Green_AgainstSim (needs the AB CIP docker fixture, red on a clean master too) and a flaky Roslyn race test (green 2/2 in isolation). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
c6a9f93a0c |
fix(localdb): re-cache the artifact on RestoreApplied so a wiped node self-heals
CacheAppliedArtifact ran only on a fresh apply (ApplyAndAck), never on the bootstrap RestoreApplied path. So a node whose cache was lost — fresh/wiped volume, disk failure — recovered its served state from central yet stayed cache-less, unable to boot-from-cache on the NEXT central outage until some future new deploy happened to land. Replication does not heal this: a peer's already-acked cache rows are pruned from its oplog, so a fully-wiped, converged node is never back-filled. RestoreApplied now writes the served artifact back to the cache (invariant: the cache holds what the node serves). Found on the docker-dev live gate wiping a node's LocalDb volume. |
||
|
Joseph Doherty
|
9137cb41eb |
fix(localdb): materialise the address space from the cached blob on boot-from-cache
Boot-from-cache told OpcUaPublishActor to RebuildAddressSpace(deploymentId), which re-loads the artifact from the ConfigDb by id — the very database that is unreachable during the outage this path exists to survive. The rebuild no-op'd, so a cache-booted node ran its drivers but served OPC UA clients an EMPTY address space. RebuildAddressSpace now carries an optional in-hand Artifact blob; ApplyCachedArtifact passes the cached bytes so materialisation happens from them directly. Found on the docker-dev live gate: healthy peer served 16 nodes, the cache-booted node served 1. |
||
|
Joseph Doherty
|
ce9fa07ff8 |
fix(localdb): re-bind ASPNETCORE_HTTP_PORTS surface, not just ASPNETCORE_URLS
The Kestrel re-bind read only urls/ASPNETCORE_URLS. The aspnet:8.0+ base images set ASPNETCORE_HTTP_PORTS=8080 (all interfaces) as the CONTAINER default in place of ASPNETCORE_URLS, so a driver node that never sets URLS fell straight through to Kestrel's localhost:5000 default — silently moving its health/metrics surface to loopback when the sync listener took over Kestrel. Now falls through URLS → HTTP_PORTS/HTTPS_PORTS (wildcard, all-interfaces) → localhost:5000, mirroring Kestrel's own source precedence. Found on the docker-dev live gate (central nodes were fine — they set ASPNETCORE_URLS=9000 explicitly). |