Ran on master @ 0b194003, post-deletion image d2a290fc5711 (provenance-guarded).
PSK matrix 7/7 incl. unset-key boot refusal; command matrix with Restart/Reconnect
driven from the real /hosts panel (#546 closed the 1A composition carry-forward by
direct evidence); exactly-once ack matrix sealed 6/6; central-kill + site-kill drills
honest (one loud WRN, +2:00 deadlines, boot-replay, no retro-ack); mid-deploy
coordinator kill: recovered coordinator fired the deadline at exactly +2:00, all
landed acks preserved, zero dupes; boundary: 3 established :4053 associations, all
pair-internal, zero site↔central; full-rig restart transport grep = 0 on all six
(one non-mesh observation: central pair reproduced the documented guard-off
cold-start split-brain, healed by single-node rejoin); MeshGrpcBoundaryTests 3/3;
terminal full suite RC=0 direct capture, 50/50 suites, 9,333/0.
I-4 cold-PSK-cache re-carried out of the program (unmeasurable on docker-dev).
No new defects.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Node→central ApplyAcks over gRPC, gated live on the docker-dev rig at
4a2a3d3a. All 7 checks PASS (the 6 planned plus the mandated 4b hang
drill).
The 4b drill is the one that earns its place: with central-1 PAUSED
rather than stopped, both site-a nodes burned a full AckTimeoutSeconds
on it as DeadlineExceeded and still reached central-2 with a live
deadline — the per-endpoint FRESH deadline from 667639e6, proven on the
failure shape a hoisted deadline could not survive. Observed failover
latency ~10 s, exactly one AckTimeoutSeconds.
It also produced the run's only duplicate ack, and that turned into
evidence rather than a defect: the frozen central had in fact received
both acks and replayed them on unpause, and the coordinator reported
2/6 acks landed, not 4/6 — the idempotent absorption the sender's
at-least-once remark asserts, observed instead of argued.
Coexistence is per pair in one deployment: site-a's rows are Applied
with matching ReportApply lines on central and zero receptionist
associations, site-b's are Applied with zero ReportApply lines and a
live receptionist association. Neither pair could have used the other's
transport.
Carry-forwards recorded: the I-4 cold-PSK-cache latency is structurally
unmeasurable on docker-dev (ClusterPsks is read ahead of the store, so
InvalidateAll clears an empty cache) and moves to a store-backed rig;
a restarted central's cached channel can stay in backoff ~2 min on the
node that watched it die, absorbed by the walk with no ack lost. The
1A finding-5 wording fix gains a second confirmation — 4 of 4 admin
boots lost the PreStart race, and every ack central-1 accepted proves
the provider it said would NEVER be published was.
Docs only; the rig flip and the four Serilog Debug overrides were
reverted and are not in this diff. Rig restored to six healthy,
all-ClusterClient, sealing 6/6.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Re-ran the 1A gate on the docker-dev rig at e54883a8 after the defect-4 boot
fix. Image rebuilt (otopcua-host:dev -> 1d8452a29018); six host nodes plus
both one-shots recreated; run 1's preserved image was stale and deliberately
not reused. VERDICT: PASS.
Boot: 6/6 nodes healthy INCLUDING the central pair, 0 Hosting failed to start,
/health/ready 200 on every node, MAIN mesh 2/2.
Check 1 PASS: one DispatchDeployment, one correlation id, accepted by all six
nodes; central logs 6 accepted / 0 failed; zero ClusterClient lines on either
central node with Akka.Cluster.Tools.Client at Debug.
Check 2 PASS: deployment Sealed with all six NodeDeploymentState rows Applied
inside ~90 ms, while the four site nodes hold receptionist associations — the
only ack path they have in 1A. Mixed transport proven.
Check 4 PASS: Ack clicked on the real /alerts page against a live ACTIVATED
row; central fanned one AlarmCommand to all six over gRPC and the condition
acknowledged 0.54 s later carrying the operator identity.
Check 7 PASS: with site-b-2's GrpcPort NULLed, both central nodes warned once
and the next dispatch reached exactly five nodes; the deployment then honestly
ended TimedOut with site-b-2 alone unapplied. Row restored, 6/6 sealed after.
Check 8 PASS and no longer vacuous: 0 denials on all four interceptors, and
this time config-serve (28 KB verified artifact fetches from :4055) and
telemetry (streams open on all six, live driver health on /hosts) both carried
real traffic.
Check 3 closed by composition, with the reason now established rather than
assumed: <DriverStatusPanel has 0 usages repo-wide, no rendered page carries a
Restart/Reconnect control, and the only non-script-analysis POST is
/api/deployments. Run 1 drove the node-side leg for both commands; check 4
drives the identical AdminOperationsActor.Dispatch -> DispatchOverGrpc origin
path from the AdminUI.
Checks 5 + 6 stand from run 1 — nothing in the fix touches the interceptor.
New finding 5, recorded because it is load-bearing for the fix's design: the
PreStart resolve LOSES its race on real boots — 2 of 3 observed admin boots
logged Failed to resolve for both dependencies with the same
AuditWriterActorKey cause. The memoize-on-first-use hardening is therefore not
theoretical, and its heal was proven live by moving the coordinator singleton
onto the node that lost the race (stop central-2) and watching central-1 fan a
dispatch to all five live nodes with no dialer error. Residual is cosmetic and
carried to Phase 2: the Error text reads as permanent, and it will trip a soak
error-grep on a healthy node.
Defect 4 marked CLOSED; run 1's record kept in full underneath the combined
table. Rig restored: flip reverted, six nodes healthy on all-ClusterClient on
the committed image, three mesh ClusterClients back on each central node.
Ran the 1A live gate on the docker-dev rig at fbf7b708 (image rebuilt from
this worktree, all eight containers recreated). VERDICT: FAIL.
Both central nodes fail to boot on this branch with
MissingActorRegistryEntryException: AuditWriterActorKey — the first WithActors
starter eagerly resolves IMeshCommandDialer/IMeshPskProvider, whose graph
reaches ActorAuditWriter, whose ctor reads an ActorRegistry key registered ~60
lines later in the same builder. Reproduced on the same image with
MeshTransport__Mode=ClusterClient, so it is mode-independent and not caused by
the gate's rig-only flip. Recorded as defect 4; no product change made.
Checks 5 (auth negative/positive on real Kestrel h2c) and 6 (wrong-key
isolation, both directions, with a positive control) PASS. Checks 3 and 4 pass
their node-side leg only, driven with real RestartDriver/ReconnectDriver/
AlarmCommand envelopes that restarted and reconnected SITEA-modbus. Checks 1,
2 and 7 are blocked outright — no central, no dispatch, no contact refresh —
and check 8 is vacuous for telemetry + config-serve because central dials both
and never booted. All five gaps are named in the carry-forward register and
must be closed by a 1A gate re-run, not carried past the phase.
Issues filed by this task: lmxopcua#544 (ConfigPublishCoordinatorTimeoutTests
deflake) and lmxopcua#545 (mesh planes are h2c-only — accept the risk or add a
UseTls option); the tracking table's two placeholder rows now carry them.
Also recorded: DriverStatusPanel.razor is referenced by no page or component,
so the AdminUI Reconnect/Restart controls are unreachable on this branch and on
master — pre-existing, out of program scope, carried forward.
Rig restored: compose flip reverted, six nodes healthy on all-ClusterClient.
The rig runs the PRE-gate image (the 1A build is kept as otopcua-host:grpc-1a);
a re-run must rebuild.
Phase 0 task 0.6 of the ClusterClient->gRPC migration: wire the dark
MeshTransport Grpc-mode keys into the docker-dev rig (GrpcPsk +
CentralGrpcEndpoints on all six nodes, ClusterPsks on the central pair
only), document the new keys in docs/Configuration.md as shipped
dark/cutover pending, and record the Phase 0 live-gate evidence
(rig boot with no validator fires, two deliberate-misconfig refusals,
full suite green).
Files gitea.dohertylan.com/dohertj2/lmxopcua#542 (FleetStatusBroadcaster
mesh-blind post-Phase-6) and #543 (three pre-existing interceptors accept
a whitespace-only key as set) as follow-ups out of Phase 0's scope.
Joseph Doherty