lmxopcua/docs/plans/2026-07-22-mesh-phase2-live-gate.md

Per-cluster mesh Phase 2 — live gate record

Date: 2026-07-22 · Rig: docker-dev (six-node single mesh) · Branch: feat/mesh-phase2 Plan: 2026-07-22-mesh-phase2-clusterclient-transport.md

Result: PASSED for the transport, with two gate steps not run (no surface exists to drive them), one step whose stated pass-signal does not exist in the code, and four findings — one of which corrects a claim this phase's own documentation makes.

The rig was flipped end-to-end with OTOPCUA_MESH_MODE=ClusterClient (no rebuild, no compose edit — the env indirection added in Task 9 is what made the A/B and the DPS control cheap enough to run).

---

Result table

#	Step	Result
1	Deploy on Mode=Dps	PASS — 48d0c7c5 Sealed, 6/6 Applied
2	Flip all six to ClusterClient, redeploy	PASS — 5db8f6ad Sealed, 6/6 Applied
3	Confirm the transport actually changed	PASS — positive log evidence, see below
4	Stop a node, deploy	PASS — 59b81248 TimedOut naming site-b-2:4053, 5/6
5	MaintenanceMode = 1, redeploy	PASS — c6cb9812 Sealed, 5 rows, contacts 6→5
6	AdminUI Restart + Reconnect	NOT RUN — no UI surface exists (finding D)
7	AdminUI alarm ack	NOT RUN — no alarms configured on the rig
8	Per-node (not singleton) discriminator	PASS — via a better discriminator than the plan's
9	Both centrals down, restart a driver node	PARTIAL — node survives; the stated ack-drop cannot occur
10	Restart central, redeploy	PASS only after operator action (finding C)
11	Frame-size canary	PASS — 0 warnings across all six nodes

Step 3 — the transport really changed

Central, both nodes, at boot under the flag:

central-1 [22:18:16 INF] Mesh ClusterClient created with 6 contact point(s)
central-2 [22:18:17 INF] Mesh ClusterClient created with 6 contact point(s)

Node side, with Serilog__MinimumLevel__Default=Debug overlaid on site-a-1 (the rig pins the default above Debug, so this line is invisible on a stock node):

site-a-1  [22:10:34 DBG] Mesh: received DispatchDeployment from central; publishing node-locally

Under Mode=Dps the same node logs nothing of the sort and central logs Mesh transport is Dps: commands are published on DistributedPubSub and no ClusterClient is created.

Step 5 — the DB-sourced contact set is live

Setting MaintenanceMode = 1 on site-b-2 shrank both centrals' contact sets on the next refresh, and clearing it restored them — independently, on each central's own 60 s timer:

central-1 [22:22:16 INF] Mesh contact set changed; replacing the ClusterClient
central-1 [22:22:16 INF] Mesh ClusterClient created with 5 contact point(s)
central-1 [22:23:16 INF] Mesh ClusterClient created with 6 contact point(s)
central-2 [22:22:16 INF] … identical, same timestamps

No InvalidActorNameException on any rebuild — the generation-suffixed client name (Task 2, ported from the sister project's shipped bug) does its job.

Step 8 — the per-node discriminator, replaced with a stronger one

The plan's pass-signal does not exist. It asked for the surviving central's comm actor to log receipt of a node ack inside the downing window. CentralCommunicationActor.HandleApplyAck logs nothing on the success path — it only warns when the coordinator is unresolvable. There is no line to observe. (Recorded as finding E.)

The evidence above is a cleaner discriminator and needs no timing race. Mesh ClusterClient created is emitted from PreStart, so it fires once per live actor instance. Both centrals logged it, and both logged their own independent contact-set rebuild at step 5. A ClusterSingletonManager runs the actor on the oldest node only — the other node hosts a manager that never starts the actor and would log nothing. Two nodes logging it proves two live instances, i.e. per-node registration. The node side is proven the same way: SendToAll collected 6/6 acks, which requires six separately registered node comm actors.

The failover half was run anyway, and passed: docker stop central-1 (the oldest, and the singleton host) → the surviving central accepted a deployment 4 seconds later and collected acks from all five live nodes (e3161c25; it TimedOut only because central-1 is itself a configured node that was down — correct Phase 1 semantics). Graceful stop hands the singleton over immediately, so no downing wait was involved.

Step 9 — the stated failure mode cannot occur

The plan expected a restarting driver node to emit a boot-time ack that gets dropped with a Warning. Driver nodes do not ack on boot. DriverHostActor.SendAck is called only from the apply paths — a node with no dispatch to answer has nothing to send, so there is no drop to observe.

What was verified: with both centrals stopped, site-a-1 restarted, came up clean, logged its association failures against both central addresses, retried, and stayed up — no crash loop. The node-side ClusterClient's forever-retry (reconnect-timeout = off) behaves as configured.

Step 11 — the canary stayed quiet

grep -ci "frame size|frame-size|maximum-frame" = 0 on all six nodes, as designed: the deploy notify carries only DeploymentId + RevisionHash + CorrelationId.

---

Finding A — a 10–30 s post-restart delivery window, pre-existing, not Phase 2

A deploy issued shortly after a driver node restarts does not reach that node, and the deployment fails at the 2-minute deadline naming it. Measured against site-a-1:

Delay after the node logs readiness	Result
~4 s	missed (f738e1de TimedOut, node row Applying)
10 s	missed (0c7a6296 TimedOut)
30 s	applied, sealed

The DPS control fails identically. Flipping the rig back to Mode=Dps and repeating the 10 s case produced e85f53c5 → TimedOut, site-a-1 unacked. So the window is a property of cluster membership convergence, not of the transport this phase introduces. Recorded rather than fixed.

Running that control is the only reason this is a note and not a Phase 2 regression. The temptation to skip it — "obviously the new transport broke it" — would have produced a false attribution and a wasted investigation.

Finding B — buffer-size = 0 does not deliver what this phase's docs claim it does

The stated rationale, in the plan, CLAUDE.md, and docs/Configuration.md, is that buffering "would replay a DispatchDeployment on reconnect and apply a deployment the coordinator already sealed as TimedOut, leaving the node running a configuration the database records as failed."

That outcome happens anyway, by a different route. Deployment 0c7a6296 timed out at 22:14:30 with site-a-1 unacked. The node's row:

NodeId          Status  StartedAtUtc              AppliedAtUtc
site-a-1:4053   1       2026-07-22 22:12:30.145   2026-07-22 22:15:14.823

It applied 44 s after the deployment was sealed TimedOut, on its next restart. The mechanism is DriverHostActor.Bootstrap(), which reads the node's most recent NodeDeploymentState, finds the orphan Applying row the coordinator pre-created at dispatch, and replays it:

case NodeDeploymentStatus.Applying:
    _log.Warning("DriverHost {Node}: found orphan Applying row for deployment {Id}; replaying", …);
    ApplyAndAck(new DeploymentId(latest.DeploymentId), revision.Value, CorrelationId.NewId());

So buffer-size = 0 removes the transport-level, timer-driven replay — a real difference: it is silent, unbounded in time, and can fire without any node event. The boot-time DB replay remains, and it is the better-behaved of the two (deliberate, Warning-logged, and bounded to a restart). The decision stands; the justification was overstated and the docs are corrected in the same change as this record.

Finding C — stopping both centrals strands the non-seed nodes permanently

After docker stop of both centrals followed by docker start, the fleet did not recover. Deployments reached 2/6 then 3/6 nodes. The cause is not the transport — the Akka cluster itself had split:

Cluster hosts:  MEMBERS 3   UP 3   UNREACHABLE 0
  central-1:4053  UP   admin,driver   leader for admin+driver
  central-2:4053  UP   admin,driver
  site-a-1:4053   UP   driver

The three site nodes that never restarted were not members at all, and their own gossip showed the stale inverse view — central-2 as Exiting, site-a-1 as Leaving — frozen since 22:29:

Gossip(members = [central-2 status = Exiting, site-a-1 status = Leaving,
                  site-a-2 Up, site-b-1 Up, site-b-2 Up])

A graceful stop puts the centrals into Leaving/Exiting; the processes died before the removal handshake completed, leaving the survivors holding members that will never confirm. auto-down does not help — these members are not Unreachable, they are stuck mid-Exiting. The restarted centrals formed a fresh cluster (with site-a-1, which had also restarted and re-joined through its seed), and the three stranded nodes had no path back: they believe they are already in a cluster, so they never re-run the join process.

Recovery required restarting the stranded nodes. After that, the next deploy sealed 6/6 (e20836ac).

This is transport-independent — membership is below both DPS and ClusterClient, and the set of nodes that acked matched the cluster view exactly. It is a downing/rejoin gap of the current single-mesh topology, in the same family as the registered outage gap the design doc already records, and it belongs to the Phase 7 drill work. Under the target per-pair topology (every node a seed of its own pair) the shape changes, and this drill must be re-run there rather than assumed fixed.

Finding D — the AdminUI's driver Restart/Reconnect buttons have no page

Step 6 could not be run, and not for a rig reason. Components/Shared/Drivers/DriverStatusPanel.razor is the only component in the codebase carrying the DriverOperator-gated Reconnect/Restart buttons, and nothing renders it — grep -rn "<DriverStatusPanel" across src/ returns nothing, and no @page route under Components/Pages/ hosts it. It was orphaned when v3 Batch 2 retired the routed /clusters/{id}/drivers flow.

So the driver-control channel has no operator surface on either transport — this is a pre-existing v3 regression that Phase 2 merely surfaced. Filed as a follow-up; not fixed here.

Residual risk for the channel: central-side dispatch is sabotage-verified by MeshRouterDispatchTests (all four AdminOperationsActor call sites go through one Dispatch helper); node-side EventStream receipt by DriverHostActor is proven live by this gate, since DispatchDeployment arrives through the identical PublishLocally path. What is unproven live is only the composition for those two message types.

Step 7 (alarm ack) was likewise not run: the rig has no scripted alarms configured (ScriptedAlarmHost: applying (enabled=0/0, depRefs=0)) and /alerts is empty, so there is nothing to acknowledge. Authoring a scripted alarm end-to-end was judged out of proportion to the residual risk above. ScriptedAlarmHostActor's AlarmCommand EventStream subscription therefore rests on its (sabotage-verified) unit test.

Finding E — the central comm actor is silent on the success path

HandleApplyAck logs only on failure. During this gate that made "did the ack reach the surviving central?" unanswerable from logs, which is exactly the question an operator asks during a failover. A Debug line on ack receipt would have made step 8 directly observable. Not added during the gate — adding code to make a gate step pass is the wrong order — but recorded as a follow-up.

---

What this gate did not cover

• Steps 6 and 7 — see findings D and E; no surface exists for driver-control, no alarms for ack.
• The alarm-commands node-side leg under a real transport, for the reason above.
• Split meshes. Everything here ran on the single mesh. The contact set was fleet-wide (6 points, one client), which is the correct and only shape until Phase 6 splits the meshes.
• A network partition (as opposed to process stops). auto-down's dual-active trade is untested here and remains Phase 7's.