# Per-cluster mesh Phase 2 — live gate record **Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase2` **Plan:** [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md) **Result: PASSED for the transport**, with two gate steps not run (no surface exists to drive them), one step whose stated pass-signal does not exist in the code, and **four findings** — one of which corrects a claim this phase's own documentation makes. The rig was flipped end-to-end with `OTOPCUA_MESH_MODE=ClusterClient` (no rebuild, no compose edit — the env indirection added in Task 9 is what made the A/B and the DPS control cheap enough to run). --- ## Result table | # | Step | Result | |---|---|---| | 1 | Deploy on `Mode=Dps` | **PASS** — `48d0c7c5` Sealed, 6/6 Applied | | 2 | Flip all six to `ClusterClient`, redeploy | **PASS** — `5db8f6ad` Sealed, 6/6 Applied | | 3 | Confirm the transport actually changed | **PASS** — positive log evidence, see below | | 4 | Stop a node, deploy | **PASS** — `59b81248` TimedOut naming `site-b-2:4053`, 5/6 | | 5 | `MaintenanceMode = 1`, redeploy | **PASS** — `c6cb9812` Sealed, 5 rows, contacts 6→5 | | 6 | AdminUI Restart + Reconnect | **NOT RUN** — no UI surface exists (finding D) | | 7 | AdminUI alarm ack | **NOT RUN** — no alarms configured on the rig | | 8 | Per-node (not singleton) discriminator | **PASS** — via a better discriminator than the plan's | | 9 | Both centrals down, restart a driver node | **PARTIAL** — node survives; the stated ack-drop cannot occur | | 10 | Restart central, redeploy | **PASS only after operator action** (finding C) | | 11 | Frame-size canary | **PASS** — 0 warnings across all six nodes | ## Step 3 — the transport really changed Central, both nodes, at boot under the flag: ``` central-1 [22:18:16 INF] Mesh ClusterClient created with 6 contact point(s) central-2 [22:18:17 INF] Mesh ClusterClient created with 6 contact point(s) ``` Node side, with `Serilog__MinimumLevel__Default=Debug` overlaid on `site-a-1` (the rig pins the default above Debug, so this line is invisible on a stock node): ``` site-a-1 [22:10:34 DBG] Mesh: received DispatchDeployment from central; publishing node-locally ``` Under `Mode=Dps` the same node logs nothing of the sort and central logs `Mesh transport is Dps: commands are published on DistributedPubSub and no ClusterClient is created`. ## Step 5 — the DB-sourced contact set is live Setting `MaintenanceMode = 1` on `site-b-2` shrank both centrals' contact sets on the next refresh, and clearing it restored them — independently, on each central's own 60 s timer: ``` central-1 [22:22:16 INF] Mesh contact set changed; replacing the ClusterClient central-1 [22:22:16 INF] Mesh ClusterClient created with 5 contact point(s) central-1 [22:23:16 INF] Mesh ClusterClient created with 6 contact point(s) central-2 [22:22:16 INF] … identical, same timestamps ``` No `InvalidActorNameException` on any rebuild — the generation-suffixed client name (Task 2, ported from the sister project's shipped bug) does its job. ## Step 8 — the per-node discriminator, replaced with a stronger one **The plan's pass-signal does not exist.** It asked for the surviving central's comm actor to *log receipt* of a node ack inside the downing window. `CentralCommunicationActor.HandleApplyAck` logs **nothing** on the success path — it only warns when the coordinator is unresolvable. There is no line to observe. (Recorded as finding E.) The evidence above is a **cleaner** discriminator and needs no timing race. `Mesh ClusterClient created` is emitted from `PreStart`, so it fires **once per live actor instance**. Both centrals logged it, and both logged their own independent contact-set rebuild at step 5. A `ClusterSingletonManager` runs the actor on the oldest node only — the other node hosts a manager that never starts the actor and would log nothing. **Two nodes logging it proves two live instances, i.e. per-node registration.** The node side is proven the same way: `SendToAll` collected 6/6 acks, which requires six separately registered node comm actors. The failover half was run anyway, and passed: `docker stop central-1` (the oldest, and the singleton host) → the surviving central accepted a deployment **4 seconds later** and collected acks from all five live nodes (`e3161c25`; it TimedOut only because `central-1` is itself a configured node that was down — correct Phase 1 semantics). Graceful stop hands the singleton over immediately, so no downing wait was involved. ## Step 9 — the stated failure mode cannot occur The plan expected a restarting driver node to emit a boot-time ack that gets dropped with a Warning. **Driver nodes do not ack on boot.** `DriverHostActor.SendAck` is called only from the apply paths — a node with no dispatch to answer has nothing to send, so there is no drop to observe. What was verified: with **both** centrals stopped, `site-a-1` restarted, came up clean, logged its association failures against both central addresses, retried, and stayed up — no crash loop. The node-side ClusterClient's forever-retry (`reconnect-timeout = off`) behaves as configured. ## Step 11 — the canary stayed quiet `grep -ci "frame size|frame-size|maximum-frame"` = **0** on all six nodes, as designed: the deploy notify carries only `DeploymentId` + `RevisionHash` + `CorrelationId`. --- ## Finding A — a 10–30 s post-restart delivery window, **pre-existing, not Phase 2** A deploy issued shortly after a driver node restarts does not reach that node, and the deployment fails at the 2-minute deadline naming it. Measured against `site-a-1`: | Delay after the node logs readiness | Result | |---|---| | ~4 s | **missed** (`f738e1de` TimedOut, node row `Applying`) | | 10 s | **missed** (`0c7a6296` TimedOut) | | 30 s | applied, sealed | **The DPS control fails identically.** Flipping the rig back to `Mode=Dps` and repeating the 10 s case produced `e85f53c5` → TimedOut, `site-a-1` unacked. So the window is a property of cluster membership convergence, not of the transport this phase introduces. Recorded rather than fixed. > Running that control is the only reason this is a note and not a Phase 2 regression. The > temptation to skip it — "obviously the new transport broke it" — would have produced a false > attribution and a wasted investigation. ## Finding B — **`buffer-size = 0` does not deliver what this phase's docs claim it does** The stated rationale, in the plan, `CLAUDE.md`, and `docs/Configuration.md`, is that buffering *"would replay a `DispatchDeployment` on reconnect and apply a deployment the coordinator already sealed as `TimedOut`, leaving the node running a configuration the database records as failed."* **That outcome happens anyway, by a different route.** Deployment `0c7a6296` timed out at 22:14:30 with `site-a-1` unacked. The node's row: ``` NodeId Status StartedAtUtc AppliedAtUtc site-a-1:4053 1 2026-07-22 22:12:30.145 2026-07-22 22:15:14.823 ``` It applied **44 s after the deployment was sealed TimedOut**, on its next restart. The mechanism is `DriverHostActor.Bootstrap()`, which reads the node's most recent `NodeDeploymentState`, finds the orphan `Applying` row the coordinator pre-created at dispatch, and replays it: ```csharp case NodeDeploymentStatus.Applying: _log.Warning("DriverHost {Node}: found orphan Applying row for deployment {Id}; replaying", …); ApplyAndAck(new DeploymentId(latest.DeploymentId), revision.Value, CorrelationId.NewId()); ``` So `buffer-size = 0` removes the **transport-level, timer-driven** replay — a real difference: it is silent, unbounded in time, and can fire without any node event. The **boot-time DB replay remains**, and it is the better-behaved of the two (deliberate, Warning-logged, and bounded to a restart). The decision stands; the *justification* was overstated and the docs are corrected in the same change as this record. ## Finding C — stopping **both** centrals strands the non-seed nodes permanently After `docker stop` of both centrals followed by `docker start`, the fleet did **not** recover. Deployments reached 2/6 then 3/6 nodes. The cause is not the transport — the Akka cluster itself had split: ``` Cluster hosts: MEMBERS 3 UP 3 UNREACHABLE 0 central-1:4053 UP admin,driver leader for admin+driver central-2:4053 UP admin,driver site-a-1:4053 UP driver ``` The three site nodes that never restarted were **not members at all**, and their own gossip showed the stale inverse view — `central-2` as `Exiting`, `site-a-1` as `Leaving` — frozen since 22:29: ``` Gossip(members = [central-2 status = Exiting, site-a-1 status = Leaving, site-a-2 Up, site-b-1 Up, site-b-2 Up]) ``` A graceful stop puts the centrals into `Leaving`/`Exiting`; the processes died before the removal handshake completed, leaving the survivors holding members that will never confirm. `auto-down` does not help — these members are not *Unreachable*, they are stuck mid-`Exiting`. The restarted centrals formed a fresh cluster (with `site-a-1`, which had also restarted and re-joined through its seed), and the three stranded nodes had no path back: they believe they are already in a cluster, so they never re-run the join process. **Recovery required restarting the stranded nodes.** After that, the next deploy sealed 6/6 (`e20836ac`). This is transport-independent — membership is below both DPS and ClusterClient, and the set of nodes that acked matched the cluster view exactly. It is a **downing/rejoin gap of the current single-mesh topology**, in the same family as the registered outage gap the design doc already records, and it belongs to the Phase 7 drill work. Under the target per-pair topology (every node a seed of its own pair) the shape changes, and this drill must be re-run there rather than assumed fixed. ## Finding D — the AdminUI's driver Restart/Reconnect buttons have no page Step 6 could not be run, and not for a rig reason. `Components/Shared/Drivers/DriverStatusPanel.razor` is the **only** component in the codebase carrying the `DriverOperator`-gated Reconnect/Restart buttons, and **nothing renders it** — `grep -rn "