docs(mesh): Phase 2 live-gate record + correct the buffer-size rationale

Flipped the docker-dev rig end-to-end to MeshTransport:Mode=ClusterClient
(OTOPCUA_MESH_MODE, no rebuild) and ran the gate. PASSED for the transport:
deploy seals on ClusterClient, a stopped node fails the deploy naming it, the
MaintenanceMode hatch shrinks the DB-sourced contact set live (6->5->6 on both
centrals independently), and the frame-size canary stayed silent.

Four findings, recorded in 2026-07-22-mesh-phase2-live-gate.md:

  A. A 10-30s post-restart delivery window exists, but the DPS control fails
     identically -- it is a cluster-convergence property, not a Phase 2
     regression. Running the control is the only reason this is a note and not
     a false attribution.

  B. buffer-size=0's stated rationale was OVERSTATED. A node applies a TimedOut
     deployment anyway -- live-observed at +44s -- via DriverHostActor.Bootstrap
     replaying the orphan Applying row on restart. Zero buffering removes the
     silent, timer-driven replay, not the boot-time DB one. Corrected in the
     plan decisions table, Configuration.md, and CLAUDE.md; the decision stands,
     the justification narrowed.

  C. Stopping BOTH centrals strands the non-seed nodes at the Akka membership
     layer (frozen mid-Exiting, not Unreachable, so auto-down does not help).
     Transport-independent; a Phase 7 downing/rejoin drill item.

  D. The AdminUI driver Restart/Reconnect buttons (DriverStatusPanel) are
     rendered by no page -- orphaned since v3 Batch 2 retired /clusters/{id}/
     drivers. So gate steps 6/7 could not run; the driver-control node-side leg
     is proven live only because DispatchDeployment shares its EventStream path.

Step 8's planned log-signal does not exist (HandleApplyAck is silent on
success); proven instead by both centrals independently logging per-instance
ClusterClient creation, which a singleton cannot.

Also adds a repo-root .dockerignore: docker-dev/Dockerfile does COPY . ., which
was pulling .claude/worktrees (19GB of agent checkouts with their own bin/obj)
into the build context and exhausting the Docker VM disk mid-COPY.

Rig restored to Mode=Dps, six members, no leftover maintenance flags.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md

@@ -15,7 +15,7 @@
 | Decision | Choice | Why |
 |---|---|---|
 | Cutover | **Config-flagged dark switch** (`MeshTransport:Mode` = `Dps` default \| `ClusterClient`) | Lets the rig gate A/B the same deploy and roll back with a config change, not a revert. Phase 1's live gate is the argument: the defect only appeared against real infrastructure. |
-| Buffering | **`buffer-size = 0`** (drop immediately) | Akka's default buffers 1000 and replays on reconnect. With Phase 1 semantics a deploy recorded `TimedOut` would still be applied minutes later when the node returns — the node runs a config the DB says failed. Fail-fast keeps the DB the single record of truth. |
+| Buffering | **`buffer-size = 0`** (drop immediately) | Akka's default buffers 1000 and replays on reconnect. With Phase 1 semantics a deploy recorded `TimedOut` would still be applied minutes later when the node returns — the node runs a config the DB says failed. Fail-fast keeps the DB the single record of truth. **CORRECTED by the live gate ([finding B](2026-07-22-mesh-phase2-live-gate.md)):** this justification was overstated. A node applies a `TimedOut` deployment *anyway*, via `DriverHostActor.Bootstrap()` replaying the orphan `Applying` row on restart (live-observed at +44 s). `buffer-size = 0` removes the *silent, timer-driven* replay, not the boot-time DB one. The decision stands; the rationale narrowed. |
 | `alarm-commands` central leg | **In scope** | Same shape as driver-control; leaves no half-migrated command plane. The topic stays alive for the node-local publisher. |
 
 ---

docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md.tasks.json

@@ -128,7 +128,8 @@
     {
       "id": 10,
       "subject": "Task 10: live gate on docker-dev (11 steps; step 4 = deploy-seal semantics, step 8 = per-node/contact-rotation discriminator [replaces the dropped Task 6 assertion \u2014 do not skip], step 9 = both-centrals drop behavior)",
-      "status": "pending",
+      "status": "completed",
+      "note": "PASSED for the transport (steps 1-5, 8, 11 green; flip end-to-end via OTOPCUA_MESH_MODE=ClusterClient). Record: 2026-07-22-mesh-phase2-live-gate.md. Steps 6/7 NOT RUN (no UI surface for driver-control \u2014 finding D, DriverStatusPanel is orphaned; no alarms configured). Step 8's stated log-signal does not exist (HandleApplyAck is silent on success \u2014 finding E); proven instead by both centrals independently logging per-instance ClusterClient creation + rebuild, which a singleton cannot. FOUR findings: A) 10-30s post-restart delivery window is PRE-EXISTING (DPS control failed identically); B) buffer-size=0 rationale OVERSTATED \u2014 a node applies a TimedOut deploy anyway via Bootstrap() orphan-row replay at +44s, corrected in 3 docs; C) stopping BOTH centrals strands non-seed nodes at the membership layer (Phase 7 drill, transport-independent); D) driver Restart/Reconnect UI orphaned since v3 Batch 2. Rig restored to Dps default.",
       "classification": "high-risk",
       "blockedBy": [
         8,

docs/plans/2026-07-22-mesh-phase2-live-gate.md

@@ -0,0 +1,228 @@
+# Per-cluster mesh Phase 2 — live gate record
+
+**Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase2`
+**Plan:** [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md)
+
+**Result: PASSED for the transport**, with two gate steps not run (no surface exists to drive them),
+one step whose stated pass-signal does not exist in the code, and **four findings** — one of which
+corrects a claim this phase's own documentation makes.
+
+The rig was flipped end-to-end with `OTOPCUA_MESH_MODE=ClusterClient` (no rebuild, no compose edit —
+the env indirection added in Task 9 is what made the A/B and the DPS control cheap enough to run).
+
+---
+
+## Result table
+
+| # | Step | Result |
+|---|---|---|
+| 1 | Deploy on `Mode=Dps` | **PASS** — `48d0c7c5` Sealed, 6/6 Applied |
+| 2 | Flip all six to `ClusterClient`, redeploy | **PASS** — `5db8f6ad` Sealed, 6/6 Applied |
+| 3 | Confirm the transport actually changed | **PASS** — positive log evidence, see below |
+| 4 | Stop a node, deploy | **PASS** — `59b81248` TimedOut naming `site-b-2:4053`, 5/6 |
+| 5 | `MaintenanceMode = 1`, redeploy | **PASS** — `c6cb9812` Sealed, 5 rows, contacts 6→5 |
+| 6 | AdminUI Restart + Reconnect | **NOT RUN** — no UI surface exists (finding D) |
+| 7 | AdminUI alarm ack | **NOT RUN** — no alarms configured on the rig |
+| 8 | Per-node (not singleton) discriminator | **PASS** — via a better discriminator than the plan's |
+| 9 | Both centrals down, restart a driver node | **PARTIAL** — node survives; the stated ack-drop cannot occur |
+| 10 | Restart central, redeploy | **PASS only after operator action** (finding C) |
+| 11 | Frame-size canary | **PASS** — 0 warnings across all six nodes |
+
+## Step 3 — the transport really changed
+
+Central, both nodes, at boot under the flag:
+
+```
+central-1 [22:18:16 INF] Mesh ClusterClient created with 6 contact point(s)
+central-2 [22:18:17 INF] Mesh ClusterClient created with 6 contact point(s)
+```
+
+Node side, with `Serilog__MinimumLevel__Default=Debug` overlaid on `site-a-1` (the rig pins the
+default above Debug, so this line is invisible on a stock node):
+
+```
+site-a-1  [22:10:34 DBG] Mesh: received DispatchDeployment from central; publishing node-locally
+```
+
+Under `Mode=Dps` the same node logs nothing of the sort and central logs
+`Mesh transport is Dps: commands are published on DistributedPubSub and no ClusterClient is created`.
+
+## Step 5 — the DB-sourced contact set is live
+
+Setting `MaintenanceMode = 1` on `site-b-2` shrank both centrals' contact sets on the next refresh,
+and clearing it restored them — independently, on each central's own 60 s timer:
+
+```
+central-1 [22:22:16 INF] Mesh contact set changed; replacing the ClusterClient
+central-1 [22:22:16 INF] Mesh ClusterClient created with 5 contact point(s)
+central-1 [22:23:16 INF] Mesh ClusterClient created with 6 contact point(s)
+central-2 [22:22:16 INF] … identical, same timestamps
+```
+
+No `InvalidActorNameException` on any rebuild — the generation-suffixed client name (Task 2, ported
+from the sister project's shipped bug) does its job.
+
+## Step 8 — the per-node discriminator, replaced with a stronger one
+
+**The plan's pass-signal does not exist.** It asked for the surviving central's comm actor to *log
+receipt* of a node ack inside the downing window. `CentralCommunicationActor.HandleApplyAck` logs
+**nothing** on the success path — it only warns when the coordinator is unresolvable. There is no
+line to observe. (Recorded as finding E.)
+
+The evidence above is a **cleaner** discriminator and needs no timing race. `Mesh ClusterClient
+created` is emitted from `PreStart`, so it fires **once per live actor instance**. Both centrals
+logged it, and both logged their own independent contact-set rebuild at step 5. A
+`ClusterSingletonManager` runs the actor on the oldest node only — the other node hosts a manager
+that never starts the actor and would log nothing. **Two nodes logging it proves two live instances,
+i.e. per-node registration.** The node side is proven the same way: `SendToAll` collected 6/6 acks,
+which requires six separately registered node comm actors.
+
+The failover half was run anyway, and passed: `docker stop central-1` (the oldest, and the singleton
+host) → the surviving central accepted a deployment **4 seconds later** and collected acks from all
+five live nodes (`e3161c25`; it TimedOut only because `central-1` is itself a configured node that
+was down — correct Phase 1 semantics). Graceful stop hands the singleton over immediately, so no
+downing wait was involved.
+
+## Step 9 — the stated failure mode cannot occur
+
+The plan expected a restarting driver node to emit a boot-time ack that gets dropped with a Warning.
+**Driver nodes do not ack on boot.** `DriverHostActor.SendAck` is called only from the apply paths —
+a node with no dispatch to answer has nothing to send, so there is no drop to observe.
+
+What was verified: with **both** centrals stopped, `site-a-1` restarted, came up clean, logged its
+association failures against both central addresses, retried, and stayed up — no crash loop. The
+node-side ClusterClient's forever-retry (`reconnect-timeout = off`) behaves as configured.
+
+## Step 11 — the canary stayed quiet
+
+`grep -ci "frame size|frame-size|maximum-frame"` = **0** on all six nodes, as designed: the deploy
+notify carries only `DeploymentId` + `RevisionHash` + `CorrelationId`.
+
+---
+
+## Finding A — a 10–30 s post-restart delivery window, **pre-existing, not Phase 2**
+
+A deploy issued shortly after a driver node restarts does not reach that node, and the deployment
+fails at the 2-minute deadline naming it. Measured against `site-a-1`:
+
+| Delay after the node logs readiness | Result |
+|---|---|
+| ~4 s | **missed** (`f738e1de` TimedOut, node row `Applying`) |
+| 10 s | **missed** (`0c7a6296` TimedOut) |
+| 30 s | applied, sealed |
+
+**The DPS control fails identically.** Flipping the rig back to `Mode=Dps` and repeating the 10 s
+case produced `e85f53c5` → TimedOut, `site-a-1` unacked. So the window is a property of cluster
+membership convergence, not of the transport this phase introduces. Recorded rather than fixed.
+
+> Running that control is the only reason this is a note and not a Phase 2 regression. The
+> temptation to skip it — "obviously the new transport broke it" — would have produced a false
+> attribution and a wasted investigation.
+
+## Finding B — **`buffer-size = 0` does not deliver what this phase's docs claim it does**
+
+The stated rationale, in the plan, `CLAUDE.md`, and `docs/Configuration.md`, is that buffering
+*"would replay a `DispatchDeployment` on reconnect and apply a deployment the coordinator already
+sealed as `TimedOut`, leaving the node running a configuration the database records as failed."*
+
+**That outcome happens anyway, by a different route.** Deployment `0c7a6296` timed out at 22:14:30
+with `site-a-1` unacked. The node's row:
+
+```
+NodeId          Status  StartedAtUtc              AppliedAtUtc
+site-a-1:4053   1       2026-07-22 22:12:30.145   2026-07-22 22:15:14.823
+```
+
+It applied **44 s after the deployment was sealed TimedOut**, on its next restart. The mechanism is
+`DriverHostActor.Bootstrap()`, which reads the node's most recent `NodeDeploymentState`, finds the
+orphan `Applying` row the coordinator pre-created at dispatch, and replays it:
+
+```csharp
+case NodeDeploymentStatus.Applying:
+    _log.Warning("DriverHost {Node}: found orphan Applying row for deployment {Id}; replaying", …);
+    ApplyAndAck(new DeploymentId(latest.DeploymentId), revision.Value, CorrelationId.NewId());
+```
+
+So `buffer-size = 0` removes the **transport-level, timer-driven** replay — a real difference: it is
+silent, unbounded in time, and can fire without any node event. The **boot-time DB replay remains**,
+and it is the better-behaved of the two (deliberate, Warning-logged, and bounded to a restart). The
+decision stands; the *justification* was overstated and the docs are corrected in the same change as
+this record.
+
+## Finding C — stopping **both** centrals strands the non-seed nodes permanently
+
+After `docker stop` of both centrals followed by `docker start`, the fleet did **not** recover.
+Deployments reached 2/6 then 3/6 nodes. The cause is not the transport — the Akka cluster itself had
+split:
+
+```
+Cluster hosts:  MEMBERS 3   UP 3   UNREACHABLE 0
+  central-1:4053  UP   admin,driver   leader for admin+driver
+  central-2:4053  UP   admin,driver
+  site-a-1:4053   UP   driver
+```
+
+The three site nodes that never restarted were **not members at all**, and their own gossip showed
+the stale inverse view — `central-2` as `Exiting`, `site-a-1` as `Leaving` — frozen since 22:29:
+
+```
+Gossip(members = [central-2 status = Exiting, site-a-1 status = Leaving,
+                  site-a-2 Up, site-b-1 Up, site-b-2 Up])
+```
+
+A graceful stop puts the centrals into `Leaving`/`Exiting`; the processes died before the removal
+handshake completed, leaving the survivors holding members that will never confirm. `auto-down`
+does not help — these members are not *Unreachable*, they are stuck mid-`Exiting`. The restarted
+centrals formed a fresh cluster (with `site-a-1`, which had also restarted and re-joined through its
+seed), and the three stranded nodes had no path back: they believe they are already in a cluster, so
+they never re-run the join process.
+
+**Recovery required restarting the stranded nodes.** After that, the next deploy sealed 6/6
+(`e20836ac`).
+
+This is transport-independent — membership is below both DPS and ClusterClient, and the set of nodes
+that acked matched the cluster view exactly. It is a **downing/rejoin gap of the current single-mesh
+topology**, in the same family as the registered outage gap the design doc already records, and it
+belongs to the Phase 7 drill work. Under the target per-pair topology (every node a seed of its own
+pair) the shape changes, and this drill must be re-run there rather than assumed fixed.
+
+## Finding D — the AdminUI's driver Restart/Reconnect buttons have no page
+
+Step 6 could not be run, and not for a rig reason.
+`Components/Shared/Drivers/DriverStatusPanel.razor` is the **only** component in the codebase
+carrying the `DriverOperator`-gated Reconnect/Restart buttons, and **nothing renders it** — `grep -rn
+"<DriverStatusPanel"` across `src/` returns nothing, and no `@page` route under `Components/Pages/`
+hosts it. It was orphaned when v3 Batch 2 retired the routed `/clusters/{id}/drivers` flow.
+
+So the `driver-control` channel has no operator surface **on either transport** — this is a
+pre-existing v3 regression that Phase 2 merely surfaced. Filed as a follow-up; not fixed here.
+
+**Residual risk for the channel:** central-side dispatch is sabotage-verified by
+`MeshRouterDispatchTests` (all four `AdminOperationsActor` call sites go through one `Dispatch`
+helper); node-side `EventStream` receipt by `DriverHostActor` is proven **live** by this gate, since
+`DispatchDeployment` arrives through the identical `PublishLocally` path. What is unproven live is
+only the composition for those two message types.
+
+Step 7 (alarm ack) was likewise not run: the rig has no scripted alarms configured
+(`ScriptedAlarmHost: applying (enabled=0/0, depRefs=0)`) and `/alerts` is empty, so there is nothing
+to acknowledge. Authoring a scripted alarm end-to-end was judged out of proportion to the residual
+risk above. `ScriptedAlarmHostActor`'s `AlarmCommand` EventStream subscription therefore rests on its
+(sabotage-verified) unit test.
+
+## Finding E — the central comm actor is silent on the success path
+
+`HandleApplyAck` logs only on failure. During this gate that made "did the ack reach the surviving
+central?" unanswerable from logs, which is exactly the question an operator asks during a failover.
+A Debug line on ack receipt would have made step 8 directly observable. Not added during the gate —
+adding code to make a gate step pass is the wrong order — but recorded as a follow-up.
+
+---
+
+## What this gate did not cover
+
+- **Steps 6 and 7** — see findings D and E; no surface exists for driver-control, no alarms for ack.
+- **The `alarm-commands` node-side leg** under a real transport, for the reason above.
+- **Split meshes.** Everything here ran on the single mesh. The contact set was fleet-wide (6 points,
+  one client), which is the correct and only shape until Phase 6 splits the meshes.
+- **A network partition** (as opposed to process stops). `auto-down`'s dual-active trade is untested
+  here and remains Phase 7's.

docs/plans/2026-07-22-per-cluster-mesh-program.md

@@ -99,10 +99,14 @@ this removes the coordinator's one genuinely mesh-bound dependency).
 set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
 missing; a node up but row absent → its ack is not expected).
 
-### Phase 2 — Comm actors + ClusterClient transport — **CODE COMPLETE 2026-07-22, live gate pending**
+### Phase 2 — Comm actors + ClusterClient transport — **DONE 2026-07-22 (live gate PASSED)**
 Shipped per [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md)
 as a **dark switch**: `MeshTransport:Mode` defaults to `Dps`, so nothing changes on any deployment
-until the flag flips. Both comm actors are spawned and receptionist-registered in **both** modes, so
+until the flag flips. The live gate ([`2026-07-22-mesh-phase2-live-gate.md`](2026-07-22-mesh-phase2-live-gate.md))
+flipped the whole rig to `ClusterClient` and passed for the transport, with four findings — most
+notably that `buffer-size = 0`'s stated rationale was overstated (a node applies a `TimedOut`
+deployment anyway via boot-time orphan-row replay) and that stopping both centrals strands non-seed
+nodes at the membership layer (a Phase 7 drill item, transport-independent). Both comm actors are spawned and receptionist-registered in **both** modes, so
 the flip is a config change rather than a redeploy. Three scope corrections landed against the
 design doc (that plan's header carries them in full):
 
@@ -224,7 +228,7 @@ resource sizing on the site VMs should be checked once both products run the ful
 | 0b oldest-Up election | DONE 2026-07-21 |
 | Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
 | 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
-| 2 ClusterClient transport | **CODE COMPLETE 2026-07-22, live gate pending** — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
+| 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
 | 3 fetch-and-cache | not started |
 | 4 cut driver ConfigDb | not started |
 | 5 gRPC telemetry | not started |