docs(mesh): Phase 2 live-gate record + correct the buffer-size rationale

Flipped the docker-dev rig end-to-end to MeshTransport:Mode=ClusterClient
(OTOPCUA_MESH_MODE, no rebuild) and ran the gate. PASSED for the transport:
deploy seals on ClusterClient, a stopped node fails the deploy naming it, the
MaintenanceMode hatch shrinks the DB-sourced contact set live (6->5->6 on both
centrals independently), and the frame-size canary stayed silent.

Four findings, recorded in 2026-07-22-mesh-phase2-live-gate.md:

  A. A 10-30s post-restart delivery window exists, but the DPS control fails
     identically -- it is a cluster-convergence property, not a Phase 2
     regression. Running the control is the only reason this is a note and not
     a false attribution.

  B. buffer-size=0's stated rationale was OVERSTATED. A node applies a TimedOut
     deployment anyway -- live-observed at +44s -- via DriverHostActor.Bootstrap
     replaying the orphan Applying row on restart. Zero buffering removes the
     silent, timer-driven replay, not the boot-time DB one. Corrected in the
     plan decisions table, Configuration.md, and CLAUDE.md; the decision stands,
     the justification narrowed.

  C. Stopping BOTH centrals strands the non-seed nodes at the Akka membership
     layer (frozen mid-Exiting, not Unreachable, so auto-down does not help).
     Transport-independent; a Phase 7 downing/rejoin drill item.

  D. The AdminUI driver Restart/Reconnect buttons (DriverStatusPanel) are
     rendered by no page -- orphaned since v3 Batch 2 retired /clusters/{id}/
     drivers. So gate steps 6/7 could not run; the driver-control node-side leg
     is proven live only because DispatchDeployment shares its EventStream path.

Step 8's planned log-signal does not exist (HandleApplyAck is silent on
success); proven instead by both centrals independently logging per-instance
ClusterClient creation, which a singleton cannot.

Also adds a repo-root .dockerignore: docker-dev/Dockerfile does COPY . ., which
was pulling .claude/worktrees (19GB of agent checkouts with their own bin/obj)
into the build context and exhausting the Docker VM disk mid-COPY.

Rig restored to Mode=Dps, six members, no leftover maintenance flags.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-22 18:40:42 -04:00
parent 986b640eea
commit 32055c2238
7 changed files with 274 additions and 8 deletions
@@ -99,10 +99,14 @@ this removes the coordinator's one genuinely mesh-bound dependency).
set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
missing; a node up but row absent → its ack is not expected).
### Phase 2 — Comm actors + ClusterClient transport — **CODE COMPLETE 2026-07-22, live gate pending**
### Phase 2 — Comm actors + ClusterClient transport — **DONE 2026-07-22 (live gate PASSED)**
Shipped per [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md)
as a **dark switch**: `MeshTransport:Mode` defaults to `Dps`, so nothing changes on any deployment
until the flag flips. Both comm actors are spawned and receptionist-registered in **both** modes, so
until the flag flips. The live gate ([`2026-07-22-mesh-phase2-live-gate.md`](2026-07-22-mesh-phase2-live-gate.md))
flipped the whole rig to `ClusterClient` and passed for the transport, with four findings — most
notably that `buffer-size = 0`'s stated rationale was overstated (a node applies a `TimedOut`
deployment anyway via boot-time orphan-row replay) and that stopping both centrals strands non-seed
nodes at the membership layer (a Phase 7 drill item, transport-independent). Both comm actors are spawned and receptionist-registered in **both** modes, so
the flip is a config change rather than a redeploy. Three scope corrections landed against the
design doc (that plan's header carries them in full):
@@ -224,7 +228,7 @@ resource sizing on the site VMs should be checked once both products run the ful
| 0b oldest-Up election | DONE 2026-07-21 |
| Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
| 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
| 2 ClusterClient transport | **CODE COMPLETE 2026-07-22, live gate pending** — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
| 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
| 3 fetch-and-cache | not started |
| 4 cut driver ConfigDb | not started |
| 5 gRPC telemetry | not started |