From 32055c2238fd24294a9c0a2df8eb14a517f3f171 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 22 Jul 2026 18:40:42 -0400 Subject: [PATCH] docs(mesh): Phase 2 live-gate record + correct the buffer-size rationale Flipped the docker-dev rig end-to-end to MeshTransport:Mode=ClusterClient (OTOPCUA_MESH_MODE, no rebuild) and ran the gate. PASSED for the transport: deploy seals on ClusterClient, a stopped node fails the deploy naming it, the MaintenanceMode hatch shrinks the DB-sourced contact set live (6->5->6 on both centrals independently), and the frame-size canary stayed silent. Four findings, recorded in 2026-07-22-mesh-phase2-live-gate.md: A. A 10-30s post-restart delivery window exists, but the DPS control fails identically -- it is a cluster-convergence property, not a Phase 2 regression. Running the control is the only reason this is a note and not a false attribution. B. buffer-size=0's stated rationale was OVERSTATED. A node applies a TimedOut deployment anyway -- live-observed at +44s -- via DriverHostActor.Bootstrap replaying the orphan Applying row on restart. Zero buffering removes the silent, timer-driven replay, not the boot-time DB one. Corrected in the plan decisions table, Configuration.md, and CLAUDE.md; the decision stands, the justification narrowed. C. Stopping BOTH centrals strands the non-seed nodes at the Akka membership layer (frozen mid-Exiting, not Unreachable, so auto-down does not help). Transport-independent; a Phase 7 downing/rejoin drill item. D. The AdminUI driver Restart/Reconnect buttons (DriverStatusPanel) are rendered by no page -- orphaned since v3 Batch 2 retired /clusters/{id}/ drivers. So gate steps 6/7 could not run; the driver-control node-side leg is proven live only because DispatchDeployment shares its EventStream path. Step 8's planned log-signal does not exist (HandleApplyAck is silent on success); proven instead by both centrals independently logging per-instance ClusterClient creation, which a singleton cannot. Also adds a repo-root .dockerignore: docker-dev/Dockerfile does COPY . ., which was pulling .claude/worktrees (19GB of agent checkouts with their own bin/obj) into the build context and exhausting the Docker VM disk mid-COPY. Rig restored to Mode=Dps, six members, no leftover maintenance flags. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- .dockerignore | 29 +++ CLAUDE.md | 6 +- docs/Configuration.md | 4 +- ...-22-mesh-phase2-clusterclient-transport.md | 2 +- ...ase2-clusterclient-transport.md.tasks.json | 3 +- .../plans/2026-07-22-mesh-phase2-live-gate.md | 228 ++++++++++++++++++ .../2026-07-22-per-cluster-mesh-program.md | 10 +- 7 files changed, 274 insertions(+), 8 deletions(-) create mode 100644 .dockerignore create mode 100644 docs/plans/2026-07-22-mesh-phase2-live-gate.md diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 00000000..bad7715e --- /dev/null +++ b/.dockerignore @@ -0,0 +1,29 @@ +# Build context exclusions for docker-dev/Dockerfile, whose `COPY . .` takes the whole repo. +# +# Without this file the context also carries every project's bin/obj AND .claude/worktrees — agent +# worktrees that are themselves full checkouts WITH their own build outputs. That reached ~19 GB and +# exhausted the Docker VM's disk mid-COPY, failing the rig build with "no space left on device" +# rather than anything a build log would attribute to the repo. +# +# Nothing excluded here is needed: the build restores and publishes inside the image, and the +# migrator stage runs `dotnet ef` against the same restored source. + +# Agent worktrees — full checkouts, build outputs and all. +.claude/ + +# Local build outputs; the image builds its own. +**/bin/ +**/obj/ +artifacts/ + +# VCS + editor state. +.git/ +.gitignore +.vs/ +.vscode/ +.idea/ + +# Docker's own state, and anything the image would never read. +**/.DS_Store +**/*.user +docker-dev/**/*.log diff --git a/CLAUDE.md b/CLAUDE.md index 50d8c4b0..160fbb52 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -258,8 +258,10 @@ change, not a redeploy. Things worth knowing before touching it: - **Comm actors are per-node, NOT singletons.** A client rotates across contact points and must find a live comm actor at whichever node answers; `akka.cluster.client.receptionist.role` is empty for the same reason. - **`buffer-size = 0` is a behaviour decision, not a tuning knob.** Akka's default buffers 1000 and replays - on reconnect, which would apply a `DispatchDeployment` for a deployment the coordinator already sealed - `TimedOut` — the node would run a config the DB records as failed. Do not raise it to quiet a flaky test. + on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not raise + it to quiet a flaky test. **But do not over-claim what it buys:** the Phase 2 live gate observed a node + applying a deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()` + replaying the orphan `Applying` row on restart. Zero buffering removes the silent replay, not every replay. Contact points are node **addresses only** (`akka.tcp://@:`); `/system/receptionist` is appended at construction and `MeshTransportOptionsValidator` refuses to start a node whose contact carries an diff --git a/docs/Configuration.md b/docs/Configuration.md index a49bfb80..b92ba78b 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -127,7 +127,9 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only **Central's contact set comes from the enabled, non-maintenance `ClusterNode` rows** — the same source Phase 1 gave the deploy ack set. A node in `MaintenanceMode` is not dialled, but note that **the contact set does not scope delivery**: a receptionist serves its whole cluster, so `SendToAll` still reaches every registered node exactly as today's DPS broadcast does. The filter is about which receptionists are worth dialling, not about who receives. -**Buffering is off by design** (`akka.cluster.client.buffer-size = 0`, set in the shipped `akka.conf`). If no receptionist is known, the command is dropped with a Warning rather than queued. Buffering would replay a `DispatchDeployment` on reconnect and apply a deployment the coordinator had already sealed as `TimedOut`, leaving the node running a configuration the database records as failed. Do not "fix" this back to Akka's default of 1000 to quiet a flaky test. +**Buffering is off by design** (`akka.cluster.client.buffer-size = 0`, set in the shipped `akka.conf`). If no receptionist is known, the command is dropped rather than queued. Akka's default of 1000 would replay a `DispatchDeployment` on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not "fix" it back to quiet a flaky test. + +> **What this does *not* buy you.** It does not guarantee a node never applies a deployment the DB records as `TimedOut`. A node that missed a dispatch keeps the orphan `Applying` row the coordinator pre-created, and `DriverHostActor.Bootstrap()` replays that row on the node's next restart — live-observed applying a deployment 44 s after it was sealed `TimedOut` (see the [Phase 2 live gate](plans/2026-07-22-mesh-phase2-live-gate.md), finding B). The boot replay is the better-behaved of the two — deliberate, Warning-logged, and bounded to a restart — but it is still a replay. `buffer-size = 0` removes the *silent, timer-driven* one only. On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`. diff --git a/docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md b/docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md index 10d9ba58..2c26ae5f 100644 --- a/docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md +++ b/docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md @@ -15,7 +15,7 @@ | Decision | Choice | Why | |---|---|---| | Cutover | **Config-flagged dark switch** (`MeshTransport:Mode` = `Dps` default \| `ClusterClient`) | Lets the rig gate A/B the same deploy and roll back with a config change, not a revert. Phase 1's live gate is the argument: the defect only appeared against real infrastructure. | -| Buffering | **`buffer-size = 0`** (drop immediately) | Akka's default buffers 1000 and replays on reconnect. With Phase 1 semantics a deploy recorded `TimedOut` would still be applied minutes later when the node returns — the node runs a config the DB says failed. Fail-fast keeps the DB the single record of truth. | +| Buffering | **`buffer-size = 0`** (drop immediately) | Akka's default buffers 1000 and replays on reconnect. With Phase 1 semantics a deploy recorded `TimedOut` would still be applied minutes later when the node returns — the node runs a config the DB says failed. Fail-fast keeps the DB the single record of truth. **CORRECTED by the live gate ([finding B](2026-07-22-mesh-phase2-live-gate.md)):** this justification was overstated. A node applies a `TimedOut` deployment *anyway*, via `DriverHostActor.Bootstrap()` replaying the orphan `Applying` row on restart (live-observed at +44 s). `buffer-size = 0` removes the *silent, timer-driven* replay, not the boot-time DB one. The decision stands; the rationale narrowed. | | `alarm-commands` central leg | **In scope** | Same shape as driver-control; leaves no half-migrated command plane. The topic stays alive for the node-local publisher. | --- diff --git a/docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md.tasks.json b/docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md.tasks.json index 3b1815d7..5d5c8a9a 100644 --- a/docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md.tasks.json +++ b/docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md.tasks.json @@ -128,7 +128,8 @@ { "id": 10, "subject": "Task 10: live gate on docker-dev (11 steps; step 4 = deploy-seal semantics, step 8 = per-node/contact-rotation discriminator [replaces the dropped Task 6 assertion \u2014 do not skip], step 9 = both-centrals drop behavior)", - "status": "pending", + "status": "completed", + "note": "PASSED for the transport (steps 1-5, 8, 11 green; flip end-to-end via OTOPCUA_MESH_MODE=ClusterClient). Record: 2026-07-22-mesh-phase2-live-gate.md. Steps 6/7 NOT RUN (no UI surface for driver-control \u2014 finding D, DriverStatusPanel is orphaned; no alarms configured). Step 8's stated log-signal does not exist (HandleApplyAck is silent on success \u2014 finding E); proven instead by both centrals independently logging per-instance ClusterClient creation + rebuild, which a singleton cannot. FOUR findings: A) 10-30s post-restart delivery window is PRE-EXISTING (DPS control failed identically); B) buffer-size=0 rationale OVERSTATED \u2014 a node applies a TimedOut deploy anyway via Bootstrap() orphan-row replay at +44s, corrected in 3 docs; C) stopping BOTH centrals strands non-seed nodes at the membership layer (Phase 7 drill, transport-independent); D) driver Restart/Reconnect UI orphaned since v3 Batch 2. Rig restored to Dps default.", "classification": "high-risk", "blockedBy": [ 8, diff --git a/docs/plans/2026-07-22-mesh-phase2-live-gate.md b/docs/plans/2026-07-22-mesh-phase2-live-gate.md new file mode 100644 index 00000000..b32de27f --- /dev/null +++ b/docs/plans/2026-07-22-mesh-phase2-live-gate.md @@ -0,0 +1,228 @@ +# Per-cluster mesh Phase 2 — live gate record + +**Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase2` +**Plan:** [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md) + +**Result: PASSED for the transport**, with two gate steps not run (no surface exists to drive them), +one step whose stated pass-signal does not exist in the code, and **four findings** — one of which +corrects a claim this phase's own documentation makes. + +The rig was flipped end-to-end with `OTOPCUA_MESH_MODE=ClusterClient` (no rebuild, no compose edit — +the env indirection added in Task 9 is what made the A/B and the DPS control cheap enough to run). + +--- + +## Result table + +| # | Step | Result | +|---|---|---| +| 1 | Deploy on `Mode=Dps` | **PASS** — `48d0c7c5` Sealed, 6/6 Applied | +| 2 | Flip all six to `ClusterClient`, redeploy | **PASS** — `5db8f6ad` Sealed, 6/6 Applied | +| 3 | Confirm the transport actually changed | **PASS** — positive log evidence, see below | +| 4 | Stop a node, deploy | **PASS** — `59b81248` TimedOut naming `site-b-2:4053`, 5/6 | +| 5 | `MaintenanceMode = 1`, redeploy | **PASS** — `c6cb9812` Sealed, 5 rows, contacts 6→5 | +| 6 | AdminUI Restart + Reconnect | **NOT RUN** — no UI surface exists (finding D) | +| 7 | AdminUI alarm ack | **NOT RUN** — no alarms configured on the rig | +| 8 | Per-node (not singleton) discriminator | **PASS** — via a better discriminator than the plan's | +| 9 | Both centrals down, restart a driver node | **PARTIAL** — node survives; the stated ack-drop cannot occur | +| 10 | Restart central, redeploy | **PASS only after operator action** (finding C) | +| 11 | Frame-size canary | **PASS** — 0 warnings across all six nodes | + +## Step 3 — the transport really changed + +Central, both nodes, at boot under the flag: + +``` +central-1 [22:18:16 INF] Mesh ClusterClient created with 6 contact point(s) +central-2 [22:18:17 INF] Mesh ClusterClient created with 6 contact point(s) +``` + +Node side, with `Serilog__MinimumLevel__Default=Debug` overlaid on `site-a-1` (the rig pins the +default above Debug, so this line is invisible on a stock node): + +``` +site-a-1 [22:10:34 DBG] Mesh: received DispatchDeployment from central; publishing node-locally +``` + +Under `Mode=Dps` the same node logs nothing of the sort and central logs +`Mesh transport is Dps: commands are published on DistributedPubSub and no ClusterClient is created`. + +## Step 5 — the DB-sourced contact set is live + +Setting `MaintenanceMode = 1` on `site-b-2` shrank both centrals' contact sets on the next refresh, +and clearing it restored them — independently, on each central's own 60 s timer: + +``` +central-1 [22:22:16 INF] Mesh contact set changed; replacing the ClusterClient +central-1 [22:22:16 INF] Mesh ClusterClient created with 5 contact point(s) +central-1 [22:23:16 INF] Mesh ClusterClient created with 6 contact point(s) +central-2 [22:22:16 INF] … identical, same timestamps +``` + +No `InvalidActorNameException` on any rebuild — the generation-suffixed client name (Task 2, ported +from the sister project's shipped bug) does its job. + +## Step 8 — the per-node discriminator, replaced with a stronger one + +**The plan's pass-signal does not exist.** It asked for the surviving central's comm actor to *log +receipt* of a node ack inside the downing window. `CentralCommunicationActor.HandleApplyAck` logs +**nothing** on the success path — it only warns when the coordinator is unresolvable. There is no +line to observe. (Recorded as finding E.) + +The evidence above is a **cleaner** discriminator and needs no timing race. `Mesh ClusterClient +created` is emitted from `PreStart`, so it fires **once per live actor instance**. Both centrals +logged it, and both logged their own independent contact-set rebuild at step 5. A +`ClusterSingletonManager` runs the actor on the oldest node only — the other node hosts a manager +that never starts the actor and would log nothing. **Two nodes logging it proves two live instances, +i.e. per-node registration.** The node side is proven the same way: `SendToAll` collected 6/6 acks, +which requires six separately registered node comm actors. + +The failover half was run anyway, and passed: `docker stop central-1` (the oldest, and the singleton +host) → the surviving central accepted a deployment **4 seconds later** and collected acks from all +five live nodes (`e3161c25`; it TimedOut only because `central-1` is itself a configured node that +was down — correct Phase 1 semantics). Graceful stop hands the singleton over immediately, so no +downing wait was involved. + +## Step 9 — the stated failure mode cannot occur + +The plan expected a restarting driver node to emit a boot-time ack that gets dropped with a Warning. +**Driver nodes do not ack on boot.** `DriverHostActor.SendAck` is called only from the apply paths — +a node with no dispatch to answer has nothing to send, so there is no drop to observe. + +What was verified: with **both** centrals stopped, `site-a-1` restarted, came up clean, logged its +association failures against both central addresses, retried, and stayed up — no crash loop. The +node-side ClusterClient's forever-retry (`reconnect-timeout = off`) behaves as configured. + +## Step 11 — the canary stayed quiet + +`grep -ci "frame size|frame-size|maximum-frame"` = **0** on all six nodes, as designed: the deploy +notify carries only `DeploymentId` + `RevisionHash` + `CorrelationId`. + +--- + +## Finding A — a 10–30 s post-restart delivery window, **pre-existing, not Phase 2** + +A deploy issued shortly after a driver node restarts does not reach that node, and the deployment +fails at the 2-minute deadline naming it. Measured against `site-a-1`: + +| Delay after the node logs readiness | Result | +|---|---| +| ~4 s | **missed** (`f738e1de` TimedOut, node row `Applying`) | +| 10 s | **missed** (`0c7a6296` TimedOut) | +| 30 s | applied, sealed | + +**The DPS control fails identically.** Flipping the rig back to `Mode=Dps` and repeating the 10 s +case produced `e85f53c5` → TimedOut, `site-a-1` unacked. So the window is a property of cluster +membership convergence, not of the transport this phase introduces. Recorded rather than fixed. + +> Running that control is the only reason this is a note and not a Phase 2 regression. The +> temptation to skip it — "obviously the new transport broke it" — would have produced a false +> attribution and a wasted investigation. + +## Finding B — **`buffer-size = 0` does not deliver what this phase's docs claim it does** + +The stated rationale, in the plan, `CLAUDE.md`, and `docs/Configuration.md`, is that buffering +*"would replay a `DispatchDeployment` on reconnect and apply a deployment the coordinator already +sealed as `TimedOut`, leaving the node running a configuration the database records as failed."* + +**That outcome happens anyway, by a different route.** Deployment `0c7a6296` timed out at 22:14:30 +with `site-a-1` unacked. The node's row: + +``` +NodeId Status StartedAtUtc AppliedAtUtc +site-a-1:4053 1 2026-07-22 22:12:30.145 2026-07-22 22:15:14.823 +``` + +It applied **44 s after the deployment was sealed TimedOut**, on its next restart. The mechanism is +`DriverHostActor.Bootstrap()`, which reads the node's most recent `NodeDeploymentState`, finds the +orphan `Applying` row the coordinator pre-created at dispatch, and replays it: + +```csharp +case NodeDeploymentStatus.Applying: + _log.Warning("DriverHost {Node}: found orphan Applying row for deployment {Id}; replaying", …); + ApplyAndAck(new DeploymentId(latest.DeploymentId), revision.Value, CorrelationId.NewId()); +``` + +So `buffer-size = 0` removes the **transport-level, timer-driven** replay — a real difference: it is +silent, unbounded in time, and can fire without any node event. The **boot-time DB replay remains**, +and it is the better-behaved of the two (deliberate, Warning-logged, and bounded to a restart). The +decision stands; the *justification* was overstated and the docs are corrected in the same change as +this record. + +## Finding C — stopping **both** centrals strands the non-seed nodes permanently + +After `docker stop` of both centrals followed by `docker start`, the fleet did **not** recover. +Deployments reached 2/6 then 3/6 nodes. The cause is not the transport — the Akka cluster itself had +split: + +``` +Cluster hosts: MEMBERS 3 UP 3 UNREACHABLE 0 + central-1:4053 UP admin,driver leader for admin+driver + central-2:4053 UP admin,driver + site-a-1:4053 UP driver +``` + +The three site nodes that never restarted were **not members at all**, and their own gossip showed +the stale inverse view — `central-2` as `Exiting`, `site-a-1` as `Leaving` — frozen since 22:29: + +``` +Gossip(members = [central-2 status = Exiting, site-a-1 status = Leaving, + site-a-2 Up, site-b-1 Up, site-b-2 Up]) +``` + +A graceful stop puts the centrals into `Leaving`/`Exiting`; the processes died before the removal +handshake completed, leaving the survivors holding members that will never confirm. `auto-down` +does not help — these members are not *Unreachable*, they are stuck mid-`Exiting`. The restarted +centrals formed a fresh cluster (with `site-a-1`, which had also restarted and re-joined through its +seed), and the three stranded nodes had no path back: they believe they are already in a cluster, so +they never re-run the join process. + +**Recovery required restarting the stranded nodes.** After that, the next deploy sealed 6/6 +(`e20836ac`). + +This is transport-independent — membership is below both DPS and ClusterClient, and the set of nodes +that acked matched the cluster view exactly. It is a **downing/rejoin gap of the current single-mesh +topology**, in the same family as the registered outage gap the design doc already records, and it +belongs to the Phase 7 drill work. Under the target per-pair topology (every node a seed of its own +pair) the shape changes, and this drill must be re-run there rather than assumed fixed. + +## Finding D — the AdminUI's driver Restart/Reconnect buttons have no page + +Step 6 could not be run, and not for a rig reason. +`Components/Shared/Drivers/DriverStatusPanel.razor` is the **only** component in the codebase +carrying the `DriverOperator`-gated Reconnect/Restart buttons, and **nothing renders it** — `grep -rn +"