From e4cb9a08e051cad4b1e369f94a16291727960617 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 22 Jul 2026 17:46:46 -0400 Subject: [PATCH] docs(mesh): MeshTransport config, Phase 2 as shipped, design-doc correction Rig: all six docker-dev nodes carry MeshTransport__Mode plus both central contact points. Mode is "${OTOPCUA_MESH_MODE:-Dps}", so the rig still comes up on the transport it has always used and the live gate can flip the whole fleet at `docker compose up` without a compose edit or a rebuild. Docs: a MeshTransport section in Configuration.md (including why contact points are addresses only, why the contact set does not scope delivery, and why buffer-size=0 is a behaviour decision rather than a tuning knob), a "Command transport" section in Redundancy.md contrasting the two modes, and a summary beside the Redundancy notes in CLAUDE.md. Corrects the design doc's claim that ScadaBridge replies with a typed failure to every unhandled message. Neither of their comm actors has a ReceiveAny or an Unhandled override; the idiom is per message type and fires on a missing registration, and an unknown type dead-letters there as it does anywhere else. Records the exit-gate deviation in the program plan: "an Ask timing out cleanly against a stopped node" cannot be run, because no Ask crosses this boundary. The honest equivalents are a stopped node failing the deploy at the apply deadline and an unreachable contact dropping the command with a Warning. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- CLAUDE.md | 39 +++++++++- docker-dev/docker-compose.yml | 78 +++++++++++++++++++ docs/Configuration.md | 21 +++++ docs/Redundancy.md | 31 ++++++++ .../2026-07-21-per-cluster-mesh-design.md | 10 ++- .../2026-07-22-per-cluster-mesh-program.md | 28 ++++++- 6 files changed, 202 insertions(+), 5 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d585b07b..50d8c4b0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -228,7 +228,44 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file. -**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2–7 not started). +**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2 code-complete behind a dark switch; 3–7 not started). + +## Mesh command transport (`MeshTransport`) + +Per-cluster mesh **Phase 2** added a second transport for the three central→node command channels +(`deployments`, `driver-control`, `alarm-commands`) and the `deployment-acks` reply, selected by +`MeshTransport:Mode`: `Dps` (the DistributedPubSub path this repo has always used, **the default**) +or `ClusterClient`, which does **not** require central and the node to share a gossip ring — the +prerequisite for splitting the fleet into one mesh per application `Cluster`. Both `CentralCommunicationActor` +(`/user/central-communication`) and `NodeCommunicationActor` (`/user/node-communication`) are spawned +and registered with the `ClusterClientReceptionist` in **both** modes, so flipping the flag is a config +change, not a redeploy. Things worth knowing before touching it: + +- **`SendToAll`, never `Send`.** Today's DPS publish reaches every `DriverHostActor`; there is no + ClusterId or node filter on the node side at all (scoping happens later, inside the artifact via + `DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered + actor — it would deploy to a single node while every other node silently kept its old configuration, + and the deployment could still seal green. +- **Exactly ONE fleet-wide ClusterClient while the fleet is one mesh.** A receptionist serves its whole + cluster, so `SendToAll` reaches every registered node-comm actor regardless of which node's address was + dialled. One client per application `Cluster` — the obviously-right-looking shape, and what Phase 6 + wants — would fan each command out once per cluster (N× duplicate dispatch *and* N× duplicate ack). + Marked `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient`. **Corollary:** the contact set does + not scope delivery; excluding a `MaintenanceMode` node from the contacts does not stop it receiving. +- **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and + central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber. + Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both. +- **Comm actors are per-node, NOT singletons.** A client rotates across contact points and must find a live + comm actor at whichever node answers; `akka.cluster.client.receptionist.role` is empty for the same reason. +- **`buffer-size = 0` is a behaviour decision, not a tuning knob.** Akka's default buffers 1000 and replays + on reconnect, which would apply a `DispatchDeployment` for a deployment the coordinator already sealed + `TimedOut` — the node would run a config the DB records as failed. Do not raise it to quiet a flaky test. + +Contact points are node **addresses only** (`akka.tcp://@:`); `/system/receptionist` is +appended at construction and `MeshTransportOptionsValidator` refuses to start a node whose contact carries an +actor-path suffix. `redundancy-state` stays on DPS in both modes (bidirectional, and built from +`Cluster.State`). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md` §"Command transport", +and `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`. **Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys: diff --git a/docker-dev/docker-compose.yml b/docker-dev/docker-compose.yml index cdfc663c..71042130 100644 --- a/docker-dev/docker-compose.yml +++ b/docker-dev/docker-compose.yml @@ -190,6 +190,19 @@ services: Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053" Cluster__Roles__0: "admin" Cluster__Roles__1: "driver" + # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", + # the transport this repo has always used, and every node keeps its ClusterClient wiring built + # and its comm actor registered with the receptionist either way. Flip the whole rig with + # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild. + # + # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction + # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an + # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in + # silence. Both central nodes are listed so a driver node's client can rotate to whichever one + # is up — that rotation is why the comm actors are per-node and NOT cluster singletons. + MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}" + MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" + MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053" Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345" Security__Jwt__Issuer: "otopcua-dev" Security__Jwt__Audience: "otopcua-dev" @@ -263,6 +276,19 @@ services: Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053" Cluster__Roles__0: "admin" Cluster__Roles__1: "driver" + # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", + # the transport this repo has always used, and every node keeps its ClusterClient wiring built + # and its comm actor registered with the receptionist either way. Flip the whole rig with + # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild. + # + # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction + # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an + # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in + # silence. Both central nodes are listed so a driver node's client can rotate to whichever one + # is up — that rotation is why the comm actors are per-node and NOT cluster singletons. + MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}" + MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" + MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053" Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345" Security__Jwt__Issuer: "otopcua-dev" Security__Jwt__Audience: "otopcua-dev" @@ -329,6 +355,19 @@ services: # is in its own seed list), so these configs are exempt rather than broken. Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053" Cluster__Roles__0: "driver" + # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", + # the transport this repo has always used, and every node keeps its ClusterClient wiring built + # and its comm actor registered with the receptionist either way. Flip the whole rig with + # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild. + # + # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction + # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an + # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in + # silence. Both central nodes are listed so a driver node's client can rotate to whichever one + # is up — that rotation is why the comm actors are per-node and NOT cluster singletons. + MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}" + MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" + MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053" <<: *secrets-env # Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/ # mem_reservation are inherited from the *otopcua-host anchor. @@ -395,6 +434,19 @@ services: Cluster__PublicHostname: "site-a-2" Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053" Cluster__Roles__0: "driver" + # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", + # the transport this repo has always used, and every node keeps its ClusterClient wiring built + # and its comm actor registered with the receptionist either way. Flip the whole rig with + # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild. + # + # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction + # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an + # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in + # silence. Both central nodes are listed so a driver node's client can rotate to whichever one + # is up — that rotation is why the comm actors are per-node and NOT cluster singletons. + MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}" + MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" + MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053" <<: *secrets-env Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning" Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning" @@ -453,6 +505,19 @@ services: Cluster__PublicHostname: "site-b-1" Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053" Cluster__Roles__0: "driver" + # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", + # the transport this repo has always used, and every node keeps its ClusterClient wiring built + # and its comm actor registered with the receptionist either way. Flip the whole rig with + # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild. + # + # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction + # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an + # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in + # silence. Both central nodes are listed so a driver node's client can rotate to whichever one + # is up — that rotation is why the comm actors are per-node and NOT cluster singletons. + MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}" + MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" + MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053" <<: *secrets-env Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning" Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning" @@ -494,6 +559,19 @@ services: Cluster__PublicHostname: "site-b-2" Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053" Cluster__Roles__0: "driver" + # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", + # the transport this repo has always used, and every node keeps its ClusterClient wiring built + # and its comm actor registered with the receptionist either way. Flip the whole rig with + # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild. + # + # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction + # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an + # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in + # silence. Both central nodes are listed so a driver node's client can rotate to whichever one + # is up — that rotation is why the comm actors are per-node and NOT cluster singletons. + MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}" + MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" + MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053" <<: *secrets-env Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning" Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning" diff --git a/docs/Configuration.md b/docs/Configuration.md index 6d7d249c..a49bfb80 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -110,6 +110,27 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only > **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode). +### `MeshTransport` (central↔node command transport) + +- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split. +- **Options class:** `MeshTransportOptions` (`SectionName = "MeshTransport"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`. Bound and validated by `AddOtOpcUaCluster(config)`; `MeshTransportOptionsValidator` runs at `ValidateOnStart`. + +| Key | Type | Default | Meaning | +|---|---|---|---| +| `Mode` | string | `Dps` | `Dps` = DistributedPubSub (the transport this repo has always used). `ClusterClient` = the mesh boundary. Any other value **refuses to start**. | +| `CentralContactPoints` | string[] | `[]` | Node-side only: central node **addresses** (`akka.tcp://@:`). Required under `ClusterClient`, ignored under `Dps`. | +| `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its ClusterClient contact set from the `ClusterNode` rows. | + +**Addresses only — no actor-path suffix.** `/system/receptionist` is appended at construction time; a contact carrying `/user/` or `/system/` is rejected at startup, because a doubled suffix produces a path that resolves to nothing and drops every command in silence. + +**It is a dark switch, and both sides are always wired.** The comm actors are spawned and registered with the `ClusterClientReceptionist` on every node in *both* modes, so flipping `Mode` is a config change rather than a redeploy. Under `Dps` the registration is simply idle. + +**Central's contact set comes from the enabled, non-maintenance `ClusterNode` rows** — the same source Phase 1 gave the deploy ack set. A node in `MaintenanceMode` is not dialled, but note that **the contact set does not scope delivery**: a receptionist serves its whole cluster, so `SendToAll` still reaches every registered node exactly as today's DPS broadcast does. The filter is about which receptionists are worth dialling, not about who receives. + +**Buffering is off by design** (`akka.cluster.client.buffer-size = 0`, set in the shipped `akka.conf`). If no receptionist is known, the command is dropped with a Warning rather than queued. Buffering would replay a `DispatchDeployment` on reconnect and apply a deployment the coordinator had already sealed as `TimedOut`, leaving the node running a configuration the database records as failed. Do not "fix" this back to Akka's default of 1000 to quiet a flaky test. + +On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`. + ### `ConnectionStrings` → `ConfigDb` - **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally. diff --git a/docs/Redundancy.md b/docs/Redundancy.md index bd8e3f07..c08d35db 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -407,6 +407,37 @@ Net effect: each alarm transition appears **once** on `/alerts` and would histor See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals. +## Command transport (central↔node) + +Everything above rides one Akka mesh: central publishes on DistributedPubSub, every node +subscribes, and the two ends must be members of the same cluster. Per-cluster mesh **Phase 2** +adds a second transport under `MeshTransport:Mode` (see +[Configuration.md § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport)) +that does not need shared membership — the prerequisite for splitting the fleet into one mesh per +application `Cluster`. + +| | `Dps` (default) | `ClusterClient` | +|---|---|---| +| Central → node commands | `Publish` on `deployments` / `driver-control` / `alarm-commands` | `ClusterClient.SendToAll` to `/user/node-communication`, re-emitted on the node's **local** `EventStream` | +| Node → central acks | `Publish` on `deployment-acks` | `ClusterClient.Send` to `/user/central-communication`, forwarded to the deploy-coordinator singleton | +| Requires shared cluster membership | **yes** | no | + +Two properties are load-bearing and easy to break: + +- **Both comm actors are registered per node, NOT as cluster singletons.** A `ClusterClient` rotates + across its contact points and must find a live comm actor at whichever node answers. As a + singleton the boundary would be reachable through exactly one node, and rotation would fail + silently against the others — acks landing or not depending on where the client happened to + settle. `akka.cluster.client.receptionist.role` is deliberately empty for the same reason. +- **Inbound commands go to the node-local `EventStream`, never back onto a DPS topic.** DPS is + mesh-wide and central `SendToAll`s to *every* node, so a DPS re-publish would deliver N copies of + every command to every subscriber. + +Phase 2 changes no redundancy semantics: the Primary gate, ServiceLevel, and the singleton layout +are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional and built from +`Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the +meshes can split). + ## Pair-local store (LocalDb — Phases 1 + 2) Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated diff --git a/docs/plans/2026-07-21-per-cluster-mesh-design.md b/docs/plans/2026-07-21-per-cluster-mesh-design.md index 1acff34d..8ccd3ed6 100644 --- a/docs/plans/2026-07-21-per-cluster-mesh-design.md +++ b/docs/plans/2026-07-21-per-cluster-mesh-design.md @@ -68,8 +68,14 @@ Other load-bearing details: caller's Ask times out. *"It keeps the central coordinator stateless with respect to site availability."* A `ConnectionStateChanged` mechanism was built for this and **deleted as dead code**. -- **Every unhandled message replies with a typed failure rather than dropping**, so a central Ask - can never stall on a missing handler. +- ~~**Every unhandled message replies with a typed failure rather than dropping**, so a central Ask + can never stall on a missing handler.~~ **CORRECTED 2026-07-22 (Phase 2 recon).** This overstates + what ScadaBridge does. Neither of their comm actors has a `ReceiveAny` or an `Unhandled` override: + the typed-failure idiom is **per message type**, and it fires only when a *registration* is missing + (no ClusterClient for the site yet, no handler registered for that type). A genuinely unknown + message type still dead-letters exactly as it would anywhere else. Do not build a catch-all failure + reply believing you are copying them. OtOpcUa's Phase 2 comm actors list each type explicitly and + deliberately let an unknown type dead-letter loudly rather than republish it blind. - **Sender preservation is the whole Ask idiom**: `_client.Tell(new ClusterClient.Send(path, msg), Sender)` routes the reply straight back to the waiting Ask, bypassing the comm actor. diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md b/docs/plans/2026-07-22-per-cluster-mesh-program.md index 89ab6453..4dc4710f 100644 --- a/docs/plans/2026-07-22-per-cluster-mesh-program.md +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md @@ -99,7 +99,31 @@ this removes the coordinator's one genuinely mesh-bound dependency). set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node missing; a node up but row absent → its ack is not expected). -### Phase 2 — Comm actors + ClusterClient transport +### Phase 2 — Comm actors + ClusterClient transport — **CODE COMPLETE 2026-07-22, live gate pending** +Shipped per [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md) +as a **dark switch**: `MeshTransport:Mode` defaults to `Dps`, so nothing changes on any deployment +until the flag flips. Both comm actors are spawned and receptionist-registered in **both** modes, so +the flip is a config change rather than a redeploy. Three scope corrections landed against the +design doc (that plan's header carries them in full): + +- the node path is `/user/node-communication`, not `/user/cluster-communication` (there is no + per-cluster scoping until Phase 6); +- the substitution for `Publish` is **`SendToAll`**, not `Send` — `Send` reaches exactly one node and + would deploy to one member of the fleet while the rest silently kept their old config; +- central creates **exactly one fleet-wide ClusterClient**. On a single mesh a receptionist serves + its whole cluster, so a client per application `Cluster` would fan every command out once per + cluster. Per-cluster clients become correct only when Phase 6 splits the meshes (`TODO(Phase 6)` + in `CentralCommunicationActor.RebuildClient`). + +**Exit-gate deviation — "an Ask timing out cleanly against a stopped node" cannot be run as written.** +There is no cross-boundary Ask in Phase 2: every migrated command is fire-and-forget on the wire. +`DispatchDeployment` is a `Tell` whose ack returns later as an independent `ApplyAck`; +`RestartDriver` / `ReconnectDriver` / `AlarmCommand` reply `Ok = true` from the admin node the moment +they are dispatched (`Ok` has always meant *dispatched*, not *applied* — the AdminUI string is +literally "Restart dispatched"). The honest equivalents, both in the live gate: a **stopped node** +fails the deploy at the apply deadline naming it (Phase 1 semantics), and **no reachable contact** +drops the command with a Warning and never buffers it. + **Scope:** one receptionist-registered actor per side (`/user/central-communication`, `/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation); ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge @@ -200,7 +224,7 @@ resource sizing on the site VMs should be checked once both products run the ful | 0b oldest-Up election | DONE 2026-07-21 | | Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. | | 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. | -| 2 ClusterClient transport | **IN PROGRESS** — plan being written 2026-07-22 | +| 2 ClusterClient transport | **CODE COMPLETE 2026-07-22, live gate pending** — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. | | 3 fetch-and-cache | not started | | 4 cut driver ConfigDb | not started | | 5 gRPC telemetry | not started |