Merge branch 'feat/mesh-phase2' — per-cluster mesh Phase 2 (ClusterClient transport)

Central↔node command transport behind the MeshTransport:Mode dark switch (Dps default | ClusterClient). Both comm actors registered with the receptionist in both modes; SendToAll fan-out, DB-sourced contacts, node-local EventStream re-emit, ApplyAck relay. Live-gated on docker-dev end-to-end. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-22 18:42:57 -04:00
parent 7654f24dab 32055c2238
commit 6eaa81cae7
38 changed files with 3917 additions and 46 deletions
@@ -0,0 +1,29 @@
 # Build context exclusions for docker-dev/Dockerfile, whose `COPY . .` takes the whole repo.
 #
 # Without this file the context also carries every project's bin/obj AND .claude/worktrees — agent
 # worktrees that are themselves full checkouts WITH their own build outputs. That reached ~19 GB and
 # exhausted the Docker VM's disk mid-COPY, failing the rig build with "no space left on device"
 # rather than anything a build log would attribute to the repo.
 #
 # Nothing excluded here is needed: the build restores and publishes inside the image, and the
 # migrator stage runs `dotnet ef` against the same restored source.
 # Agent worktrees — full checkouts, build outputs and all.
 .claude/
 # Local build outputs; the image builds its own.
 **/bin/
 **/obj/
 artifacts/
 # VCS + editor state.
 .git/
 .gitignore
 .vs/
 .vscode/
 .idea/
 # Docker's own state, and anything the image would never read.
 **/.DS_Store
 **/*.user
 docker-dev/**/*.log
@@ -228,7 +228,46 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
 Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
-**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2–7 not started).
+**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2 code-complete behind a dark switch; 3–7 not started).
 ## Mesh command transport (`MeshTransport`)
 Per-cluster mesh **Phase 2** added a second transport for the three central→node command channels
 (`deployments`, `driver-control`, `alarm-commands`) and the `deployment-acks` reply, selected by
 `MeshTransport:Mode`: `Dps` (the DistributedPubSub path this repo has always used, **the default**)
 or `ClusterClient`, which does **not** require central and the node to share a gossip ring — the
 prerequisite for splitting the fleet into one mesh per application `Cluster`. Both `CentralCommunicationActor`
 (`/user/central-communication`) and `NodeCommunicationActor` (`/user/node-communication`) are spawned
 and registered with the `ClusterClientReceptionist` in **both** modes, so flipping the flag is a config
 change, not a redeploy. Things worth knowing before touching it:
 - **`SendToAll`, never `Send`.** Today's DPS publish reaches every `DriverHostActor`; there is no
  ClusterId or node filter on the node side at all (scoping happens later, inside the artifact via
  `DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered
  actor — it would deploy to a single node while every other node silently kept its old configuration,
  and the deployment could still seal green.
 - **Exactly ONE fleet-wide ClusterClient while the fleet is one mesh.** A receptionist serves its whole
  cluster, so `SendToAll` reaches every registered node-comm actor regardless of which node's address was
  dialled. One client per application `Cluster` — the obviously-right-looking shape, and what Phase 6
  wants — would fan each command out once per cluster (N× duplicate dispatch *and* N× duplicate ack).
  Marked `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient`. **Corollary:** the contact set does
  not scope delivery; excluding a `MaintenanceMode` node from the contacts does not stop it receiving.
 - **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and
  central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber.
  Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both.
 - **Comm actors are per-node, NOT singletons.** A client rotates across contact points and must find a live
  comm actor at whichever node answers; `akka.cluster.client.receptionist.role` is empty for the same reason.
 - **`buffer-size = 0` is a behaviour decision, not a tuning knob.** Akka's default buffers 1000 and replays
  on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not raise
  it to quiet a flaky test. **But do not over-claim what it buys:** the Phase 2 live gate observed a node
  applying a deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()`
  replaying the orphan `Applying` row on restart. Zero buffering removes the silent replay, not every replay.
 Contact points are node **addresses only** (`akka.tcp://<system>@<host>:<port>`); `/system/receptionist` is
 appended at construction and `MeshTransportOptionsValidator` refuses to start a node whose contact carries an
 actor-path suffix. `redundancy-state` stays on DPS in both modes (bidirectional, and built from
 `Cluster.State`). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md` §"Command transport",
 and `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`.
 **Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
@@ -190,6 +190,19 @@ services:
      Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
      Cluster__Roles__0: "admin"
      Cluster__Roles__1: "driver"
      # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
      # the transport this repo has always used, and every node keeps its ClusterClient wiring built
      # and its comm actor registered with the receptionist either way. Flip the whole rig with
      # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
      #
      # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
      # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
      # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
      # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
      # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
      MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
      MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
      MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
      Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
      Security__Jwt__Issuer: "otopcua-dev"
      Security__Jwt__Audience: "otopcua-dev"
@@ -263,6 +276,19 @@ services:
      Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
      Cluster__Roles__0: "admin"
      Cluster__Roles__1: "driver"
      # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
      # the transport this repo has always used, and every node keeps its ClusterClient wiring built
      # and its comm actor registered with the receptionist either way. Flip the whole rig with
      # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
      #
      # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
      # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
      # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
      # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
      # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
      MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
      MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
      MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
      Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
      Security__Jwt__Issuer: "otopcua-dev"
      Security__Jwt__Audience: "otopcua-dev"
@@ -329,6 +355,19 @@ services:
      # is in its own seed list), so these configs are exempt rather than broken.
      Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
      Cluster__Roles__0: "driver"
      # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
      # the transport this repo has always used, and every node keeps its ClusterClient wiring built
      # and its comm actor registered with the receptionist either way. Flip the whole rig with
      # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
      #
      # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
      # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
      # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
      # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
      # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
      MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
      MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
      MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
      <<: *secrets-env
      # Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
      # mem_reservation are inherited from the *otopcua-host anchor.
@@ -395,6 +434,19 @@ services:
      Cluster__PublicHostname: "site-a-2"
      Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
      Cluster__Roles__0: "driver"
      # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
      # the transport this repo has always used, and every node keeps its ClusterClient wiring built
      # and its comm actor registered with the receptionist either way. Flip the whole rig with
      # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
      #
      # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
      # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
      # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
      # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
      # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
      MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
      MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
      MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
      <<: *secrets-env
      Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
      Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
@@ -453,6 +505,19 @@ services:
      Cluster__PublicHostname: "site-b-1"
      Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
      Cluster__Roles__0: "driver"
      # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
      # the transport this repo has always used, and every node keeps its ClusterClient wiring built
      # and its comm actor registered with the receptionist either way. Flip the whole rig with
      # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
      #
      # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
      # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
      # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
      # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
      # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
      MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
      MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
      MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
      <<: *secrets-env
      Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
      Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
@@ -494,6 +559,19 @@ services:
      Cluster__PublicHostname: "site-b-2"
      Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
      Cluster__Roles__0: "driver"
      # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
      # the transport this repo has always used, and every node keeps its ClusterClient wiring built
      # and its comm actor registered with the receptionist either way. Flip the whole rig with
      # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
      #
      # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
      # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
      # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
      # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
      # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
      MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
      MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
      MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
      <<: *secrets-env
      Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
      Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
@@ -110,6 +110,29 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
 > **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
 ### `MeshTransport` (central↔node command transport)
 - **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split.
 - **Options class:** `MeshTransportOptions` (`SectionName = "MeshTransport"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`. Bound and validated by `AddOtOpcUaCluster(config)`; `MeshTransportOptionsValidator` runs at `ValidateOnStart`.
 | Key | Type | Default | Meaning |
 |---|---|---|---|
 | `Mode` | string | `Dps` | `Dps` = DistributedPubSub (the transport this repo has always used). `ClusterClient` = the mesh boundary. Any other value **refuses to start**. |
 | `CentralContactPoints` | string[] | `[]` | Node-side only: central node **addresses** (`akka.tcp://<system>@<host>:<port>`). Required under `ClusterClient`, ignored under `Dps`. |
 | `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its ClusterClient contact set from the `ClusterNode` rows. |
 **Addresses only — no actor-path suffix.** `/system/receptionist` is appended at construction time; a contact carrying `/user/` or `/system/` is rejected at startup, because a doubled suffix produces a path that resolves to nothing and drops every command in silence.
 **It is a dark switch, and both sides are always wired.** The comm actors are spawned and registered with the `ClusterClientReceptionist` on every node in *both* modes, so flipping `Mode` is a config change rather than a redeploy. Under `Dps` the registration is simply idle.
 **Central's contact set comes from the enabled, non-maintenance `ClusterNode` rows** — the same source Phase 1 gave the deploy ack set. A node in `MaintenanceMode` is not dialled, but note that **the contact set does not scope delivery**: a receptionist serves its whole cluster, so `SendToAll` still reaches every registered node exactly as today's DPS broadcast does. The filter is about which receptionists are worth dialling, not about who receives.
 **Buffering is off by design** (`akka.cluster.client.buffer-size = 0`, set in the shipped `akka.conf`). If no receptionist is known, the command is dropped rather than queued. Akka's default of 1000 would replay a `DispatchDeployment` on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not "fix" it back to quiet a flaky test.
 > **What this does *not* buy you.** It does not guarantee a node never applies a deployment the DB records as `TimedOut`. A node that missed a dispatch keeps the orphan `Applying` row the coordinator pre-created, and `DriverHostActor.Bootstrap()` replays that row on the node's next restart — live-observed applying a deployment 44 s after it was sealed `TimedOut` (see the [Phase 2 live gate](plans/2026-07-22-mesh-phase2-live-gate.md), finding B). The boot replay is the better-behaved of the two — deliberate, Warning-logged, and bounded to a restart — but it is still a replay. `buffer-size = 0` removes the *silent, timer-driven* one only.
 On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`.
 ### `ConnectionStrings` → `ConfigDb`
 - **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally.
@@ -407,6 +407,37 @@ Net effect: each alarm transition appears **once** on `/alerts` and would histor
 See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.
 ## Command transport (central↔node)
 Everything above rides one Akka mesh: central publishes on DistributedPubSub, every node
 subscribes, and the two ends must be members of the same cluster. Per-cluster mesh **Phase 2**
 adds a second transport under `MeshTransport:Mode` (see
 [Configuration.md § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport))
 that does not need shared membership — the prerequisite for splitting the fleet into one mesh per
 application `Cluster`.
 | | `Dps` (default) | `ClusterClient` |
 |---|---|---|
 | Central → node commands | `Publish` on `deployments` / `driver-control` / `alarm-commands` | `ClusterClient.SendToAll` to `/user/node-communication`, re-emitted on the node's **local** `EventStream` |
 | Node → central acks | `Publish` on `deployment-acks` | `ClusterClient.Send` to `/user/central-communication`, forwarded to the deploy-coordinator singleton |
 | Requires shared cluster membership | **yes** | no |
 Two properties are load-bearing and easy to break:
 - **Both comm actors are registered per node, NOT as cluster singletons.** A `ClusterClient` rotates
  across its contact points and must find a live comm actor at whichever node answers. As a
  singleton the boundary would be reachable through exactly one node, and rotation would fail
  silently against the others — acks landing or not depending on where the client happened to
  settle. `akka.cluster.client.receptionist.role` is deliberately empty for the same reason.
 - **Inbound commands go to the node-local `EventStream`, never back onto a DPS topic.** DPS is
  mesh-wide and central `SendToAll`s to *every* node, so a DPS re-publish would deliver N copies of
  every command to every subscriber.
 Phase 2 changes no redundancy semantics: the Primary gate, ServiceLevel, and the singleton layout
 are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional and built from
 `Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the
 meshes can split).
 ## Pair-local store (LocalDb — Phases 1 + 2)
 Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated
@@ -68,8 +68,14 @@ Other load-bearing details:
  caller's Ask times out. *"It keeps the central coordinator stateless with respect to site
  availability."* A `ConnectionStateChanged` mechanism was built for this and **deleted as dead
  code**.
- **Every unhandled message replies with a typed failure rather than dropping**, so a central Ask
+- ~~**Every unhandled message replies with a typed failure rather than dropping**, so a central Ask
-  can never stall on a missing handler.
+  can never stall on a missing handler.~~ **CORRECTED 2026-07-22 (Phase 2 recon).** This overstates
  what ScadaBridge does. Neither of their comm actors has a `ReceiveAny` or an `Unhandled` override:
  the typed-failure idiom is **per message type**, and it fires only when a *registration* is missing
  (no ClusterClient for the site yet, no handler registered for that type). A genuinely unknown
  message type still dead-letters exactly as it would anywhere else. Do not build a catch-all failure
  reply believing you are copying them. OtOpcUa's Phase 2 comm actors list each type explicitly and
  deliberately let an unknown type dead-letter loudly rather than republish it blind.
 - **Sender preservation is the whole Ask idiom**: `_client.Tell(new ClusterClient.Send(path, msg),
  Sender)` routes the reply straight back to the waiting Ask, bypassing the comm actor.
@@ -0,0 +1,141 @@
 {
  "planPath": "docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md",
  "note": "Per-cluster mesh Phase 2. Decisions pre-made: config-flagged dark switch (MeshTransport:Mode), buffer-size=0, alarm-commands central leg in scope. Three design-doc corrections and the single-mesh duplicate-delivery trap are documented in the plan header \u2014 read them before Task 3.",
  "tasks": [
    {
      "id": 0,
      "subject": "Task 0: MeshTransportOptions + validator + registration",
      "status": "completed",
      "classification": "small",
      "parallelizableWith": [
        1
      ],
      "commit": "5439f148"
    },
    {
      "id": 1,
      "subject": "Task 1: receptionist extension, buffer-size=0, frame-size logging (effective-config test)",
      "status": "completed",
      "classification": "small",
      "parallelizableWith": [
        0
      ],
      "commit": "7b71a6a3",
      "note": "First draft of the buffering test was VACUOUS -- Config.GetInt returns 0 for a missing key, so it passed before the feature existed. Rewritten to assert through ClusterClientSettings/ClusterReceptionistSettings."
    },
    {
      "id": 2,
      "subject": "Task 2: MeshCommand envelope + generation-suffixed ClusterClient factory",
      "status": "completed",
      "classification": "small",
      "blockedBy": [
        0
      ],
      "commit": "0d6669d5",
      "note": "Dropped a third test that asserted ClusterClientSettings directly and never touched the factory."
    },
    {
      "id": 3,
      "subject": "Task 3: CentralCommunicationActor \u2014 dark-switched SendToAll fan-out, DB-sourced contacts, ApplyAck forward",
      "status": "completed",
      "classification": "high-risk",
      "blockedBy": [
        1,
        2
      ],
      "parallelizableWith": [
        4
      ],
      "commit": "d5b5cb6e",
      "note": "SendToAll sabotage-verified. Two test-harness bugs found: SubscribeAck goes to the SENDER, and EventFilter matches case-INSENSITIVELY (a 'was NOT' filter also caught 'was not created')."
    },
    {
      "id": 4,
      "subject": "Task 4: NodeCommunicationActor \u2014 inbound to node-local EventStream, outbound ApplyAck",
      "status": "completed",
      "classification": "standard",
      "blockedBy": [
        1,
        2
      ],
      "parallelizableWith": [
        3
      ],
      "commit": "915beec1",
      "note": "No Ask crosses the boundary in Phase 2, so this actor is far smaller than the sister project's."
    },
    {
      "id": 5,
      "subject": "Task 5: node subscribers accept EventStream too; rename _coordinatorOverride -> _ackRouter",
      "status": "completed",
      "classification": "standard",
      "blockedBy": [
        4
      ],
      "commit": "5cb0e721",
      "note": "Two wiring tests added -- unit tests either side of the seam both pass with the subscription deleted. First draft raced (ActorOf returns before PreStart; EventStream publish to nobody is dropped silently); alarm version was vacuous until the warm-up moved outside the assertion window."
    },
    {
      "id": 6,
      "subject": "Task 6: wire both sides, receptionist registration, pin the wire-contract actor paths",
      "status": "completed",
      "classification": "standard",
      "blockedBy": [
        3,
        5
      ],
      "commit": "16d59856",
      "note": "Dropped a planned 'not a singleton' assertion: two candidate discriminators were both invalid (a SingletonManager also sits at /user/<name>; /singleton is null even for the known singleton config-publish). A sabotage re-registering via WithSingleton left both green, which exposed them. Property deferred to Task 8."
    },
    {
      "id": 7,
      "subject": "Task 7: route the five publish sites through the mesh router (nullable, mediator fallback)",
      "status": "completed",
      "classification": "standard",
      "blockedBy": [
        6
      ],
      "note": "Sabotage-verified by inverting the branch: all 6 tests red. Comm actor registration moved above the singletons so registry.Get works; an earlier ResolveOne().GetAwaiter().GetResult() blocked inside a props factory."
    },
    {
      "id": 8,
      "subject": "Task 8: real two-ActorSystem ClusterClient boundary test + falsifiability control",
      "status": "completed",
      "classification": "high-risk",
      "blockedBy": [
        7
      ],
      "parallelizableWith": [
        9
      ],
      "commit": "5468b79d",
      "note": "Not flaky (3/3 clean repeats; 4 tests, ~16s). Both legs sabotage-verified against production code, and the control was itself falsified by restoring the receptionist. The control necessarily removes RegisterService too -- registering would materialise the extension it is removing. Task 6's dropped per-node/singleton property does NOT land here (one node per side cannot discriminate); it is live-gate step 8, and the MeshCommActorPathTests comment was corrected to say so."
    },
    {
      "id": 9,
      "subject": "Task 9: docker-dev rig config + docs + design-doc correction",
      "status": "completed",
      "commit": "e4cb9a08",
      "note": "Rig Mode is ${OTOPCUA_MESH_MODE:-Dps} rather than a hardcoded Dps, so Task 10 can flip all six nodes at `docker compose up` with no compose edit and no rebuild. Verified via `docker compose config` (6 services resolved).",
      "classification": "small",
      "blockedBy": [
        7
      ],
      "parallelizableWith": [
        8
      ]
    },
    {
      "id": 10,
      "subject": "Task 10: live gate on docker-dev (11 steps; step 4 = deploy-seal semantics, step 8 = per-node/contact-rotation discriminator [replaces the dropped Task 6 assertion \u2014 do not skip], step 9 = both-centrals drop behavior)",
      "status": "completed",
      "note": "PASSED for the transport (steps 1-5, 8, 11 green; flip end-to-end via OTOPCUA_MESH_MODE=ClusterClient). Record: 2026-07-22-mesh-phase2-live-gate.md. Steps 6/7 NOT RUN (no UI surface for driver-control \u2014 finding D, DriverStatusPanel is orphaned; no alarms configured). Step 8's stated log-signal does not exist (HandleApplyAck is silent on success \u2014 finding E); proven instead by both centrals independently logging per-instance ClusterClient creation + rebuild, which a singleton cannot. FOUR findings: A) 10-30s post-restart delivery window is PRE-EXISTING (DPS control failed identically); B) buffer-size=0 rationale OVERSTATED \u2014 a node applies a TimedOut deploy anyway via Bootstrap() orphan-row replay at +44s, corrected in 3 docs; C) stopping BOTH centrals strands non-seed nodes at the membership layer (Phase 7 drill, transport-independent); D) driver Restart/Reconnect UI orphaned since v3 Batch 2. Rig restored to Dps default.",
      "classification": "high-risk",
      "blockedBy": [
        8,
        9
      ]
    }
  ],
  "lastUpdated": "2026-07-22T00:00:00Z"
 }
@@ -0,0 +1,228 @@
 # Per-cluster mesh Phase 2 — live gate record
 **Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase2`
 **Plan:** [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md)
 **Result: PASSED for the transport**, with two gate steps not run (no surface exists to drive them),
 one step whose stated pass-signal does not exist in the code, and **four findings** — one of which
 corrects a claim this phase's own documentation makes.
 The rig was flipped end-to-end with `OTOPCUA_MESH_MODE=ClusterClient` (no rebuild, no compose edit —
 the env indirection added in Task 9 is what made the A/B and the DPS control cheap enough to run).
 ---
 ## Result table
 | # | Step | Result |
 |---|---|---|
 | 1 | Deploy on `Mode=Dps` | **PASS** — `48d0c7c5` Sealed, 6/6 Applied |
 | 2 | Flip all six to `ClusterClient`, redeploy | **PASS** — `5db8f6ad` Sealed, 6/6 Applied |
 | 3 | Confirm the transport actually changed | **PASS** — positive log evidence, see below |
 | 4 | Stop a node, deploy | **PASS** — `59b81248` TimedOut naming `site-b-2:4053`, 5/6 |
 | 5 | `MaintenanceMode = 1`, redeploy | **PASS** — `c6cb9812` Sealed, 5 rows, contacts 6→5 |
 | 6 | AdminUI Restart + Reconnect | **NOT RUN** — no UI surface exists (finding D) |
 | 7 | AdminUI alarm ack | **NOT RUN** — no alarms configured on the rig |
 | 8 | Per-node (not singleton) discriminator | **PASS** — via a better discriminator than the plan's |
 | 9 | Both centrals down, restart a driver node | **PARTIAL** — node survives; the stated ack-drop cannot occur |
 | 10 | Restart central, redeploy | **PASS only after operator action** (finding C) |
 | 11 | Frame-size canary | **PASS** — 0 warnings across all six nodes |
 ## Step 3 — the transport really changed
 Central, both nodes, at boot under the flag:
 ```
 central-1 [22:18:16 INF] Mesh ClusterClient created with 6 contact point(s)
 central-2 [22:18:17 INF] Mesh ClusterClient created with 6 contact point(s)
 ```
 Node side, with `Serilog__MinimumLevel__Default=Debug` overlaid on `site-a-1` (the rig pins the
 default above Debug, so this line is invisible on a stock node):
 ```
 site-a-1  [22:10:34 DBG] Mesh: received DispatchDeployment from central; publishing node-locally
 ```
 Under `Mode=Dps` the same node logs nothing of the sort and central logs
 `Mesh transport is Dps: commands are published on DistributedPubSub and no ClusterClient is created`.
 ## Step 5 — the DB-sourced contact set is live
 Setting `MaintenanceMode = 1` on `site-b-2` shrank both centrals' contact sets on the next refresh,
 and clearing it restored them — independently, on each central's own 60 s timer:
 ```
 central-1 [22:22:16 INF] Mesh contact set changed; replacing the ClusterClient
 central-1 [22:22:16 INF] Mesh ClusterClient created with 5 contact point(s)
 central-1 [22:23:16 INF] Mesh ClusterClient created with 6 contact point(s)
 central-2 [22:22:16 INF] … identical, same timestamps
 ```
 No `InvalidActorNameException` on any rebuild — the generation-suffixed client name (Task 2, ported
 from the sister project's shipped bug) does its job.
 ## Step 8 — the per-node discriminator, replaced with a stronger one
 **The plan's pass-signal does not exist.** It asked for the surviving central's comm actor to *log
 receipt* of a node ack inside the downing window. `CentralCommunicationActor.HandleApplyAck` logs
 **nothing** on the success path — it only warns when the coordinator is unresolvable. There is no
 line to observe. (Recorded as finding E.)
 The evidence above is a **cleaner** discriminator and needs no timing race. `Mesh ClusterClient
 created` is emitted from `PreStart`, so it fires **once per live actor instance**. Both centrals
 logged it, and both logged their own independent contact-set rebuild at step 5. A
 `ClusterSingletonManager` runs the actor on the oldest node only — the other node hosts a manager
 that never starts the actor and would log nothing. **Two nodes logging it proves two live instances,
 i.e. per-node registration.** The node side is proven the same way: `SendToAll` collected 6/6 acks,
 which requires six separately registered node comm actors.
 The failover half was run anyway, and passed: `docker stop central-1` (the oldest, and the singleton
 host) → the surviving central accepted a deployment **4 seconds later** and collected acks from all
 five live nodes (`e3161c25`; it TimedOut only because `central-1` is itself a configured node that
 was down — correct Phase 1 semantics). Graceful stop hands the singleton over immediately, so no
 downing wait was involved.
 ## Step 9 — the stated failure mode cannot occur
 The plan expected a restarting driver node to emit a boot-time ack that gets dropped with a Warning.
 **Driver nodes do not ack on boot.** `DriverHostActor.SendAck` is called only from the apply paths —
 a node with no dispatch to answer has nothing to send, so there is no drop to observe.
 What was verified: with **both** centrals stopped, `site-a-1` restarted, came up clean, logged its
 association failures against both central addresses, retried, and stayed up — no crash loop. The
 node-side ClusterClient's forever-retry (`reconnect-timeout = off`) behaves as configured.
 ## Step 11 — the canary stayed quiet
 `grep -ci "frame size|frame-size|maximum-frame"` = **0** on all six nodes, as designed: the deploy
 notify carries only `DeploymentId` + `RevisionHash` + `CorrelationId`.
 ---
 ## Finding A — a 10–30 s post-restart delivery window, **pre-existing, not Phase 2**
 A deploy issued shortly after a driver node restarts does not reach that node, and the deployment
 fails at the 2-minute deadline naming it. Measured against `site-a-1`:
 | Delay after the node logs readiness | Result |
 |---|---|
 | ~4 s | **missed** (`f738e1de` TimedOut, node row `Applying`) |
 | 10 s | **missed** (`0c7a6296` TimedOut) |
 | 30 s | applied, sealed |
 **The DPS control fails identically.** Flipping the rig back to `Mode=Dps` and repeating the 10 s
 case produced `e85f53c5` → TimedOut, `site-a-1` unacked. So the window is a property of cluster
 membership convergence, not of the transport this phase introduces. Recorded rather than fixed.
 > Running that control is the only reason this is a note and not a Phase 2 regression. The
 > temptation to skip it — "obviously the new transport broke it" — would have produced a false
 > attribution and a wasted investigation.
 ## Finding B — **`buffer-size = 0` does not deliver what this phase's docs claim it does**
 The stated rationale, in the plan, `CLAUDE.md`, and `docs/Configuration.md`, is that buffering
 *"would replay a `DispatchDeployment` on reconnect and apply a deployment the coordinator already
 sealed as `TimedOut`, leaving the node running a configuration the database records as failed."*
 **That outcome happens anyway, by a different route.** Deployment `0c7a6296` timed out at 22:14:30
 with `site-a-1` unacked. The node's row:
 ```
 NodeId          Status  StartedAtUtc              AppliedAtUtc
 site-a-1:4053   1       2026-07-22 22:12:30.145   2026-07-22 22:15:14.823
 ```
 It applied **44 s after the deployment was sealed TimedOut**, on its next restart. The mechanism is
 `DriverHostActor.Bootstrap()`, which reads the node's most recent `NodeDeploymentState`, finds the
 orphan `Applying` row the coordinator pre-created at dispatch, and replays it:
 ```csharp
 case NodeDeploymentStatus.Applying:
    _log.Warning("DriverHost {Node}: found orphan Applying row for deployment {Id}; replaying", …);
    ApplyAndAck(new DeploymentId(latest.DeploymentId), revision.Value, CorrelationId.NewId());
 ```
 So `buffer-size = 0` removes the **transport-level, timer-driven** replay — a real difference: it is
 silent, unbounded in time, and can fire without any node event. The **boot-time DB replay remains**,
 and it is the better-behaved of the two (deliberate, Warning-logged, and bounded to a restart). The
 decision stands; the *justification* was overstated and the docs are corrected in the same change as
 this record.
 ## Finding C — stopping **both** centrals strands the non-seed nodes permanently
 After `docker stop` of both centrals followed by `docker start`, the fleet did **not** recover.
 Deployments reached 2/6 then 3/6 nodes. The cause is not the transport — the Akka cluster itself had
 split:
 ```
 Cluster hosts:  MEMBERS 3   UP 3   UNREACHABLE 0
  central-1:4053  UP   admin,driver   leader for admin+driver
  central-2:4053  UP   admin,driver
  site-a-1:4053   UP   driver
 ```
 The three site nodes that never restarted were **not members at all**, and their own gossip showed
 the stale inverse view — `central-2` as `Exiting`, `site-a-1` as `Leaving` — frozen since 22:29:
 ```
 Gossip(members = [central-2 status = Exiting, site-a-1 status = Leaving,
                  site-a-2 Up, site-b-1 Up, site-b-2 Up])
 ```
 A graceful stop puts the centrals into `Leaving`/`Exiting`; the processes died before the removal
 handshake completed, leaving the survivors holding members that will never confirm. `auto-down`
 does not help — these members are not *Unreachable*, they are stuck mid-`Exiting`. The restarted
 centrals formed a fresh cluster (with `site-a-1`, which had also restarted and re-joined through its
 seed), and the three stranded nodes had no path back: they believe they are already in a cluster, so
 they never re-run the join process.
 **Recovery required restarting the stranded nodes.** After that, the next deploy sealed 6/6
 (`e20836ac`).
 This is transport-independent — membership is below both DPS and ClusterClient, and the set of nodes
 that acked matched the cluster view exactly. It is a **downing/rejoin gap of the current single-mesh
 topology**, in the same family as the registered outage gap the design doc already records, and it
 belongs to the Phase 7 drill work. Under the target per-pair topology (every node a seed of its own
 pair) the shape changes, and this drill must be re-run there rather than assumed fixed.
 ## Finding D — the AdminUI's driver Restart/Reconnect buttons have no page
 Step 6 could not be run, and not for a rig reason.
 `Components/Shared/Drivers/DriverStatusPanel.razor` is the **only** component in the codebase
 carrying the `DriverOperator`-gated Reconnect/Restart buttons, and **nothing renders it** — `grep -rn
 "<DriverStatusPanel"` across `src/` returns nothing, and no `@page` route under `Components/Pages/`
 hosts it. It was orphaned when v3 Batch 2 retired the routed `/clusters/{id}/drivers` flow.
 So the `driver-control` channel has no operator surface **on either transport** — this is a
 pre-existing v3 regression that Phase 2 merely surfaced. Filed as a follow-up; not fixed here.
 **Residual risk for the channel:** central-side dispatch is sabotage-verified by
 `MeshRouterDispatchTests` (all four `AdminOperationsActor` call sites go through one `Dispatch`
 helper); node-side `EventStream` receipt by `DriverHostActor` is proven **live** by this gate, since
 `DispatchDeployment` arrives through the identical `PublishLocally` path. What is unproven live is
 only the composition for those two message types.
 Step 7 (alarm ack) was likewise not run: the rig has no scripted alarms configured
 (`ScriptedAlarmHost: applying (enabled=0/0, depRefs=0)`) and `/alerts` is empty, so there is nothing
 to acknowledge. Authoring a scripted alarm end-to-end was judged out of proportion to the residual
 risk above. `ScriptedAlarmHostActor`'s `AlarmCommand` EventStream subscription therefore rests on its
 (sabotage-verified) unit test.
 ## Finding E — the central comm actor is silent on the success path
 `HandleApplyAck` logs only on failure. During this gate that made "did the ack reach the surviving
 central?" unanswerable from logs, which is exactly the question an operator asks during a failover.
 A Debug line on ack receipt would have made step 8 directly observable. Not added during the gate —
 adding code to make a gate step pass is the wrong order — but recorded as a follow-up.
 ---
 ## What this gate did not cover
 - **Steps 6 and 7** — see findings D and E; no surface exists for driver-control, no alarms for ack.
 - **The `alarm-commands` node-side leg** under a real transport, for the reason above.
 - **Split meshes.** Everything here ran on the single mesh. The contact set was fleet-wide (6 points,
  one client), which is the correct and only shape until Phase 6 splits the meshes.
 - **A network partition** (as opposed to process stops). `auto-down`'s dual-active trade is untested
  here and remains Phase 7's.
@@ -99,7 +99,35 @@ this removes the coordinator's one genuinely mesh-bound dependency).
 set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
 missing; a node up but row absent → its ack is not expected).
-### Phase 2 — Comm actors + ClusterClient transport
+### Phase 2 — Comm actors + ClusterClient transport — **DONE 2026-07-22 (live gate PASSED)**
 Shipped per [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md)
 as a **dark switch**: `MeshTransport:Mode` defaults to `Dps`, so nothing changes on any deployment
 until the flag flips. The live gate ([`2026-07-22-mesh-phase2-live-gate.md`](2026-07-22-mesh-phase2-live-gate.md))
 flipped the whole rig to `ClusterClient` and passed for the transport, with four findings — most
 notably that `buffer-size = 0`'s stated rationale was overstated (a node applies a `TimedOut`
 deployment anyway via boot-time orphan-row replay) and that stopping both centrals strands non-seed
 nodes at the membership layer (a Phase 7 drill item, transport-independent). Both comm actors are spawned and receptionist-registered in **both** modes, so
 the flip is a config change rather than a redeploy. Three scope corrections landed against the
 design doc (that plan's header carries them in full):
 - the node path is `/user/node-communication`, not `/user/cluster-communication` (there is no
  per-cluster scoping until Phase 6);
 - the substitution for `Publish` is **`SendToAll`**, not `Send` — `Send` reaches exactly one node and
  would deploy to one member of the fleet while the rest silently kept their old config;
 - central creates **exactly one fleet-wide ClusterClient**. On a single mesh a receptionist serves
  its whole cluster, so a client per application `Cluster` would fan every command out once per
  cluster. Per-cluster clients become correct only when Phase 6 splits the meshes (`TODO(Phase 6)`
  in `CentralCommunicationActor.RebuildClient`).
 **Exit-gate deviation — "an Ask timing out cleanly against a stopped node" cannot be run as written.**
 There is no cross-boundary Ask in Phase 2: every migrated command is fire-and-forget on the wire.
 `DispatchDeployment` is a `Tell` whose ack returns later as an independent `ApplyAck`;
 `RestartDriver` / `ReconnectDriver` / `AlarmCommand` reply `Ok = true` from the admin node the moment
 they are dispatched (`Ok` has always meant *dispatched*, not *applied* — the AdminUI string is
 literally "Restart dispatched"). The honest equivalents, both in the live gate: a **stopped node**
 fails the deploy at the apply deadline naming it (Phase 1 semantics), and **no reachable contact**
 drops the command with a Warning and never buffers it.
 **Scope:** one receptionist-registered actor per side (`/user/central-communication`,
 `/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation);
 ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
@@ -198,9 +226,9 @@ resource sizing on the site VMs should be checked once both products run the ful
 |---|---|
 | 0a downing strategy | DONE 2026-07-21 (live gate → Phase 7) |
 | 0b oldest-Up election | DONE 2026-07-21 |
-| Prereq: selfform-fallback + manual-failover plan | plan written 2026-07-22, not executed |
+| Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
-| 1 ClusterNode columns + DB ack set | **detailed plan ALREADY WRITTEN** (`2026-07-21-per-cluster-mesh-phase1.md`, from the design session — do NOT write a new one), not executed. ⚠ Carries a decision gate: DB-derived expected-ack set changes deploy-seal semantics (a stopped-but-enabled node now FAILS the deploy at the apply deadline instead of sealing green without it; escape hatch = `ClusterNode.Enabled=0`) — confirm with the owner before its Task 3. |
+| 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
-| 2 ClusterClient transport | not started |
+| 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
 | 3 fetch-and-cache | not started |
 | 4 cut driver ConfigDb | not started |
 | 5 gRPC telemetry | not started |
@@ -2,9 +2,9 @@
  "planPath": "docs/plans/2026-07-22-per-cluster-mesh-program.md",
  "note": "PROGRAM plan: each task = write that phase's detailed plan (writing-plans), execute it (executing-plans), run its exit gate, update the tracking tables. Prereq task 0 is the separate selfform-fallback plan in this repo.",
  "tasks": [
-    {"id": 0, "subject": "Prereq: execute 2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks)", "status": "pending"},
+    {"id": 0, "subject": "Prereq: execute 2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks)", "status": "completed", "note": "Shipped as self-first seed ordering + AkkaClusterOptionsValidator, NOT the planned SelfFormAfter watchdog (live gate caught it islanding a failed-over node). Merged a78425ea."},
-    {"id": 1, "subject": "Phase 1: ClusterNode Akka+gRPC address columns; ConfigPublishCoordinator ack set from DB", "status": "pending"},
+    {"id": 1, "subject": "Phase 1: ClusterNode Akka+gRPC address columns; ConfigPublishCoordinator ack set from DB", "status": "completed", "note": "Merged 7654f24d. Escape hatch is ClusterNode.MaintenanceMode, not Enabled=0."},
-    {"id": 2, "subject": "Phase 2: comm actors + receptionist + ClusterClient transport (deploy notify/acks, driver-control)", "status": "pending", "blockedBy": [1]},
+    {"id": 2, "subject": "Phase 2: comm actors + receptionist + ClusterClient transport (deploy notify/acks, driver-control)", "status": "in_progress", "blockedBy": [1]},
    {"id": 3, "subject": "Phase 3: config fetch-and-cache from central; LocalDb steady-state config store (live gate)", "status": "pending", "blockedBy": [2]},
    {"id": 4, "subject": "Phase 4: cut driver-side ConfigDb — EfAlarmConditionStateStore to LocalDb, DbHealthProbeActor ServiceLevel input, OpcUaPublishActor audit (live gate)", "status": "pending", "blockedBy": [3]},
    {"id": 5, "subject": "Phase 5: gRPC oneof stream contract; migrate 7 observability topics; AdminUI reconnect story", "status": "pending", "blockedBy": [2]},
@@ -0,0 +1,61 @@
 namespace ZB.MOM.WW.OtOpcUa.Cluster;
 /// <summary>
 ///     Selects and configures the transport carrying central→node commands and node→central
 ///     deployment acks (per-cluster mesh Phase 2).
 /// </summary>
 /// <remarks>
 ///     <para>
 ///         <see cref="Mode"/> is a deliberate <b>dark switch</b>, not a permanent knob: both
 ///         transports live in the tree for one phase so the docker-dev rig can run the same
 ///         deployment over each and compare, and so a bad Phase 2 rolls back with a config change
 ///         rather than a revert. Phase 6 deletes the DistributedPubSub branch along with the single
 ///         mesh that made it viable.
 ///     </para>
 ///     <para>
 ///         <b>Discovery is deliberately asymmetric</b>, matching the sister project. Central
 ///         discovers nodes <i>from the database</i> — enabled, non-maintenance <c>ClusterNode</c>
 ///         rows, rebuilt every <see cref="ContactRefreshSeconds"/>. Nodes discover central <i>from
 ///         this section</i>: static, restart to change. The asymmetry follows the rate of change —
 ///         operators add and retire nodes, but central's own address is part of the deployment.
 ///     </para>
 /// </remarks>
 public sealed class MeshTransportOptions
 {
    /// <summary>Configuration section name.</summary>
    public const string SectionName = "MeshTransport";
    /// <summary>Every command stays on DistributedPubSub — the pre-Phase-2 behaviour.</summary>
    public const string ModeDps = "Dps";
    /// <summary>Commands cross the ClusterClient receptionist boundary.</summary>
    public const string ModeClusterClient = "ClusterClient";
    /// <summary>
    ///     <see cref="ModeDps"/> (default) or <see cref="ModeClusterClient"/>. Any other value fails
    ///     the host at startup rather than silently falling back — a transport that is neither of
    ///     these delivers nothing, and the symptom would be commands that vanish.
    /// </summary>
    public string Mode { get; set; } = ModeDps;
    /// <summary>
    ///     Akka remoting addresses of the central nodes, e.g.
    ///     <c>akka.tcp://otopcua@central-1:4053</c>. <b>Node addresses only</b> — the
    ///     <c>/system/receptionist</c> path is appended at construction time.
    /// </summary>
    /// <remarks>
    ///     Each entry must be a <i>central</i> node's remoting endpoint, never this node's own port.
    ///     The sister project shipped a site template whose second contact was the site's own
    ///     remoting port; it is a permanent failure in the initial-contact rotation and stays
    ///     invisible until a command goes missing. <see cref="MeshTransportOptionsValidator"/>
    ///     rejects the malformed shapes at boot instead.
    /// </remarks>
    public string[] CentralContactPoints { get; set; } = [];
    /// <summary>
    ///     How often central re-reads <c>ClusterNode</c> rows to rebuild its contact points.
    ///     Refreshing on a timer (rather than only on admin change) means a row edited by any
    ///     process — migration, direct SQL, a second admin node — is picked up without a restart.
    /// </summary>
    public int ContactRefreshSeconds { get; set; } = 60;
 }
@@ -0,0 +1,83 @@
 using Microsoft.Extensions.Options;
 namespace ZB.MOM.WW.OtOpcUa.Cluster;
 /// <summary>
 ///     Fails the host at startup on a <see cref="MeshTransportOptions"/> shape that would leave
 ///     commands silently undelivered.
 /// </summary>
 /// <remarks>
 ///     Every fault caught here otherwise surfaces as an <i>absence</i> — a deployment that never
 ///     arrives, an alarm acknowledgement that does nothing, a node whose ack is dropped without a
 ///     trace. An absence has no stack trace and no failing node to point at, which makes it the
 ///     most expensive class of misconfiguration to diagnose in the field. Refusing to start is
 ///     cheaper for everyone.
 /// </remarks>
 public sealed class MeshTransportOptionsValidator : IValidateOptions<MeshTransportOptions>
 {
    /// <inheritdoc />
    public ValidateOptionsResult Validate(string? name, MeshTransportOptions options)
    {
        ArgumentNullException.ThrowIfNull(options);
        var errors = new List<string>();
        var isDps = string.Equals(options.Mode, MeshTransportOptions.ModeDps, StringComparison.OrdinalIgnoreCase);
        var isClusterClient = string.Equals(
            options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
        if (!isDps && !isClusterClient)
        {
            errors.Add(
                $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)} is "
                + $"'{options.Mode}'. Expected '{MeshTransportOptions.ModeDps}' or "
                + $"'{MeshTransportOptions.ModeClusterClient}'.");
        }
        if (options.ContactRefreshSeconds <= 0)
        {
            errors.Add(
                $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.ContactRefreshSeconds)} "
                + $"must be greater than 0 (was {options.ContactRefreshSeconds}). Central rebuilds its "
                + "ClusterClient contact points on this interval; a non-positive value would leave the "
                + "contact set frozen at whatever it held when the node booted.");
        }
        // Contact points are only load-bearing under ClusterClient mode. Requiring them under the
        // default would make the section mandatory everywhere for a feature that is switched off —
        // including on admin-only nodes, which have no reason to carry them at all.
        if (isClusterClient)
        {
            if (options.CentralContactPoints.Length == 0)
            {
                errors.Add(
                    $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.CentralContactPoints)} "
                    + "is empty. Under ClusterClient mode a driver node cannot reach central without at "
                    + "least one contact point, and its deployment acks would be dropped silently — the "
                    + "deployment would then time out naming a node that is running perfectly.");
            }
            foreach (var contactPoint in options.CentralContactPoints)
            {
                if (!contactPoint.StartsWith("akka.tcp://", StringComparison.Ordinal))
                {
                    errors.Add(
                        $"Contact point '{contactPoint}' must start with 'akka.tcp://' — it is an Akka "
                        + "remoting address (akka.tcp://<system>@<host>:<port>), not a bare host:port.");
                }
                else if (contactPoint.Contains("/user/", StringComparison.Ordinal)
                         || contactPoint.Contains("/system/", StringComparison.Ordinal))
                {
                    errors.Add(
                        $"Contact point '{contactPoint}' carries an actor-path suffix. Configure the "
                        + "node address only — '/system/receptionist' is appended at construction time, "
                        + "and a doubled suffix produces a path that never resolves.");
                }
            }
        }
        return errors.Count == 0
            ? ValidateOptionsResult.Success
            : ValidateOptionsResult.Fail(string.Join(" ", errors));
    }
 }
@@ -15,6 +15,12 @@ akka {
    extensions = [
        "Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider, Akka.Cluster.Tools"
        # Per-cluster mesh Phase 2 — the central<->node command boundary. NOT auto-started: without
        # this entry the receptionist materialises only if some code path happens to call
        # ClusterClientReceptionist.Get(system) first, which makes "is the boundary listening?"
        # depend on startup ordering rather than on configuration.
        "Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"
    ]
    actor {
@@ -25,6 +31,20 @@ akka {
        dot-netty.tcp {
            hostname = "0.0.0.0"
            port = 4053
            # FRAME SIZE. maximum-frame-size restates Akka's own default; what actually changes is
            # log-frame-size-exceeding, which defaults to `off`. Over the limit the transport drops
            # that ONE message without tearing down the association — heartbeats keep flowing, the
            # node still reports healthy, and the only symptom is a command that never arrived. The
            # sister project lost a deploy path to exactly this and could not attribute it; their
            # postmortem also notes the default JSON serializer escapes an already-serialized
            # payload a second time, so ~60-70 KB of real JSON is enough to blow 128 KB.
            #
            # OtOpcUa's deploy notify is payload-free by design (DispatchDeployment carries only
            # DeploymentId + RevisionHash + CorrelationId), so this threshold should stay silent.
            # It is a canary, and a quiet one is the point.
            maximum-frame-size = 128000b
            log-frame-size-exceeding = 32000b
        }
        transport-failure-detector {
            heartbeat-interval = 2s
@@ -74,6 +94,31 @@ akka {
        down-removal-margin = 15s
        run-coordinated-shutdown-when-down = on
        # CLUSTERCLIENT — the central<->node command boundary (per-cluster mesh Phase 2).
        client {
            # DELIBERATE, and NOT what the sister project runs: they never wrote this section, so
            # they inherit buffer-size = 1000. Zero means "if the receptionist is unknown, drop the
            # message now". Buffering would replay a DispatchDeployment on reconnect and apply a
            # deployment ConfigPublishCoordinator had already sealed as TimedOut — the node would
            # then be running a configuration the database says failed. Fail-fast keeps the DB the
            # single record of truth, and reproduces today's DPS behaviour exactly: a node that is
            # down misses the publish and self-heals from the DB on restart.
            buffer-size = 0
            # Retry forever rather than self-stopping the client. Restates Akka's default; stated
            # explicitly because the behaviour is load-bearing — a central node down for an hour
            # must not require a driver-node restart to become reachable again.
            reconnect-timeout = off
            receptionist {
                # Empty = every member hosts a receptionist. This is what makes the comm actors
                # "registered per node, not as a singleton" work: contact rotation reaches whichever
                # node answers. Restates Akka's default; do NOT scope this to a role, or the
                # boundary becomes reachable through one node only and rotation silently fails.
                role = ""
            }
        }
        singleton {
            singleton-name = "singleton"
        }
@@ -34,6 +34,12 @@ public static class ServiceCollectionExtensions
        services.AddValidatedOptions<AkkaClusterOptions, AkkaClusterOptionsValidator>(
            configuration, AkkaClusterOptions.SectionName);
        // Per-cluster mesh Phase 2: which transport carries central→node commands. Validated at
        // startup for the same reason the cluster options are — a contact point that cannot resolve
        // produces silence, not an error, and silence is indistinguishable from "nothing to do".
        services.AddValidatedOptions<MeshTransportOptions, MeshTransportOptionsValidator>(
            configuration, MeshTransportOptions.SectionName);
        services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
        return services;
@@ -0,0 +1,28 @@
 namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 /// <summary>
 ///     A central→node command, plus the DistributedPubSub topic it would have been published on.
 /// </summary>
 /// <param name="Topic">
 ///     The legacy DPS topic. Carried so the router can publish under
 ///     <c>MeshTransport:Mode = Dps</c> without a second dispatch table; ignored entirely under
 ///     <c>ClusterClient</c> mode, where the destination is an actor path rather than a topic.
 /// </param>
 /// <param name="Message">
 ///     The command itself, never wrapped. Node-side handlers see exactly the type they see today,
 ///     which is what lets the dark switch touch publishers only.
 /// </param>
 /// <remarks>
 ///     <para>
 ///         This envelope exists so the admin singletons stop knowing which transport is in force.
 ///         Before per-cluster mesh Phase 2 each publisher held its own
 ///         <c>DistributedPubSub.Get(Context.System).Mediator</c> call, so the switch would have had
 ///         to be threaded through five call sites across two actors, and each would have been free
 ///         to drift.
 ///     </para>
 ///     <para>
 ///         Central-side only — it never crosses the wire, so it carries no serialization contract
 ///         and needs no additive-evolution discipline.
 ///     </para>
 /// </remarks>
 public sealed record MeshCommand(string Topic, object Message);
@@ -0,0 +1,33 @@
 namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 /// <summary>
 ///     Actor paths the two mesh comm actors register under. <b>These strings are the wire
 ///     contract</b> of the central↔node boundary.
 /// </summary>
 /// <remarks>
 ///     <para>
 ///         Declared once, here, rather than as literals on each side. The <c>deployments</c> /
 ///         <c>deployment-acks</c> topic names are declared twice — once in
 ///         <c>ConfigPublishCoordinator</c> and again in <c>DriverHostActor</c> — and agree only by
 ///         convention; renaming one would compile, deploy, and deliver nothing. A ClusterClient path
 ///         has the same failure mode and no compiler to catch it, so the constant is shared.
 ///     </para>
 ///     <para>
 ///         Nothing else pins these values, which is why <c>MeshCommActorPathTests</c> boots a real
 ///         host and <c>Identify</c>-probes both paths.
 ///     </para>
 /// </remarks>
 public static class MeshPaths
 {
    /// <summary>Central-side comm actor — where a driver node sends its deployment acks.</summary>
    public const string CentralCommunication = "/user/central-communication";
    /// <summary>Node-side comm actor — where central sends commands.</summary>
    public const string NodeCommunication = "/user/node-communication";
    /// <summary>Actor name of <see cref="CentralCommunication"/> (the path's final segment).</summary>
    public const string CentralCommunicationName = "central-communication";
    /// <summary>Actor name of <see cref="NodeCommunication"/> (the path's final segment).</summary>
    public const string NodeCommunicationName = "node-communication";
 }
@@ -4,6 +4,7 @@ using Akka.Event;
 using Microsoft.EntityFrameworkCore;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
 using ZB.MOM.WW.OtOpcUa.Commons.Types;
 using ZB.MOM.WW.OtOpcUa.Configuration;
@@ -26,6 +27,13 @@ public sealed class AdminOperationsActor : ReceiveActor
    private readonly IActorRef _coordinator;
    private readonly IReadOnlyDictionary<string, IDriverProbe> _probesByType;
    private readonly TagConfigValidationMode _tagConfigValidationMode;
    /// <summary>
    ///     Where central→node commands are handed off (per-cluster mesh Phase 2), or
    ///     <see langword="null"/> to publish on the mediator directly.
    /// </summary>
    private readonly IActorRef? _meshRouter;
    private readonly ILoggingAdapter _log = Context.GetLogger();
    /// <summary>Creates actor props for the admin operations actor.</summary>
@@ -33,29 +41,38 @@ public sealed class AdminOperationsActor : ReceiveActor
    /// <param name="coordinator">Reference to the deployment coordinator actor.</param>
    /// <param name="probes">Driver probes registered in DI; keyed by DriverType (case-insensitive).</param>
    /// <param name="tagConfigValidationMode">Deploy-gate TagConfig strictness mode (Warn default | Error opt-in).</param>
    /// <param name="meshRouter">
    ///     Central comm actor commands are handed to, or <see langword="null"/> to publish on the
    ///     DistributedPubSub mediator directly (the pre-Phase-2 behaviour).
    /// </param>
    /// <returns>Props configured to create an AdminOperationsActor.</returns>
    public static Props Props(
        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
        IActorRef coordinator,
        IEnumerable<IDriverProbe> probes,
-        TagConfigValidationMode tagConfigValidationMode = TagConfigValidationMode.Warn) =>
+        TagConfigValidationMode tagConfigValidationMode = TagConfigValidationMode.Warn,
-        Akka.Actor.Props.Create(() => new AdminOperationsActor(dbFactory, coordinator, probes, tagConfigValidationMode));
+        IActorRef? meshRouter = null) =>
        Akka.Actor.Props.Create(() =>
            new AdminOperationsActor(dbFactory, coordinator, probes, tagConfigValidationMode, meshRouter));
    /// <summary>Initializes a new instance of the AdminOperationsActor.</summary>
    /// <param name="dbFactory">Factory for creating config database contexts.</param>
    /// <param name="coordinator">Reference to the deployment coordinator actor.</param>
    /// <param name="probes">Driver probes registered in DI; keyed by DriverType (case-insensitive).</param>
    /// <param name="tagConfigValidationMode">Deploy-gate TagConfig strictness mode (Warn default | Error opt-in).</param>
    /// <param name="meshRouter">Central comm actor, or <see langword="null"/> for the mediator.</param>
    public AdminOperationsActor(
        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
        IActorRef coordinator,
        IEnumerable<IDriverProbe> probes,
-        TagConfigValidationMode tagConfigValidationMode = TagConfigValidationMode.Warn)
+        TagConfigValidationMode tagConfigValidationMode = TagConfigValidationMode.Warn,
        IActorRef? meshRouter = null)
    {
        _dbFactory = dbFactory;
        _coordinator = coordinator;
        _probesByType = probes.ToDictionary(p => p.DriverType, StringComparer.OrdinalIgnoreCase);
        _tagConfigValidationMode = tagConfigValidationMode;
        _meshRouter = meshRouter;
        ReceiveAsync<StartDeployment>(HandleStartDeploymentAsync);
        ReceiveAsync<TestDriverConnect>(HandleTestDriverConnectAsync);
@@ -65,6 +82,23 @@ public sealed class AdminOperationsActor : ReceiveActor
        Receive<ShelveAlarmCommand>(HandleShelveAlarm);
    }
    /// <summary>
    ///     Hands a central→node command to the mesh transport router, or publishes it directly when
    ///     no router is wired.
    /// </summary>
    /// <remarks>
    ///     The fallback keeps ~12 existing tests (and any node wired before per-cluster mesh Phase 2)
    ///     on exactly the pre-Phase-2 behaviour. It is not a permanent seam: Phase 6 deletes it along
    ///     with the DistributedPubSub branch and the single mesh.
    /// </remarks>
    /// <param name="topic">The legacy DistributedPubSub topic.</param>
    /// <param name="message">The command.</param>
    private void Dispatch(string topic, object message)
    {
        if (_meshRouter is not null) _meshRouter.Tell(new MeshCommand(topic, message));
        else DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(topic, message));
    }
    /// <summary>
    ///     AdminUI Acknowledge path. Maps the control-plane command to a
    ///     <see cref="AlarmCommand"/> (<c>Operation = "Acknowledge"</c>) and publishes it onto the
@@ -85,7 +119,7 @@ public sealed class AdminOperationsActor : ReceiveActor
                Comment: msg.Comment,
                UnshelveAtUtc: null);
-            DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(AlarmCommandsTopic.Name, alarmCmd));
+            Dispatch(AlarmCommandsTopic.Name, alarmCmd);
            _log.Info("AdminOps: Acknowledge published for alarm {AlarmId} by {User}", msg.AlarmId, msg.User);
            replyTo.Tell(new AcknowledgeAlarmResult(true, null, msg.CorrelationId));
@@ -132,7 +166,7 @@ public sealed class AdminOperationsActor : ReceiveActor
                Comment: msg.Comment,
                UnshelveAtUtc: unshelveAt);
-            DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(AlarmCommandsTopic.Name, alarmCmd));
+            Dispatch(AlarmCommandsTopic.Name, alarmCmd);
            _log.Info("AdminOps: {Operation} published for alarm {AlarmId} by {User}", operation, msg.AlarmId, msg.User);
            replyTo.Tell(new ShelveAlarmResult(true, null, msg.CorrelationId));
@@ -394,7 +428,7 @@ public sealed class AdminOperationsActor : ReceiveActor
        {
            // Broadcast to every DriverHostActor on every node via the driver-control DPS topic.
            // Only the host that owns the instance will act; others ignore it (id not found in _children).
-            DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(DriverControlTopic.Name, msg));
+            Dispatch(DriverControlTopic.Name, msg);
            await using var db = await _dbFactory.CreateDbContextAsync();
            db.ConfigEdits.Add(new ConfigEdit
@@ -424,7 +458,7 @@ public sealed class AdminOperationsActor : ReceiveActor
        try
        {
            // Broadcast to every DriverHostActor; only the one owning the instance reacts.
-            DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(DriverControlTopic.Name, msg));
+            Dispatch(DriverControlTopic.Name, msg);
            await using var db = await _dbFactory.CreateDbContextAsync();
            db.ConfigEdits.Add(new ConfigEdit
@@ -0,0 +1,308 @@
 using System.Collections.Immutable;
 using Akka.Actor;
 using Akka.Cluster.Tools.Client;
 using Akka.Cluster.Tools.PublishSubscribe;
 using Akka.Event;
 using Microsoft.EntityFrameworkCore;
 using ZB.MOM.WW.OtOpcUa.Cluster;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 using ZB.MOM.WW.OtOpcUa.Configuration;
 namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
 /// <summary>
 ///     Central-side end of the central↔node command boundary (per-cluster mesh Phase 2). Routes
 ///     every central→node command out — over DistributedPubSub or ClusterClient, per
 ///     <see cref="MeshTransportOptions.Mode"/> — and forwards inbound <see cref="ApplyAck"/>s to the
 ///     deploy coordinator singleton.
 /// </summary>
 /// <remarks>
 ///     <para>
 ///         <b>Registered per node, NOT as a cluster singleton.</b> A driver node's ClusterClient
 ///         rotates across contact points and must find a live comm actor at whichever central node
 ///         answers. A singleton would be reachable only through the one node hosting it, and the
 ///         rotation would fail silently against the other.
 ///     </para>
 ///     <para>
 ///         <b>Exactly one ClusterClient, fleet-wide.</b> See <see cref="RebuildClient"/> — this is
 ///         the single most important thing to understand before changing this actor.
 ///     </para>
 /// </remarks>
 public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
 {
    private const string RefreshTimerKey = "mesh-contact-refresh";
    private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
    private readonly MeshTransportOptions _options;
    private readonly IMeshClusterClientFactory _clientFactory;
    private readonly Func<IActorRef?> _coordinator;
    private readonly ILoggingAdapter _log = Context.GetLogger();
    private readonly bool _useClusterClient;
    private IActorRef? _client;
    private ImmutableHashSet<string> _currentContacts = ImmutableHashSet<string>.Empty;
    private CancellationTokenSource _lifecycle = new();
    /// <summary>Gets the timer scheduler driving the periodic contact refresh.</summary>
    public ITimerScheduler Timers { get; set; } = null!;
    /// <summary>Creates the props for this actor.</summary>
    /// <param name="dbFactory">Factory for the config database holding <c>ClusterNode</c> rows.</param>
    /// <param name="options">The bound mesh-transport options.</param>
    /// <param name="clientFactory">Creates the ClusterClient; substituted in tests.</param>
    /// <param name="coordinator">
    ///     Resolves the deploy-coordinator singleton proxy. A <see cref="Func{TResult}"/> rather than
    ///     an <see cref="IActorRef"/> so this actor never depends on registration order at wiring
    ///     time — it resolves on first use and caches.
    /// </param>
    /// <returns>The props.</returns>
    public static Props Props(
        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
        MeshTransportOptions options,
        IMeshClusterClientFactory clientFactory,
        Func<IActorRef?> coordinator) =>
        Akka.Actor.Props.Create(() =>
            new CentralCommunicationActor(dbFactory, options, clientFactory, coordinator));
    /// <summary>Initializes a new instance of the <see cref="CentralCommunicationActor"/> class.</summary>
    /// <param name="dbFactory">Factory for the config database.</param>
    /// <param name="options">The bound mesh-transport options.</param>
    /// <param name="clientFactory">Creates the ClusterClient.</param>
    /// <param name="coordinator">Lazily resolves the deploy-coordinator singleton proxy.</param>
    public CentralCommunicationActor(
        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
        MeshTransportOptions options,
        IMeshClusterClientFactory clientFactory,
        Func<IActorRef?> coordinator)
    {
        _dbFactory = dbFactory;
        _options = options;
        _clientFactory = clientFactory;
        _coordinator = coordinator;
        _useClusterClient = string.Equals(
            options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
        Receive<MeshCommand>(HandleMeshCommand);
        Receive<ApplyAck>(HandleApplyAck);
        Receive<RefreshContacts>(_ => LoadContactsFromDb());
        Receive<ContactsLoaded>(HandleContactsLoaded);
        // A faulted LoadContactsFromDb task is piped here as a Status.Failure. Without this handler
        // the failure is an unhandled message (debug level only) and the refresh fails silently —
        // an operator cannot distinguish "no nodes configured" from "the database is down", and the
        // contact set silently freezes at whatever it last held.
        Receive<Status.Failure>(f => _log.Warning(
            f.Cause,
            "Failed to load ClusterNode contact points; the mesh ClusterClient contact set was NOT "
            + "refreshed and may be stale or empty. Commands may be dropped until the next refresh"));
        Receive<SubscribeAck>(_ => { /* DPS subscribe confirmation */ });
    }
    /// <inheritdoc />
    protected override void PreStart()
    {
        if (!_useClusterClient)
        {
            _log.Info(
                "Mesh transport is {Mode}: commands are published on DistributedPubSub and no "
                + "ClusterClient is created", _options.Mode);
            return;
        }
        Timers.StartPeriodicTimer(
            RefreshTimerKey,
            new RefreshContacts(),
            TimeSpan.Zero,
            TimeSpan.FromSeconds(_options.ContactRefreshSeconds));
    }
    /// <inheritdoc />
    protected override void PostStop()
    {
        _lifecycle.Cancel();
        _lifecycle.Dispose();
    }
    private void HandleMeshCommand(MeshCommand cmd)
    {
        if (!_useClusterClient)
        {
            DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(cmd.Topic, cmd.Message));
            return;
        }
        if (_client is null)
        {
            // App-level drop, matching the sister project's decision and this repo's buffer-size = 0.
            // The command is NOT queued and will NOT be retried: a deploy fails at the coordinator's
            // apply deadline naming the silent nodes, which is a better operator signal than a
            // command arriving minutes late against a config the DB already recorded as failed.
            _log.Warning(
                "No mesh ClusterClient — dropping {MessageType}. The last contact refresh produced "
                + "no usable contact point from the enabled, non-maintenance ClusterNode rows; the "
                + "command is not buffered and will not be retried",
                cmd.Message.GetType().Name);
            return;
        }
        // SendToAll, NEVER Send. Today's DPS publish reaches EVERY DriverHostActor — there is no
        // ClusterId or node filter on the node side at all; scoping happens later, inside the
        // artifact (DeploymentArtifact.ResolveClusterScope). ClusterClient.Send delivers to exactly
        // ONE registered actor, which would deploy to a single node of the fleet while every other
        // node silently kept its old configuration — and the deployment would still seal green if
        // the ack set happened to be satisfied.
        _client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, cmd.Message));
    }
    private void HandleApplyAck(ApplyAck ack)
    {
        var coordinator = _coordinator();
        if (coordinator is null)
        {
            _log.Warning(
                "Received ApplyAck for {DeploymentId} from {NodeId} but the deploy coordinator is "
                + "not resolvable on this node; the ack is lost and the deployment will time out "
                + "naming a node that applied successfully",
                ack.DeploymentId, ack.NodeId);
            return;
        }
        // Forward, not Tell: preserves the original sender across the hop so the coordinator sees
        // the node rather than this relay.
        coordinator.Forward(ack);
    }
    private void LoadContactsFromDb()
    {
        var self = Self;
        CancellationToken ct;
        try
        {
            ct = _lifecycle.Token;
        }
        catch (ObjectDisposedException)
        {
            return; // Stopping.
        }
        var systemName = Context.System.Name;
        var dbFactory = _dbFactory;
        // Off the actor thread, result piped back as a message. A synchronous DB read here would
        // block every command dispatch behind a slow or unreachable SQL Server.
        Task.Run(async () =>
        {
            await using var db = await dbFactory.CreateDbContextAsync(ct).ConfigureAwait(false);
            var rows = await db.ClusterNodes
                .AsNoTracking()
                .Where(n => n.Enabled && !n.MaintenanceMode)
                .Select(n => new { n.NodeId, n.Host, n.AkkaPort })
                .ToListAsync(ct)
                .ConfigureAwait(false);
            var contacts = new List<string>(rows.Count);
            var malformed = new List<string>();
            foreach (var row in rows)
            {
                var address = $"akka.tcp://{systemName}@{row.Host}:{row.AkkaPort}/system/receptionist";
                // Parse up front, per row, inside the loop's own guard. A single malformed row must
                // not abort the whole refresh and leave the contact set half-built — the sister
                // project shipped that bug and its regression test deliberately orders the bad row
                // first.
                if (ActorPath.TryParse(address, out _)) contacts.Add(address);
                else malformed.Add($"{row.NodeId} -> {address}");
            }
            return new ContactsLoaded(contacts, malformed);
        }, ct).PipeTo(self);
    }
    private void HandleContactsLoaded(ContactsLoaded msg)
    {
        foreach (var bad in msg.Malformed)
        {
            _log.Warning(
                "ClusterNode {Row} does not yield a usable receptionist address; skipping it in this "
                + "refresh (other nodes are unaffected). Check the row's Host and AkkaPort", bad);
        }
        var contacts = msg.Contacts.ToImmutableHashSet();
        if (contacts.IsEmpty)
        {
            _log.Warning(
                "No usable ClusterNode contact points; the mesh ClusterClient was not created and "
                + "every central→node command will be dropped until a refresh finds one");
            return;
        }
        if (_client is not null && _currentContacts.SetEquals(contacts)) return;
        RebuildClient(contacts);
    }
    /// <summary>Stops any existing client and creates a replacement for the new contact set.</summary>
    /// <remarks>
    ///     <para>
    ///         <b>TODO(Phase 6): one client per application <c>Cluster</c>.</b> Today this actor
    ///         creates exactly <b>one</b> client whose contacts are the whole fleet, and that is not
    ///         a compromise — it is the only correct shape while the fleet is a single Akka mesh.
    ///     </para>
    ///     <para>
    ///         A <c>ClusterClientReceptionist</c> serves its entire cluster, so
    ///         <c>SendToAll</c> reaches every registered node-comm actor in the mesh <i>regardless of
    ///         which node's address was used as the contact point</i>. One client per Cluster would
    ///         therefore fan each command out once per cluster — N× duplicate
    ///         <c>DispatchDeployment</c> and N× duplicate <c>ApplyAck</c>. Phase 6 splits the meshes,
    ///         at which point per-cluster clients become both correct and necessary.
    ///     </para>
    ///     <para>
    ///         Corollary worth stating because it is easy to assume otherwise: <b>the contact set
    ///         does not scope delivery.</b> Excluding a maintenance-mode node from the contacts does
    ///         not stop it receiving commands — it still receives them, exactly as it does under
    ///         today's DPS broadcast. The filter is about which receptionists are worth <i>dialling</i>
    ///         (a node in maintenance may be switched off), not about who gets the message.
    ///     </para>
    /// </remarks>
    /// <param name="contacts">The receptionist addresses to dial.</param>
    private void RebuildClient(ImmutableHashSet<string> contacts)
    {
        if (_client is not null)
        {
            _log.Info("Mesh contact set changed; replacing the ClusterClient");
            Context.Stop(_client);
            // Cleared now: if the create below throws, a stale ref would route commands into a
            // stopping actor and they would vanish without a warning.
            _client = null;
            _currentContacts = ImmutableHashSet<string>.Empty;
        }
        try
        {
            var paths = contacts.Select(ActorPath.Parse).ToImmutableHashSet();
            _client = _clientFactory.Create(Context.System, paths);
            _currentContacts = contacts;
            _log.Info("Mesh ClusterClient created with {Count} contact point(s)", contacts.Count);
        }
        catch (Exception ex)
        {
            _log.Error(ex,
                "Failed to create the mesh ClusterClient; central→node commands are dropped until "
                + "the next contact refresh succeeds");
        }
    }
    /// <summary>Internal tick asking for a contact-point refresh.</summary>
    public sealed record RefreshContacts;
    /// <summary>Result of a contact-point load, piped back from the DB read.</summary>
    /// <param name="Contacts">Receptionist addresses that parsed cleanly.</param>
    /// <param name="Malformed">Rows that did not, described for the log.</param>
    public sealed record ContactsLoaded(
        IReadOnlyList<string> Contacts,
        IReadOnlyList<string> Malformed);
 }
@@ -0,0 +1,47 @@
 using System.Collections.Immutable;
 using Akka.Actor;
 using Akka.Cluster.Tools.Client;
 namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
 /// <summary>
 ///     Creates the <c>ClusterClient</c> central uses to reach driver nodes. Exists as a seam so
 ///     <c>CentralCommunicationActor</c> can be tested against a <c>TestProbe</c> without standing up
 ///     a second cluster.
 /// </summary>
 public interface IMeshClusterClientFactory
 {
    /// <summary>Creates a client for the given receptionist contact points.</summary>
    /// <param name="system">The actor system to spawn the client in.</param>
    /// <param name="contacts">Receptionist actor paths, one per reachable node.</param>
    /// <returns>The new client's <see cref="IActorRef"/>.</returns>
    IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts);
 }
 /// <inheritdoc />
 public sealed class DefaultMeshClusterClientFactory : IMeshClusterClientFactory
 {
    /// <summary>
    ///     Per-incarnation generation counter, appended to the actor name.
    /// </summary>
    /// <remarks>
    ///     <c>Context.Stop</c> of the previous client is <b>asynchronous</b> — its actor name stays
    ///     reserved until termination completes — so recreating the same name while handling a single
    ///     message throws <see cref="InvalidActorNameException"/>. That is exactly what a contact-set
    ///     change does: stop the old client, create the replacement, both in one handler. A
    ///     generation suffix makes every incarnation's name unique by construction. Ported from the
    ///     sister project, where it was a shipped bug fix rather than foresight.
    /// </remarks>
    private long _generation;
    /// <inheritdoc />
    public IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts)
    {
        ArgumentNullException.ThrowIfNull(system);
        ArgumentNullException.ThrowIfNull(contacts);
        var settings = ClusterClientSettings.Create(system).WithInitialContacts(contacts);
        var name = $"mesh-client-{Interlocked.Increment(ref _generation)}";
        return system.ActorOf(ClusterClient.Props(settings), name);
    }
 }
@@ -3,6 +3,7 @@ using Akka.Cluster.Tools.PublishSubscribe;
 using Akka.Event;
 using Microsoft.EntityFrameworkCore;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 using ZB.MOM.WW.OtOpcUa.Commons.Types;
 using ZB.MOM.WW.OtOpcUa.Configuration;
 using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
@@ -60,6 +61,17 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
    private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
    private readonly TimeSpan _applyDeadline;
    /// <summary>
    ///     Where central→node commands are handed off (per-cluster mesh Phase 2), or
    ///     <see langword="null"/> to publish on the mediator directly.
    /// </summary>
    /// <remarks>
    ///     Null in tests and on any node wired before Phase 2, where the fallback reproduces the
    ///     pre-Phase-2 behaviour exactly. The fallback is not a permanent seam — Phase 6 deletes it
    ///     along with the DistributedPubSub branch and the single mesh.
    /// </remarks>
    private readonly IActorRef? _meshRouter;
    private readonly ILoggingAdapter _log = Context.GetLogger();
    // NodeId equality here is case-insensitive (by Value) to match the case-insensitive ClusterId/
    // NodeId scoping in DeploymentArtifact.ResolveClusterScope — so an ack from a node whose address
@@ -78,22 +90,31 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
    /// <param name="dbFactory">The database context factory for accessing configuration data.</param>
    /// <param name="applyDeadline">The timeout for waiting for apply acknowledgments from cluster nodes.</param>
    /// <returns>The Akka.NET <see cref="Akka.Actor.Props"/> used to spawn this actor.</returns>
    /// <param name="meshRouter">
    ///     Central comm actor the dispatch is handed to, or <see langword="null"/> to publish on the
    ///     DistributedPubSub mediator directly (the pre-Phase-2 behaviour).
    /// </param>
    public static Props Props(
        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
-        TimeSpan? applyDeadline = null) =>
+        TimeSpan? applyDeadline = null,
-        Akka.Actor.Props.Create(() => new ConfigPublishCoordinator(dbFactory, applyDeadline ?? DefaultApplyDeadline));
+        IActorRef? meshRouter = null) =>
        Akka.Actor.Props.Create(() =>
            new ConfigPublishCoordinator(dbFactory, applyDeadline ?? DefaultApplyDeadline, meshRouter));
    /// <summary>
    ///     Initializes a new instance of the <see cref="ConfigPublishCoordinator"/> class.
    /// </summary>
    /// <param name="dbFactory">The database context factory for persisting deployment state.</param>
    /// <param name="applyDeadline">The timeout for waiting for per-node apply acknowledgments.</param>
    /// <param name="meshRouter">Central comm actor, or <see langword="null"/> for the mediator.</param>
    public ConfigPublishCoordinator(
        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
-        TimeSpan applyDeadline)
+        TimeSpan applyDeadline,
        IActorRef? meshRouter = null)
    {
        _dbFactory = dbFactory;
        _applyDeadline = applyDeadline;
        _meshRouter = meshRouter;
        Receive<DispatchDeployment>(HandleDispatch);
        Receive<ApplyAck>(HandleAck);
        Receive<DeadlineElapsed>(HandleDeadline);
@@ -172,7 +193,10 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
            db.SaveChanges();
        }
-        DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(DeploymentsTopic, msg));
+        // Per-cluster mesh Phase 2: hand off to the transport router rather than publishing
        // directly, so this actor no longer knows which transport is in force.
        if (_meshRouter is not null) _meshRouter.Tell(new MeshCommand(DeploymentsTopic, msg));
        else DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(DeploymentsTopic, msg));
        Timers.StartSingleTimer(DeadlineTimerKey, new DeadlineElapsed(msg.DeploymentId), _applyDeadline);
        if (_expectedAcks.Count == 0)
@@ -1,12 +1,17 @@
 using Akka.Actor;
 using Akka.Cluster.Hosting;
 using Akka.Cluster.Tools.Client;
 using Akka.Hosting;
 using Microsoft.EntityFrameworkCore;
 using Microsoft.Extensions.Configuration;
 using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.OtOpcUa.Configuration;
 using ZB.MOM.WW.OtOpcUa.Cluster;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.AdminOperations;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Audit;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
@@ -48,12 +53,49 @@ public static class ServiceCollectionExtensions
        var singletonOptions = new ClusterSingletonOptions { Role = AdminRole };
        var proxyOptions = new ClusterSingletonOptions { Role = AdminRole };
        // Per-cluster mesh Phase 2: the central end of the central↔node command boundary.
        //
        // A plain WithActors, NOT a singleton — and that is the load-bearing part. A driver node's
        // ClusterClient rotates across contact points and must find a live comm actor at whichever
        // central node answers. As a singleton it would exist on one central node only, and the
        // rotation would fail silently against the other: acks would land sometimes, depending on
        // which contact the client happened to settle on.
        builder.WithActors((system, registry, resolver) =>
        {
            var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
            var options = resolver.GetService<IOptions<MeshTransportOptions>>().Value;
            var actor = system.ActorOf(
                CentralCommunicationActor.Props(
                    dbFactory,
                    options,
                    new DefaultMeshClusterClientFactory(),
                    // Resolved lazily, per message, so this actor never depends on the order in
                    // which Akka.Hosting materialises the singleton proxy relative to this block.
                    () => registry.TryGet<ConfigPublishCoordinatorKey>(out var coordinator)
                        ? coordinator
                        : null),
                MeshPaths.CentralCommunicationName);
            // Registered unconditionally, in both transport modes. Under Dps nothing dials it, and
            // an idle registration costs nothing — whereas registering only under ClusterClient
            // would mean flipping the flag on a running fleet requires a central restart before any
            // node can ack, turning a config change back into a deployment.
            ClusterClientReceptionist.Get(system).RegisterService(actor);
            registry.Register<CentralCommunicationActorKey>(actor);
        });
        builder.WithSingleton<ConfigPublishCoordinatorKey>(
            ConfigPublishSingletonName,
            (system, registry, resolver) =>
            {
                var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
-                return ConfigPublishCoordinator.Props(dbFactory);
+                // Registered above, so it is already in the registry — the same
                // ordering-by-registration the AdminOperations singleton relies on to resolve the
                // coordinator. Deliberately NOT an ActorSelection.ResolveOne here: that blocks
                // inside a props factory and races the spawn it is waiting for.
                var meshRouter = registry.Get<CentralCommunicationActorKey>();
                return ConfigPublishCoordinator.Props(dbFactory, applyDeadline: null, meshRouter: meshRouter);
            },
            singletonOptions);
@@ -69,7 +111,8 @@ public static class ServiceCollectionExtensions
                var mode = Enum.TryParse<TagConfigValidationMode>(
                    resolver.GetService<IConfiguration>()?["Deployment:TagConfigValidationMode"],
                    ignoreCase: true, out var m) ? m : TagConfigValidationMode.Warn;
-                return AdminOperationsActor.Props(dbFactory, coordinator, probes, mode);
+                var meshRouter = registry.Get<CentralCommunicationActorKey>();
                return AdminOperationsActor.Props(dbFactory, coordinator, probes, mode, meshRouter);
            },
            singletonOptions);
@@ -123,3 +166,4 @@ public sealed class AuditWriterActorKey { }
 public sealed class FleetStatusBroadcasterKey { }
 public sealed class RedundancyStateActorKey { }
 public sealed class ClusterNodeAddressReconcilerKey { }
 public sealed class CentralCommunicationActorKey { }
@@ -0,0 +1,94 @@
 using Akka.Actor;
 using Akka.Cluster.Tools.Client;
 using Akka.Event;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
 namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
 /// <summary>
 ///     Node-side end of the central↔node command boundary (per-cluster mesh Phase 2), registered
 ///     with the <c>ClusterClientReceptionist</c> at <see cref="MeshPaths.NodeCommunication"/>.
 /// </summary>
 /// <remarks>
 ///     <para>
 ///         <b>Registered per node, NOT as a cluster singleton</b> — central's ClusterClient rotates
 ///         across contact points and must find a live comm actor at whichever node answers.
 ///     </para>
 ///     <para>
 ///         <b>Inbound commands are re-emitted on the node's <see cref="EventStream"/>, not on their
 ///         DistributedPubSub topic.</b> DPS is mesh-wide, and central <c>SendToAll</c>s to every
 ///         node's comm actor — so a DPS re-publish would deliver N copies of every command to every
 ///         subscriber, where N is the node count. The EventStream is node-local by construction,
 ///         which is exactly the needed semantics. It also avoids a registration handshake with
 ///         actors spawned later: <c>ScriptedAlarmHostActor</c> is a <i>child of</i>
 ///         <c>DriverHostActor</c> and has no registry key, so there is no ref to hand in at wiring
 ///         time.
 ///     </para>
 ///     <para>
 ///         <b>Outbound is <see cref="ApplyAck"/> only.</b> There is no Ask across this boundary in
 ///         Phase 2 — every migrated command is fire-and-forget, and the deploy ack returns as an
 ///         independent message the coordinator already subscribes for. Nothing here needs sender
 ///         preservation, which is why this actor is far smaller than the sister project's.
 ///     </para>
 /// </remarks>
 public sealed class NodeCommunicationActor : ReceiveActor
 {
    private readonly IActorRef? _centralClient;
    private readonly ILoggingAdapter _log = Context.GetLogger();
    /// <summary>Creates the props for this actor.</summary>
    /// <param name="centralClient">
    ///     ClusterClient dialling central's receptionist, or <see langword="null"/> under
    ///     <c>MeshTransport:Mode = Dps</c>, where acks still go out over DistributedPubSub from
    ///     <c>DriverHostActor</c> and this actor never sees them.
    /// </param>
    /// <returns>The props.</returns>
    public static Props Props(IActorRef? centralClient) =>
        Akka.Actor.Props.Create(() => new NodeCommunicationActor(centralClient));
    /// <summary>Initializes a new instance of the <see cref="NodeCommunicationActor"/> class.</summary>
    /// <param name="centralClient">ClusterClient dialling central, or <see langword="null"/>.</param>
    public NodeCommunicationActor(IActorRef? centralClient)
    {
        _centralClient = centralClient;
        // Inbound from central. Each type is listed explicitly rather than caught by a ReceiveAny:
        // an unknown type must dead-letter loudly during development rather than be republished
        // blind onto the node's EventStream, where it would be silently ignored by every subscriber.
        Receive<DispatchDeployment>(PublishLocally);
        Receive<RestartDriver>(PublishLocally);
        Receive<ReconnectDriver>(PublishLocally);
        Receive<AlarmCommand>(PublishLocally);
        // Outbound to central.
        Receive<ApplyAck>(HandleApplyAck);
    }
    private void PublishLocally(object msg)
    {
        _log.Debug("Mesh: received {MessageType} from central; publishing node-locally", msg.GetType().Name);
        Context.System.EventStream.Publish(msg);
    }
    private void HandleApplyAck(ApplyAck ack)
    {
        if (_centralClient is null)
        {
            _log.Warning(
                "No central ClusterClient — dropping ApplyAck for {DeploymentId}. The ack is not "
                + "buffered and will not be retried, so the deployment will time out at central "
                + "naming this node even though it applied successfully. Check "
                + "MeshTransport:CentralContactPoints",
                ack.DeploymentId);
            return;
        }
        // Send, not SendToAll: the deploy coordinator is a single singleton, and the central comm
        // actor forwards to its proxy. SendToAll would deliver one ack per central node, and the
        // coordinator would count the same node twice.
        _centralClient.Tell(new ClusterClient.Send(MeshPaths.CentralCommunication, ack));
    }
 }
@@ -86,7 +86,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
    private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
    private readonly CommonsNodeId _localNode;
-    private readonly IActorRef? _coordinatorOverride;
+    private readonly IActorRef? _ackRouter;
    private readonly IDriverFactory _driverFactory;
    /// <summary>Builds the per-driver-instance <see cref="IDriverCapabilityInvoker"/> attached to each
@@ -328,7 +328,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
    /// <summary>Creates props for a DriverHostActor with the specified dependencies.</summary>
    /// <param name="dbFactory">Database context factory for configuration database access.</param>
    /// <param name="localNode">The local cluster node identifier.</param>
-    /// <param name="coordinator">Optional coordinator actor reference for deployment coordination.</param>
+    /// <param name="ackRouter">Where ApplyAcks are sent. A direct coordinator handle in tests; under
    /// <c>MeshTransport:Mode = ClusterClient</c> the node's <c>NodeCommunicationActor</c>, which relays
    /// across the boundary. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
    /// <param name="driverFactory">Optional driver factory; defaults to null factory if not provided.</param>
    /// <param name="localRoles">Optional set of roles assigned to the local node.</param>
    /// <param name="dependencyMux">Optional actor reference for dependency multiplexing.</param>
@@ -364,7 +366,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
    public static Props Props(
        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
        CommonsNodeId localNode,
-        IActorRef? coordinator = null,
+        IActorRef? ackRouter = null,
        IDriverFactory? driverFactory = null,
        IReadOnlySet<string>? localRoles = null,
        IActorRef? dependencyMux = null,
@@ -387,7 +389,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
        // the wrong dependency at runtime. New parameters go LAST in all three places — this
        // signature, the constructor's, and this call — and nothing else moves.
        Akka.Actor.Props.Create(() => new DriverHostActor(
-            dbFactory, localNode, coordinator, driverFactory, localRoles, dependencyMux, opcUaPublishActor,
+            dbFactory, localNode, ackRouter, driverFactory, localRoles, dependencyMux, opcUaPublishActor,
            healthPublisher, virtualTagEvaluator, historyWriter, virtualTagHostOverride,
            loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider,
            deploymentArtifactCache, redundancyRoleView, replicationPeerHost));
@@ -395,7 +397,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
    /// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
    /// <param name="dbFactory">Database context factory for configuration database access.</param>
    /// <param name="localNode">The local cluster node identifier.</param>
-    /// <param name="coordinator">Optional coordinator actor reference for deployment coordination.</param>
+    /// <param name="ackRouter">Where ApplyAcks are sent. A direct coordinator handle in tests; under
    /// <c>MeshTransport:Mode = ClusterClient</c> the node's <c>NodeCommunicationActor</c>, which relays
    /// across the boundary. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
    /// <param name="driverFactory">Optional driver factory; defaults to null factory if not provided.</param>
    /// <param name="localRoles">Optional set of roles assigned to the local node.</param>
    /// <param name="dependencyMux">Optional actor reference for dependency multiplexing.</param>
@@ -426,7 +430,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
    public DriverHostActor(
        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
        CommonsNodeId localNode,
-        IActorRef? coordinator,
+        IActorRef? ackRouter,
        IDriverFactory? driverFactory = null,
        IReadOnlySet<string>? localRoles = null,
        IActorRef? dependencyMux = null,
@@ -449,7 +453,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
        _replicationPeerHost = replicationPeerHost;
        _dbFactory = dbFactory;
        _localNode = localNode;
-        _coordinatorOverride = coordinator;
+        _ackRouter = ackRouter;
        _driverFactory = driverFactory ?? NullDriverFactory.Instance;
        _invokerFactory = invokerFactory ?? NullDriverCapabilityInvokerFactory.Instance;
        _driverMemberCountProvider = driverMemberCountProvider;
@@ -483,6 +487,16 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
        // uses so only the Primary services operator writes (the secondary keeps state warm for failover).
        _mediator.Tell(
            new Subscribe(ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RedundancyStateTopic, Self));
        // Per-cluster mesh Phase 2 dark switch. Under MeshTransport:Mode = ClusterClient these same
        // commands arrive from NodeCommunicationActor on the node-local EventStream instead of the
        // mesh-wide DPS topic. Subscribing to BOTH unconditionally is deliberate: exactly one of the
        // two ever publishes, so the transport flag lives entirely on the publishing side and the
        // switch can be flipped (or reverted) without touching any subscriber.
        Context.System.EventStream.Subscribe(Self, typeof(DispatchDeployment));
        Context.System.EventStream.Subscribe(Self, typeof(RestartDriver));
        Context.System.EventStream.Subscribe(Self, typeof(ReconnectDriver));
        // Spawn the VirtualTag host BEFORE Bootstrap so the bootstrap-restore path (which routes
        // through PushDesiredSubscriptions and Tells ApplyVirtualTags) has a live host to target.
        SpawnVirtualTagHost();
@@ -2410,15 +2424,20 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
    private void SendAck(DeploymentId deploymentId, ApplyAckOutcome outcome, string? failureReason, CorrelationId correlation)
    {
        var ack = new ApplyAck(deploymentId, _localNode, outcome, failureReason, correlation);
-        if (_coordinatorOverride is not null)
+        if (_ackRouter is not null)
        {
-            _coordinatorOverride.Tell(ack);
+            // Either a direct coordinator handle (tests) or, under
            // MeshTransport:Mode = ClusterClient, the NodeCommunicationActor that relays this
            // across the boundary. Renamed from _coordinatorOverride in per-cluster mesh Phase 2:
            // under ClusterClient mode this ref is emphatically NOT a coordinator, and the old
            // name would send the next reader looking for one.
            _ackRouter.Tell(ack);
        }
        else
        {
-            // No direct coordinator handle — publish on the dedicated ACK topic. The coordinator
+            // No router — publish on the dedicated ACK topic. The coordinator singleton subscribes
-            // singleton subscribes there in PreStart so the ACK reaches whichever admin node hosts
+            // there in PreStart so the ACK reaches whichever admin node hosts it without an
-            // it without an actor-path lookup.
+            // actor-path lookup.
            DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(DeploymentAcksTopic, ack));
        }
    }
@@ -528,6 +528,15 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
        // RedundancyStateChanged and cache this node's role — OnEngineEmission gates the cluster-wide
        // alerts publish to the Primary so a 2-node single-cluster deploy doesn't double-emit transitions.
        _mediator.Tell(new Subscribe(OpcUaPublishActor.RedundancyStateTopic, Self));
        // Per-cluster mesh Phase 2 dark switch. An AdminUI-originated ack/shelve arrives from
        // NodeCommunicationActor on the node-local EventStream under
        // MeshTransport:Mode = ClusterClient; the DPS subscribe above still carries the node-LOCAL
        // publisher (OtOpcUaServerHostedService, the OPC UA Part 9 method calls), which never
        // crosses the boundary and stays on DPS in both modes. Subscribing to both is safe: exactly
        // one publisher feeds each path.
        Context.System.EventStream.Subscribe(Self, typeof(AlarmCommand));
        base.PreStart();
    }
@@ -1,4 +1,6 @@
 using System.Collections.Immutable;
 using Akka.Actor;
 using Akka.Cluster.Tools.Client;
 using Akka.Hosting;
 using Microsoft.EntityFrameworkCore;
 using Microsoft.Extensions.Configuration;
@@ -6,6 +8,10 @@ using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.DependencyInjection.Extensions;
 using Microsoft.Extensions.Logging;
 using Microsoft.Extensions.Logging.Abstractions;
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.OtOpcUa.Cluster;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
 using ZB.MOM.WW.OtOpcUa.Commons.Engines;
 using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
 using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
@@ -37,6 +43,7 @@ public static class ServiceCollectionExtensions
    public const string OpcUaPublishActorName = "opcua-publish";
    public const string PeerProbeSupervisorName = "peer-probe-supervisor";
    public const string ContinuousHistorizationRecorderActorName = "continuous-historization-recorder";
    public const string CentralClusterClientName = "central-cluster-client";
    /// <summary>
    /// Registers shared runtime services. Currently binds <see cref="IAlarmHistorianSink"/>
@@ -319,6 +326,40 @@ public static class ServiceCollectionExtensions
            var mux = system.ActorOf(DependencyMuxActor.Props(), DependencyMuxActorName);
            registry.Register<DependencyMuxActorKey>(mux);
            // Per-cluster mesh Phase 2: the node end of the central↔node command boundary. Spawned
            // BEFORE DriverHostActor because it is the host's ackRouter under ClusterClient mode.
            // It has no dependency of its own on the host — inbound commands travel via the node's
            // EventStream — so this ordering costs nothing.
            var meshOptions = resolver.GetService<IOptions<MeshTransportOptions>>().Value;
            var useClusterClient = string.Equals(
                meshOptions.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
            IActorRef? centralClient = null;
            if (useClusterClient)
            {
                // Contacts come from appsettings, not the database — the deliberate asymmetry:
                // central's node set changes as operators add and retire nodes, but central's own
                // address is part of the deployment. MeshTransportOptionsValidator has already
                // rejected an empty or malformed list at boot, so this cannot silently produce a
                // client with no contacts.
                var contacts = meshOptions.CentralContactPoints
                    .Select(cp => ActorPath.Parse($"{cp}/system/receptionist"))
                    .ToImmutableHashSet();
                centralClient = system.ActorOf(
                    ClusterClient.Props(ClusterClientSettings.Create(system).WithInitialContacts(contacts)),
                    CentralClusterClientName);
            }
            var nodeComm = system.ActorOf(
                NodeCommunicationActor.Props(centralClient),
                MeshPaths.NodeCommunicationName);
            // Registered with the receptionist in BOTH modes: under Dps nothing dials it, and an
            // idle registration costs nothing — whereas registering only under ClusterClient would
            // make flipping the flag require a node restart before central could reach it.
            ClusterClientReceptionist.Get(system).RegisterService(nodeComm);
            registry.Register<NodeCommunicationActorKey>(nodeComm);
            // Continuous-historization recorder — gated on ContinuousHistorization:Enabled AND the
            // gateway-backed IHistorianValueWriter + the durable IHistorizationOutbox being registered
            // (the Host registers both ONLY when historization is enabled and the ServerHistorian gateway
@@ -398,7 +439,11 @@ public static class ServiceCollectionExtensions
            registry.Register<PeerProbeSupervisorKey>(peerProbes);
            var driverHost = system.ActorOf(
-                DriverHostActor.Props(dbFactory, roleInfo.LocalNode, coordinator: null,
+                // ackRouter: under ClusterClient mode the node comm actor relays ApplyAcks across
                // the boundary; under Dps it stays null and the host publishes on the
                // deployment-acks topic exactly as before.
                DriverHostActor.Props(dbFactory, roleInfo.LocalNode,
                    ackRouter: useClusterClient ? nodeComm : null,
                    driverFactory: driverFactory, localRoles: roleInfo.LocalRoles,
                    dependencyMux: mux,
                    opcUaPublishActor: publishActor,
@@ -430,6 +475,7 @@ public sealed class DbHealthProbeActorKey { }
 public sealed class HistorianAdapterActorKey { }
 public sealed class DependencyMuxActorKey { }
 public sealed class OpcUaPublishActorKey { }
 public sealed class NodeCommunicationActorKey { }
 /// <summary>Marker key for the per-node ContinuousHistorizationRecorder (spawned only when
 /// <c>ContinuousHistorization:Enabled=true</c> and the gateway value-writer + outbox are registered).</summary>
@@ -0,0 +1,100 @@
 using Akka.Actor;
 using Akka.Cluster.Tools.Client;
 using Akka.Configuration;
 using Shouldly;
 using Xunit;
 namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
 /// <summary>
 ///     Asserts the ClusterClient / receptionist / frame-size settings that are actually <b>in
 ///     force</b> on a real <see cref="ActorSystem"/> built from the shipped base config.
 /// </summary>
 /// <remarks>
 ///     Asserting the <c>akka.conf</c> text instead would pass while a competing HOCON fragment
 ///     silently overrode the value — the failure mode recorded at length in
 ///     <c>ServiceCollectionExtensions.BuildDowningHocon</c> and pinned the same way by
 ///     <c>SplitBrainResolverActivationTests</c>. In this codebase the effective value is the only
 ///     one worth asserting.
 /// </remarks>
 public class MeshTransportHoconTests : IDisposable
 {
    private readonly ActorSystem _sys;
    public MeshTransportHoconTests()
    {
        var config = ConfigurationFactory
            .ParseString("akka.remote.dot-netty.tcp.port = 0\nakka.cluster.seed-nodes = []")
            .WithFallback(HoconLoader.LoadBaseConfig());
        _sys = ActorSystem.Create("mesh-hocon-probe", config);
    }
    [Fact]
    public void Cluster_client_buffering_is_disabled()
    {
        // Asserted through ClusterClientSettings — the object the client is actually built from —
        // NOT through Config.GetInt. GetInt returns 0 for a MISSING key, so a config assertion here
        // passes identically whether the setting is explicitly 0 or absent with Akka's default of
        // 1000 in force. It was green before this feature existed; that is the whole defect class.
        //
        // buffer-size = 0 means "drop now if the receptionist is unknown". Buffering would replay a
        // DispatchDeployment on reconnect and apply a deployment the coordinator already sealed as
        // TimedOut, leaving the node running a configuration the database says failed. This is a
        // behaviour decision, not a tuning knob.
        ClusterClientSettings.Create(_sys).BufferSize.ShouldBe(0);
    }
    [Fact]
    public void Cluster_client_retries_forever_rather_than_self_stopping()
    {
        // reconnect-timeout = off surfaces as a null TimeSpan on the settings object. A central node
        // down for an hour must not require a driver-node restart to become reachable again.
        ClusterClientSettings.Create(_sys).ReconnectTimeout.ShouldBeNull();
    }
    [Fact]
    public void Receptionist_serves_every_member_regardless_of_role()
    {
        // An empty role is what makes "registered per node, not as a singleton" work: every member
        // hosts a receptionist, so ClusterClient contact rotation reaches whichever node answers.
        // Scoping this to a role would make the boundary reachable through one node only.
        ClusterReceptionistSettings.Create(_sys).Role.ShouldBeNull();
    }
    [Fact]
    public void Oversized_frames_are_logged_rather_than_dropped_silently()
    {
        // The sister project's most expensive incident. A message over the frame limit is dropped
        // WITHOUT tearing down the association: heartbeats keep flowing, the node still reports
        // healthy, and the only symptom is a command that never arrived. log-frame-size-exceeding
        // converts that silence into a log line naming the offending size.
        _sys.Settings.Config.GetByteSize("akka.remote.dot-netty.tcp.maximum-frame-size").ShouldBe(128000);
        _sys.Settings.Config.GetByteSize("akka.remote.dot-netty.tcp.log-frame-size-exceeding").ShouldBe(32000);
    }
    [Fact]
    public void Receptionist_extension_is_registered_at_startup()
    {
        // Not auto-started. Without this entry the receptionist materialises only if some code path
        // happens to call ClusterClientReceptionist.Get(system) first, which makes "is the boundary
        // listening?" depend on startup ordering rather than on configuration.
        _sys.Settings.Config.GetStringList("akka.extensions")
            .ShouldContain(s => s.Contains("ClusterClientReceptionistExtensionProvider"));
    }
    [Fact]
    public void Distributed_pub_sub_extension_survives_alongside_the_receptionist()
    {
        // The dark switch keeps both transports live for one phase, and DPS additionally carries
        // traffic Phase 2 does not touch at all — node-local alarm-command publishes and the
        // node<->node redundancy-state probe leg. Losing it here would break far more than deploys.
        _sys.Settings.Config.GetStringList("akka.extensions")
            .ShouldContain(s => s.Contains("DistributedPubSubExtensionProvider"));
    }
    public void Dispose()
    {
        _sys.Terminate().Wait(TimeSpan.FromSeconds(5));
        GC.SuppressFinalize(this);
    }
 }
@@ -0,0 +1,138 @@
 using Microsoft.Extensions.Options;
 using Shouldly;
 using Xunit;
 namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
 /// <summary>
 ///     Every failure this validator catches otherwise surfaces as an <i>absence</i> — a deployment
 ///     that never arrives, an alarm ack that does nothing, a node whose ack is dropped without a
 ///     trace. Absences are the hardest class of fault to attribute in the field, so the whole point
 ///     of these tests is that the host refuses to start instead.
 /// </summary>
 public class MeshTransportOptionsValidatorTests
 {
    private static ValidateOptionsResult Validate(MeshTransportOptions o) =>
        new MeshTransportOptionsValidator().Validate(MeshTransportOptions.SectionName, o);
    [Fact]
    public void Default_options_are_valid()
    {
        // The default is Dps — the pre-Phase-2 transport. A node that never sets this section must
        // behave exactly as it did before, which is the whole premise of the dark switch.
        Validate(new MeshTransportOptions()).Succeeded.ShouldBeTrue();
    }
    [Fact]
    public void Unknown_mode_fails()
    {
        var result = Validate(new MeshTransportOptions { Mode = "grpc" });
        result.Failed.ShouldBeTrue();
        result.FailureMessage.ShouldContain("grpc");
    }
    [Fact]
    public void Mode_matching_is_case_insensitive()
    {
        Validate(new MeshTransportOptions
        {
            Mode = "clusterclient",
            CentralContactPoints = ["akka.tcp://otopcua@central-1:4053"],
        }).Succeeded.ShouldBeTrue();
    }
    [Fact]
    public void Non_positive_contact_refresh_fails()
    {
        var result = Validate(new MeshTransportOptions { ContactRefreshSeconds = 0 });
        result.Failed.ShouldBeTrue();
        result.FailureMessage.ShouldContain(nameof(MeshTransportOptions.ContactRefreshSeconds));
    }
    [Fact]
    public void ClusterClient_mode_requires_central_contact_points()
    {
        var result = Validate(new MeshTransportOptions
        {
            Mode = MeshTransportOptions.ModeClusterClient,
            CentralContactPoints = [],
        });
        result.Failed.ShouldBeTrue();
        result.FailureMessage.ShouldContain(nameof(MeshTransportOptions.CentralContactPoints));
    }
    [Fact]
    public void Dps_mode_does_not_require_central_contact_points()
    {
        // An admin-only node has no reason to carry them, and requiring them under the default mode
        // would make the section mandatory everywhere for a feature that is switched off.
        Validate(new MeshTransportOptions
        {
            Mode = MeshTransportOptions.ModeDps,
            CentralContactPoints = [],
        }).Succeeded.ShouldBeTrue();
    }
    [Fact]
    public void A_contact_point_that_is_not_an_akka_address_fails()
    {
        var result = Validate(new MeshTransportOptions
        {
            Mode = MeshTransportOptions.ModeClusterClient,
            CentralContactPoints = ["central-1:4053"],
        });
        result.Failed.ShouldBeTrue();
        result.FailureMessage.ShouldContain("akka.tcp://");
    }
    [Fact]
    public void A_contact_point_carrying_an_actor_path_suffix_fails()
    {
        // The sister project's shipped site template listed a full receptionist path, and separately
        // listed the site's OWN remoting port as the second contact — "a permanent failure in the
        // initial-contact rotation". Both are invisible until a command goes missing.
        var result = Validate(new MeshTransportOptions
        {
            Mode = MeshTransportOptions.ModeClusterClient,
            CentralContactPoints = ["akka.tcp://otopcua@central-1:4053/system/receptionist"],
        });
        result.Failed.ShouldBeTrue();
        result.FailureMessage.ShouldContain("/system/receptionist");
    }
    [Fact]
    public void One_bad_contact_point_among_good_ones_still_fails()
    {
        var result = Validate(new MeshTransportOptions
        {
            Mode = MeshTransportOptions.ModeClusterClient,
            CentralContactPoints =
            [
                "akka.tcp://otopcua@central-1:4053",
                "central-2:4053",
            ],
        });
        result.Failed.ShouldBeTrue();
        result.FailureMessage.ShouldContain("central-2:4053");
    }
    [Fact]
    public void Well_formed_ClusterClient_options_succeed()
    {
        Validate(new MeshTransportOptions
        {
            Mode = MeshTransportOptions.ModeClusterClient,
            CentralContactPoints =
            [
                "akka.tcp://otopcua@central-1:4053",
                "akka.tcp://otopcua@central-2:4053",
            ],
        }).Succeeded.ShouldBeTrue();
    }
 }
@@ -0,0 +1,271 @@
 using System.Collections.Immutable;
 using Akka.Actor;
 using Akka.Cluster.Tools.Client;
 using Akka.Cluster.Tools.PublishSubscribe;
 using Akka.TestKit;
 using Microsoft.EntityFrameworkCore;
 using Shouldly;
 using Xunit;
 using ZB.MOM.WW.OtOpcUa.Cluster;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 using ZB.MOM.WW.OtOpcUa.Commons.Types;
 using ZB.MOM.WW.OtOpcUa.Configuration;
 using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
 using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Harness;
 namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
 public class CentralCommunicationActorTests : ControlPlaneActorTestBase
 {
    private const string DeploymentsTopic = "deployments";
    /// <summary>Records every client the actor asks for, and hands back a probe in its place.</summary>
    private sealed class RecordingClientFactory : IMeshClusterClientFactory
    {
        private readonly Func<TestProbe> _newProbe;
        public RecordingClientFactory(Func<TestProbe> newProbe) => _newProbe = newProbe;
        public List<ImmutableHashSet<ActorPath>> Calls { get; } = [];
        public List<TestProbe> Probes { get; } = [];
        public IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts)
        {
            Calls.Add(contacts);
            var probe = _newProbe();
            Probes.Add(probe);
            return probe.Ref;
        }
    }
    private static MeshTransportOptions ClusterClientMode(int refreshSeconds = 60) => new()
    {
        Mode = MeshTransportOptions.ModeClusterClient,
        CentralContactPoints = ["akka.tcp://otopcua@central-1:4053"],
        ContactRefreshSeconds = refreshSeconds,
    };
    private static void SeedCluster(
        IDbContextFactory<OtOpcUaConfigDbContext> factory,
        string clusterId,
        params (string NodeId, string Host, int AkkaPort, bool Enabled, bool Maintenance)[] nodes)
    {
        using var db = factory.CreateDbContext();
        db.ServerClusters.Add(new ServerCluster
        {
            ClusterId = clusterId,
            Name = clusterId,
            Enterprise = "ZB",
            Site = "Site",
            // Warm/Hot require NodeCount = 2 (CK_ServerCluster_RedundancyMode_NodeCount); these
            // fixtures only need the row to exist, so use the standalone shape.
            RedundancyMode = RedundancyMode.None,
            NodeCount = 1,
            CreatedBy = "test",
        });
        foreach (var n in nodes)
        {
            db.ClusterNodes.Add(new ClusterNode
            {
                NodeId = n.NodeId,
                ClusterId = clusterId,
                Host = n.Host,
                AkkaPort = n.AkkaPort,
                ApplicationUri = $"urn:{n.NodeId}",
                Enabled = n.Enabled,
                MaintenanceMode = n.Maintenance,
                CreatedBy = "test",
            });
        }
        db.SaveChanges();
    }
    private IActorRef Spawn(
        IDbContextFactory<OtOpcUaConfigDbContext> factory,
        MeshTransportOptions options,
        IMeshClusterClientFactory clientFactory,
        IActorRef? coordinator = null) =>
        Sys.ActorOf(CentralCommunicationActor.Props(factory, options, clientFactory, () => coordinator));
    [Fact]
    public void Dps_mode_publishes_on_the_carried_topic_and_creates_no_client()
    {
        var factory = NewInMemoryDbFactory();
        var clients = new RecordingClientFactory(() => CreateTestProbe());
        var mediatorProbe = CreateTestProbe();
        // Subscribe on the probe's behalf: SubscribeAck goes to the SENDER, so telling the mediator
        // without an explicit sender would ack to TestActor and this wait would hang.
        DistributedPubSub.Get(Sys).Mediator.Tell(
            new Subscribe(DeploymentsTopic, mediatorProbe.Ref), mediatorProbe.Ref);
        mediatorProbe.ExpectMsg<SubscribeAck>(TimeSpan.FromSeconds(5));
        var actor = Spawn(factory, new MeshTransportOptions(), clients);
        var dispatch = new DispatchDeployment(
            new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), new CorrelationId(Guid.NewGuid()));
        actor.Tell(new MeshCommand(DeploymentsTopic, dispatch));
        mediatorProbe.ExpectMsg<DispatchDeployment>(TimeSpan.FromSeconds(5)).ShouldBe(dispatch);
        clients.Calls.ShouldBeEmpty();
    }
    [Fact]
    public void ClusterClient_mode_sends_to_all_not_to_one()
    {
        // THE assertion of this phase. ClusterClient.Send delivers to exactly ONE registered actor;
        // today's DPS publish reaches EVERY DriverHostActor, and the node side has no ClusterId or
        // node filter to compensate. Send would deploy to a single node while every other node
        // silently kept its old configuration.
        var factory = NewInMemoryDbFactory();
        SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false));
        var clients = new RecordingClientFactory(() => CreateTestProbe());
        var actor = Spawn(factory, ClusterClientMode(), clients);
        AwaitCondition(() => clients.Probes.Count == 1, TimeSpan.FromSeconds(5));
        var dispatch = new DispatchDeployment(
            new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), new CorrelationId(Guid.NewGuid()));
        actor.Tell(new MeshCommand(DeploymentsTopic, dispatch));
        var sent = clients.Probes[0].ExpectMsg<ClusterClient.SendToAll>(TimeSpan.FromSeconds(5));
        sent.Path.ShouldBe(MeshPaths.NodeCommunication);
        sent.Message.ShouldBe(dispatch);
    }
    [Fact]
    public void Exactly_one_client_is_created_for_a_multi_cluster_fleet()
    {
        // A receptionist serves its whole mesh, so one client per Cluster would fan each command out
        // once per cluster while the fleet is still a single mesh — N x duplicate deployments.
        var factory = NewInMemoryDbFactory();
        SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false));
        SeedCluster(factory, "C2", ("host-b:4053", "host-b", 4053, true, false));
        var clients = new RecordingClientFactory(() => CreateTestProbe());
        Spawn(factory, ClusterClientMode(), clients);
        AwaitCondition(() => clients.Calls.Count == 1, TimeSpan.FromSeconds(5));
        // ...and that one client dials BOTH clusters' nodes.
        clients.Calls[0].Count.ShouldBe(2);
        // Give a second refresh a chance to (wrongly) create another.
        ExpectNoMsg(TimeSpan.FromMilliseconds(300));
        clients.Calls.Count.ShouldBe(1);
    }
    [Fact]
    public void Disabled_and_maintenance_nodes_are_not_dialled()
    {
        var factory = NewInMemoryDbFactory();
        SeedCluster(
            factory,
            "C1",
            ("host-a:4053", "host-a", 4053, true, false),
            ("host-b:4053", "host-b", 4053, false, false),
            ("host-c:4053", "host-c", 4053, true, true));
        var clients = new RecordingClientFactory(() => CreateTestProbe());
        Spawn(factory, ClusterClientMode(), clients);
        AwaitCondition(() => clients.Calls.Count == 1, TimeSpan.FromSeconds(5));
        clients.Calls[0].Count.ShouldBe(1);
        clients.Calls[0].Single().ToString().ShouldContain("host-a");
    }
    [Fact]
    public void A_malformed_node_row_does_not_abort_the_refresh()
    {
        // The bad row is ordered FIRST on purpose: the failure this guards against is an exception
        // that aborts the loop and leaves the contact set half-built.
        var factory = NewInMemoryDbFactory();
        SeedCluster(
            factory,
            "C1",
            ("bad", "not a host!!", 4053, true, false),
            ("host-a:4053", "host-a", 4053, true, false));
        var clients = new RecordingClientFactory(() => CreateTestProbe());
        Spawn(factory, ClusterClientMode(), clients);
        AwaitCondition(() => clients.Calls.Count == 1, TimeSpan.FromSeconds(5));
        clients.Calls[0].ShouldContain(p => p.ToString().Contains("host-a"));
    }
    [Fact]
    public void A_command_with_no_client_is_dropped_with_a_warning()
    {
        // No ClusterNode rows at all: nothing to dial, so the command must be dropped loudly rather
        // than queued. buffer-size = 0 covers the case where a client exists but cannot connect;
        // this covers the case where there is no client at all.
        var factory = NewInMemoryDbFactory();
        var clients = new RecordingClientFactory(() => CreateTestProbe());
        var actor = Spawn(factory, ClusterClientMode(), clients);
        EventFilter.Warning(contains: "dropping").ExpectOne(TimeSpan.FromSeconds(5), () =>
            actor.Tell(new MeshCommand(DeploymentsTopic, "anything")));
    }
    [Fact]
    public void ApplyAck_is_forwarded_to_the_coordinator_preserving_the_sender()
    {
        var factory = NewInMemoryDbFactory();
        var clients = new RecordingClientFactory(() => CreateTestProbe());
        var coordinator = CreateTestProbe();
        var node = CreateTestProbe();
        var actor = Spawn(factory, ClusterClientMode(), clients, coordinator.Ref);
        var ack = new ApplyAck(
            new DeploymentId(Guid.NewGuid()),
            new NodeId("host-a:4053"),
            ApplyAckOutcome.Applied,
            null,
            new CorrelationId(Guid.NewGuid()));
        actor.Tell(ack, node.Ref);
        coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).ShouldBe(ack);
        // Forward, not Tell: the coordinator must see the node, not this relay.
        coordinator.LastSender.ShouldBe(node.Ref);
    }
    [Fact]
    public void An_unresolvable_coordinator_logs_rather_than_losing_the_ack_silently()
    {
        var factory = NewInMemoryDbFactory();
        var clients = new RecordingClientFactory(() => CreateTestProbe());
        var actor = Spawn(factory, ClusterClientMode(), clients, coordinator: null);
        var ack = new ApplyAck(
            new DeploymentId(Guid.NewGuid()),
            new NodeId("host-a:4053"),
            ApplyAckOutcome.Applied,
            null,
            new CorrelationId(Guid.NewGuid()));
        EventFilter.Warning(contains: "not resolvable").ExpectOne(TimeSpan.FromSeconds(5), () =>
            actor.Tell(ack));
    }
    [Fact]
    public void A_faulted_contact_load_warns_and_leaves_the_actor_alive()
    {
        var factory = NewInMemoryDbFactory();
        var clients = new RecordingClientFactory(() => CreateTestProbe());
        var actor = Spawn(factory, ClusterClientMode(), clients);
        // Filter on a phrase unique to the DB-failure warning. EventFilter matches case-INSENSITIVELY,
        // so a looser "was NOT" also caught this actor's other warning ("the mesh ClusterClient was
        // not created") and the assertion failed with two matches instead of one.
        EventFilter.Warning(contains: "contact set was NOT refreshed").ExpectOne(
            TimeSpan.FromSeconds(5),
            () => actor.Tell(new Status.Failure(new InvalidOperationException("db down"))));
        // Still serving: a transient DB outage must not take the command boundary down with it.
        actor.Tell(new MeshCommand(DeploymentsTopic, "anything"));
        ExpectNoMsg(TimeSpan.FromMilliseconds(200));
    }
 }
@@ -0,0 +1,67 @@
 using System.Collections.Immutable;
 using Akka.Actor;
 using Akka.Cluster.Tools.Client;
 using Akka.Configuration;
 using Akka.TestKit.Xunit2;
 using Shouldly;
 using Xunit;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
 namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
 /// <summary>
 ///     Guards the one non-obvious part of client creation: recreating a client with the same name
 ///     inside a single message handler throws, because <c>Context.Stop</c> is asynchronous and the
 ///     name stays reserved until termination completes.
 /// </summary>
 public class MeshClusterClientFactoryTests : TestKit
 {
    private static readonly Config TestConfig = ConfigurationFactory.ParseString(@"
        akka.actor.provider = cluster
        akka.remote.dot-netty.tcp.port = 0
        akka.remote.dot-netty.tcp.hostname = 127.0.0.1")
        .WithFallback(ClusterClientReceptionist.DefaultConfig());
    public MeshClusterClientFactoryTests()
        : base(TestConfig)
    {
    }
    // A real ClusterClient pointed at an unreachable contact is safe in TestKit — it simply retries
    // the initial-contact rotation and never fails the test.
    private static ImmutableHashSet<ActorPath> UnreachableContact => ImmutableHashSet.Create(
        ActorPath.Parse("akka.tcp://otopcua@127.0.0.1:65501/system/receptionist"));
    [Fact]
    public void Recreating_a_client_in_one_pass_does_not_collide_on_the_actor_name()
    {
        var factory = new DefaultMeshClusterClientFactory();
        var first = factory.Create(Sys, UnreachableContact);
        Sys.Stop(first);
        // Stop is asynchronous: `first`'s name is still reserved right here. Without the generation
        // suffix this throws InvalidActorNameException — which is precisely what a contact-set
        // change would do in production, where stop-then-recreate happens in one handler.
        var second = factory.Create(Sys, UnreachableContact);
        second.Path.Name.ShouldNotBe(first.Path.Name);
    }
    [Fact]
    public void Every_incarnation_gets_a_distinct_name()
    {
        var factory = new DefaultMeshClusterClientFactory();
        var names = Enumerable.Range(0, 5)
            .Select(_ => factory.Create(Sys, UnreachableContact).Path.Name)
            .ToList();
        names.Distinct().Count().ShouldBe(5);
    }
    // Contact-set propagation is deliberately NOT asserted here: reading initial contacts back
    // off a created ClusterClient is not possible, and asserting ClusterClientSettings directly
    // would test Akka rather than this factory. It is covered where it can actually fail — the
    // real two-ActorSystem boundary test, which only delivers if the contacts reached the client.
 }
@@ -0,0 +1,134 @@
 using Akka.Actor;
 using Akka.Cluster.Tools.PublishSubscribe;
 using Shouldly;
 using Xunit;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
 using ZB.MOM.WW.OtOpcUa.Commons.Types;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.AdminOperations;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Harness;
 using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
 namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
 /// <summary>
 ///     Proves the five central→node publish sites go through the transport router when one is
 ///     wired, and fall back to the mediator when it is not.
 /// </summary>
 /// <remarks>
 ///     The fallback is what keeps the ~12 pre-existing coordinator and admin-operations tests
 ///     meaningful: they construct these actors without a router and must still exercise the real
 ///     DistributedPubSub path, not a silently-disabled one.
 /// </remarks>
 public class MeshRouterDispatchTests : ControlPlaneActorTestBase
 {
    private const string DeploymentsTopic = "deployments";
    [Fact]
    public void Deploy_dispatch_goes_to_the_router_when_one_is_wired()
    {
        var router = CreateTestProbe();
        var dbFactory = NewInMemoryDbFactory();
        var coordinator = Sys.ActorOf(
            ConfigPublishCoordinator.Props(dbFactory, applyDeadline: null, meshRouter: router.Ref));
        var deploymentId = new DeploymentId(Guid.NewGuid());
        coordinator.Tell(new Commons.Messages.Deploy.DispatchDeployment(
            deploymentId, new RevisionHash("rev-1"), CorrelationId.NewId()));
        var cmd = router.ExpectMsg<MeshCommand>(TimeSpan.FromSeconds(5));
        cmd.Topic.ShouldBe(DeploymentsTopic);
        cmd.Message.ShouldBeOfType<Commons.Messages.Deploy.DispatchDeployment>();
    }
    [Fact]
    public void Deploy_dispatch_falls_back_to_the_mediator_when_no_router_is_wired()
    {
        var subscriber = CreateTestProbe();
        DistributedPubSub.Get(Sys).Mediator.Tell(
            new Subscribe(DeploymentsTopic, subscriber.Ref), subscriber.Ref);
        subscriber.ExpectMsg<SubscribeAck>(TimeSpan.FromSeconds(5));
        var coordinator = Sys.ActorOf(ConfigPublishCoordinator.Props(NewInMemoryDbFactory()));
        var deploymentId = new DeploymentId(Guid.NewGuid());
        coordinator.Tell(new Commons.Messages.Deploy.DispatchDeployment(
            deploymentId, new RevisionHash("rev-1"), CorrelationId.NewId()));
        subscriber.ExpectMsg<Commons.Messages.Deploy.DispatchDeployment>(TimeSpan.FromSeconds(5));
    }
    [Fact]
    public void Restart_driver_goes_to_the_router_on_the_driver_control_topic()
    {
        var router = CreateTestProbe();
        var admin = Sys.ActorOf(AdminOperationsActor.Props(
            NewInMemoryDbFactory(),
            CreateTestProbe().Ref,
            Enumerable.Empty<IDriverProbe>(),
            TagConfigValidationMode.Warn,
            router.Ref));
        admin.Tell(new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid()));
        var cmd = router.ExpectMsg<MeshCommand>(TimeSpan.FromSeconds(5));
        cmd.Topic.ShouldBe(DriverControlTopic.Name);
        cmd.Message.ShouldBeOfType<RestartDriver>();
    }
    [Fact]
    public void Reconnect_driver_goes_to_the_router_on_the_driver_control_topic()
    {
        var router = CreateTestProbe();
        var admin = Sys.ActorOf(AdminOperationsActor.Props(
            NewInMemoryDbFactory(),
            CreateTestProbe().Ref,
            Enumerable.Empty<IDriverProbe>(),
            TagConfigValidationMode.Warn,
            router.Ref));
        admin.Tell(new ReconnectDriver("C1", "driver-1", "alice", Guid.NewGuid()));
        var cmd = router.ExpectMsg<MeshCommand>(TimeSpan.FromSeconds(5));
        cmd.Topic.ShouldBe(DriverControlTopic.Name);
        cmd.Message.ShouldBeOfType<ReconnectDriver>();
    }
    [Fact]
    public void Alarm_acknowledge_goes_to_the_router_on_the_alarm_commands_topic()
    {
        var router = CreateTestProbe();
        var admin = Sys.ActorOf(AdminOperationsActor.Props(
            NewInMemoryDbFactory(),
            CreateTestProbe().Ref,
            Enumerable.Empty<IDriverProbe>(),
            TagConfigValidationMode.Warn,
            router.Ref));
        admin.Tell(new AcknowledgeAlarmCommand("alarm-1", "alice", "c", CorrelationId.NewId()));
        var cmd = router.ExpectMsg<MeshCommand>(TimeSpan.FromSeconds(5));
        cmd.Topic.ShouldBe(AlarmCommandsTopic.Name);
        cmd.Message.ShouldBeOfType<AlarmCommand>();
    }
    [Fact]
    public void Alarm_shelve_goes_to_the_router_on_the_alarm_commands_topic()
    {
        var router = CreateTestProbe();
        var admin = Sys.ActorOf(AdminOperationsActor.Props(
            NewInMemoryDbFactory(),
            CreateTestProbe().Ref,
            Enumerable.Empty<IDriverProbe>(),
            TagConfigValidationMode.Warn,
            router.Ref));
        admin.Tell(new ShelveAlarmCommand("alarm-1", "alice", ShelveKind.OneShot, null, "c", CorrelationId.NewId()));
        var cmd = router.ExpectMsg<MeshCommand>(TimeSpan.FromSeconds(5));
        cmd.Topic.ShouldBe(AlarmCommandsTopic.Name);
        cmd.Message.ShouldBeOfType<AlarmCommand>();
    }
 }
@@ -0,0 +1,302 @@
 using System.Collections.Immutable;
 using Akka.Actor;
 using Akka.Cluster;
 using Akka.Cluster.Tools.Client;
 using Akka.Configuration;
 using Shouldly;
 using Xunit;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 using ZB.MOM.WW.OtOpcUa.Commons.Types;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
 using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
 using AkkaCluster = Akka.Cluster.Cluster;
 namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
 /// <summary>
 ///     Drives a real <c>ClusterClient</c> across two genuinely separate <see cref="ActorSystem"/>s —
 ///     the only test in the repo where the mesh boundary carries a message over the wire.
 /// </summary>
 /// <remarks>
 ///     <para>
 ///         The sister project substitutes an in-process relay that unwraps
 ///         <see cref="ClusterClient.Send"/> and forwards. That tests actor <i>logic</i> and cannot
 ///         fail for any transport reason: not a missing receptionist extension, not a service that
 ///         was never registered, not a malformed contact path, not a serialization break. Those are
 ///         precisely the Phase 2 failure modes, so they are what this test exercises.
 ///     </para>
 ///     <para>
 ///         <b>Two clusters, one ActorSystem name, joined ONLY by ClusterClient.</b> The shared name
 ///         is mandatory — Akka.Remote matches on the full address, so a contact path naming
 ///         <c>otopcua</c> cannot reach a system called anything else. Each system seeds itself and
 ///         neither lists the other, and
 ///         <see cref="The_two_systems_are_separate_clusters_joined_only_by_the_client"/> asserts the
 ///         separation outright: if they ever gossiped into one mesh, every delivery below could be
 ///         plain local messaging and the whole class would be testing nothing.
 ///     </para>
 ///     <para>
 ///         <b>Scope limit.</b> With exactly one node per side, "registered per node" and "registered
 ///         as a cluster singleton" are indistinguishable by construction, so this class does NOT
 ///         cover the per-node property whose structural assertions <c>MeshCommActorPathTests</c>
 ///         dropped. That property's discriminating coverage is live-gate step 8: stop the OLDEST
 ///         central and the survivor's comm actor must still receive a node ack, which a singleton
 ///         hosted on the stopped node cannot.
 ///     </para>
 /// </remarks>
 public sealed class MeshClusterClientBoundaryTests : IAsyncLifetime
 {
    private const string SharedSystemName = "otopcua";
    private static readonly TimeSpan Arrival = TimeSpan.FromSeconds(30);
    /// <summary>
    ///     How long the falsifiability control waits before concluding nothing arrived. Deliberately
    ///     shorter than <see cref="Arrival"/> but far longer than a successful delivery takes in
    ///     practice, so a pass means "the boundary is shut", not "the clock ran out first".
    /// </summary>
    private static readonly TimeSpan Silence = TimeSpan.FromSeconds(10);
    /// <summary>
    ///     One factory for every client this class creates, because the generation counter that keeps
    ///     client actor names unique is <b>per factory instance</b>. A second factory on the same
    ///     system restarts at generation 1 and collides with the first factory's client — which is
    ///     exactly how this test first failed. Production has one instance per
    ///     <c>CentralCommunicationActor</c>, so the real code path is safe.
    /// </summary>
    private readonly DefaultMeshClusterClientFactory _clientFactory = new();
    private ActorSystem _central = null!;
    private ActorSystem _node = null!;
    private IActorRef _nodeComm = null!;
    private IActorRef _centralClient = null!;
    private TaskCompletionSource<object> _centralInbox = null!;
    /// <inheritdoc />
    public async ValueTask InitializeAsync()
    {
        _central = StartSelfSeededSystem(withReceptionist: true);
        _node = StartSelfSeededSystem(withReceptionist: true);
        await Task.WhenAll(AwaitSingleMemberUpAsync(_central), AwaitSingleMemberUpAsync(_node));
        // Central: the ack landing pad, at the path the node's comm actor sends to.
        _centralInbox = NewInbox();
        var centralComm = _central.ActorOf(
            CaptureActor.Props(_centralInbox), MeshPaths.CentralCommunicationName);
        ClusterClientReceptionist.Get(_central).RegisterService(centralComm);
        // Node: the REAL comm actor, dialling central through the REAL production client factory.
        var nodeToCentral = _clientFactory.Create(_node, ReceptionistOf(_central));
        _nodeComm = _node.ActorOf(
            NodeCommunicationActor.Props(nodeToCentral), MeshPaths.NodeCommunicationName);
        ClusterClientReceptionist.Get(_node).RegisterService(_nodeComm);
        _centralClient = _clientFactory.Create(_central, ReceptionistOf(_node));
    }
    /// <inheritdoc />
    public async ValueTask DisposeAsync()
    {
        await ShutdownAsync(_central);
        await ShutdownAsync(_node);
    }
    [Fact]
    public void The_two_systems_are_separate_clusters_joined_only_by_the_client()
    {
        // The load-bearing precondition of every other test here. Two systems sharing a name in one
        // process are one gossip accident away from being one cluster, at which point ClusterClient
        // is no longer the thing under test.
        AkkaCluster.Get(_central).State.Members.Count.ShouldBe(1,
            "central must be its own single-member cluster, or the boundary is not being crossed");
        AkkaCluster.Get(_node).State.Members.Count.ShouldBe(1,
            "the node must be its own single-member cluster, or the boundary is not being crossed");
    }
    [Fact]
    public async Task Central_send_to_all_reaches_the_node_comm_actor_and_lands_on_its_event_stream()
    {
        var inbox = NewInbox();
        _node.EventStream.Subscribe(_node.ActorOf(CaptureActor.Props(inbox)), typeof(RestartDriver));
        var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
        var delivered = await SendUntilDeliveredAsync(
            _centralClient,
            () => new ClusterClient.SendToAll(MeshPaths.NodeCommunication, command),
            inbox,
            Arrival);
        delivered.ShouldNotBeNull(
            "SendToAll must reach the node's registered comm actor and be re-emitted node-locally; "
            + "this is the whole central→node leg of the mesh transport");
        delivered.ShouldBeOfType<RestartDriver>().DriverInstanceId.ShouldBe("driver-1");
    }
    [Fact]
    public async Task A_node_ack_reaches_central_through_the_node_comm_actors_own_client()
    {
        // Told directly rather than routed from central, so a failure here can only be the outbound
        // leg. The inbound leg has its own test above.
        var ack = new ApplyAck(
            new DeploymentId(Guid.NewGuid()),
            new NodeId("node-a"),
            ApplyAckOutcome.Applied,
            null,
            CorrelationId.NewId());
        var delivered = await SendUntilDeliveredAsync(
            _nodeComm, () => ack, _centralInbox, Arrival);
        delivered.ShouldNotBeNull(
            "the node's ApplyAck must reach central's comm actor, or every deployment times out at "
            + "the apply deadline naming nodes that applied successfully");
        delivered.ShouldBeOfType<ApplyAck>().NodeId.Value.ShouldBe("node-a");
    }
    [Fact]
    public async Task Falsifiability_control_a_node_without_the_receptionist_extension_receives_nothing()
    {
        // Without this, the two tests above cannot distinguish "the ClusterClient boundary works"
        // from "something else in the process delivered the message". Same message, same client
        // factory, same paths, same comm actor — and it must NOT arrive.
        //
        // HONEST SCOPE: this removes the receptionist extension AND, unavoidably, the RegisterService
        // call — resolving ClusterClientReceptionist.Get to register would materialise the very
        // extension whose absence is the control. So it falsifies "delivery happens without a
        // receptionist boundary", not the extension line in isolation. Verified falsifiable: giving
        // this node the extension and the registration turns the test red, which also proves the send
        // below is live rather than silently misaddressed.
        var deaf = StartSelfSeededSystem(withReceptionist: false);
        try
        {
            await AwaitSingleMemberUpAsync(deaf);
            var inbox = NewInbox();
            deaf.EventStream.Subscribe(deaf.ActorOf(CaptureActor.Props(inbox)), typeof(RestartDriver));
            // Deliberately NOT ClusterClientReceptionist.Get(deaf) — resolving the extension would
            // materialise the very receptionist whose absence is the control.
            var comm = deaf.ActorOf(NodeCommunicationActor.Props(null), MeshPaths.NodeCommunicationName);
            comm.ShouldNotBeNull();
            var client = _clientFactory.Create(_central, ReceptionistOf(deaf));
            var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
            var deadline = DateTime.UtcNow + Silence;
            while (DateTime.UtcNow < deadline && !inbox.Task.IsCompleted)
            {
                client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, command));
                await Task.Delay(500, TestContext.Current.CancellationToken);
            }
            inbox.Task.IsCompleted.ShouldBeFalse(
                "delivery survived removing the receptionist extension, so the two positive tests "
                + "above are not measuring the ClusterClient boundary at all");
        }
        finally
        {
            await ShutdownAsync(deaf);
        }
    }
    /// <summary>
    ///     Starts a single-member cluster on a dynamic port, optionally without the receptionist.
    /// </summary>
    /// <param name="withReceptionist">
    ///     <see langword="false"/> drops the receptionist provider from <c>akka.extensions</c>,
    ///     leaving DistributedPubSub in place so only the one variable changes.
    /// </param>
    /// <returns>The started system.</returns>
    private static ActorSystem StartSelfSeededSystem(bool withReceptionist)
    {
        var extensions = withReceptionist
            ? string.Empty
            : "akka.extensions = [\"Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider, "
              + "Akka.Cluster.Tools\"]\n";
        // Port 0 + a post-start self-Join, rather than a seed-nodes list: the address is not known
        // until the transport binds, and hard-coding ports makes the test collide with itself under
        // parallel execution.
        var config = ConfigurationFactory.ParseString(
                "akka.remote.dot-netty.tcp.hostname = \"127.0.0.1\"\n"
                + "akka.remote.dot-netty.tcp.public-hostname = \"127.0.0.1\"\n"
                + "akka.remote.dot-netty.tcp.port = 0\n"
                + "akka.cluster.seed-nodes = []\n"
                + "akka.loglevel = \"WARNING\"\n"
                + extensions)
            .WithFallback(ZB.MOM.WW.OtOpcUa.Cluster.HoconLoader.LoadBaseConfig());
        var system = ActorSystem.Create(SharedSystemName, config);
        var cluster = AkkaCluster.Get(system);
        cluster.Join(cluster.SelfAddress);
        return system;
    }
    private static async Task AwaitSingleMemberUpAsync(ActorSystem system)
    {
        var cluster = AkkaCluster.Get(system);
        var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
        while (DateTime.UtcNow < deadline)
        {
            if (cluster.State.Members.Any(m => m.Status == MemberStatus.Up)) return;
            await Task.Delay(100, TestContext.Current.CancellationToken);
        }
        cluster.State.Members.Any(m => m.Status == MemberStatus.Up).ShouldBeTrue(
            $"{cluster.SelfAddress} never reached Up as its own seed");
    }
    /// <summary>
    ///     Re-sends until the inbox completes or the deadline passes.
    /// </summary>
    /// <remarks>
    ///     A ClusterClient establishes contact asynchronously and <c>buffer-size = 0</c> means anything
    ///     sent before that lands is dropped on the floor — by design. A single send would therefore
    ///     race the handshake and fail intermittently for a reason that is not a defect. Re-sending
    ///     weakens nothing: the control test re-sends on exactly the same schedule and must still
    ///     receive nothing.
    /// </remarks>
    /// <param name="target">Actor to send to.</param>
    /// <param name="message">Builds the message for each attempt.</param>
    /// <param name="inbox">Completion source the receiving capture actor resolves.</param>
    /// <param name="timeout">How long to keep trying.</param>
    /// <returns>The delivered message, or <see langword="null"/> if none arrived.</returns>
    private static async Task<object?> SendUntilDeliveredAsync(
        IActorRef target,
        Func<object> message,
        TaskCompletionSource<object> inbox,
        TimeSpan timeout)
    {
        var deadline = DateTime.UtcNow + timeout;
        while (DateTime.UtcNow < deadline)
        {
            if (inbox.Task.IsCompleted) return await inbox.Task;
            target.Tell(message());
            await Task.Delay(250, TestContext.Current.CancellationToken);
        }
        return inbox.Task.IsCompleted ? await inbox.Task : null;
    }
    private static TaskCompletionSource<object> NewInbox() =>
        new(TaskCreationOptions.RunContinuationsAsynchronously);
    private static ImmutableHashSet<ActorPath> ReceptionistOf(ActorSystem system) =>
        ImmutableHashSet.Create<ActorPath>(
            new RootActorPath(AkkaCluster.Get(system).SelfAddress) / "system" / "receptionist");
    private static async Task ShutdownAsync(ActorSystem? system)
    {
        if (system is null) return;
        await system.Terminate().WaitAsync(TimeSpan.FromSeconds(15));
    }
    /// <summary>Completes an inbox with the first message it receives.</summary>
    private sealed class CaptureActor : ReceiveActor
    {
        public static Props Props(TaskCompletionSource<object> sink) =>
            Akka.Actor.Props.Create(() => new CaptureActor(sink));
        public CaptureActor(TaskCompletionSource<object> sink) => ReceiveAny(m => sink.TrySetResult(m));
    }
 }
@@ -0,0 +1,66 @@
 using Akka.Actor;
 using Shouldly;
 using Xunit;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
 /// <summary>
 ///     Pins the two mesh comm-actor paths against a <b>really booted</b> host.
 /// </summary>
 /// <remarks>
 ///     <para>
 ///         <c>/user/central-communication</c> and <c>/user/node-communication</c> are the wire
 ///         contract of the central↔node boundary: central sends to a path string, and the node
 ///         registers under one. Nothing else in the codebase asserts they agree. Renaming either
 ///         actor would compile, pass every unit test, deploy — and deliver nothing, with no error
 ///         on either side, because a ClusterClient send to an unregistered path is simply dropped.
 ///     </para>
 ///     <para>
 ///         Booting the real host is the point. A test that spawned the actors itself would pin the
 ///         constant against itself and prove nothing about the wiring in
 ///         <c>WithOtOpcUaControlPlaneSingletons</c> / <c>WithOtOpcUaRuntimeActors</c>.
 ///     </para>
 /// </remarks>
 public class MeshCommActorPathTests
 {
    private static async Task AssertActorExistsAsync(ActorSystem system, string path)
    {
        var identity = await system.ActorSelection(path)
            .Ask<ActorIdentity>(new Identify(path), TimeSpan.FromSeconds(10));
        identity.Subject.ShouldNotBeNull(
            $"{path} is the ClusterClient wire contract for the central↔node boundary; nothing else "
            + "pins it, and a mismatch delivers nothing without erroring on either side");
    }
    [Fact]
    public async Task Both_mesh_comm_actors_exist_at_their_contract_paths()
    {
        await using var harness = await TwoNodeClusterHarness.StartAsync();
        // Harness nodes carry admin+driver, so BOTH comm actors live on each of them.
        await AssertActorExistsAsync(harness.NodeASystem, MeshPaths.CentralCommunication);
        await AssertActorExistsAsync(harness.NodeASystem, MeshPaths.NodeCommunication);
        await AssertActorExistsAsync(harness.NodeBSystem, MeshPaths.CentralCommunication);
        await AssertActorExistsAsync(harness.NodeBSystem, MeshPaths.NodeCommunication);
    }
    // NOT TESTED HERE: that the comm actors are registered per node rather than as cluster
    // singletons. Two candidate discriminators were tried and BOTH were invalid:
    //
    //   * "the path resolves on both nodes" — a ClusterSingletonManager is also created on every
    //     node in the role, so this passes under either shape.
    //   * "no /singleton child exists" — measured against the KNOWN singleton /user/config-publish,
    //     that child is null there too, so Akka.Hosting does not lay singletons out this way and
    //     the assertion proves nothing.
    //
    // A sabotage that re-registered CentralCommunicationActor via WithSingleton left both green,
    // which is what exposed them. Rather than ship an assertion I could not make fail, the property
    // is left to live-gate step 8 (stop the OLDEST central; the survivor's comm actor must still
    // receive a node ack, which a singleton hosted on the stopped node cannot).
    //
    // NOT MeshClusterClientBoundaryTests: that class proves a send only arrives if the target is
    // registered with the receptionist, which is necessary but not discriminating — with one node
    // per side, per-node and singleton registration look identical.
 }
@@ -0,0 +1,161 @@
 using Akka.Actor;
 using Akka.TestKit.Xunit2;
 using Shouldly;
 using Xunit;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
 using ZB.MOM.WW.OtOpcUa.Commons.Types;
 using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
 using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
 using Serilog;
 using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
 using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
 using ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms;
 using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
 using ZB.MOM.WW.OtOpcUa.Core.Scripting;
 namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Communication;
 /// <summary>
 ///     Proves the Phase 2 node-side leg joins up: a command handed to
 ///     <see cref="NodeCommunicationActor"/> reaches the <b>real</b> <see cref="DriverHostActor"/>,
 ///     not just a test probe.
 /// </summary>
 /// <remarks>
 ///     <para>
 ///         The unit tests either side of this seam both pass with the wiring broken.
 ///         <c>NodeCommunicationActorTests</c> asserts a probe subscribed to the EventStream receives
 ///         the message; the driver-host tests drive the actor by direct <c>Tell</c>. Neither notices
 ///         if <c>DriverHostActor.PreStart</c> never subscribes. That gap is why this file exists.
 ///     </para>
 ///     <para>
 ///         <b>Ordering matters, and it bit this test first time out.</b> <c>ActorOf</c> returns
 ///         before <c>PreStart</c> has run, and an <c>EventStream.Publish</c> with no subscriber is
 ///         dropped silently — there is no buffering and no dead-letter. Relaying immediately after
 ///         spawning therefore fails intermittently. The warm-up below is not ceremony: it is the
 ///         only available proof that <c>PreStart</c> has completed. (In production the driver host
 ///         is spawned at startup, long before any deployment, so the same race does not arise.)
 ///     </para>
 /// </remarks>
 public class EventStreamCommandDeliveryTests : RuntimeActorTestBase
 {
    private static DispatchDeployment NewDispatch() => new(
        new DeploymentId(Guid.NewGuid()),
        new RevisionHash("rev-does-not-exist"),
        new CorrelationId(Guid.NewGuid()));
    [Fact]
    public void A_dispatch_relayed_by_the_comm_actor_reaches_the_real_driver_host()
    {
        var ackRouter = CreateTestProbe();
        var driverHost = Sys.ActorOf(
            DriverHostActor.Props(NewInMemoryDbFactory(), new NodeId("127.0.0.1:4053"), ackRouter: ackRouter.Ref),
            "driver-host");
        var comm = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null), "node-communication");
        // Warm-up: a direct Tell whose ack proves PreStart has completed, and with it the
        // EventStream subscription. Without this the relay below races the subscription.
        var warmup = NewDispatch();
        driverHost.Tell(warmup);
        ackRouter.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(15)).DeploymentId.ShouldBe(warmup.DeploymentId);
        // Now the real assertion — relayed exactly as central would send it. The comm actor only
        // publishes node-locally, so the host must be subscribed for this to arrive at all.
        var relayed = NewDispatch();
        comm.Tell(relayed);
        // The deployment id does not exist, so the host answers Failed. That is the stronger
        // assertion, not a weaker one: an ack at all proves the command was received and handled,
        // and only PreStart's EventStream subscription can have delivered it.
        ackRouter.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(15)).DeploymentId.ShouldBe(relayed.DeploymentId);
    }
    [Fact]
    public void CONTROL_the_driver_host_acks_a_direct_tell()
    {
        // Falsifiability control. If the test above fails, this one says which half broke: green
        // here means the host is healthy and the EventStream relay is at fault; red here means the
        // host would not have acked either way and the relay is not implicated.
        var ackRouter = CreateTestProbe();
        var driverHost = Sys.ActorOf(
            DriverHostActor.Props(NewInMemoryDbFactory(), new NodeId("127.0.0.1:4053"), ackRouter: ackRouter.Ref),
            "driver-host-control");
        var dispatch = NewDispatch();
        driverHost.Tell(dispatch);
        ackRouter.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(15)).DeploymentId.ShouldBe(dispatch.DeploymentId);
    }
 }
 /// <summary>
 ///     The alarm leg of the same wiring proof, in its own ActorSystem because the shared
 ///     <see cref="RuntimeActorTestBase"/> harness pins <c>loglevel = WARNING</c> and the only
 ///     observable signal <c>ScriptedAlarmHostActor</c> gives for an unowned alarm is a Debug line.
 /// </summary>
 public class AlarmCommandEventStreamDeliveryTests : TestKit
 {
    private static string DebugLevelHocon => @"
 akka {
    loglevel = ""DEBUG""
    extensions = [
        ""Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider, Akka.Cluster.Tools""
    ]
    actor {
        provider = ""Akka.Cluster.ClusterActorRefProvider, Akka.Cluster""
    }
    remote.dot-netty.tcp {
        hostname = ""127.0.0.1""
        port = 0
    }
    cluster {
        seed-nodes = []
        roles = [""driver""]
        min-nr-of-members = 1
        run-coordinated-shutdown-when-down = off
    }
 }";
    public AlarmCommandEventStreamDeliveryTests()
        : base(DebugLevelHocon)
    {
        var cluster = Akka.Cluster.Cluster.Get(Sys);
        cluster.Join(cluster.SelfAddress);
        AwaitCondition(
            () => cluster.State.Members.Any(m => m.Status == Akka.Cluster.MemberStatus.Up),
            TimeSpan.FromSeconds(5));
    }
    [Fact]
    public void An_alarm_command_relayed_by_the_comm_actor_reaches_the_real_scripted_alarm_host()
    {
        // Needed for the same reason as the deploy leg: the comm actor's unit test only proves a
        // probe receives it, and the alarm-host tests drive the actor by direct Tell. Neither
        // notices a missing PreStart subscription.
        var upstream = new DependencyMuxTagUpstreamSource();
        var logger = new LoggerConfiguration().CreateLogger();
        var engine = new ScriptedAlarmEngine(
            upstream, new InMemoryAlarmStateStore(), new ScriptLoggerFactory(logger), logger);
        var host = Sys.ActorOf(
            ScriptedAlarmHostActor.Props(
                CreateTestProbe().Ref, CreateTestProbe().Ref, upstream, engine, new NodeId("node-A")),
            "scripted-alarm-host");
        var comm = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null), "node-communication");
        var cmd = new AlarmCommand("alarm-unowned", "Acknowledge", "alice", "c", null);
        // Warm-up, OUTSIDE the real assertion: a direct Tell whose debug line proves PreStart has
        // completed and the EventStream subscription is registered. An earlier draft did the direct
        // Tell INSIDE the assertion window alongside the relay, which made the test vacuous — the
        // direct Tell alone produced the log the assertion was waiting for.
        EventFilter.Debug(contains: "ignoring AlarmCommand")
            .ExpectOne(TimeSpan.FromSeconds(15), () => host.Tell(cmd));
        // The real assertion: relayed ONLY through the comm actor, which publishes node-locally.
        // The host owns no alarms, so it logs and stops — still a positive delivery signal, because
        // that line is only reached if the message arrived.
        EventFilter.Debug(contains: "ignoring AlarmCommand")
            .ExpectOne(TimeSpan.FromSeconds(15), () => comm.Tell(cmd));
    }
 }
@@ -0,0 +1,130 @@
 using Akka.Actor;
 using Akka.Cluster.Tools.Client;
 using Shouldly;
 using Xunit;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
 using ZB.MOM.WW.OtOpcUa.Commons.Types;
 using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
 using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
 namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Communication;
 public class NodeCommunicationActorTests : RuntimeActorTestBase
 {
    private static DispatchDeployment NewDispatch() => new(
        new DeploymentId(Guid.NewGuid()),
        new RevisionHash("rev-1"),
        new CorrelationId(Guid.NewGuid()));
    [Fact]
    public void DispatchDeployment_from_central_lands_on_the_node_local_event_stream()
    {
        var subscriber = CreateTestProbe();
        Sys.EventStream.Subscribe(subscriber.Ref, typeof(DispatchDeployment));
        var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null));
        var dispatch = NewDispatch();
        actor.Tell(dispatch);
        subscriber.ExpectMsg<DispatchDeployment>(TimeSpan.FromSeconds(5)).ShouldBe(dispatch);
    }
    [Fact]
    public void A_command_is_published_exactly_once()
    {
        // The reason inbound commands do NOT go back onto their DPS topic: central SendToAll-s to
        // every node's comm actor, so a mesh-wide republish would deliver N copies to every
        // subscriber. One publish per inbound command is the whole point.
        var subscriber = CreateTestProbe();
        Sys.EventStream.Subscribe(subscriber.Ref, typeof(DispatchDeployment));
        var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null));
        actor.Tell(NewDispatch());
        subscriber.ExpectMsg<DispatchDeployment>(TimeSpan.FromSeconds(5));
        subscriber.ExpectNoMsg(TimeSpan.FromMilliseconds(300));
    }
    [Fact]
    public void RestartDriver_from_central_lands_on_the_event_stream()
    {
        var subscriber = CreateTestProbe();
        Sys.EventStream.Subscribe(subscriber.Ref, typeof(RestartDriver));
        var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null));
        var msg = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
        actor.Tell(msg);
        subscriber.ExpectMsg<RestartDriver>(TimeSpan.FromSeconds(5)).ShouldBe(msg);
    }
    [Fact]
    public void ReconnectDriver_from_central_lands_on_the_event_stream()
    {
        var subscriber = CreateTestProbe();
        Sys.EventStream.Subscribe(subscriber.Ref, typeof(ReconnectDriver));
        var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null));
        var msg = new ReconnectDriver("C1", "driver-1", "alice", Guid.NewGuid());
        actor.Tell(msg);
        subscriber.ExpectMsg<ReconnectDriver>(TimeSpan.FromSeconds(5)).ShouldBe(msg);
    }
    [Fact]
    public void AlarmCommand_from_central_lands_on_the_event_stream()
    {
        // ScriptedAlarmHostActor is a CHILD of DriverHostActor with no registry key, so there is no
        // ref to hand this actor at wiring time — the EventStream is what makes the alarm-command
        // leg possible without a registration handshake.
        var subscriber = CreateTestProbe();
        Sys.EventStream.Subscribe(subscriber.Ref, typeof(AlarmCommand));
        var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null));
        var msg = new AlarmCommand("alarm-1", "Acknowledge", "alice", "comment", null);
        actor.Tell(msg);
        subscriber.ExpectMsg<AlarmCommand>(TimeSpan.FromSeconds(5)).ShouldBe(msg);
    }
    [Fact]
    public void ApplyAck_goes_to_central_as_a_Send_not_a_SendToAll()
    {
        // Send, not SendToAll: the deploy coordinator is one singleton behind the central comm
        // actor's proxy. SendToAll would deliver one ack per central node and the coordinator would
        // count this node twice.
        var client = CreateTestProbe();
        var actor = Sys.ActorOf(NodeCommunicationActor.Props(client.Ref));
        var ack = new ApplyAck(
            new DeploymentId(Guid.NewGuid()),
            new NodeId("host-a:4053"),
            ApplyAckOutcome.Applied,
            null,
            new CorrelationId(Guid.NewGuid()));
        actor.Tell(ack);
        var sent = client.ExpectMsg<ClusterClient.Send>(TimeSpan.FromSeconds(5));
        sent.Path.ShouldBe(MeshPaths.CentralCommunication);
        sent.Message.ShouldBe(ack);
    }
    [Fact]
    public void An_ack_with_no_central_client_is_dropped_with_a_warning()
    {
        var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null));
        var ack = new ApplyAck(
            new DeploymentId(Guid.NewGuid()),
            new NodeId("host-a:4053"),
            ApplyAckOutcome.Applied,
            null,
            new CorrelationId(Guid.NewGuid()));
        EventFilter.Warning(contains: "dropping ApplyAck").ExpectOne(
            TimeSpan.FromSeconds(5), () => actor.Tell(ack));
    }
 }
@@ -41,7 +41,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
        var cache = new StubArtifactCache(cached);
        var actor = Sys.ActorOf(DriverHostActor.Props(
-            new ThrowingDbFactory(), TestNode, coordinator: null,
+            new ThrowingDbFactory(), TestNode, ackRouter: null,
            localRoles: new HashSet<string> { "driver" },
            deploymentArtifactCache: cache));
@@ -70,7 +70,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
        var publishProbe = CreateTestProbe();
        Sys.ActorOf(DriverHostActor.Props(
-            new ThrowingDbFactory(), TestNode, coordinator: null,
+            new ThrowingDbFactory(), TestNode, ackRouter: null,
            localRoles: new HashSet<string> { "driver" },
            opcUaPublishActor: publishProbe.Ref,
            deploymentArtifactCache: new StubArtifactCache(cached)));
@@ -85,7 +85,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
    {
        // The pre-cache behaviour, unchanged. A cache miss must not invent a configuration.
        var actor = Sys.ActorOf(DriverHostActor.Props(
-            new ThrowingDbFactory(), TestNode, coordinator: null,
+            new ThrowingDbFactory(), TestNode, ackRouter: null,
            localRoles: new HashSet<string> { "driver" },
            deploymentArtifactCache: new StubArtifactCache(current: null)));
@@ -100,7 +100,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
    {
        // Admin-only graphs and older harnesses pass null; behaviour is identical to a cache miss.
        var actor = Sys.ActorOf(DriverHostActor.Props(
-            new ThrowingDbFactory(), TestNode, coordinator: null,
+            new ThrowingDbFactory(), TestNode, ackRouter: null,
            localRoles: new HashSet<string> { "driver" }));
        var snapshot = AskDiagnostics(actor);
@@ -118,7 +118,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
            DeploymentId.NewId().ToString(), CachedRev.Value, EmptyArtifact(), DateTimeOffset.UtcNow));
        var actor = Sys.ActorOf(DriverHostActor.Props(
-            NewInMemoryDbFactory(), TestNode, coordinator: null,
+            NewInMemoryDbFactory(), TestNode, ackRouter: null,
            localRoles: new HashSet<string> { "driver" },
            deploymentArtifactCache: cache));
@@ -140,7 +140,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
            deploymentId.ToString(), CachedRev.Value, EmptyArtifact(), DateTimeOffset.UtcNow));
        var actor = Sys.ActorOf(DriverHostActor.Props(
-            db, TestNode, coordinator: null,
+            db, TestNode, ackRouter: null,
            localRoles: new HashSet<string> { "driver" },
            deploymentArtifactCache: cache));
@@ -156,7 +156,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
    {
        // A cache fault must leave the node exactly where it would have been without a cache.
        var actor = Sys.ActorOf(DriverHostActor.Props(
-            new ThrowingDbFactory(), TestNode, coordinator: null,
+            new ThrowingDbFactory(), TestNode, ackRouter: null,
            localRoles: new HashSet<string> { "driver" },
            deploymentArtifactCache: new ThrowingReadCache()));
@@ -183,7 +183,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
        var cache = new StubArtifactCache(current: null);
        var actor = Sys.ActorOf(DriverHostActor.Props(
-            db, TestNode, coordinator: null,
+            db, TestNode, ackRouter: null,
            localRoles: new HashSet<string> { "driver" },
            deploymentArtifactCache: cache));
@@ -183,7 +183,7 @@ public sealed class DriverHostActorRoleViewTests : RuntimeActorTestBase
        Sys.ActorOf(DriverHostActor.Props(
            NewInMemoryDbFactory(),
            TestNode,
-            coordinator: null,
+            ackRouter: null,
            driverMemberCountProvider: () => driverMemberCount,
            redundancyRoleView: view,
            replicationPeerHost: PeerHost));
@@ -139,7 +139,7 @@ public sealed class DependencyMuxActorTests : RuntimeActorTestBase
        var hostActor = Sys.ActorOf(DriverHostActor.Props(
            dbFactory: NewInMemoryDbFactory(),
            localNode: ZB.MOM.WW.OtOpcUa.Commons.Types.NodeId.Parse("host-1"),
-            coordinator: hostProbe.Ref,
+            ackRouter: hostProbe.Ref,
            dependencyMux: mux));
        // Tell the host an AttributeValuePublished — it should fan out to the mux + subscriber.