docs(mesh): sweep for ClusterClient deletion — Dps|Grpc only, CentralContactPoints retired

Task 3.2 of the ClusterClient->gRPC migration. Sweeps the docs that still
described the Akka ClusterClient transport as live, following its Phase 3
deletion (this branch): MeshTransport:Mode is now Dps (pair-local DPS) or
Grpc (cross-mesh unary calls) only, ClusterClient fails validation loudly,
CentralContactPoints is deleted (not superseded), and the split-topology
validator requires exactly Grpc rather than ClusterClient.

- docs/Configuration.md: rewrite the MeshTransport section — table,
  dark-switch/PSK-seeding prose, listener sharing (:4056 driver-role,
  :4055 admin-role), remove the "shipped dark, cutover pending" framing.
- docs/Redundancy.md: replace the Dps/ClusterClient transport table with
  Dps/Grpc, fix the Phase 6 CentralCommunicationActor + validator bullets.
- docs/Telemetry.md: fix stale ClusterClient cross-refs, add a note that
  the driver-role telemetry listener also hosts NodeCommandService.
- docs/v2/config-db-schema.md: one-line "(retired: ...)" annotation on the
  historical AkkaPort/GrpcPort note — left as a historical record otherwise.
- CLAUDE.md: rewrite "## Mesh command transport (MeshTransport)" for the
  post-deletion world (no ClusterClient/receptionist, per-node gRPC dial,
  fire-and-forget semantics unchanged) and fix two Phase 6 bullets.

deferment.md has no mesh-transport mentions; left untouched.

docs-only change; `dotnet build ZB.MOM.WW.OtOpcUa.slnx` stays green (0
errors) after the edit.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1

CLAUDE.md

@@ -371,8 +371,8 @@ Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBr
 
 - **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.** ⚠️ **Wiring landmine (fixed `3a4ed8dd`, caught by the live gate):** the singleton's cluster-role *scope* is computed at `AddAkka`-configurator time, so it must come from `AkkaClusterOptions` (config) — **never** by resolving `IClusterRoleInfo` from the SP there. `ClusterRoleInfo` depends on the `ActorSystem`, and that lambda runs *while the ActorSystem is being built*, so resolving it recurses into ActorSystem construction and **stack-overflows every node at boot** (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep `WithOtOpcUaClusterRedundancySingleton`/`BuildClusterRedundancySingletonOptions` taking `AkkaClusterOptions`.
 - `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh.
-- `CentralCommunicationActor` creates **one `ClusterClient` per application `Cluster`** and fans commands across them (closing the `TODO(Phase 6)` Phase 2 left behind), rather than one fleet-wide client.
+- `CentralCommunicationActor` dials each node directly over gRPC — one `NodeCommandService.Execute` call per enabled `ClusterNode` row carrying a `GrpcPort` — rather than fanning through a client/receptionist object. **Superseded by the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md):** the Akka `ClusterClient` this bullet originally described (one per application `Cluster`, closing the `TODO(Phase 6)` Phase 2 left behind) is deleted; the per-node gRPC dial is its replacement, not an evolution of it.
-- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=ClusterClient`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
+- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=Grpc`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
 - **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
 - **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).
 
@@ -380,51 +380,74 @@ See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.m
 
 ## Mesh command transport (`MeshTransport`)
 
-Per-cluster mesh **Phase 2** added a second transport for the three central→node command channels
+**The Akka `ClusterClient` transport described in this section's earlier history is DELETED**
-(`deployments`, `driver-control`, `alarm-commands`) and the `deployment-acks` reply, selected by
+(branch `feat/grpc-phase3-deletion`, [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md),
-`MeshTransport:Mode`: `Dps` (the DistributedPubSub path this repo has always used, **the default**)
+Phase 3) — code, the `akka.cluster.client` HOCON block, the receptionist extension,
-or `ClusterClient`, which does **not** require central and the node to share a gossip ring — the
+`MeshTransport:CentralContactPoints`, and the `ClusterClient` mode value itself, which now **fails
-prerequisite for splitting the fleet into one mesh per application `Cluster`. Both `CentralCommunicationActor`
+validation loudly** at boot (`MeshTransportOptionsValidator` names the key and the allowed values)
-(`/user/central-communication`) and `NodeCommunicationActor` (`/user/node-communication`) are spawned
+rather than silently falling back to `Dps`. What remains is exactly two transports, selected by
-and registered with the `ClusterClientReceptionist` in **both** modes, so flipping the flag is a config
+`MeshTransport:Mode`:
-change, not a redeploy. Things worth knowing before touching it:
 
-- **`SendToAll`, never `Send`.** Today's DPS publish reaches every `DriverHostActor`; there is no
+- **`Dps`** (default) — DistributedPubSub, the path this repo has always used. **Pair-local only**:
-  ClusterId or node filter on the node side at all (scoping happens later, inside the artifact via
+  it works only when central and the node share a gossip ring, which after Phase 6 means within one
-  `DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered
+  2-node mesh.
-  actor — it would deploy to a single node while every other node silently kept its old configuration,
+- **`Grpc`** — unary gRPC calls on the `mesh_transport.v1` contract. **Required** on any node
-  and the deployment could still seal green.
+  carrying a `cluster-{ClusterId}` role; `SplitTopologyTransportValidator` refuses to boot such a
-- **RESOLVED by Phase 6 — one `ClusterClient` per application `Cluster`, not one fleet-wide.** When
+  node on anything else.
-  this was written the fleet was still one mesh, so a receptionist served the whole cluster and
-  `SendToAll` reached every registered node-comm actor regardless of which node's address was dialled;
-  the `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient` asked for one client per application
-  `Cluster` instead. Phase 6 split the fleet into one mesh per `Cluster` and shipped exactly that — see
-  the [Redundancy](#redundancy) section's Phase 6 paragraph above. The `SendToAll`-within-a-mesh
-  behaviour below is otherwise unchanged; a `MaintenanceMode` node still isn't excluded from delivery
-  within its own mesh.
-- **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and
-  central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber.
-  Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both.
-- **Comm actors are per-node, NOT singletons.** A client rotates across contact points and must find a live
-  comm actor at whichever node answers; `akka.cluster.client.receptionist.role` is empty for the same reason.
-- **`buffer-size = 0` is a behaviour decision, not a tuning knob.** Akka's default buffers 1000 and replays
-  on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not raise
-  it to quiet a flaky test. **But do not over-claim what it buys:** the Phase 2 live gate observed a node
-  applying a deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()`
-  replaying the orphan `Applying` row on restart. Zero buffering removes the silent replay, not every replay.
 
-Contact points are node **addresses only** (`akka.tcp://<system>@<host>:<port>`); `/system/receptionist` is
+Central→node commands (`deployments`, `driver-control`, `alarm-commands`) ride `NodeCommandService.Execute`,
-appended at construction and `MeshTransportOptionsValidator` refuses to start a node whose contact carries an
+served on each node's driver-role telemetry h2c listener (`Telemetry:GrpcListenPort`, the docker-dev
-actor-path suffix. `redundancy-state` stays on DPS in both modes (bidirectional, and built from
+rig's `:4056` — it shares that listener with `TelemetryStreamGrpcService` rather than claiming a port
-`Cluster.State`). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md` §"Command transport",
+of its own). `CentralCommunicationActor` dials it **directly, one call per node** — the target set is
-and `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`.
+every enabled, non-maintenance `ClusterNode` row carrying a `GrpcPort`, dialled at `Host`+`GrpcPort`
+and refreshed every `MeshTransport:ContactRefreshSeconds`. There is no client/receptionist object and
+no rotation: each command goes to the specific node dialled, which is what makes per-node delivery
+provable — the old `ClusterClient.SendToAll`/`Send` distinction this section used to explain no longer
+applies to either direction.
+
+Node→central deploy acks ride `CentralCommandService.ReportApply`, served on central's admin-role
+config-serve h2c listener (`ConfigServe:GrpcListenPort`, the rig's `:4055` — again sharing rather than
+claiming a new port). `GrpcApplyAckSender` walks `MeshTransport:CentralGrpcEndpoints` in order and
+stops at the first success.
+
+Auth is a **per-cluster PSK bearer**, both directions: `MeshTransport:GrpcPsk` (node-side, one key per
+application `Cluster`) and `MeshTransport:ClusterPsks` (central-side map, consulted before the Secrets
+store). `MeshCommandAuthInterceptor` (node) and `CentralCommandAuthInterceptor` (central) fail closed
+on every unresolvable case — constant-time compare, no key material logged. Production keys are
+`${secret:OT-MESH-PSK-<clusterId>}`; the docker-dev rig supplies three literal dev keys via env
+(`dev-mesh-psk-MAIN`/`-SITE-A`/`-SITE-B`).
+
+Things that carried over unchanged from the `ClusterClient` era, because they were never
+`ClusterClient`-specific to begin with:
+
+- **Fire-and-forget, no retry, on both transports.** A failed gRPC dispatch is logged (naming the node
+  and the gRPC status) and dropped in the dispatch second. No queue, no ordering guarantee across
+  nodes — the deploy coordinator's apply deadline and a node's boot-time replay are the only recovery.
+  The old `buffer-size = 0` decision this section used to describe is gone with the HOCON block that
+  set it; the gRPC transport has nothing to buffer in the first place — a dial either lands or doesn't.
+- **Inbound commands re-emit on the node's `EventStream`, not back onto DPS.** DPS is mesh-wide, so a
+  DPS re-publish would deliver N copies to every subscriber. Node-side subscribers (`DriverHostActor`,
+  `ScriptedAlarmHostActor`) accept a command from either transport identically.
+- **Do not over-claim what fire-and-forget buys.** The Phase 2 live gate observed a node applying a
+  deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()`
+  replaying the orphan `Applying` row on restart. That backstop is unaffected by the ClusterClient
+  deletion — it was always about the deploy path, not the transport.
+
+`redundancy-state` stays on DPS in both modes (bidirectional, and built from `Cluster.State`, which
+makes it genuinely mesh-bound). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md`
+§"Command transport", and the migration's [design doc](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) /
+[plan](docs/plans/2026-08-06-clusterclient-to-grpc.md) / [tracking doc](docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md) /
+[live gate](docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the phase-by-phase history —
+those are historical records, not updated further by this sweep. `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`
+is the original Phase 2 design doc and is likewise left as a historical record of the transport this
+section now describes as deleted.
 
 **Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
 
 - **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
 - **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
 - **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
-- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever.
+- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever. **Resolved, and not the way this bullet originally expected:** the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) deleted the Akka `ClusterClient` transport that would have dialled `Host`+`AkkaPort`; the Phase 2 command dial now uses `Host`+`GrpcPort`, the same column Phase 5 telemetry already dialled — `AkkaPort` is Akka remoting only again.
 
 ## Config source (`ConfigSource` / `ConfigServe`)
 
@@ -488,8 +511,9 @@ touching it:
   `Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an
   observability panel); `fleet-status` is central-internal (`FleetStatusBroadcaster` builds it from
   the admin node's own membership events, `Fleet.razor` polls the Config DB and ignores the feed —
-  never a node→central stream); `deployment-acks` already rides the Phase 2 `ClusterClient`
+  never a node→central stream); `deployment-acks` already rides the Phase 2 gRPC command transport
-  transport as a command-plane reply, not telemetry.
+  (`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a command-plane reply, not
+  telemetry.
 - **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits
   into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC
   server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only

docs/Configuration.md

@@ -142,7 +142,7 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
 - **`SplitTopologyTransportValidator`** (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`, wired at
   `ValidateOnStart` alongside the other `Cluster`-family validators) fails host start on any node
   carrying a `cluster-{ClusterId}` role unless it is also configured for the split-safe transports:
-  `MeshTransport:Mode = ClusterClient`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode =
+  `MeshTransport:Mode = Grpc`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode =
   Grpc` on an admin node. A DPS-mode transport on a cluster-role node would try to gossip across a
   partition boundary that no longer exists, so the validator fails closed rather than let commands or
   telemetry silently vanish. A legacy node with no `cluster-{ClusterId}` role is exempt and may stay on
@@ -155,45 +155,38 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
   actually carries a `cluster-{ClusterId}` role must set the split-safe modes explicitly.
 - See [`Redundancy.md` § Per-cluster meshes (Phase 6)](Redundancy.md#per-cluster-meshes-phase-6) for the
   full runtime picture (one Primary per pair, the cluster-scoped `RedundancyStateActor` singleton,
-  `ClusterNodeAddressReconciler` own-cluster-scoping, and central's one-`ClusterClient`-per-`Cluster`
+  `ClusterNodeAddressReconciler` own-cluster-scoping, and central's per-node gRPC command dial —
-  fan-out).
+  one `NodeCommandService.Execute` call per enabled `ClusterNode` row carrying a `GrpcPort`).
 
 ### `MeshTransport` (central↔node command transport)
 
-- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split.
+- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub (pair-local only), or over unary gRPC calls that do **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs, now that the meshes are split. The Akka `ClusterClient` transport that used to be the second option here is **deleted** ([ClusterClient→gRPC migration](plans/2026-08-06-clusterclient-to-grpc-design.md), Phase 3) — `Grpc` is its replacement, not a third option alongside it.
 - **Options class:** `MeshTransportOptions` (`SectionName = "MeshTransport"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`. Bound and validated by `AddOtOpcUaCluster(config)`; `MeshTransportOptionsValidator` runs at `ValidateOnStart`.
 
 | Key | Type | Default | Meaning |
 |---|---|---|---|
-| `Mode` | string | `Dps` | `Dps` = DistributedPubSub (the transport this repo has always used). `ClusterClient` = the mesh boundary. `Grpc` = the `mesh_transport.v1` gRPC contract — **shipped dark, cutover pending** (see below). Any other value **refuses to start**. |
+| `Mode` | string | `Dps` | `Dps` = DistributedPubSub, pair-local only — cannot cross an application-`Cluster` boundary. `Grpc` = the `mesh_transport.v1` gRPC contract (below), required on any node carrying a `cluster-{ClusterId}` role. Any other value **refuses to start**, including the retired `ClusterClient` value — a node still configured for it is refused at boot rather than silently demoted to `Dps`, which on a split-topology node would deliver nothing at all. |
-| `CentralContactPoints` | string[] | `[]` | Node-side only: central node **addresses** (`akka.tcp://<system>@<host>:<port>`). Required under `ClusterClient`, ignored under `Dps`/`Grpc`. |
+| `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its node dispatch set — the gRPC dial targets and the known-cluster PSK gate — from the `ClusterNode` rows. Runs in every mode; under `Dps` it still keeps the inbound-ack endpoint's known-cluster gate current even though nothing dials out. |
-| `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its ClusterClient contact set from the `ClusterNode` rows. |
 
-**Addresses only — no actor-path suffix.** `/system/receptionist` is appended at construction time; a contact carrying `/user/` or `/system/` is rejected at startup, because a doubled suffix produces a path that resolves to nothing and drops every command in silence.
+**`Grpc` mode: central dials each node, a node posts its ack to central.** `CentralCommunicationActor` fans one `NodeCommandService.Execute` call per node — the target set is every enabled, non-maintenance `ClusterNode` row that carries a `GrpcPort` (dialled at `Host`+`GrpcPort`), rebuilt every `ContactRefreshSeconds`. There is no receptionist and no single fleet-wide (or per-cluster) client object: it is a plain per-node dial. A node posts its `ApplyAck` with `GrpcApplyAckSender`, which walks `CentralGrpcEndpoints` in order and stops at the first success — an availability list, not a fan-out list, so central normally sees one ack per node per dispatch (the walk as a whole is at-least-once on a slow-then-retried endpoint; `ConfigPublishCoordinator.HandleAck` absorbs a duplicate idempotently). Both directions are **fire-and-forget, with no retry**: a failed dispatch is logged (naming the node and the gRPC status) and dropped in the dispatch second — the deploy path's 2-minute apply deadline and a node's boot-time replay (`DriverHostActor.Bootstrap()` re-applying an orphan `Applying` row) are the only recovery mechanisms, exactly as they were under `ClusterClient`.
-
-**It is a dark switch, and both sides are always wired.** The comm actors are spawned and registered with the `ClusterClientReceptionist` on every node in *both* modes, so flipping `Mode` is a config change rather than a redeploy. Under `Dps` the registration is simply idle.
-
-**Central's contact set comes from the enabled, non-maintenance `ClusterNode` rows** — the same source Phase 1 gave the deploy ack set. A node in `MaintenanceMode` is not dialled, but note that **the contact set does not scope delivery**: a receptionist serves its whole cluster, so `SendToAll` still reaches every registered node exactly as today's DPS broadcast does. The filter is about which receptionists are worth dialling, not about who receives.
-
-**Buffering is off by design** (`akka.cluster.client.buffer-size = 0`, set in the shipped `akka.conf`). If no receptionist is known, the command is dropped rather than queued. Akka's default of 1000 would replay a `DispatchDeployment` on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not "fix" it back to quiet a flaky test.
-
-> **What this does *not* buy you.** It does not guarantee a node never applies a deployment the DB records as `TimedOut`. A node that missed a dispatch keeps the orphan `Applying` row the coordinator pre-created, and `DriverHostActor.Bootstrap()` replays that row on the node's next restart — live-observed applying a deployment 44 s after it was sealed `TimedOut` (see the [Phase 2 live gate](plans/2026-07-22-mesh-phase2-live-gate.md), finding B). The boot replay is the better-behaved of the two — deliberate, Warning-logged, and bounded to a restart — but it is still a replay. `buffer-size = 0` removes the *silent, timer-driven* one only.
-
-On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`.
-
-**`Grpc` mode — shipped dark, cutover pending** ([ClusterClient→gRPC migration design](plans/2026-08-06-clusterclient-to-grpc-design.md)). A third `Mode` value replaces `ClusterClient` with unary gRPC calls on the `mesh_transport.v1` contract: central dials each node's `NodeCommandService`, a node posts its deployment ack to central's `CentralCommandService`. Unlike `ClusterClient`, the command plane is authenticated end to end (`GrpcPsk` below) rather than relying on the mesh boundary alone. As of Phase 0 only the contract, DTO mapper, options and the two auth interceptors exist — the compiled default and the docker-dev rig both stay on `ClusterClient`, and setting `Mode=Grpc` today validates the keys below but nothing yet dials or serves over the contract.
 
 | Key | Type | Default | Meaning |
 |---|---|---|---|
 | `GrpcPsk` | string? | `null` (unset) | Node-side only, required under `Grpc` on a driver-role node: this node's pre-shared key for its own `Cluster`, presented as the bearer on outbound acks and required on inbound `NodeCommandService` calls. Production form is `${secret:OT-MESH-PSK-<clusterId>}`; supply a literal only on a rig, via `MeshTransport__GrpcPsk` — never commit one. |
-| `CentralGrpcEndpoints` | string[] | `[]` | Node-side only, required under `Grpc` on a driver-role node: full gRPC base URLs (`scheme://host:port`, e.g. `http://central-1:4055` for h2c) of the central pair's `CentralCommandService`, tried in order, first success wins. The `Grpc` counterpart of `CentralContactPoints`, and replaces it — a node on `Grpc` never reads the Akka contact points. |
+| `CentralGrpcEndpoints` | string[] | `[]` | Node-side only, required under `Grpc` on a driver-role node: full gRPC base URLs (`scheme://host:port`, e.g. `http://central-1:4055` for h2c) of the central pair's `CentralCommandService`, tried in order, first success wins. There is no Akka-side counterpart any more — the old `MeshTransport:CentralContactPoints` (the `ClusterClient` contact-point list) is deleted along with `ClusterClient` itself, not merely superseded. |
 | `DispatchTimeoutSeconds` | int | `10` | Central-side per-node deadline for one `NodeCommandService.Execute` call. Must be positive under `Grpc`; validated on driver-role nodes (covers the central pair, which is always fused). |
 | `AckTimeoutSeconds` | int | `10` | Node-side per-endpoint deadline for one `CentralCommandService.ReportApply` call. Must be positive under `Grpc`. Applied per endpoint walked in `CentralGrpcEndpoints`, not per ack, so a dead first endpoint costs this deadline before the second is tried. |
 | `ClusterPsks` | `Dictionary<string,string>` | `{}` (case-insensitive keys) | **Central-side only** override map, keyed by `Cluster` id: `IMeshPskProvider` consults this map *before* the Secrets store when central builds a channel to (or verifies an ack from) a given cluster. Production leaves this empty and resolves `OT-MESH-PSK-{clusterId}` from Secrets instead; the map exists so the docker-dev rig and integration tests can supply literals via the environment (`MeshTransport__ClusterPsks__SITE-A`) without standing up a KEK. |
 
+**Auth is per-cluster PSK bearer, both directions, fail-closed.** `GrpcPsk` (node-side) and `ClusterPsks` (central-side) are two ends of the same key, one per application `Cluster` — not a fleet-wide shared bearer, because this channel carries `RestartDriver`, `AlarmCommand` and deployment dispatch, and a single fleet key would let one compromised site node command every other site or forge acks fleet-wide. Both `MeshCommandAuthInterceptor` (node-side, gates `NodeCommandService`) and `CentralCommandAuthInterceptor` (central-side, gates `CentralCommandService`) reject every call when no key is configured for the relevant cluster, compare in constant time, and never log key material. **PSK seeding:** production form is a secret reference, `${secret:OT-MESH-PSK-<clusterId>}` (e.g. `${secret:OT-MESH-PSK-SITE-A}`), resolved by the Secrets store so it works on a driver-only node with no ConfigDb; the docker-dev rig instead supplies three literal dev keys via environment — `dev-mesh-psk-MAIN`, `dev-mesh-psk-SITE-A`, `dev-mesh-psk-SITE-B` — one per cluster, node-side via `MeshTransport__GrpcPsk` and central-side via `MeshTransport__ClusterPsks__<clusterId>`. Never commit a literal key.
+
 **`ClusterPsks` / `OT-MESH-PSK-<clusterId>` and `SecretName`'s allow-list.** `SecretName` (`ZB.MOM.WW.Secrets.Abstractions`) lowercases the secret name and restricts it to `[a-z0-9._/-]` — a `Cluster` id that carries any character outside that set cannot be expressed as `OT-MESH-PSK-<clusterId>` at all, so `MeshPskProvider.TryGetPsk` fails closed for that cluster on every call that reaches the store (`SecretName`'s constructor throws, `MeshPskProvider.ResolveFromStore` catches it, logs a warning, and returns no key) — `MeshTransport:ClusterPsks` is the only path that still works for such an id. The rig's three ids (`MAIN`, `SITE-A`, `SITE-B`) are all within the allow-list and resolve normally either way.
 
-The `Grpc` mode's node-side comm actors and central-side services are not yet wired to `Mode` — Phase 0 lands only the contract, mapper, options and the two auth interceptors (`MeshCommandAuthInterceptor` node-side, `CentralCommandAuthInterceptor` central-side), both dark. See the [design doc](plans/2026-08-06-clusterclient-to-grpc-design.md) and [tracking doc](plans/2026-08-06-clusterclient-to-grpc-tracking.md) for the remaining phases.
+**Both command-plane gRPC services ride an existing listener — neither claims a port of its own.** `NodeCommandService` is mapped alongside `TelemetryStreamGrpcService` on the driver-role node's `Telemetry:GrpcListenPort` listener (the docker-dev rig's `:4056`) whenever that port is bound, deliberately NOT gated on `MeshTransport:Mode` — mapping it under `Dps` costs nothing (nothing dials it), whereas gating it would mean a running fleet needs a central restart before any node could ack after flipping the mode. `CentralCommandService` (the node→central ack ingress) is mapped alongside `DeploymentArtifactService` on the admin-role node's `ConfigServe:GrpcListenPort` listener (the rig's `:4055`), for the same reason — see [`docs/Telemetry.md`](Telemetry.md) for the shared driver-role listener and [§ `ConfigSource`/`ConfigServe`](#configsource--configserve-config-fetch-and-cache) for the shared admin-role one.
+
+On the docker-dev rig every node carries these keys already; the rig's compose default is `Grpc` (`OTOPCUA_MESH_MODE:-Grpc`, cutover commit `caf2eec6`) — the compiled class default in `MeshTransportOptions` stays `Dps`, so an unconfigured deployment outside the rig still needs `Mode=Grpc` set explicitly. Every rig node carries a `cluster-{ClusterId}` role (Phase 6), so `OTOPCUA_MESH_MODE=Dps` is not a usable override on this rig as shipped — `SplitTopologyTransportValidator` refuses to boot any of them on `Dps`.
+
+See the [ClusterClient→gRPC migration design](plans/2026-08-06-clusterclient-to-grpc-design.md), [implementation plan](plans/2026-08-06-clusterclient-to-grpc.md), [tracking doc](plans/2026-08-06-clusterclient-to-grpc-tracking.md) and [live gate](plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the migration's phase-by-phase history — those are historical records of the cutover and are not updated further by this sweep.
 
 ### `ConfigSource` / `ConfigServe` (config fetch-and-cache)
 
@@ -284,7 +277,7 @@ non-empty `ApiKey`** — fail-closed, same as `ConfigServe`/`LocalDb:Replication
 `redundancy-state` (pair-local, built from `Cluster.State`, stays on DPS in both modes — see
 [`Redundancy.md`](Redundancy.md#command-transport-centralnode)), `fleet-status` (central-internal;
 `Fleet.razor` polls the Config DB, not any stream), and `deployment-acks` (already rides the Phase 2
-`ClusterClient` transport as a command-plane reply). See
+gRPC command transport, `CentralCommandService.ReportApply`, as a command-plane reply). See
 [`docs/Telemetry.md`](Telemetry.md) for the full architecture, the reconnect/generation-stamped
 dialer story, and the rationale for the node-hosts/central-dials inversion.
 

docs/Redundancy.md

@@ -139,13 +139,16 @@ Consequences:
 - **`ClusterNodeAddressReconciler` is own-cluster-scoped** — an admin node can only reconcile
   `ClusterNode` rows belonging to its own cluster; it has no gossip visibility into another mesh to
   reconcile against.
-- **`CentralCommunicationActor` creates one `ClusterClient` per application `Cluster`** and fans
+- **`CentralCommunicationActor` dials each node directly over gRPC** — one `NodeCommandService.Execute`
-  commands across them, rather than one fleet-wide client. Central reaches a site mesh over
+  call per enabled, non-maintenance `ClusterNode` row carrying a `GrpcPort` — rather than fanning
-  `ClusterClient` (commands), gRPC (telemetry — see `docs/Telemetry.md`), and `ConfigServe` (config
+  through any client/receptionist object (the Akka `ClusterClient` transport this used to run on is
-  fetch — see [`ConfigSource`/`ConfigServe`](../CLAUDE.md#config-source-configsource--configserve)),
+  **deleted**; see [ClusterClient→gRPC migration](plans/2026-08-06-clusterclient-to-grpc-design.md)).
-  never over gossip.
+  Central reaches a site mesh over gRPC for commands (`MeshTransport`), telemetry (see
+  `docs/Telemetry.md`), and config fetch (`ConfigServe` — see
+  [`ConfigSource`/`ConfigServe`](../CLAUDE.md#config-source-configsource--configserve)), never over
+  gossip.
 - **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}`
-  role unless it is also configured for the split-safe transports: `MeshTransport:Mode = ClusterClient`,
+  role unless it is also configured for the split-safe transports: `MeshTransport:Mode = Grpc`,
   `Telemetry:Mode = Grpc` (driver nodes), `TelemetryDial:Mode = Grpc` (admin nodes). A DPS-mode transport
   on a cluster-role node would silently try to gossip across a partition that no longer exists, so the
   validator fails closed instead. Legacy/non-cluster-role nodes are exempt.
@@ -538,27 +541,34 @@ See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.
 
 Everything above rides one Akka mesh: central publishes on DistributedPubSub, every node
 subscribes, and the two ends must be members of the same cluster. Per-cluster mesh **Phase 2**
+(cross-mesh cutover: [ClusterClient→gRPC migration](plans/2026-08-06-clusterclient-to-grpc-design.md))
 adds a second transport under `MeshTransport:Mode` (see
 [Configuration.md § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport))
 that does not need shared membership — the prerequisite for splitting the fleet into one mesh per
-application `Cluster`.
+application `Cluster`. The Akka `ClusterClient` transport that originally filled this second slot is
+**deleted**; `Grpc` is its replacement, not an alternative alongside it.
 
-| | `Dps` (default) | `ClusterClient` |
+| | `Dps` (default, pair-local) | `Grpc` (cross-mesh) |
 |---|---|---|
-| Central → node commands | `Publish` on `deployments` / `driver-control` / `alarm-commands` | `ClusterClient.SendToAll` to `/user/node-communication`, re-emitted on the node's **local** `EventStream` |
+| Central → node commands | `Publish` on `deployments` / `driver-control` / `alarm-commands` | `NodeCommandService.Execute`, one unary call per node dialled directly from `ClusterNode.Host`+`GrpcPort`; decoded and handed to the node's `NodeCommunicationActor`, which re-emits it on the node's **local** `EventStream` exactly as the DPS path does |
-| Node → central acks | `Publish` on `deployment-acks` | `ClusterClient.Send` to `/user/central-communication`, forwarded to the deploy-coordinator singleton |
+| Node → central acks | `Publish` on `deployment-acks` | `CentralCommandService.ReportApply`, walking `MeshTransport:CentralGrpcEndpoints` in order and stopping at the first success, forwarded to the deploy-coordinator singleton |
 | Requires shared cluster membership | **yes** | no |
+| Auth | none (mesh membership is the boundary) | per-cluster PSK bearer, both directions, fail-closed |
 
 Two properties are load-bearing and easy to break:
 
-- **Both comm actors are registered per node, NOT as cluster singletons.** A `ClusterClient` rotates
+- **The gRPC dialer fans node by node, not through any shared client or receptionist object.** There
-  across its contact points and must find a live comm actor at whichever node answers. As a
+  is no rotation and no "whichever node answers" — each command is dialled at the specific node's own
-  singleton the boundary would be reachable through exactly one node, and rotation would fail
+  endpoint, so a node left out of the `ClusterNode` dispatch set (no `GrpcPort`, disabled, or in
-  silently against the others — acks landing or not depending on where the client happened to
+  `MaintenanceMode`) silently keeps its old configuration while the deployment still seals green
-  settle. `akka.cluster.client.receptionist.role` is deliberately empty for the same reason.
+  around it, the same failure shape `ClusterClient`'s `SendToAll` had.
 - **Inbound commands go to the node-local `EventStream`, never back onto a DPS topic.** DPS is
-  mesh-wide and central `SendToAll`s to *every* node, so a DPS re-publish would deliver N copies of
+  mesh-wide and central fans out to *every* node, so a DPS re-publish would deliver N copies of
   every command to every subscriber.
+- **Both transports are fire-and-forget, with no retry or queue.** A failed gRPC dispatch is logged
+  (naming the node and the gRPC status) and dropped in the dispatch second — the deploy path's
+  2-minute apply deadline and a node's boot-time replay (`DriverHostActor.Bootstrap()` re-applying an
+  orphan `Applying` row) are the only recovery mechanisms, unchanged from the DPS/`ClusterClient` era.
 
 Phase 2 changes no redundancy semantics: the Primary gate, ServiceLevel, and the singleton layout
 are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional and built from

docs/Telemetry.md

@@ -39,10 +39,20 @@ mirrors the inversion ScadaBridge already uses for the same problem (its `SiteSt
 A fused `admin,driver` node both hosts (as driver) and dials (as admin) — it dials itself plus its
 redundant pair peer, the same way central dials site nodes.
 
+**This same driver-role listener also hosts the mesh gRPC command-plane ingress.** Whenever
+`Telemetry:GrpcListenPort > 0`, the node maps both `TelemetryStreamGrpcService` (this stream) and
+`NodeCommandService` (central→node commands under `MeshTransport:Mode=Grpc` — see
+[`docs/Configuration.md` § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport))
+on the one port. They share it rather than each claiming its own, since both are dialled by the same
+central admin nodes and a second cleartext h2c port would buy nothing but another firewall rule.
+`NodeCommandService` is gated by its own `MeshCommandAuthInterceptor`/`GrpcPsk` bearer, independent
+of this stream's `Telemetry:ApiKey`.
+
 Central discovers driver-node telemetry endpoints from the `ClusterNode` table (`Host` +
 `GrpcPort`, the latter added in Phase 1 for exactly this purpose) — the same DB-sourced discovery
-Phase 1's ack set and Phase 2's `ClusterClient` contact set already use. No shared gossip
+Phase 1's ack set and Phase 2's gRPC command dial already use (see
-membership is required to find or dial a node.
+[`docs/Configuration.md` § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport)).
+No shared gossip membership is required to find or dial a node.
 
 ## The dark switch
 
@@ -139,8 +149,8 @@ each for a distinct, settled reason:
   nothing here for a per-node gRPC stream to carry. Revisit in Phase 6, once central genuinely
   loses gossip visibility of site nodes and needs another way to know a pair's membership.
 - **`deployment-acks`** — already migrated, but onto a different transport: it rides the Phase 2
-  `ClusterClient` transport (`MeshTransport:Mode=ClusterClient`) as a command-plane reply, not an
+  gRPC command transport (`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a
-  observability broadcast. It was never a Phase 5 candidate.
+  command-plane reply, not an observability broadcast. It was never a Phase 5 candidate.
 
 See [`docs/Redundancy.md`](Redundancy.md#command-transport-centralnode) for how `redundancy-state`
 and the command transports fit together.

docs/v2/config-db-schema.md

@@ -177,7 +177,11 @@ from `Cluster:Port` in its own appsettings; these columns duplicate that value f
 cannot see the node's config. Today central shares a gossip ring with every node and does not need
 them — [per-cluster mesh Phase 2](../plans/2026-07-21-per-cluster-mesh-design.md) splits the fleet
 into one mesh per `Cluster`, after which `Host` + `AkkaPort` is how central builds its ClusterClient
-contact points, and `GrpcPort` is the Phase 5 telemetry stream.
+contact points (**retired**: the Akka `ClusterClient` transport was deleted by the
+[ClusterClient→gRPC migration](../plans/2026-08-06-clusterclient-to-grpc-design.md) — central now
+dials `NodeCommandService` at `Host`+`GrpcPort` instead, and `AkkaPort` is Akka remoting only), and
+`GrpcPort` is the Phase 5 telemetry stream **and**, since that migration, the Phase 2 gRPC command
+dial target too — see [`Configuration.md` § `MeshTransport`](../Configuration.md#meshtransport-centralnode-command-transport).
 
 `AkkaPort` is `NOT NULL DEFAULT 4053` because every node listens on a remoting port — `0` is never a
 truthful value, and rows predating the column must migrate to something real. `GrpcPort` is nullable