docs(plans): record Phase 1A merge SHA b360239e

merge: ClusterClient→gRPC Phase 1A — central→node command plane over gRPC (NodeCommandService on the telemetry listener, MeshGrpcCommandDialer, third mode branch in CentralCommunicationActor; security-register items 1-2 closed, gate follows the endpoint; live gate PASS 8/8 on re-run after the starter-order boot defect was found live and fixed; #544/#545/#546 filed)
docs(plans): Phase 1A close-out — carry-forwards from the gate, #546 filed
2026-08-06 08:41:42 -04:00 · 2026-08-06 08:41:30 -04:00 · 2026-08-06 08:39:45 -04:00 · 2026-08-06 08:37:48 -04:00 · 2026-08-06 08:07:34 -04:00 · 2026-08-06 07:55:31 -04:00
19 changed files with 3543 additions and 92 deletions
@@ -154,22 +154,498 @@ Rig left running on `ClusterClient` (its normal state) after the gate.

 ---

-## Phase 1A — central→node command plane — **PENDING**
+## Phase 1A — central→node command plane — **PASS (on the re-run at `e54883a8`)**

-Branch `feat/grpc-node-command` @ `________`. Rig-only flip: central-1/-2 →
-`MeshTransport__Mode=Grpc`; **site nodes remain `ClusterClient`** (coexistence is the point).
-Flip reverted after the gate, never committed.
+Two runs. **Run 1 @ `fbf7b708` FAILED** at the central boot boundary (defect 4) and could not
+drive checks 1/2/7 or the central halves of 3/4; its checks 5 + 6 and the node-side halves of
+3 + 4 stand as recorded and were not repeated. **Run 2 @ `e54883a8`** — after the boot fix
+(`c0475f2d`) and its review-hardening follow-up (`e54883a8`) — drove every carried check. The
+result table below is the **combined, current** verdict; run 1's failure record is preserved
+underneath it because the defect it caught is the most valuable thing this gate produced.

-| # | Check | Result |
-|---|---|---|
-| 1 | Deploy dispatch from AdminUI reaches **all six** driver nodes over gRPC (per-node `Execute` log lines on every node; zero `SendToAll` lines on central) | PENDING |
-| 2 | Acks still return over ClusterClient (site mode unchanged) and the deployment **seals** — mixed-transport deploy proven end to end | PENDING |
-| 3 | `RestartDriver` + `ReconnectDriver` from AdminUI land on the target site node (EventStream delivery observed via driver restart in logs) | PENDING |
-| 4 | `AlarmCommand` (ack a scripted alarm from AdminUI) lands and the condition acknowledges | PENDING |
-| 5 | Unauthenticated `grpcurl` to a site node `:4056` `mesh_transport.v1.NodeCommandService/Execute` ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | PENDING |
-| 6 | **Wrong-key isolation:** site-b's PSK presented to site-a ⇒ `PermissionDenied` (per-cluster scoping is real, not decorative) | PENDING |
-| 7 | A node with NULL `GrpcPort` is skipped with the warn-once, fan continues to the others | PENDING |
-| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures on their interceptors across the run | PENDING |
+| # | Check | Verdict | Run |
+|---|---|---|---|
+| — | **Boot:** all six nodes healthy INCLUDING the central pair, on the branch image | **PASS** — 6/6 `running restarts=0`, 0 `Hosting failed to start`, `/health/ready` 200 ×6, MAIN mesh `memberCount=2 unreachableCount=0`. **But the `Resolve` Error line the fix logs DID fire — see finding 5** | 2 |
+| 1 | Deploy dispatch reaches **all six** driver nodes over gRPC; zero `SendToAll`/ClusterClient on central | **PASS** — 6/6 nodes accepted one `DispatchDeployment` under a single shared `correlationId=cf499b42…`; central's own per-node outcome log records 6 accepted / 0 failed; `grep -ci clusterclient` = **0** on both central nodes | 2 |
+| 2 | Acks return over ClusterClient (site mode unchanged) and the deployment **seals** | **PASS** — deployment `8DF9AAE0…` `Status=2` (Sealed) with all six `NodeDeploymentState` rows `Status=1` (Applied) inside ~90 ms; all four site nodes hold live `Connected to [akka.tcp://…/system/receptionist]` associations, which is their only ack path in 1A | 2 |
+| 3 | `RestartDriver` + `ReconnectDriver` land on the target site node | **PASS (composed)** — node-side leg driven live in run 1 (both commands acted on site-a-1). The AdminUI half **cannot be driven and this is now proven, not assumed**: `<DriverStatusPanel` has **0** usages repo-wide, seven rendered pages contain no Restart/Reconnect text or button, and the only non-script-analysis HTTP POST in the whole app is `/api/deployments`. Closed by composition — check 4 proves the *same* `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path live from central for an admin-originated command | 1 + 2 |
+| 4 | `AlarmCommand` from the AdminUI lands and the condition acknowledges | **PASS** — Ack clicked on the real `/alerts` page against a live `gate481-alarm` ACTIVATED row; central fanned **one** `AlarmCommand` (`correlationId=ccafe375…`, minted once) to all six over gRPC, and the condition acknowledged — a new `ACKNOWLEDGED` row appeared 0.54 s later carrying the operator's identity (`multi-role-test`) | 2 |
+| 5 | Unauthenticated ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | **PASS** — 3/3 probes as designed (run 1; not repeated — nothing in the fix touches the interceptor) | 1 |
+| 6 | **Wrong-key isolation** | **PASS** — both directions rejected, with a positive control proving the rejection is the key and not reachability (run 1) | 1 |
+| 7 | NULL `GrpcPort` node skipped with the warn-once; fan continues | **PASS** — with `site-b-2:4053` NULLed, both central nodes logged the warn **once** each and the next dispatch (`correlationId=88a9e0a2…`) was accepted by exactly **5** nodes, site-b-2 absent; the deployment then honestly ended `Status=4` (TimedOut) with site-b-2 alone at `Status=0`. Row restored; a later dispatch sealed 6/6 | 2 |
+| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures | **PASS, and no longer vacuous** — 0 denials on all four interceptors across all six nodes, **and all three planes carried real traffic this time**: site nodes fetched + SHA-verified 28 KB artifacts from `http://central-1:4055`, `Telemetry stream opened` on every node with live driver health rendering on `/hosts`, LocalDb sync active on the site pairs | 2 |
+
+### Run 2 provenance
+
+`docker compose -f docker-dev/docker-compose.yml up -d --build` from the worktree, **exit 0**;
+image `otopcua-host:dev` rebuilt to `1d8452a29018` at 12:15 UTC (run 1's preserved
+`otopcua-host:grpc-1a` was stale and deliberately **not** reused). The six host nodes plus
+`migrator` and `cluster-seed` were recreated (both one-shots exit 0); `sql` and its volume were
+kept, and `traefik` was reused unchanged (its service definition did not change). Nothing else
+touched the stack during the run, so it is not concurrent-rebuild-invalidated. Same rig-only
+flip and same three Serilog Debug overrides as run 1; both reverted afterwards, never committed.
+
+```
+$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
+    --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
+central-1  MeshTransport__Mode=Grpc
+central-2  MeshTransport__Mode=Grpc
+site-a-1   MeshTransport__Mode=ClusterClient
+site-b-1   MeshTransport__Mode=ClusterClient
+```
+
+### Run 2 evidence
+
+**Boot — the fix works, and the race it was hardened against is real.**
+
+```
+$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
+central-1  running restarts=0 mode=Grpc hostingFailed=0 resolveErr=2
+central-2  running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0
+site-a-1   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
+site-a-2   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
+site-b-1   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
+site-b-2   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
+```
+
+See **finding 5** below for the two `resolveErr` lines on central-1 — they are the PreStart
+race losing on a real boot, and the memoize retry healing it was verified live rather than
+argued.
+
+**Check 1 — the fan, both ends.** One envelope, one correlation id, six recipients:
+
+```
+$ for c in …; do docker logs otopcua-dev-$c-1 | grep "Mesh: accepted DispatchDeployment" | tail -1; done
+central-1  [12:16:49 DBG] Mesh: accepted DispatchDeployment (correlationId=cf499b42c6914a578035b2b5bc1020f7) over gRPC; handed to the node communication actor.
+central-2  [12:16:49 DBG] …same correlationId…
+site-a-1   [12:16:49 DBG] …same correlationId…
+site-a-2   [12:16:49 DBG] …same correlationId…
+site-b-1   [12:16:49 DBG] …same correlationId…
+site-b-2   [12:16:49 DBG] …same correlationId…
+
+$ docker logs otopcua-dev-central-2-1 | grep "accepted by"        # the node hosting the coordinator
+[12:16:49 DBG] Mesh: DispatchDeployment (correlationId=cf499b42…) accepted by central-2:4053 over gRPC
+[12:16:49 DBG] … accepted by site-b-1:4053 …   [12:16:49 DBG] … accepted by site-a-2:4053 …
+[12:16:49 DBG] … accepted by site-a-1:4053 …   [12:16:49 DBG] … accepted by central-1:4053 …
+[12:16:49 DBG] … accepted by site-b-2:4053 …
+$ docker logs … | grep -c "was NOT delivered to"   → 0 on both central nodes
+
+# Zero ClusterClient on the command side, both central nodes:
+central-1  MeshClusterClientCreated=0 clusterclientAnyLine=0
+central-2  MeshClusterClientCreated=0 clusterclientAnyLine=0
+```
+
+The `clusterclientAnyLine=0` count is over the whole log with
+`Serilog:MinimumLevel:Override:Akka.Cluster.Tools.Client=Debug` in force, so it covers Akka's
+own ClusterClient internals and not merely this repo's creation message.
+
+**Check 2 — mixed transport, end to end.** Commands out over gRPC (above), acks back over
+ClusterClient:
+
+```
+$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='8df9aae0-…'
+2                                    -- Sealed
+$ … SELECT NodeId, Status, AppliedAtUtc FROM dbo.NodeDeploymentState WHERE DeploymentId='8df9aae0-…'
+central-1:4053|1|2026-08-06 12:16:49.939     site-a-1:4053|1|2026-08-06 12:16:49.977
+central-2:4053|1|2026-08-06 12:16:49.915     site-a-2:4053|1|2026-08-06 12:16:49.950
+site-b-1:4053|1|2026-08-06 12:16:49.957      site-b-2:4053|1|2026-08-06 12:16:49.967
+
+$ for c in site-a-1 site-a-2 site-b-1 site-b-2; do docker logs … | grep "receptionist" | tail -1; done
+site-a-1   [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
+site-a-2   [12:15:20 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
+site-b-1   [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
+site-b-2   [12:15:25 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
+$ grep -c "No central ClusterClient" → 0 on all six      # no ack was dropped for want of a client
+```
+
+A site node has **no** gRPC ack path in 1A (that is 1B), so the receptionist association above
+is not merely present, it is the only route those four Applied rows could have taken.
+
+**Check 3 — why the AdminUI half is undrivable, established rather than asserted:**
+
+```
+$ grep -rn "<DriverStatusPanel" --include="*.razor" --include="*.cs" src/ | wc -l
+0                                    # the component holding the buttons is referenced nowhere
+$ grep -rn "DriverStatusPanel" --include="*.razor" src/ | grep -v "Shared/Drivers/DriverStatusPanel.razor:"
+…5 hits, all of them code COMMENTS in Alerts.razor / Hosts.razor ("mirrors DriverStatusPanel…")
+
+$ node scan-ui.js          # every page a Designer/Administrator/Operator can reach
+/            | reconnect/restart text: false | buttons: ["Sign out"]
+/hosts       | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
+/fleet       | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
+/clusters    | reconnect/restart text: false | buttons: ["Sign out"]
+/clusters/SITE-A | reconnect/restart text: false | buttons: ["Sign out"]
+/alerts      | reconnect/restart text: false | buttons: ["Sign out","Clear"]
+/deployments | reconnect/restart text: false | buttons: ["Sign out","Deploy current configuration"]
+
+$ grep -rn "MapPost|MapPut|MapDelete" src/…/AdminUI src/…/Host
+…/Api/DeployApiEndpoints.cs:43:  app.MapPost("/api/deployments", …)
+…/ScriptAnalysis/ScriptAnalysisEndpoints.cs:19-24:  /diagnostics /completions /hover /signature-help /format /inlay-hints
+```
+
+There is no button and no endpoint. What closes the check is that `RestartDriver`,
+`ReconnectDriver` and `AlarmCommand` all leave central through the **same three lines** —
+`AdminOperationsActor.Dispatch` → `MeshCommand` → `CentralCommunicationActor.DispatchOverGrpc`
+— and check 4 drove that path live from the AdminUI for `AlarmCommand`, while run 1 drove the
+node-side decode-and-act for the two driver commands. The only per-type difference left is
+`MeshTransportDtoMapper.ToEnvelope`, which the T0.2 goldens cover member-for-member.
+
+**Check 4 — AdminUI ack, fanned by central, acknowledged by the owner.** The rig's one
+scripted alarm sits Normal indefinitely (`hr200 > 70000`), so a transition was manufactured by
+editing its predicate to an always-true form and deploying; the `/alerts` page was already open
+when it fired, the Ack button was clicked, and the predicate + hash were restored afterwards
+(verified byte-identical by `SELECT`) with a further deploy:
+
+```
+page open at 2026-08-06T12:20:59.036Z        initial: empty window
+ROWS:  12:21:00.017  gate481-alarm | PressHigh | EQ-6e09bcc2d907 | ACTIVATED | 750 | system | press high
+ROW BUTTONS: ["Ack","Shelve","Shelve (timed)","Unshelve"]
+clicking Acknowledge at 2026-08-06T12:21:00.362Z
+after click:  12:21:00.557 | gate481-alarm | EQ-6e09bcc2d907 | ACKNOWLEDGED | 750 | multi-role-test | press high
+
+$ docker logs otopcua-dev-central-2-1 | grep AlarmCommand
+[12:21:00 DBG] Mesh: AlarmCommand (correlationId=ccafe3754aca40cfa8aa0a0602c86cbb) accepted by site-b-2:4053 over gRPC
+[12:21:00 DBG] … accepted by central-1:4053 …   [12:21:00 DBG] … accepted by central-2:4053 …
+[12:21:00 DBG] … accepted by site-a-1:4053 …    [12:21:00 DBG] … accepted by site-b-1:4053 …
+[12:21:00 DBG] … accepted by site-a-2:4053 …
+$ for c in …; do grep "Mesh: accepted AlarmCommand" …; done   → all six, same correlationId
+```
+
+One correlation id across all six recipients is the "encode once, before the fan" rule holding
+for the one command whose CLR record has no correlation member of its own — the case the design
+called out as the reason encoding is hoisted out of the loop.
+
+**Check 7 — the skip, the warn, and the honest deadline.**
+
+```
+$ … UPDATE dbo.ClusterNode SET GrpcPort=NULL WHERE NodeId='site-b-2:4053'    # 12:23:14
+
+# one warn per central node, once each, within one ContactRefreshSeconds:
+central-2  [12:23:20 WRN] ClusterNode site-b-2:4053 has no GrpcPort; it exposes no command endpoint
+           and is skipped in this dispatch set (further skips of this node are silent). Commands
+           will not reach it until the row carries a port
+central-1  [12:23:19 WRN] …identical…
+# (each node also logs the TelemetryNodeSource twin — "exposes no telemetry stream … skipped in
+#  this dial refresh" — expected collateral: both target sets read the same column)
+
+# dispatch at 12:23:36 — accepted by five, site-b-2 absent:
+central-2  [12:23:37 DBG] … (correlationId=88a9e0a2…) accepted by central-2 / site-a-2 / site-b-1
+                          / site-a-1 / central-1 :4053 over gRPC
+# cumulative per-node accept counts after this dispatch — site-b-2 one behind everyone:
+central-1 4   central-2 4   site-a-1 4   site-a-2 4   site-b-1 4   site-b-2 3
+
+$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='5c8bdc9e-…'   → 4  (TimedOut)
+$ … SELECT NodeId, Status FROM dbo.NodeDeploymentState …
+central-1|1  central-2|1  site-a-1|1  site-a-2|1  site-b-1|1  site-b-2|0
+```
+
+Row restored to `4056` and a later dispatch sealed 6/6 (`080EBB0B…`, `Status=2`, every node
+`Status=1`). One operational detail worth carrying: **the skip persists for up to a full
+`ContactRefreshSeconds` after the row is repaired** — a deploy issued 5 s after the restore
+(`DF1C0B74…`) was still skipped and still TimedOut, because the dispatch set is only rebuilt on
+the refresh tick.
+
+**Check 8 — denials, and proof the planes were actually used.**
+
+```
+$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
+central-1  telemetry=0 configserve=0 localdbsync=0 mesh=0
+central-2  telemetry=0 configserve=0 localdbsync=0 mesh=0
+site-a-1   telemetry=0 configserve=0 localdbsync=0 mesh=0
+site-a-2   telemetry=0 configserve=0 localdbsync=0 mesh=0
+site-b-1   telemetry=0 configserve=0 localdbsync=0 mesh=0
+site-b-2   telemetry=0 configserve=0 localdbsync=0 mesh=0
+
+# config-serve carried real, authenticated traffic:
+site-a-1  [12:20:59 INF] Fetched + verified artifact for 8d486c6b… from http://central-1:4055 (28313 bytes).
+# telemetry likewise, end to end into the UI:
+$ for c in …; do grep -c "Telemetry stream opened"; done → 3,4,3,3,3,5
+/hosts →  CLUSTER SITE-A … sa-modbus2 SITEA-modbus Modbus HEALTHY 1 host 1 tag  12:30:49 UTC  0 errors
+```
+
+The five telemetry `failed … reconnecting` warnings in the run are all accounted for: three at
+12:15:20 (central dialing nodes that had not finished binding `:4056`) and two at 12:28:50 /
+12:29:30 (the deliberate central-2 stop in finding 5). None is an auth failure.
+
+### Phase 1A finding 5 — the PreStart race loses on real boots, and the memoize is what saves it
+
+The hardening commit's premise is not hypothetical. On the run-2 boot, **central-1 lost the
+race on both dependencies**:
+
+```
+$ docker logs otopcua-dev-central-1-1 | grep -A3 "Failed to resolve"
+[12:15:19 ERR] Failed to resolve IMeshCommandDialer for the mesh command boundary; continuing
+               without it. The guards below report what that costs
+Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
+    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
+   at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
+[12:15:19 ERR] Failed to resolve IMeshPskProvider for the mesh command boundary; …same cause…
+```
+
+It is not rare: across the three admin boots observed in run 2 (central-1 and central-2 at
+12:15, central-2 again at 12:29 after the stop/start), **two of three lost it**. Had the fix
+stopped at PreStart-only resolution, those nodes would have held a permanent silent `null` —
+no dialer, and a never-published known-cluster gate — and the failure would only have surfaced
+the first time the coordinator singleton moved onto such a node.
+
+**The memoize heals it, verified on the node that actually lost the race.** Run 2's dispatches
+were all fanned by central-2 (the singleton host), so central-1's dialer was never exercised
+by them. Central-2 was therefore stopped to move the singleton, and a dispatch driven while
+central-1 alone held it:
+
+```
+$ docker compose stop central-2                                   # 12:28:39
+$ curl … /health/ready → self akka.tcp://otopcua@central-1:4053, members 1     # 12:28:57
+$ curl -X POST /api/deployments  -d '{"createdBy":"1a-gate-central1-heal"}'    # 12:29:01
+$ docker logs otopcua-dev-central-1-1 | grep -E "accepted by|no command dialer is wired"
+[12:29:02 DBG] Mesh: DispatchDeployment (correlationId=ad97d386…) accepted by site-b-1:4053 over gRPC
+[12:29:02 DBG] … accepted by site-a-2 … site-b-2 … site-a-1 … central-1 …      # 5 live nodes
+# no "no command dialer is wired" line, and no further "Failed to resolve"
+$ docker compose start central-2  → members 2, unreachable 0, status Healthy   # 12:29:51
+```
+
+So the boot-time Error is **cosmetically alarming but functionally healed**, and that is now an
+observed property rather than a design claim. Two follow-ups for Phase 2 rather than for 1A:
+the Error text says "continuing without it", which reads as permanent and should say that the
+first use retries; and a `Failed to resolve` line on a healthy node is exactly the kind of
+signal a soak's PSK-drift/error grep will trip over.
+
+**One incidental observation, not a check.** The deployment `RevisionHash` is deterministic
+(three separate no-change dispatches all produced `642b9d6f79b9…`), but restoring the alarm
+predicate's two text columns did **not** restore the pre-gate hash `6178a8f302aa…` — so the
+artifact covers a column the `UPDATE` bumped (`Script.RowVersion` is the obvious candidate).
+The rig's configuration content is byte-identical to its pre-gate state; only the hash moved.
+Not investigated further — out of this program's scope.
+
+### Run 1 record — the defect that blocked it
+
+Branch `feat/grpc-node-command` @ `fbf7b708`. Image rebuilt from this worktree
+(`/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a`) so it carries this branch;
+`docker compose -f docker-dev/docker-compose.yml up -d --build` exited 0 and **all eight
+containers were recreated** — `migrator` and `cluster-seed` re-ran to completion (exit 0), the
+six host nodes and traefik were recreated, the `sql` container and its named volume were kept.
+Nothing else touched the stack afterwards, so this run is not a concurrent-rebuild-invalidated
+one. Rig-only flip: central-1/-2 `MeshTransport__Mode: "Grpc"` (hard value); **site nodes left
+on the compose default `ClusterClient`** (coexistence is the point). Three Serilog
+`MinimumLevel:Override` Debug scopes were added on all six nodes for this run only
+(`ZB.MOM.WW.OtOpcUa.Host.Grpc`, `ZB.MOM.WW.OtOpcUa.ControlPlane.Communication`,
+`Akka.Cluster.Tools.Client`) — the accept/dispatch lines the checks read are `Debug` and the
+rig's default Serilog level is `Information`, so without them checks 1/3 would have had no
+observable signal. Flip and overrides reverted after the gate, never committed.
+
+**Run 1 verdict: FAIL.** Neither central node booted, so nothing that originates at central
+could be driven: checks 1, 2 and 7 were **BLOCKED** outright, 3 and 4 reached only their
+node-side legs, and check 8's zero-denial result was vacuous for telemetry and config-serve
+(central dials both and never booted). Checks 5 and 6 **PASSED** and are the run-1 evidence
+carried into the combined table above. Root cause was defect 4 — resolving `IMeshCommandDialer`
+/ `IMeshPskProvider` inside the first `WithActors` starter drags in `ISecretResolver` →
+`IAuditWriter` → `ActorAuditWriter`, whose constructor reads `AuditWriterActorKey` out of the
+`ActorRegistry` before the singleton that registers it has run. It was **mode-independent**
+(reproduced on the same image with `MeshTransport__Mode=ClusterClient`), so it was a defect of
+the branch, not of the gate's flip, and it would have failed a `ClusterClient` rig equally.
+
+### Run 1 evidence
+
+**Provenance** — flip in effect on central only, sites untouched:
+
+```
+$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
+    --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
+central-1  MeshTransport__Mode=Grpc
+central-2  MeshTransport__Mode=Grpc
+site-a-1   MeshTransport__Mode=ClusterClient
+site-b-1   MeshTransport__Mode=ClusterClient
+
+$ docker images --format '{{.Repository}}:{{.Tag}}|{{.ID}}|{{.CreatedAt}}' | grep otopcua-host
+otopcua-host:dev|69360a173f1f|2026-08-06 07:20:14 -0400 EDT     # rebuilt for this run
+```
+
+**Defect 4 — the boot failure (checks 1/2/7 blocked by it).** Both central nodes, identical:
+
+```
+$ docker logs otopcua-dev-central-1-1 2>&1 | sed -n '25,45p'
+[11:21:27 INF] OPC UA server started on opc.tcp://0.0.0.0:4840
+[11:21:27 FTL] Unable to start AkkaHostedService - shutting down application
+Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
+    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
+   at Akka.Hosting.ActorRegistry.Get[TKey]()
+   at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
+        in /src/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Audit/ActorAuditWriter.cs:line 24
+   ...
+   at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_5(IServiceProvider sp)
+   ...
+   at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_6(IServiceProvider sp)
+   ...
+   at ZB.MOM.WW.OtOpcUa.ControlPlane.ServiceCollectionExtensions.<>c.<WithOtOpcUaControlPlaneSingletons>b__7_0(
+        ActorSystem system, IActorRegistry registry, IDependencyResolver resolver)
+        in /src/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs:line 74
+   at Akka.Hosting.AkkaConfigurationBuilder.StartAsync(ActorSystem sys)
+   at Akka.Hosting.AkkaHostedService.StartAsync(CancellationToken cancellationToken)
+```
+
+**Mode-independence, proven rather than argued** — same image, central-1's exact environment
+with `MeshTransport__Mode=ClusterClient`, own hostname + self-seed so it cannot disturb the rig:
+
+```
+$ docker run -d --name scratch-cc-mode --network otopcua-dev_default --env-file c1-env.txt \
+    -e Cluster__PublicHostname=scratch-cc-mode \
+    -e Cluster__SeedNodes__0=akka.tcp://otopcua@scratch-cc-mode:4053 otopcua-host:dev
+$ docker logs scratch-cc-mode 2>&1 | grep -m4 -E "MissingActorRegistry|Hosting failed|Host starting"
+[11:28:45 INF] OtOpcUa.Host starting with roles=[admin,driver,cluster-main] (admin=True, driver=True)
+Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
+    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
+[11:28:47 ERR] Hosting failed to start
+```
+
+Scratch container removed after capture (`docker rm -f scratch-cc-mode`); the six real nodes
+were untouched by it.
+
+**It presents as a healthy container.** Worth carrying into Phase 2/4 readings — the failed
+process does not exit, so `docker ps` shows `Up`, `RestartCount=0`, and only an HTTP probe or
+the log tells the truth:
+
+```
+$ docker inspect otopcua-dev-central-1-1 -f '{{.State.Status}} restarts={{.RestartCount}}'
+running restarts=0
+$ docker run --rm --network otopcua-dev_default curlimages/curl -s -o /dev/null \
+    -w '%{http_code}\n' --max-time 5 http://central-1:9000/health/ready
+000
+$ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9200/health/ready
+503
+```
+
+**Checks 5 + 6 — the wire-probe auth matrix** (`-import-path` onto the in-repo proto; the node
+has no reflection). An empty `CommandEnvelope` has no `oneof` arm set, so `FromEnvelope` throws
+and the service answers `InvalidArgument` — **after** the interceptor ran, which is exactly why
+it is the positive signal here: `PermissionDenied` means the key was refused, `InvalidArgument`
+means the key was accepted and only the payload was rejected:
+
+```
+$ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos
+$ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \
+     -plaintext -import-path /protos -proto mesh_transport.proto"
+
+# 5a  no bearer                        -> site-a-1
+Code: PermissionDenied   Message: Node command authentication failed.
+# 5b  -H 'authorization: Bearer dev-mesh-psk-SITE-A'   -> site-a-1
+Code: InvalidArgument    Message: Command envelope could not be decoded.
+# 6a  -H 'authorization: Bearer dev-mesh-psk-SITE-B'   -> site-a-1
+Code: PermissionDenied   Message: Node command authentication failed.
+# 6b  -H 'authorization: Bearer dev-mesh-psk-SITE-A'   -> site-b-1
+Code: PermissionDenied   Message: Node command authentication failed.
+# 6c  -H 'authorization: Bearer dev-mesh-psk-SITE-B'   -> site-b-1   (positive control)
+Code: InvalidArgument    Message: Command envelope could not be decoded.
+```
+
+**Checks 3 + 4 — the node-side leg, driven with real envelopes** over the same authenticated
+h2c path central would use. This is the first live proof that `MeshCommandAuthInterceptor` is
+actually invoked on a real Kestrel pipeline (register item 4) and that
+`NodeCommandGrpcService` → `NodeCommunicationActor` → EventStream → `DriverHostActor` works
+end to end:
+
+```
+$ $G -H 'authorization: Bearer dev-mesh-psk-SITE-A' -H 'x-otopcua-cluster: SITE-A' \
+     -d '{"correlationId":"8461008e...","restartDriver":{"clusterId":"SITE-A",
+          "driverInstanceId":"SITEA-modbus","actorByUserName":"1a-gate",
+          "correlationId":"8461008e..."}}' \
+     site-a-1:4056 mesh_transport.v1.NodeCommandService/Execute
+{}
+
+$ docker logs otopcua-dev-site-a-1-1 2>&1 | tail
+[11:31:01 DBG] Mesh: accepted RestartDriver (correlationId=8461008e37654724a292219202d78dfa) over gRPC; handed to the node communication actor.
+[11:31:01 INF] DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate
+[11:31:01 INF] DriverHost site-a-1:4053: spawned Modbus driver SITEA-modbus (stub=False)
+[11:31:55 DBG] Mesh: accepted ReconnectDriver (correlationId=29c5dc77283f4eb7869d0c7d6686cadd) over gRPC; handed to the node communication actor.
+[11:31:55 INF] DriverHost site-a-1:4053: reconnecting driver SITEA-modbus by request of 1a-gate
+[11:31:55 INF] DriverInstance SITEA-modbus: ForceReconnect requested by admin; re-entering Reconnecting
+[11:31:56 DBG] Mesh: accepted AlarmCommand (correlationId=39ba451173954229a71b9dd1e00ac1b9) over gRPC; handed to the node communication actor.
+```
+
+**Check 8 — interceptor denial counts across every node, whole run:**
+
+```
+$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
+central-1  telemetry=0 configserve=0 localdbsync=0 mesh=0
+central-2  telemetry=0 configserve=0 localdbsync=0 mesh=0
+site-a-1   telemetry=0 configserve=0 localdbsync=0 mesh=2   # the two deliberate negatives
+site-a-2   telemetry=0 configserve=0 localdbsync=0 mesh=0
+site-b-1   telemetry=0 configserve=0 localdbsync=0 mesh=1   # the one deliberate negative
+site-b-2   telemetry=0 configserve=0 localdbsync=0 mesh=0
+```
+
+The `TwoNodeClusterHarness` I-1 null-provider Error (a known test-harness artifact) did **not**
+appear in any rig log — `grep -c "IMeshPskProvider"` = 0 on all six nodes, as expected for
+production wiring where every admin node registers the provider.
+
+### Phase 1A defect — **CLOSED by `c0475f2d` + `e54883a8`, re-gated in run 2**
+
+**Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter.**
+`WithOtOpcUaControlPlaneSingletons`'s `WithActors` block resolves `IMeshCommandDialer` and
+`IMeshPskProvider` from `IServiceProvider` to hand to `CentralCommunicationActor.Props`.
+`MeshGrpcCommandDialer` takes `IMeshPskProvider`, which takes `ISecretResolver`, which
+`AddZbSecrets` builds by resolving `IAuditWriter`, which on an admin node is
+`ActorAuditWriter` — and its constructor does `registry.Get<AuditWriterActorKey>()`. That
+singleton is registered ~60 lines later in the same builder, so at this point the key is
+absent and `ActorRegistry.Get` throws `MissingActorRegistryEntryException`, failing
+`AkkaHostedService.StartAsync` and the whole host.
+
+Three things make this worth a pinned regression rather than a one-line fix note:
+
+- **It is not mode-gated.** The two `GetService` calls run in every transport mode by design
+  (so a mode flip needs no restart), so the eager graph is walked on every admin boot. Proven
+  above with a `ClusterClient` scratch container on the same image.
+- **It is invisible to the suite.** The 1A.2 container-activation test
+  (`It_activates_from_the_container_despite_the_optional_client_factory_parameter`) resolves
+  the dialer from a service collection that does not include `AddZbSecrets`, and
+  `TwoNodeClusterHarness` registers no `IMeshPskProvider` at all — so no existing test walks
+  the real graph in starter order. This is the same class as scadaproj#1 ("only visible with
+  the DI graph built in a real host") and as mesh Phase 6's `IClusterRoleInfo` boot crash.
+- **The container lies about it.** The process does not exit, so `docker ps` reports `Up` with
+  `restarts=0` while the node serves no HTTP at all. Any Phase 2/4 readiness check that reads
+  container status rather than `/health/ready` would score this as healthy.
+
+The shape of the fix is the one this file already uses two lines above for the coordinator
+proxy — hand the actor a lazy accessor (`Func<IMeshCommandDialer?>` / `Func<IMeshPskProvider?>`,
+or resolve on first use) so nothing is constructed while the `ActorSystem` is still building.
+
+**That is what shipped.** `c0475f2d` moved both dependencies behind `Func<>` invoked in
+`PreStart`, pinned by `ControlPlaneStarterOrderTests` (boots a real admin host through the
+production starter order and asserts `StartAsync`; verified RED against the eager resolve in
+3/3 modes). `e54883a8` then made the resolution memoize-on-first-use rather than PreStart-only,
+because `PreStart` runs concurrently with the remaining starters and can still lose the race —
+which **run 2 observed happening on 2 of 3 real admin boots**, so that hardening was not
+theoretical. Both are re-gated above; see finding 5 for the live self-heal proof.
+
+### Rig state after run 1 (superseded by the run-2 restoration below)
+
+The compose flip was reverted and the six nodes recreated on the committed all-`ClusterClient`
+compose. Because defect 4 was mode-independent, the branch image could not produce a healthy
+central pair on *any* mode, so the rig was returned to its pre-gate image rather than left with
+a dead central pair: the run-1 build was preserved as `otopcua-host:grpc-1a` (`69360a173f1f`)
+and the pre-gate build (`8bd81d8aea9b`) re-tagged back to `otopcua-host:dev`.
+
+### Rig state after run 2 — **restored, six healthy, on the branch image**
+
+The flip was reverted (`git checkout docker-dev/docker-compose.yml`) and the six nodes
+recreated on the committed all-`ClusterClient` compose. No image substitution was needed this
+time: `otopcua-host:dev` is the run-2 build of the committed source (`1d8452a29018`), and it
+boots a healthy central pair, so the rig runs the code under test. The run-1 workaround tag
+`otopcua-host:grpc-1a` was deleted to stop anyone reusing a stale image.
+
+```
+central-1  running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
+central-2  running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
+site-a-1   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
+site-a-2   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
+site-b-1   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
+site-b-2   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
+$ curl … /health/ready → status Healthy, members 2, unreachable 0
+$ grep -c "Mesh ClusterClient for cluster"  → 3 on each central node (MAIN + SITE-A + SITE-B)
+```
+
+The three ClusterClients reappearing on the rollback is an incidental but useful result: the
+dark switch still rolls back cleanly on the fixed branch, which is what Phase 2 will depend on
+if its cutover has to be reversed.

 ---

@@ -228,8 +704,21 @@ checks 6–7.
 ### Carry-forwards

 Anything a phase gate could not drive gets named here with why + what proves it instead, and
-is re-checked in the next gate (the ScadaBridge proportionate-gate rule). Currently: none.
+is re-checked in the next gate (the ScadaBridge proportionate-gate rule).
+
+| Item | Deferred from | Why it could not be driven | Proven instead by | Status |
+|---|---|---|---|---|
+| Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-`GrpcPort` skip | 1A run 1 | Both central nodes failed to boot (defect 4) | — | **CLOSED in 1A run 2** — all three PASS live |
+| Check 4 acknowledge half — the condition actually acknowledging | 1A run 1 | The rig's one `ScriptedAlarm` sits Normal and its owning central node was down | — | **CLOSED in 1A run 2** — acked from the real `/alerts` page, `ACKNOWLEDGED` row observed |
+| Check 8 for telemetry + config-serve | 1A run 1 | Central dials both planes and never booted, so neither carried a call — 0 denials was vacuous | — | **CLOSED in 1A run 2** — both planes carried authenticated traffic with 0 denials |
+| Check 3 central half — `RestartDriver`/`ReconnectDriver` **originating at the AdminUI** | 1A runs 1 + 2 | **Structurally undrivable, not a rig problem:** `<DriverStatusPanel` has 0 usages repo-wide (the 5 name hits are comments), no rendered page carries a Restart/Reconnect control, and the only non-script-analysis HTTP POST is `/api/deployments`. Pre-existing on `master`, not caused by this program | Composition the controller accepted: run 1's live node-side leg for both commands + run 2's check 4, which drives the identical `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path from the AdminUI for another command type; per-type difference is `ToEnvelope`, covered by the T0.2 goldens | **CLOSED by composition.** The orphaned panel itself is out of this program's scope — file separately if the control is meant to exist |
+| `Failed to resolve` Error text says "continuing without it", which reads permanent although first use retries; and it will trip a soak's error grep on a healthy node | 1A run 2 (finding 5) | Cosmetic/observability, not behaviour | The self-heal proven live (finding 5) | Phase 2 |

 ### Defects the gates caught that the suite did not

 Recorded per phase as they occur, with root cause + the pinned regression test.
+
+| # | Phase | Defect | Root cause | Status |
+|---|---|---|---|---|
+| 4 | 1A gate run 1 | **An admin node cannot boot at all** — `MissingActorRegistryEntryException: AuditWriterActorKey`, both central nodes, every transport mode | The first `WithActors` starter eagerly resolves `IMeshCommandDialer`/`IMeshPskProvider`, whose graph reaches `ActorAuditWriter`, whose ctor reads an `ActorRegistry` key registered ~60 lines later in the same builder | **CLOSED** — `c0475f2d` (`Func<>` + `PreStart`, pinned red-first by `ControlPlaneStarterOrderTests`) + `e54883a8` (memoize-on-first-use). Re-gated in run 2: 6/6 nodes boot, 0 `Hosting failed to start` |
+| 5 | 1A gate run 2 | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider` with the same `AuditWriterActorKey` cause | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede `AuditWriterActorKey`'s registration | **Not a defect — it is the case `e54883a8` was written for, and the memoize demonstrably heals it.** Proven live by moving the coordinator singleton onto the node that lost the race (stop central-2) and watching it fan a dispatch to all five live nodes with no dialer error. Residual is cosmetic: the Error text and its effect on soak greps → Phase 2 |
@@ -23,8 +23,8 @@ posture).
 | Phase | Scope | Branch | Status | PR / merge SHA | Live gate |
 |---|---|---|---|---|---|
 | 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **MERGED to `master`** @ `10b6d209` (2026-08-06) — all 6 tasks complete @ `23555ed0`; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed | `10b6d209` | **PASS 4/4** |
-| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | NOT STARTED | — | PENDING |
-| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebases on 1A | `feat/grpc-central-ack` | NOT STARTED | — | PENDING |
+| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **COMPLETE @ `e54883a8` — gate re-run PASSED 2026-08-06.** Run 1 @ `fbf7b708` FAILED on defect 4 (no admin node could boot, mode-independent); fixed by `c0475f2d` + `e54883a8` and re-gated. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅. **MERGED to `master`** @ `b360239e` (2026-08-06) | `b360239e` | **PASS 8/8** (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards) |
+| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebases on 1A | `feat/grpc-central-ack` | **1B.1 ✅ `14dd07d2` + 1B.2 ✅ `a5a789d4` (+fixes `e54711a2`/`4d607575`)** — all reviews ✅ (1B.2 on Opus: 1 Critical found+fixed, see defect log); awaiting 1A merge → rebase → 1B.3 gate | — | PENDING |
 | 2 | Baseline reproduction on pre-flip build, then full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills | (rig commit) | NOT STARTED | — | PENDING |
 | 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — |
 | 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | NOT STARTED | — | PENDING |
@@ -46,11 +46,14 @@ phase DoD — PR + merge are user-owned.**
 | 2026-08-06 | No transport interface — third mode branch inside the two choke-point actors (ScadaBridge §7.1 economy) | design §4.3 |
 | 2026-08-06 | Coexistence granularity: central's mode governs commands, each node's mode governs its acks (per-pair flips possible — better than ScadaBridge's central-wide flag) | design §3.6 |
 | 2026-08-06 | `AlarmCommandMessage.operation` is a proto `string`, NOT an enum — the CLR `AlarmCommand.Operation` is an unconstrained string, and an enum would silently decode unknown operations to `UNSPECIFIED` (the drop class this migration removes). Plan T0.1's "AlarmCommand verb" enum wording was a plan defect; only `ApplyOutcome` carries the enum-0 rule. T0.2's goldens must pin pass-through of an unrecognized operation string | proto comment + this log (T0.1 review) |
+| 2026-08-06 | **Gate follows the endpoint, not the mode** (1A.2 I-2): known-cluster set published on every contact load in all three modes, from ALL `ClusterNode` rows; fan set unchanged. Survives dark-switch rollback with nothing for 1B to remember | tracking §register refinement + actor docs |
+| 2026-08-06 | `CentralCommandService` mapped by PORT, not mode-gated (1B.1): a mode flip on a running fleet must not need a central restart before acks flow; matches every sibling service | `Program.cs` comment + this log |
+| 2026-08-06 | Ack deadline is PER ENDPOINT, not per walk; the walk is at-least-once on the deadline path, absorbed idempotently by the coordinator's NodeId-keyed upsert | `MeshTransportOptions` docs + defect log #1 |
 | 2026-08-06 | `correlation_id` lives on the envelope AND as inner fields on the three commands whose CLR record carries one — each command round-trips independently in the T0.2 mapper; the envelope covers `AlarmCommand` (no CLR correlation member). For the three, the inner field is authoritative on decode | proto comment + this log (T0.1 review) |

 ## Phase 1A security register (from the Batch B Opus code review — HARD GATE: all four close BEFORE `CentralCommandService` is mapped)

-1. **Known-cluster-set membership gate in `TryGetPsk`** — central rebuilds its cluster set from
+1. **CLOSED (1A.2 @ `48dc2120`, refined — see below).** Known-cluster-set membership gate in `TryGetPsk` — central rebuilds its cluster set from
   `ClusterNode` rows every `ContactRefreshSeconds`; refuse a cluster id outside that set in
   O(1) with no store touch. Closes BOTH open review findings at once: (a) the timing oracle
   that currently defeats the uniform-denial anti-enumeration text (unknown cluster pays a
@@ -61,13 +64,33 @@ phase DoD — PR + merge are user-owned.**
   audit singleton, attacker-chosen `Target` values, before any credential check). Namespace
   confinement (`OT-MESH-PSK-` prefix + `SecretName` allow-list) means no disclosure risk —
   this is flood/oracle only.
-2. **`bool Matches(string clusterId, string presented)` on `IMeshPskProvider`** — keep
+2. **CLOSED (1A.2 @ `14d21496`).** `bool Matches(string clusterId, string presented)` on `IMeshPskProvider` — keep
   `FixedTimeEquals` inside the provider so a future verify-side caller cannot write `==` and
   have it compile. Do before 1B adds the dial-side caller.
 3. **DI-shape pin test** — `IMeshPskProvider` is `hasAdmin`-only and its pairing with the
   central interceptor is by convention; a driver-only node injecting it throws on first call,
   not at boot. Add a Host.IntegrationTests check (admin host resolves it; driver-only host
   does NOT) when Phase 1 maps the services.
+**Refinement on item 1 (decision, 2026-08-06):** the gate follows the ENDPOINT, not the mode —
+published on every contact load in every transport mode (the interceptor and service are
+port-mapped, not mode-gated, so a dark-switch rollback must not deactivate the gate), and
+built from ALL `ClusterNode` rows (gate answers "does central know this cluster"; the fan set
+stays `Enabled && !MaintenanceMode`, so gate ⊇ fan — this also dissolved the
+maintenance/ack-auth coupling). Pinned by the every-mode Theory + gate-⊋-fan fact + the Dps
+no-ClusterClient guard pin (`86633f14`). Residuals accepted + visible: gate is fail-open until
+the first successful contact load (and stays stale if central boots with SQL down — warned
+every tick); gate derives from the node table, so a `ServerCluster` with zero rows is outside
+it.
+
+**Follow-ups from the 1A.2/1B.2 Opus reviews (tracked, not blocking):** per-tick
+`InvalidateAll` puts a cold-cache 3 s store read on the inbound auth path — MEASURE on the
+1B.3 gate (ack burst inside ~3 s after a refresh tick), then consider diff-based
+invalidation; the command plane is h2c-only so the per-cluster PSK crosses the network in
+cleartext on a write surface — precedent-consistent (all sibling gRPC planes), record as
+accepted risk + TLS-option issue (queued below); dialer minors (Lazy channel wrapper,
+retired-endpoint eviction, gRPC deadline vs token cancellation, pre-existing
+`LoadContactsFromDb` Task.Run(ct) pattern).
+
 4. **Integration gate over real Kestrel h2c** — both new interceptors have unit coverage only;
   the one-public-ctor reflection test approximates but cannot prove the interceptor is
   actually invoked. Wire-probe in the 1A/1B live gates (already in the live-gate tables).
@@ -104,7 +127,11 @@ Defects found during execution (live-gate or review), with root cause + pinned t

 | # | Phase | Defect | Root cause | Fix / pin |
 |---|---|---|---|---|
-| — | — | *(none yet)* | | |
+| 1 | 1B.2 | **Failover silently defeated on a hanging central** (CRITICAL, caught by Opus code review): the ack deadline was one absolute `DateTime` shared across the endpoint walk — a hung (not refused) central-1 burned the whole budget and central-2 was dialed with an expired deadline | deadline hoisted outside the `foreach`; a kill-style drill can't see it (refused = fast-fail), only a hang discriminates | `e54711a2`; pinned by `Each_endpoint_gets_a_FRESH_deadline` (mutation-verified); 1B.3 gains a `docker pause` hang drill |
+| 2 | 1A.2 | **Dark-switch rollback deactivated the known-cluster gate** (Opus review, I-2): gate published only under `Mode=Grpc` while the interceptor + service are port-mapped in every mode — a rollback to ClusterClient/Dps would silently restore the timing oracle + pre-auth flood | gate activation coupled to transport mode instead of the endpoint it protects | `48dc2120` — gate follows the endpoint; pinned by the every-mode Theory + Dps guard pin `86633f14` |
+| 5 | 1A.3 (live gate, run 2) | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider`, same `AuditWriterActorKey` cause. NOT a defect: it is exactly the case the review-prescribed hardening was written for | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede starter #4 | **Closed by `e54883a8`, and the heal is PROVEN LIVE** — the coordinator singleton was moved onto the node that lost the race (stop central-2) and it fanned a dispatch to all five live nodes with no dialer error. Residual is cosmetic (Error text reads permanent; will trip a soak error-grep on a healthy node) → Phase 2 |
+| 4 | 1A.3 (live gate, run 1) | **An admin node cannot boot** (BLOCKED 1A): both central nodes die at startup with `MissingActorRegistryEntryException: ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey`, taking `AkkaHostedService` and the whole host with them. Mode-INDEPENDENT — reproduced on the same image with `MeshTransport__Mode=ClusterClient`, so it is not the gate's flip. Driver-only nodes are unaffected (they register neither `IMeshPskProvider` nor `ActorAuditWriter`) | `WithOtOpcUaControlPlaneSingletons`'s first `WithActors` starter eagerly resolves `IMeshCommandDialer` + `IMeshPskProvider` to pass into `CentralCommunicationActor.Props`. That graph is dialer → `IMeshPskProvider` → `ISecretResolver` → (`AddZbSecrets`) → `IAuditWriter` → `ActorAuditWriter`, whose ctor calls `registry.Get<AuditWriterActorKey>()` — and `WithSingleton<AuditWriterActorKey>` is registered ~60 lines later in the same builder. Same class as scadaproj#1 and mesh Phase 6's `IClusterRoleInfo` crash: only visible with the DI graph built in a real host, which is why the container-activation test (no `AddZbSecrets` in its collection) and `TwoNodeClusterHarness` (no provider at all) both miss it. Aggravating: the failed process does not exit, so `docker ps` reports `Up restarts=0` while the node serves no HTTP | **OPEN.** Fix + red-first pin belong to 1A (the gate made no product change). Suggested shape: hand the actor lazy accessors, mirroring the `Func<>` the same block already uses for the coordinator proxy. Pin must build the graph in starter order (a Host.IntegrationTests boot of an admin-role host), not merely activate the dialer |
+| 3 | 1A.2 | **Fan parked the actor mailbox** (self-caught by the implementer): dialing from the receive ran `TryGetPsk`'s cold-cache 3 s block pre-await (cold after EVERY refresh due to `InvalidateAll`), stalling deployment acks behind the fan | async methods run synchronously to the first await | `Task.Run` hop (token None — a canceled scheduling would read as a DB failure via PipeTo); pinned red-first by `A_slow_dial_does_not_park_the_actors_mailbox` |

 ## Carry-forward register

@@ -112,13 +139,26 @@ Checks a gate could not drive, named + carried to the next gate (proportionate-g

 | Item | Deferred from | Proven instead by | Closed in |
 |---|---|---|---|
-| — | — | *(none yet)* | |
+| Hang-style central failover drill (`docker pause`, not `docker stop` — only a hang discriminates the per-endpoint-deadline fix) | 1B.2 review | unit fact `Each_endpoint_gets_a_FRESH_deadline` | 1B.3 gate (MUST include) |
+| I-4 measurement: ack burst within ~3 s after a contact-refresh tick (cold PSK cache on the inbound auth path) | 1A.2 review | bounded 3 s store read + `Lazy` stampede suppression | 1B.3 gate (observe) |
+| `TwoNodeClusterHarness` logs the I-1 null-provider Error at every spawn (harness registers no `IMeshPskProvider`) — EXPECTED, not a rig fault | 1A.2 re-review | production pairing is `hasAdmin`-unconditional in one block | note for 1B.3/Phase 2 readers |
+| Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter`; starter-order half now covered by `ControlPlaneStarterOrderTests` | Phase 2 (or 1B.3) — note defect 4 shows the missing pin is not merely role-scoping but **starter-order**: the container-activation test resolves from a collection without `AddZbSecrets` |
+| `Resolve`-failure Error wording says "continuing without it" (reads permanent — the memoize heals it at first use) and the line fires on real boots (2/3 in the gate) — reword or whitelist it before the Phase 2 soak error-grep, else a healthy central scores as failing the soak | 1A.3 re-run | heal proven live: singleton moved onto the race-losing node, fan clean, no dialer-missing error | Phase 2 (soak grep design) |
+| A repaired `GrpcPort` takes effect only at the next contact refresh (≤60 s): a deploy 5 s post-restore was still skipped and honestly TimedOut — by design, operator-visible; document if it surprises in Phase 2 | 1A.3 re-run | check 7 evidence | Phase 2 (observe) |
+| `RevisionHash` moved on byte-identical config after a direct SQL `UPDATE` bumped `Script.RowVersion` (hash covers a column content-identity does not); determinism itself holds (3 no-change dispatches → identical hash) | 1A.3 re-run | incidental, out of program scope | note only |
+| 1A checks 1/2/7 (deploy fan over gRPC, mixed-transport seal, NULL-`GrpcPort` skip) and the central half of 4 | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — all driven live and PASS |
+| Check 8 is vacuous for telemetry + config-serve — central never booted, so neither plane carried a call | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — both planes carried authenticated traffic (28 KB artifact fetches from `:4055`, telemetry streams open on all six) with 0 denials |
+| **`DriverStatusPanel.razor` is referenced by no page or component** — the Reconnect/Restart buttons are unreachable in the running AdminUI (pre-existing on `master`, not caused by this program; re-verified in run 2: 0 `<DriverStatusPanel` usages, no Restart/Reconnect control on any rendered page, no such HTTP endpoint) | 1A.3 gate | Check 3 **closed by composition**: run 1's live node-side leg for both commands + run 2's check 4 driving the identical `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path from the AdminUI | Out of program scope — file separately if the control is meant to exist |
+| `Failed to resolve` Error text reads as permanent though first use retries, and will trip a soak error-grep on a healthy node | 1A.3 gate run 2 (defect 5) | Self-heal proven live | Phase 2 |

 ## Issues filed by this program

 | Issue | What | Status |
 |---|---|---|
 | [lmxopcua#542](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/542) | `FleetStatusBroadcaster` derives "fleet" from central's own gossip ring — silently central-pair-blind since mesh Phase 6; pre-existing, transport-independent, out of this program's scope | FILED 2026-08-06 |
+| [lmxopcua#544](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/544) | `ConfigPublishCoordinatorTimeoutTests.DeadlineElapsed_for_current_deployment_marks_TimedOut` is racy on its own terms (150 ms wall-clock deadline; row inserted after spawn) — flaked 2/~11 full runs once 1A.2 grew the suite; green at base. Fix the test (inject clock / raise deadline) | FILED 2026-08-06 (1A.3) |
+| [lmxopcua#545](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/545) | Mesh command/ack planes are h2c-only — the per-cluster PSK crosses the network in cleartext on a write surface (precedent: every sibling gRPC plane; trusted-site-link assumption). Decide: accepted risk, or a `UseTls`/scheme option | FILED 2026-08-06 (1A.3) |
+| [lmxopcua#546](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/546) | AdminUI `DriverStatusPanel` (the only Restart/Reconnect control) is referenced by no page — the commands have no reachable UI; pre-existing, found by the 1A gate | FILED 2026-08-06 |
 | [lmxopcua#543](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/543) | The three pre-existing auth interceptors (`TelemetryStream`/`ConfigServe`/`LocalDbSync`) treat a whitespace-only key as SET (`IsNullOrEmpty`), so a whitespace `Telemetry:ApiKey` etc. opens the endpoint to a whitespace bearer; the two new mesh interceptors use `IsNullOrWhiteSpace`. Backport is a behaviour change to shipped endpoints — own issue, out of this program's scope | FILED 2026-08-06 |

 ## Known constraints carried from the research (do not rediscover)
@@ -248,7 +248,7 @@ user-owned.**
 - Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Grpc/NodeCommandGrpcServiceTests.cs`

 Follow the alias convention (`using GeneratedServiceBase = …NodeCommandService.NodeCommandServiceBase;`,
-class suffixed `GrpcService`). `Execute`: decode via `MeshTransportDtoMapper.FromProto`, Tell
+class suffixed `GrpcService`). `Execute`: decode via `MeshTransportDtoMapper.FromEnvelope`, Tell
 the registered `NodeCommunicationActor` (resolve via `NodeCommunicationActorKey` registry,
 `Runtime/ServiceCollectionExtensions.cs:583`), return `ExecuteAck`. Actor not yet registered
 (boot window) ⇒ `RpcException(Unavailable)` — the readiness convention. Decode failure ⇒
@@ -3,22 +3,233 @@
  "executionModel": "opus-subagents; sequential within a worktree, 1A and 1B concurrent in separate worktrees (git worktree add ../OtOpcUa-1B feat/grpc-central-ack); 1A merges first, 1B rebases; stop at each phase DoD (PR/merge user-owned)",
  "worktrees": {},
  "tasks": [
-    {"id": 0, "subject": "Task 0.1: mesh_transport.proto + contract lock", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "standard", "parallelizableWith": ["0.3 (batch, same agent)"], "status": "completed", "commit": "c0ccfa27", "review": "spec ✅ + code ✅"},
-    {"id": 1, "subject": "Task 0.2: DTO mapper + round-trip goldens", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "high-risk", "parallelizableWith": [], "status": "completed", "blockedBy": [0], "commit": "545413a8", "review": "spec ✅ + code ✅ (serial)"},
-    {"id": 2, "subject": "Task 0.3: MeshTransportOptions + validators + SplitTopology relaxation", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "standard", "parallelizableWith": ["0.1 (batch, same agent)"], "status": "completed", "commit": "aa21f61f", "review": "spec ✅ + code ✅"},
-    {"id": 3, "subject": "Task 0.4: node-side MeshCommandAuthInterceptor", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "high-risk", "parallelizableWith": ["0.5 (batch, same agent)"], "status": "completed", "blockedBy": [2], "commit": "fbda015c + fixes 24e55d33/a4631410", "review": "spec ✅ + code ✅ (opus, fix loop closed)"},
-    {"id": 4, "subject": "Task 0.5: IMeshPskProvider + CentralCommandAuthInterceptor", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "high-risk", "parallelizableWith": ["0.4 (batch, same agent)"], "status": "completed", "blockedBy": [2], "commit": "68a416e4 + fixes 24e55d33/a4631410", "review": "spec ✅ + code ✅ (opus, fix loop closed); 4 items gated on Phase 1A — see tracking §Phase 1A security register"},
-    {"id": 5, "subject": "Task 0.6: rig config + FleetStatus issue + docs stub + Phase 0 gate", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "small", "parallelizableWith": [], "status": "completed", "blockedBy": [0, 1, 2, 3, 4], "commit": "23555ed0", "review": "code ✅", "gate": "PASS 4/4; issues #542 #543 filed"},
-    {"id": 6, "subject": "Task 1A.1: NodeCommandGrpcService on the telemetry listener", "phase": "1A", "branch": "feat/grpc-node-command", "classification": "standard", "parallelizableWith": ["1B.1 (other worktree)"], "status": "pending", "blockedBy": [5]},
-    {"id": 7, "subject": "Task 1A.2: MeshGrpcCommandDialer + third branch in CentralCommunicationActor", "phase": "1A", "branch": "feat/grpc-node-command", "classification": "high-risk", "parallelizableWith": ["1B.2 (other worktree)"], "status": "pending", "blockedBy": [6]},
-    {"id": 8, "subject": "Task 1A.3: 1A live gate (central pair flipped, rig-only, reverted)", "phase": "1A", "branch": "feat/grpc-node-command", "classification": "small", "parallelizableWith": [], "status": "pending", "blockedBy": [7]},
-    {"id": 9, "subject": "Task 1B.1: CentralCommandGrpcService + admin validator rule", "phase": "1B", "branch": "feat/grpc-central-ack", "classification": "standard", "parallelizableWith": ["1A.1 (other worktree)"], "status": "pending", "blockedBy": [5]},
-    {"id": 10, "subject": "Task 1B.2: GrpcApplyAckSender + node-side wiring", "phase": "1B", "branch": "feat/grpc-central-ack", "classification": "high-risk", "parallelizableWith": ["1A.2 (other worktree)"], "status": "pending", "blockedBy": [9]},
-    {"id": 11, "subject": "Task 1B.3: 1B live gate (site-a flipped, rig-only, reverted; rebase on 1A first)", "phase": "1B", "branch": "feat/grpc-central-ack", "classification": "small", "parallelizableWith": [], "status": "pending", "blockedBy": [8, 10]},
-    {"id": 12, "subject": "Phase 2: baseline reproduction + full-rig cutover + soak + kill drills", "phase": "2", "branch": "(rig commit)", "classification": "high-risk", "parallelizableWith": [], "status": "pending", "blockedBy": [8, 11]},
-    {"id": 13, "subject": "Task 3.1: delete ClusterClient (code + HOCON + config + tests, ordered)", "phase": "3", "branch": "feat/grpc-phase3-deletion", "classification": "high-risk", "parallelizableWith": [], "status": "pending", "blockedBy": [12]},
-    {"id": 14, "subject": "Task 3.2: docs sweep (Configuration/Redundancy/Telemetry/CLAUDE.md + scadaproj index)", "phase": "3", "branch": "feat/grpc-phase3-deletion", "classification": "small", "parallelizableWith": [], "status": "pending", "blockedBy": [13]},
-    {"id": 15, "subject": "Phase 4: terminal live gate on post-deletion image (8 checks, provenance-guarded)", "phase": "4", "branch": "(master)", "classification": "high-risk", "parallelizableWith": [], "status": "pending", "blockedBy": [14]}
+    {
+      "id": 0,
+      "subject": "Task 0.1: mesh_transport.proto + contract lock",
+      "phase": "0",
+      "branch": "feat/grpc-phase0-contract-auth",
+      "classification": "standard",
+      "parallelizableWith": [
+        "0.3 (batch, same agent)"
+      ],
+      "status": "completed",
+      "commit": "c0ccfa27",
+      "review": "spec ✅ + code ✅"
+    },
+    {
+      "id": 1,
+      "subject": "Task 0.2: DTO mapper + round-trip goldens",
+      "phase": "0",
+      "branch": "feat/grpc-phase0-contract-auth",
+      "classification": "high-risk",
+      "parallelizableWith": [],
+      "status": "completed",
+      "blockedBy": [
+        0
+      ],
+      "commit": "545413a8",
+      "review": "spec ✅ + code ✅ (serial)"
+    },
+    {
+      "id": 2,
+      "subject": "Task 0.3: MeshTransportOptions + validators + SplitTopology relaxation",
+      "phase": "0",
+      "branch": "feat/grpc-phase0-contract-auth",
+      "classification": "standard",
+      "parallelizableWith": [
+        "0.1 (batch, same agent)"
+      ],
+      "status": "completed",
+      "commit": "aa21f61f",
+      "review": "spec ✅ + code ✅"
+    },
+    {
+      "id": 3,
+      "subject": "Task 0.4: node-side MeshCommandAuthInterceptor",
+      "phase": "0",
+      "branch": "feat/grpc-phase0-contract-auth",
+      "classification": "high-risk",
+      "parallelizableWith": [
+        "0.5 (batch, same agent)"
+      ],
+      "status": "completed",
+      "blockedBy": [
+        2
+      ],
+      "commit": "fbda015c + fixes 24e55d33/a4631410",
+      "review": "spec ✅ + code ✅ (opus, fix loop closed)"
+    },
+    {
+      "id": 4,
+      "subject": "Task 0.5: IMeshPskProvider + CentralCommandAuthInterceptor",
+      "phase": "0",
+      "branch": "feat/grpc-phase0-contract-auth",
+      "classification": "high-risk",
+      "parallelizableWith": [
+        "0.4 (batch, same agent)"
+      ],
+      "status": "completed",
+      "blockedBy": [
+        2
+      ],
+      "commit": "68a416e4 + fixes 24e55d33/a4631410",
+      "review": "spec ✅ + code ✅ (opus, fix loop closed); 4 items gated on Phase 1A — see tracking §Phase 1A security register"
+    },
+    {
+      "id": 5,
+      "subject": "Task 0.6: rig config + FleetStatus issue + docs stub + Phase 0 gate",
+      "phase": "0",
+      "branch": "feat/grpc-phase0-contract-auth",
+      "classification": "small",
+      "parallelizableWith": [],
+      "status": "completed",
+      "blockedBy": [
+        0,
+        1,
+        2,
+        3,
+        4
+      ],
+      "commit": "23555ed0",
+      "review": "code ✅",
+      "gate": "PASS 4/4; issues #542 #543 filed"
+    },
+    {
+      "id": 6,
+      "subject": "Task 1A.1: NodeCommandGrpcService on the telemetry listener",
+      "phase": "1A",
+      "branch": "feat/grpc-node-command",
+      "classification": "standard",
+      "parallelizableWith": [
+        "1B.1 (other worktree)"
+      ],
+      "status": "completed",
+      "blockedBy": [
+        5
+      ],
+      "commit": "64774652",
+      "review": "spec ✅ + code ✅"
+    },
+    {
+      "id": 7,
+      "subject": "Task 1A.2: MeshGrpcCommandDialer + third branch in CentralCommunicationActor (+ security register items 1-2)",
+      "phase": "1A",
+      "branch": "feat/grpc-node-command",
+      "classification": "high-risk",
+      "parallelizableWith": [
+        "1B.2 (other worktree)"
+      ],
+      "status": "completed",
+      "blockedBy": [
+        6
+      ],
+      "commit": "14d21496+6a862011 + fixes 48dc2120/86633f14",
+      "review": "spec ✅ + code ✅ (opus; I-2 fixed via gate-follows-endpoint decision)"
+    },
+    {
+      "id": 8,
+      "subject": "Task 1A.3: 1A live gate (central pair flipped, rig-only, reverted)",
+      "phase": "1A",
+      "branch": "feat/grpc-node-command",
+      "classification": "small",
+      "parallelizableWith": [],
+      "status": "completed",
+      "blockedBy": [
+        7
+      ],
+      "commit": "gate records 410c2e29/e3a0ddbc; boot fixes c0475f2d/e54883a8",
+      "gate": "run 1 FAIL (defect 4) -> run 2 PASS 8/8; issues #544 #545 #546 filed"
+    },
+    {
+      "id": 9,
+      "subject": "Task 1B.1: CentralCommandGrpcService + admin validator rule",
+      "phase": "1B",
+      "branch": "feat/grpc-central-ack",
+      "classification": "standard",
+      "parallelizableWith": [
+        "1A.1 (other worktree)"
+      ],
+      "status": "pending",
+      "blockedBy": [
+        5
+      ]
+    },
+    {
+      "id": 10,
+      "subject": "Task 1B.2: GrpcApplyAckSender + node-side wiring",
+      "phase": "1B",
+      "branch": "feat/grpc-central-ack",
+      "classification": "high-risk",
+      "parallelizableWith": [
+        "1A.2 (other worktree)"
+      ],
+      "status": "pending",
+      "blockedBy": [
+        9
+      ]
+    },
+    {
+      "id": 11,
+      "subject": "Task 1B.3: 1B live gate (site-a flipped, rig-only, reverted; rebase on 1A first)",
+      "phase": "1B",
+      "branch": "feat/grpc-central-ack",
+      "classification": "small",
+      "parallelizableWith": [],
+      "status": "pending",
+      "blockedBy": [
+        8,
+        10
+      ]
+    },
+    {
+      "id": 12,
+      "subject": "Phase 2: baseline reproduction + full-rig cutover + soak + kill drills",
+      "phase": "2",
+      "branch": "(rig commit)",
+      "classification": "high-risk",
+      "parallelizableWith": [],
+      "status": "pending",
+      "blockedBy": [
+        8,
+        11
+      ]
+    },
+    {
+      "id": 13,
+      "subject": "Task 3.1: delete ClusterClient (code + HOCON + config + tests, ordered)",
+      "phase": "3",
+      "branch": "feat/grpc-phase3-deletion",
+      "classification": "high-risk",
+      "parallelizableWith": [],
+      "status": "pending",
+      "blockedBy": [
+        12
+      ]
+    },
+    {
+      "id": 14,
+      "subject": "Task 3.2: docs sweep (Configuration/Redundancy/Telemetry/CLAUDE.md + scadaproj index)",
+      "phase": "3",
+      "branch": "feat/grpc-phase3-deletion",
+      "classification": "small",
+      "parallelizableWith": [],
+      "status": "pending",
+      "blockedBy": [
+        13
+      ]
+    },
+    {
+      "id": 15,
+      "subject": "Phase 4: terminal live gate on post-deletion image (8 checks, provenance-guarded)",
+      "phase": "4",
+      "branch": "(master)",
+      "classification": "high-risk",
+      "parallelizableWith": [],
+      "status": "pending",
+      "blockedBy": [
+        14
+      ]
+    }
  ],
  "lastUpdated": "2026-08-06"
-}
+}
@@ -3,17 +3,24 @@ using Akka.Actor;
 using Akka.Cluster.Tools.Client;
 using Akka.Cluster.Tools.PublishSubscribe;
 using Akka.Event;
+using Grpc.Core;
 using Microsoft.EntityFrameworkCore;
 using ZB.MOM.WW.OtOpcUa.Cluster;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
+using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
+using ZB.MOM.WW.OtOpcUa.Commons.Types;
 using ZB.MOM.WW.OtOpcUa.Configuration;
+// Grpc.Core (imported for RpcException on the dial path) declares a Status of its own. This file's
+// bare Status is Akka's — the Status.Failure a faulted piped task arrives as; the gRPC one is only
+// ever reached through an RpcException instance.
+using Status = Akka.Actor.Status;

 namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;

 /// <summary>
 ///     Central-side end of the central↔node command boundary (per-cluster mesh Phase 2). Routes
-///     every central→node command out — over DistributedPubSub or ClusterClient, per
+///     every central→node command out — over DistributedPubSub, ClusterClient or gRPC, per
 ///     <see cref="MeshTransportOptions.Mode"/> — and forwards inbound <see cref="ApplyAck"/>s to the
 ///     deploy coordinator singleton.
 /// </summary>
@@ -31,6 +38,23 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
 ///         and fans <c>SendToAll</c> across all of them. See <see cref="RebuildClient"/> — this is the
 ///         single most important thing to understand before changing this actor.
 ///     </para>
+///     <para>
+///         <b>The gRPC branch fans over the same recipient set, node by node</b>
+///         (ClusterClient→gRPC Phase 1A). <c>SendToAll</c> reaches every node registered on a mesh's
+///         receptionist, and the contact load already selects exactly the enabled, non-maintenance
+///         rows those nodes come from; the gRPC branch dials that same row set directly, one call per
+///         node, so the recipient set is identical by construction rather than by coincidence. It is
+///         deliberately NOT narrower: the node side has no <c>ClusterId</c> or node filter at all —
+///         scoping happens later, inside the artifact — so a node left out here silently keeps its
+///         old configuration while the deployment seals green.
+///     </para>
+///     <para>
+///         <b>Fire-and-forget, on every transport.</b> The gRPC branch pipes per-node results back to
+///         itself for logging ONLY: a failure names the node and the status and is dropped. No retry,
+///         no queue, no ordering guarantee across nodes. That is not an omission — it is the
+///         ClusterClient semantic preserved exactly, and the backstops are unchanged (the deploy
+///         coordinator's apply deadline, and a node's boot-time replay).
+///     </para>
 /// </remarks>
 public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
 {
@@ -40,8 +64,30 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
    private readonly MeshTransportOptions _options;
    private readonly IMeshClusterClientFactory _clientFactory;
    private readonly Func<IActorRef?> _coordinator;
+
+    // Resolved in PreStart, never at construction. See the Props doc: the DI graph behind these two
+    // reaches actor-registry entries that do not exist yet while the Akka builder's starters are
+    // still running, and touching it there is what stopped every central node from booting.
+    private readonly Func<IMeshCommandDialer?>? _dialerFactory;
+    private readonly Func<IMeshPskProvider?>? _pskProviderFactory;
+    private IMeshCommandDialer? _dialer;
+    private IMeshPskProvider? _pskProvider;
+
+    // Memoize-on-first-use, NOT PreStart-only: PreStart runs on a dispatcher thread concurrently
+    // with the builder's remaining starters, so its resolve can lose a millisecond-wide race
+    // against AuditWriterActorKey's registration (starter #4) and Resolve() would swallow that
+    // into a PERMANENT silent null — the known-cluster gate never published, fail-open. These
+    // properties retry on first use instead; the first RefreshContacts arrives as a message,
+    // long after every starter finished, so a lost boot race self-heals there. Actor-thread
+    // only, so ??= needs no synchronisation; both are container singletons, so a repeat resolve
+    // is a dictionary hit.
+    private IMeshPskProvider? PskProvider => _pskProvider ??= Resolve(_pskProviderFactory, nameof(IMeshPskProvider));
+
+    private IMeshCommandDialer? Dialer => _dialer ??= Resolve(_dialerFactory, nameof(IMeshCommandDialer));
+
    private readonly ILoggingAdapter _log = Context.GetLogger();
    private readonly bool _useClusterClient;
+    private readonly bool _useGrpc;

    // One client per application Cluster, keyed by ClusterId. A ClusterClient reaches only the mesh
    // whose receptionist it dialled, and after Phase 6 every cluster is its own mesh — so central must
@@ -49,6 +95,15 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
    private readonly Dictionary<string, (IActorRef Client, ImmutableHashSet<string> Contacts)> _clients =
        new();

+    // Node ids already warned about for a null GrpcPort. A persistently portless row would otherwise
+    // produce one warning per refresh forever; same warn-once shape as TelemetryNodeSource. Actor
+    // thread only.
+    private readonly HashSet<string> _warnedPortlessNodes = new(StringComparer.Ordinal);
+
+    // The gRPC dispatch set, replaced wholesale on every contact load. Actor thread only; the fan
+    // captures the list reference, so a refresh mid-fan cannot mutate a dispatch already under way.
+    private IReadOnlyList<MeshNodeTarget> _targets = [];
+
    private CancellationTokenSource _lifecycle = new();

    /// <summary>Gets the timer scheduler driving the periodic contact refresh.</summary>
@@ -63,46 +118,104 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
    ///     an <see cref="IActorRef"/> so this actor never depends on registration order at wiring
    ///     time — it resolves on first use and caches.
    /// </param>
+    /// <param name="dialer">
+    ///     Resolves the thing that places one command call on one node, under
+    ///     <see cref="MeshTransportOptions.ModeGrpc"/>. Optional so the two older transports need
+    ///     nothing new wired; a <see langword="null"/> dialer under <c>Grpc</c> is a mis-wiring and
+    ///     is reported loudly rather than silently dropping every command.
+    /// </param>
+    /// <param name="pskProvider">
+    ///     Resolves central's per-cluster key lookup, told on every contact refresh which clusters
+    ///     exist (the known-cluster gate) and to drop its cache (so a rotated key takes effect within
+    ///     one refresh). Optional for the same reason as <paramref name="dialer"/>.
+    /// </param>
    /// <returns>The props.</returns>
+    /// <remarks>
+    ///     <para>
+    ///         <b>Both dependencies arrive as <see cref="Func{TResult}"/>, and that is not a style
+    ///         choice.</b> Passing instances meant the caller resolved them while the Akka builder's
+    ///         starters were still running, and the DI graph behind them reaches back into the actor
+    ///         registry: dialer → <c>IMeshPskProvider</c> → <c>ISecretResolver</c> → <c>AddZbSecrets</c>
+    ///         → <c>IAuditWriter</c> → <c>ActorAuditWriter</c>, whose constructor calls
+    ///         <c>registry.Get&lt;AuditWriterActorKey&gt;()</c> — a singleton registered dozens of
+    ///         lines LATER in the very same builder. Every central node failed to boot, in every
+    ///         transport mode, on a <c>MissingActorRegistryEntryException</c>. Deferring to
+    ///         <see cref="PreStart"/> moves the first resolve to a point where the host is fully
+    ///         built, and matches the <see cref="Func{TResult}"/> the coordinator proxy already uses
+    ///         one argument earlier for the same class of reason.
+    ///     </para>
+    ///     <para>
+    ///         This is the third instance of the same failure mode in this family — eager resolution
+    ///         inside an <c>AddAkka</c> configurator, previously <c>scadaproj#1</c> (a DI cycle that
+    ///         deadlocked) and mesh Phase 6's <c>IClusterRoleInfo</c> boot <c>StackOverflow</c>. The
+    ///         rule it keeps re-teaching: inside an Akka builder, resolve nothing you can defer.
+    ///     </para>
+    /// </remarks>
    public static Props Props(
        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
        MeshTransportOptions options,
        IMeshClusterClientFactory clientFactory,
-        Func<IActorRef?> coordinator) =>
+        Func<IActorRef?> coordinator,
+        Func<IMeshCommandDialer?>? dialer = null,
+        Func<IMeshPskProvider?>? pskProvider = null) =>
        Akka.Actor.Props.Create(() =>
-            new CentralCommunicationActor(dbFactory, options, clientFactory, coordinator));
+            new CentralCommunicationActor(
+                dbFactory, options, clientFactory, coordinator, dialer, pskProvider));

    /// <summary>Initializes a new instance of the <see cref="CentralCommunicationActor"/> class.</summary>
    /// <param name="dbFactory">Factory for the config database.</param>
    /// <param name="options">The bound mesh-transport options.</param>
    /// <param name="clientFactory">Creates the ClusterClient.</param>
    /// <param name="coordinator">Lazily resolves the deploy-coordinator singleton proxy.</param>
+    /// <param name="dialer">Resolves the gRPC command dialer; invoked in <see cref="PreStart"/>.</param>
+    /// <param name="pskProvider">Resolves the PSK provider; invoked in <see cref="PreStart"/>.</param>
    public CentralCommunicationActor(
        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
        MeshTransportOptions options,
        IMeshClusterClientFactory clientFactory,
-        Func<IActorRef?> coordinator)
+        Func<IActorRef?> coordinator,
+        Func<IMeshCommandDialer?>? dialer = null,
+        Func<IMeshPskProvider?>? pskProvider = null)
    {
        _dbFactory = dbFactory;
        _options = options;
        _clientFactory = clientFactory;
        _coordinator = coordinator;
+        _dialerFactory = dialer;
+        _pskProviderFactory = pskProvider;
        _useClusterClient = string.Equals(
            options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
+        _useGrpc = string.Equals(
+            options.Mode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase);

        Receive<MeshCommand>(HandleMeshCommand);
        Receive<ApplyAck>(HandleApplyAck);
        Receive<RefreshContacts>(_ => LoadContactsFromDb());
        Receive<ContactsLoaded>(HandleContactsLoaded);
+        Receive<CommandDispatched>(HandleCommandDispatched);

        // A faulted LoadContactsFromDb task is piped here as a Status.Failure. Without this handler
        // the failure is an unhandled message (debug level only) and the refresh fails silently —
        // an operator cannot distinguish "no nodes configured" from "the database is down", and the
        // contact set silently freezes at whatever it last held.
-        Receive<Status.Failure>(f => _log.Warning(
-            f.Cause,
-            "Failed to load ClusterNode contact points; the mesh ClusterClient contact set was NOT "
-            + "refreshed and may be stale or empty. Commands may be dropped until the next refresh"));
+        //
+        // The freeze extends to the known-cluster gate, deliberately: a failed load must neither
+        // widen the gate nor narrow it, and the previous set is the last thing central actually knew.
+        // The second line exists because that freeze is INVISIBLE otherwise, and it is fail-OPEN
+        // before the first successful load — a central that has never reached its ConfigDb refuses
+        // no cluster at all.
+        Receive<Status.Failure>(f =>
+        {
+            _log.Warning(
+                f.Cause,
+                "Failed to load ClusterNode contact points; the mesh contact set was NOT refreshed "
+                + "and may be stale or empty. Commands may be dropped until the next refresh");
+            _log.Warning(
+                "The known-cluster gate was NOT refreshed either: it still holds the set from the "
+                + "last successful load, and if none has succeeded yet (boot, or central SQL Server "
+                + "unreachable since start) it is INACTIVE — every cluster id reaches the secret "
+                + "store, which is the pre-authentication cost the gate exists to remove");
+        });

        Receive<SubscribeAck>(_ => { /* DPS subscribe confirmation */ });
    }
@@ -110,14 +223,51 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
    /// <inheritdoc />
    protected override void PreStart()
    {
-        if (!_useClusterClient)
+        // First point at which the host is fully built — see the Props remark. Re-runs on every
+        // actor restart, which re-reads the container rather than caching a stale reference; both
+        // dependencies are container singletons, so a restart gets the same instances back.
+        _dialer = Resolve(_dialerFactory, nameof(IMeshCommandDialer));
+        _pskProvider = Resolve(_pskProviderFactory, nameof(IMeshPskProvider));
+
+        if (!_useClusterClient && !_useGrpc)
        {
            _log.Info(
                "Mesh transport is {Mode}: commands are published on DistributedPubSub and no "
-                + "ClusterClient is created", _options.Mode);
-            return;
+                + "ClusterClient is created. The contact refresh still runs — it is what keeps the "
+                + "known-cluster gate current, and that gate belongs to the inbound ack endpoint, "
+                + "which is mapped by port and does not care about this mode", _options.Mode);
        }

+        if (_useGrpc && _dialer is null)
+        {
+            // Not a fatal throw: the ack path (HandleApplyAck) is transport-independent and still
+            // works, so taking the actor down would turn a command-plane mis-wiring into an ack
+            // outage as well. Every dispatch logs its own error, so this cannot go unnoticed.
+            _log.Error(
+                "Mesh transport is Grpc but no command dialer was supplied to this actor; EVERY "
+                + "central→node command will be dropped. Check the ControlPlane registration wires "
+                + "IMeshCommandDialer");
+        }
+
+        if (_pskProvider is null)
+        {
+            // Mode-independent, like the gate itself. A silent absence here is the worst of the
+            // three outcomes in this method: nothing malfunctions, so nothing is ever investigated,
+            // while the gate stays permanently inactive and a rotated key is never picked up.
+            _log.Error(
+                "No IMeshPskProvider was supplied to this actor; the known-cluster gate will NEVER "
+                + "be published and cached per-cluster keys will NEVER be dropped. Inbound mesh acks "
+                + "stay exposed to the pre-authentication secret-store cost the gate removes, and a "
+                + "rotated key needs a central restart. Check the ControlPlane registration");
+        }
+
+        // Started in EVERY mode. The refresh is not transport state: it is central's ClusterNode
+        // inventory, and the known-cluster gate it feeds protects an endpoint mapped by PORT, not by
+        // MeshTransport:Mode. Gating the timer on the mode would mean a dark-switch rollback to
+        // ClusterClient or Dps left that endpoint live with the gate permanently inactive — silently
+        // reopening both the timing oracle and the pre-auth store/audit flood. Under Dps this is a
+        // ConfigDb query every ContactRefreshSeconds that did not run before; central always has a
+        // ConfigDb, and one indexed read a minute is the cheaper half of the trade.
        Timers.StartPeriodicTimer(
            RefreshTimerKey,
            new RefreshContacts(),
@@ -125,18 +275,64 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
            TimeSpan.FromSeconds(_options.ContactRefreshSeconds));
    }

+    /// <summary>
+    ///     Invokes a dependency factory, converting a resolution failure into a logged absence.
+    /// </summary>
+    /// <typeparam name="T">The dependency's type.</typeparam>
+    /// <param name="factory">The factory, or <see langword="null"/> when nothing was supplied.</param>
+    /// <param name="name">The dependency's name, for the log.</param>
+    /// <returns>The resolved dependency, or <see langword="null"/>.</returns>
+    /// <remarks>
+    ///     Catching is the point. A throw out of <see cref="PreStart"/> is an actor failure, and the
+    ///     supervisor's answer to that is to restart — which calls <see cref="PreStart"/> again, and
+    ///     a broken DI graph is not a transient condition. That is an unbounded restart loop on the
+    ///     node's command boundary, taking the transport-independent ack path down with it. Logging
+    ///     and continuing leaves the ack path serving and hands the two null guards below a genuine
+    ///     absence to report, which is the same outcome as never having been wired.
+    /// </remarks>
+    private T? Resolve<T>(Func<T?>? factory, string name)
+        where T : class
+    {
+        if (factory is null)
+            return null;
+
+        try
+        {
+            return factory();
+        }
+        catch (Exception ex)
+        {
+            _log.Error(
+                ex,
+                "Failed to resolve {Dependency} for the mesh command boundary; continuing without "
+                + "it. The guards below report what that costs", name);
+            return null;
+        }
+    }
+
    /// <inheritdoc />
    protected override void PostStop()
    {
        foreach (var (client, _) in _clients.Values) Context.Stop(client);
        _clients.Clear();

+        // Cancelling the lifecycle token also cancels every gRPC dial in flight. The dialer itself is
+        // deliberately NOT disposed here: it is a container-owned singleton shared with anything else
+        // that may dial, and an actor restart must not tear its channel pool down.
        _lifecycle.Cancel();
        _lifecycle.Dispose();
    }

    private void HandleMeshCommand(MeshCommand cmd)
    {
+        // Checked before the DPS arm: _useClusterClient is false under Grpc too, so the order is
+        // what keeps a Grpc-mode command off DistributedPubSub.
+        if (_useGrpc)
+        {
+            DispatchOverGrpc(cmd);
+            return;
+        }
+
        if (!_useClusterClient)
        {
            DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(cmd.Topic, cmd.Message));
@@ -168,6 +364,199 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
            client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, cmd.Message));
    }

+    /// <summary>
+    ///     Encodes one command once and fans it to every target node, one unary gRPC call each.
+    /// </summary>
+    /// <param name="cmd">The command envelope carrying the domain record.</param>
+    /// <remarks>
+    ///     <para>
+    ///         <b>Encoded once, before the fan.</b> Not per node — partly because the wire form is
+    ///         identical for every recipient, but mostly because <c>AlarmCommand</c> has no
+    ///         correlation member of its own and the envelope mints one: minting inside the loop
+    ///         would give one operator action N unrelated correlation ids and make it untraceable.
+    ///     </para>
+    ///     <para>
+    ///         <b>Nothing captured from the actor.</b> Each dial is a static call taking only locals,
+    ///         so no per-node continuation can touch actor state off the actor's thread. Results come
+    ///         back as ordinary messages via <c>PipeTo</c> and are handled on the actor thread like
+    ///         anything else.
+    ///     </para>
+    ///     <para>
+    ///         <b>Every dial starts on the thread pool, not here.</b> An <c>async</c> method runs
+    ///         synchronously up to its first <c>await</c>, and the dialer's pre-await work includes
+    ///         <c>IMeshPskProvider.TryGetPsk</c>, which blocks on the secret store (bounded, but
+    ///         seconds) the first time a cluster is looked up — and again after every refresh, since
+    ///         the refresh drops the cache to pick up rotations. Calling straight into it would park
+    ///         this actor's mailbox, and deployment acks queue behind that mailbox.
+    ///     </para>
+    /// </remarks>
+    private void DispatchOverGrpc(MeshCommand cmd)
+    {
+        var commandType = cmd.Message.GetType().Name;
+
+        var dialer = Dialer;
+        if (dialer is null)
+        {
+            _log.Error(
+                "Mesh transport is Grpc but no command dialer is wired — dropping {MessageType}. The "
+                + "command is not buffered and will not be retried",
+                commandType);
+            return;
+        }
+
+        // Captured before the fan: a contact refresh between here and the last dial must not change
+        // the recipient set of a dispatch already under way.
+        var targets = _targets;
+        if (targets.Count == 0)
+        {
+            // App-level drop, exactly as the ClusterClient arm does with no client. The command is
+            // NOT queued and will NOT be retried: a deploy fails at the coordinator's apply deadline
+            // naming the silent nodes, which is a better operator signal than a command arriving
+            // minutes late against a config the DB already recorded as failed.
+            _log.Warning(
+                "No mesh gRPC targets — dropping {MessageType}. The last contact refresh produced no "
+                + "enabled, non-maintenance ClusterNode row carrying a GrpcPort in any cluster; the "
+                + "command is not buffered and will not be retried",
+                commandType);
+            return;
+        }
+
+        CommandEnvelope envelope;
+        try
+        {
+            envelope = MeshTransportDtoMapper.ToEnvelope(cmd.Message, new CorrelationId(Guid.NewGuid()));
+        }
+        catch (Exception ex)
+        {
+            // The mapper refuses a payload the far side could not decode. Dropping here is the same
+            // outcome the old transport gave for such a message (nothing on the node matched it), but
+            // loud instead of silent.
+            _log.Error(
+                ex,
+                "Cannot encode {MessageType} for the gRPC command plane; the command is dropped and "
+                + "was NOT delivered to any node",
+                commandType);
+            return;
+        }
+
+        CancellationToken ct;
+        try
+        {
+            ct = _lifecycle.Token;
+        }
+        catch (ObjectDisposedException)
+        {
+            return; // Stopping.
+        }
+
+        var self = Self;
+        var correlationId = envelope.CorrelationId;
+        foreach (var target in targets)
+        {
+            var node = target;
+
+            // CancellationToken.None on Task.Run, deliberately: cancelling the SCHEDULING would make
+            // the task complete as canceled, which PipeTo converts into a Status.Failure — the one
+            // message this actor already reads as "the contact load failed". The lifecycle token is
+            // passed into the dial instead, where it cancels the call and comes back as an ordinary
+            // per-node outcome.
+            Task.Run(
+                    () => DispatchOneAsync(dialer, node, envelope, commandType, correlationId, ct),
+                    CancellationToken.None)
+                .PipeTo(self);
+        }
+    }
+
+    /// <summary>
+    ///     Dials one node and turns every possible outcome into a <see cref="CommandDispatched"/>.
+    /// </summary>
+    /// <param name="dialer">The dialer seam.</param>
+    /// <param name="target">The node to dial.</param>
+    /// <param name="envelope">The already-encoded command.</param>
+    /// <param name="commandType">CLR type name of the command, for the log.</param>
+    /// <param name="correlationId">Transport correlation, for the log.</param>
+    /// <param name="ct">The actor's lifecycle token.</param>
+    /// <returns>The outcome, always — this method does not fault.</returns>
+    /// <remarks>
+    ///     Catching everything is what keeps the result on ONE channel. A faulting task piped to an
+    ///     actor arrives as <c>Status.Failure</c>, which this actor already handles as "the contact
+    ///     load failed" — so an unhandled dial fault would log a database-outage warning about a node
+    ///     that simply did not answer.
+    /// </remarks>
+    private static async Task<CommandDispatched> DispatchOneAsync(
+        IMeshCommandDialer dialer,
+        MeshNodeTarget target,
+        CommandEnvelope envelope,
+        string commandType,
+        string correlationId,
+        CancellationToken ct)
+    {
+        try
+        {
+            await dialer.DispatchAsync(target.Endpoint, envelope, target.ClusterId, ct)
+                .ConfigureAwait(false);
+            return new CommandDispatched(target.NodeId, commandType, correlationId, null, null);
+        }
+        catch (RpcException ex)
+        {
+            return new CommandDispatched(
+                target.NodeId,
+                commandType,
+                correlationId,
+                ex.StatusCode.ToString(),
+                Truncate(ex.Status.Detail));
+        }
+        catch (OperationCanceledException)
+        {
+            // Named rather than swept into the generic arm: this is the ordinary shape of a node that
+            // did not answer in time, and "Unknown / TaskCanceledException" would send an operator
+            // looking for a bug instead of at DispatchTimeoutSeconds.
+            return new CommandDispatched(
+                target.NodeId,
+                commandType,
+                correlationId,
+                "Cancelled",
+                "the dial did not complete within MeshTransport:DispatchTimeoutSeconds, or central "
+                + "is shutting down");
+        }
+        catch (Exception ex)
+        {
+            // Type name only, never the message: this path also covers whatever the dialer's channel
+            // construction can throw, and a message is the one place a misconfigured endpoint string
+            // could carry something an operator did not mean to put in a log.
+            return new CommandDispatched(
+                target.NodeId, commandType, correlationId, "Unknown", ex.GetType().Name);
+        }
+    }
+
+    /// <summary>
+    ///     Bounds a status detail before it reaches the log. The string comes from the far side, and
+    ///     the far side is a node — trusted, but not a reason to let it choose how many lines of
+    ///     central's log one failed command occupies.
+    /// </summary>
+    /// <param name="detail">The received status detail.</param>
+    /// <returns>The detail, at most 200 characters.</returns>
+    private static string Truncate(string? detail)
+        => detail is null || detail.Length <= 200 ? detail ?? "" : detail[..200] + "…";
+
+    private void HandleCommandDispatched(CommandDispatched result)
+    {
+        if (result.FailureStatus is null)
+        {
+            _log.Debug(
+                "Mesh: {CommandType} (correlationId={CorrelationId}) accepted by {NodeId} over gRPC",
+                result.CommandType, result.CorrelationId, result.NodeId);
+            return;
+        }
+
+        _log.Warning(
+            "Mesh: {CommandType} (correlationId={CorrelationId}) was NOT delivered to {NodeId} "
+            + "({Status}: {Detail}). It is dropped and will NOT be retried — the deployment apply "
+            + "deadline and the node's boot-time replay are the backstops",
+            result.CommandType, result.CorrelationId, result.NodeId,
+            result.FailureStatus, result.FailureDetail);
+    }
+
    private void HandleApplyAck(ApplyAck ack)
    {
        var coordinator = _coordinator();
@@ -207,10 +596,18 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
        Task.Run(async () =>
        {
            await using var db = await dbFactory.CreateDbContextAsync(ct).ConfigureAwait(false);
+            // UNFILTERED, with the eligibility flags projected and applied per row below. The two
+            // outputs of this read want different row sets: the contact/dispatch sets are the
+            // enabled, non-maintenance nodes central talks TO, while the known-cluster gate answers
+            // "is this a cluster central has ever heard of" and must include a cluster whose every
+            // node is disabled or in maintenance — such a cluster can still send an ack, and denying
+            // it would make maintenance mode an authentication switch.
            var rows = await db.ClusterNodes
                .AsNoTracking()
-                .Where(n => n.Enabled && !n.MaintenanceMode)
-                .Select(n => new { n.NodeId, n.ClusterId, n.Host, n.AkkaPort })
+                .Select(n => new
+                {
+                    n.NodeId, n.ClusterId, n.Host, n.AkkaPort, n.GrpcPort, n.Enabled, n.MaintenanceMode,
+                })
                .ToListAsync(ct)
                .ConfigureAwait(false);

@@ -219,8 +616,34 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
            // client.
            var byCluster = new Dictionary<string, List<string>>();
            var malformed = new List<string>();
+
+            // The gRPC dispatch set, built from the SAME row set in the SAME pass. Sharing the read is
+            // what makes "the gRPC fan reaches exactly the ClusterClient recipient set" structural
+            // rather than a claim two separate queries would have to keep agreeing on.
+            var targets = new List<MeshNodeTarget>(rows.Count);
+            var portless = new List<string>();
+
+            // Every cluster with ANY ClusterNode row — the known-cluster gate's membership set, and
+            // deliberately a superset of the fan set. Independent of Enabled, MaintenanceMode and
+            // GrpcPort alike: none of those describe whether the cluster exists.
+            var clusterIds = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
+
            foreach (var row in rows)
            {
+                clusterIds.Add(row.ClusterId);
+
+                // The eligibility filter that used to be the query's WHERE clause. Everything below
+                // this line — receptionist contacts and gRPC targets — sees exactly the rows it saw
+                // before.
+                if (!row.Enabled || row.MaintenanceMode)
+                    continue;
+
+                if (row.GrpcPort is { } grpcPort)
+                    targets.Add(new MeshNodeTarget(
+                        row.NodeId, row.ClusterId, $"http://{row.Host}:{grpcPort}"));
+                else
+                    portless.Add(row.NodeId);
+
                var address = $"akka.tcp://{systemName}@{row.Host}:{row.AkkaPort}/system/receptionist";

                // Parse up front, per row, inside the loop's own guard. A single malformed row must
@@ -241,12 +664,24 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers

            var contactsByCluster = byCluster.ToDictionary(
                kv => kv.Key, kv => (IReadOnlyList<string>)kv.Value);
-            return new ContactsLoaded(contactsByCluster, malformed);
+            return new ContactsLoaded(
+                contactsByCluster, malformed, targets, portless, clusterIds.ToList());
        }, ct).PipeTo(self);
    }

    private void HandleContactsLoaded(ContactsLoaded msg)
    {
+        // Order is load-bearing and is pinned by test: the dispatch set is adopted BEFORE the gate is
+        // published, so anything that observes the gate can rely on the targets already being current.
+        if (_useGrpc)
+            AdoptGrpcTargets(msg);
+
+        // EVERY mode. See PreStart — the gate protects an endpoint mapped by port, not by mode.
+        PublishKnownClusters(msg);
+
+        if (!_useClusterClient)
+            return; // Dps or Grpc: no ClusterClient to rebuild.
+
        foreach (var bad in msg.Malformed)
        {
            _log.Warning(
@@ -305,6 +740,78 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
        }
    }

+    /// <summary>
+    ///     Adopts a contact load's gRPC dispatch set. <see cref="MeshTransportOptions.ModeGrpc"/> only
+    ///     — the other two transports have no use for it.
+    /// </summary>
+    /// <param name="msg">The load result.</param>
+    /// <remarks>
+    ///     The malformed-receptionist-address warnings are deliberately NOT logged here: nothing on
+    ///     this transport dials a receptionist, and reporting an unusable Akka address as a
+    ///     command-plane problem would send an operator after the wrong column. A host that is
+    ///     unusable as a gRPC endpoint surfaces instead as a named per-node dial failure on every
+    ///     dispatch, which is a truer signal.
+    /// </remarks>
+    private void AdoptGrpcTargets(ContactsLoaded msg)
+    {
+        // Rebuilt against this refresh's skip list, not merely added to. A node that gained a port
+        // and later lost it again must warn a second time — the operator's mental model is "the last
+        // refresh told me", not "you were told once, in a log that has since rotated away".
+        _warnedPortlessNodes.IntersectWith(msg.PortlessNodes);
+
+        foreach (var nodeId in msg.PortlessNodes)
+        {
+            // Warn-once per node id per continuous skip run: a persistently portless row is
+            // re-skipped every refresh, and that is the repetition TelemetryNodeSource suppresses.
+            if (_warnedPortlessNodes.Add(nodeId))
+            {
+                _log.Warning(
+                    "ClusterNode {NodeId} has no GrpcPort; it exposes no command endpoint and is "
+                    + "skipped in this dispatch set (further skips of this node are silent). Commands "
+                    + "will not reach it until the row carries a port", nodeId);
+            }
+        }
+
+        _targets = msg.Targets;
+
+        if (_targets.Count == 0)
+        {
+            _log.Warning(
+                "No enabled, non-maintenance ClusterNode row carries a GrpcPort in any cluster; there "
+                + "are no gRPC command targets and every central→node command will be dropped until a "
+                + "refresh finds one");
+        }
+    }
+
+    /// <summary>
+    ///     Hands the PSK provider this refresh's cluster inventory and drops its key cache. Runs in
+    ///     <b>every</b> transport mode.
+    /// </summary>
+    /// <param name="msg">The load result.</param>
+    /// <remarks>
+    ///     <para>
+    ///         <b>Not gated on <see cref="MeshTransportOptions.Mode"/>, on purpose.</b> The gate feeds
+    ///         <c>CentralCommandAuthInterceptor</c>, which is registered on every admin node, and the
+    ///         ack service it protects is mapped by listener port — neither consults the mode. Gating
+    ///         this on <c>Grpc</c> would mean a dark-switch rollback to <c>ClusterClient</c> or
+    ///         <c>Dps</c> left that endpoint serving with a permanently inactive gate: the timing
+    ///         oracle and the pre-authentication store/audit flood would both come back, silently,
+    ///         as a side effect of a rollback nobody would connect to authentication.
+    ///     </para>
+    ///     <para>
+    ///         <b><c>InvalidateAll</c>, not per-cluster <c>Invalidate</c>.</b> The requirement is that
+    ///         a rotated key takes effect within one refresh, and only a whole-cache drop delivers
+    ///         that: a key can rotate without the cluster's membership changing at all, so
+    ///         invalidating only departed clusters would serve the pre-rotation key forever. The cost
+    ///         is one store read per cluster per refresh — the cache holds a single entry per cluster.
+    ///     </para>
+    /// </remarks>
+    private void PublishKnownClusters(ContactsLoaded msg)
+    {
+        PskProvider?.SetKnownClusters(msg.ClusterIds);
+        PskProvider?.InvalidateAll();
+    }
+
    /// <summary>Stops a cluster's existing client and creates a replacement for its new contact set.</summary>
    /// <remarks>
    ///     <para>
@@ -371,7 +878,56 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
    ///     <c>Cluster</c>'s id. One <c>ClusterClient</c> is built per key.
    /// </param>
    /// <param name="Malformed">Rows that did not parse, described for the log.</param>
+    /// <param name="Targets">
+    ///     The gRPC dispatch set: one entry per enabled, non-maintenance row that carries a
+    ///     <c>GrpcPort</c>. Built from the same rows as <paramref name="ContactsByCluster"/>.
+    /// </param>
+    /// <param name="PortlessNodes">
+    ///     Node ids skipped because their row has no <c>GrpcPort</c>. Carried rather than logged in
+    ///     the load task so the warn-once bookkeeping stays on the actor's thread.
+    /// </param>
+    /// <param name="ClusterIds">
+    ///     Every application <c>Cluster</c> with <b>any</b> <c>ClusterNode</c> row — the PSK
+    ///     provider's known-cluster gate membership, and deliberately a superset of
+    ///     <paramref name="Targets"/>: a cluster whose nodes are all disabled or in maintenance can
+    ///     still send an ack, and refusing it would make maintenance mode an authentication switch.
+    /// </param>
    public sealed record ContactsLoaded(
        IReadOnlyDictionary<string, IReadOnlyList<string>> ContactsByCluster,
-        IReadOnlyList<string> Malformed);
+        IReadOnlyList<string> Malformed,
+        IReadOnlyList<MeshNodeTarget> Targets,
+        IReadOnlyList<string> PortlessNodes,
+        IReadOnlyList<string> ClusterIds);
+
+    /// <summary>One node the gRPC command plane can dial.</summary>
+    /// <param name="NodeId">The <c>ClusterNode</c> row's id — what a failure log names.</param>
+    /// <param name="ClusterId">
+    ///     The node's application <c>Cluster</c>. Selects the pre-shared key presented to it: a node
+    ///     dialled with another cluster's id is graded against the wrong key and refuses the command.
+    /// </param>
+    /// <param name="Endpoint">The h2c base address, <c>http://{Host}:{GrpcPort}</c>.</param>
+    public sealed record MeshNodeTarget(string NodeId, string ClusterId, string Endpoint);
+
+    /// <summary>
+    ///     The outcome of one node's dial, piped back to the actor <b>for logging only</b>. Nothing
+    ///     retries, counts or waits on these — a deployment's completion is judged by the acks nodes
+    ///     send back, never by whether central's outbound call succeeded.
+    /// </summary>
+    /// <param name="NodeId">The node dialled.</param>
+    /// <param name="CommandType">CLR type name of the command.</param>
+    /// <param name="CorrelationId">The envelope's transport correlation.</param>
+    /// <param name="FailureStatus">
+    ///     <see langword="null"/> on success; otherwise the gRPC status code's name, or
+    ///     <c>Unknown</c> for a non-gRPC fault.
+    /// </param>
+    /// <param name="FailureDetail">
+    ///     The gRPC status detail, or the exception's type name for a non-gRPC fault. Never a key,
+    ///     and never a raw exception message.
+    /// </param>
+    public sealed record CommandDispatched(
+        string NodeId,
+        string CommandType,
+        string CorrelationId,
+        string? FailureStatus,
+        string? FailureDetail);
 }
@@ -1,5 +1,7 @@
 using System.Collections.Concurrent;
 using System.Diagnostics.CodeAnalysis;
+using System.Security.Cryptography;
+using System.Text;
 using Microsoft.Extensions.Logging;
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.OtOpcUa.Cluster;
@@ -21,10 +23,21 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
 ///         proceed unauthenticated; a missing key must cost delivery, not authentication.
 ///     </para>
 ///     <para>
-///         The <c>Invalidate</c> hooks exist because central's cluster set is DB-dynamic. They are
-///         deliberately unconsumed at Phase 0 — Phase 1A wires them to the contact refresh, which is
-///         the point at which central learns a cluster's membership (and so possibly its key) has
-///         changed.
+///         The <c>Invalidate</c> hooks exist because central's cluster set is DB-dynamic. Phase 1A
+///         wires them to <c>CentralCommunicationActor</c>'s contact refresh, which is the point at
+///         which central learns a cluster's membership (and so possibly its key) has changed — so a
+///         rotated key is picked up within one <c>ContactRefreshSeconds</c> without a restart.
+///     </para>
+///     <para>
+///         <b>The known-cluster gate is a real security control, not an optimization</b> (Phase 1A
+///         security register item 1). <see cref="SetKnownClusters"/> publishes the cluster ids
+///         central just read from <c>ClusterNode</c>; <see cref="TryGetPsk"/> consults that set
+///         first and refuses anything outside it in O(1), touching neither the config map nor the
+///         secret store. Without it, an unauthenticated caller can pick an arbitrary
+///         <c>x-otopcua-cluster</c> value and make central perform a secret-store read plus an audit
+///         write per probe — and, because negatives are deliberately never cached, the cost
+///         difference between a known and an unknown cluster is measurable from the wire, which
+///         defeats the uniform denial text on the central interceptor.
 ///     </para>
 ///     <para>
 ///         <b><c>Invalidate</c> is not a revocation barrier.</b> It drops the cached entry; it does
@@ -43,6 +56,50 @@ public interface IMeshPskProvider
    /// <returns><c>true</c> when a usable key was found; otherwise <c>false</c>.</returns>
    bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk);

+    /// <summary>
+    ///     Publishes the set of application <c>Cluster</c> ids central currently knows about.
+    ///     <see cref="TryGetPsk"/> refuses anything outside it before consulting any source.
+    /// </summary>
+    /// <param name="clusterIds">
+    ///     The known cluster ids, matched case-insensitively. <b>An empty collection deactivates the
+    ///     gate</b> rather than refusing everything: central publishes this from its DB contact
+    ///     refresh, and "the DB told me nothing" must not become a fleet-wide authentication outage.
+    ///     Never having called this leaves the gate inactive for the same reason — the central
+    ///     interceptor is live from the first request, before the first refresh has run.
+    /// </param>
+    /// <remarks>
+    ///     Implementations must take a copy: the caller owns its collection and may reuse it. The
+    ///     swap must also be atomic with respect to concurrent <see cref="TryGetPsk"/> callers,
+    ///     which run on arbitrary gRPC threads.
+    /// </remarks>
+    void SetKnownClusters(IReadOnlyCollection<string> clusterIds);
+
+    /// <summary>
+    ///     Verifies a presented bearer token against one cluster's key, in constant time with
+    ///     respect to the key's content.
+    /// </summary>
+    /// <param name="clusterId">The application <c>Cluster</c> the caller claims to be.</param>
+    /// <param name="presented">The bearer token the caller supplied.</param>
+    /// <returns><c>true</c> only when a key exists for that cluster and the token equals it.</returns>
+    /// <remarks>
+    ///     <para>
+    ///         <b>This is the only comparison on the verify path</b> (Phase 1A security register
+    ///         item 2). Handing the raw key back so a caller can compare it is how <c>==</c>
+    ///         eventually appears in an interceptor and compiles; keeping the comparison behind this
+    ///         member means the verify side never holds key material at all. <see cref="TryGetPsk"/>
+    ///         survives for the <i>dial</i> side, which has to send the key rather than compare it.
+    ///     </para>
+    ///     <para>
+    ///         Declared without a default implementation on purpose: a security member a new
+    ///         implementer can inherit silently is a member a new implementer never considers.
+    ///     </para>
+    ///     <para>
+    ///         Length differences remain observable through timing; that is inherent to comparing
+    ///         byte arrays of different lengths and is not sensitive. Content is not.
+    ///     </para>
+    /// </remarks>
+    bool Matches(string clusterId, string presented);
+
    /// <summary>Drops any cached key for one cluster, so the next lookup re-reads the store.</summary>
    /// <param name="clusterId">The application <c>Cluster</c> id, matched case-insensitively.</param>
    void Invalidate(string clusterId);
@@ -87,6 +144,18 @@ public sealed class MeshPskProvider : IMeshPskProvider
    private readonly ConcurrentDictionary<string, Lazy<string?>> _cache =
        new(StringComparer.OrdinalIgnoreCase);

+    /// <summary>
+    ///     The known-cluster gate's membership set, or <see langword="null"/> when the gate is
+    ///     inactive (never published, or published empty — see <see cref="SetKnownClusters"/>).
+    /// </summary>
+    /// <remarks>
+    ///     Swapped wholesale, never mutated in place, so a reader either sees the previous set or the
+    ///     next one and never a half-built one. <c>volatile</c> is what makes the publish visible to
+    ///     the gRPC threads doing the reading; no lock is needed because there is exactly one writer
+    ///     (the contact-refresh handler, on the actor's own thread).
+    /// </remarks>
+    private volatile IReadOnlySet<string>? _knownClusters;
+
    /// <summary>Creates the provider.</summary>
    /// <param name="options">Mesh transport options; <c>ClusterPsks</c> is the override map.</param>
    /// <param name="secretResolver">The shared secret store read seam.</param>
@@ -118,6 +187,19 @@ public sealed class MeshPskProvider : IMeshPskProvider
            return false;
        }

+        // The known-cluster gate, ahead of BOTH sources. Deliberately silent: the caller that
+        // rejected the call already logs the denial, and a second line here would be a log flood
+        // driven by exactly the unauthenticated traffic this gate exists to make cheap.
+        //
+        // Ahead of the config map too, not just the store. That has a visible consequence — a
+        // MeshTransport:ClusterPsks entry for a cluster with no enabled, non-maintenance ClusterNode
+        // row stops resolving once central publishes its set — and it is the right one: central
+        // would neither dial such a cluster nor accept an ack from it, so serving a key for it only
+        // widens what an unauthenticated caller can probe.
+        var known = _knownClusters;
+        if (known is not null && !known.Contains(clusterId))
+            return false;
+
        // Config map first, on every call. Blank counts as absent: an empty
        // MeshTransport__ClusterPsks__<id> is an unset environment variable, not an operator
        // deciding this cluster has an empty key.
@@ -168,6 +250,34 @@ public sealed class MeshPskProvider : IMeshPskProvider
        return true;
    }

+    /// <inheritdoc />
+    public bool Matches(string clusterId, string presented)
+    {
+        // Blank is refused on its own terms rather than by relying on the fact that a blank key
+        // never survives resolution: this is an authentication gate, and "both sides happened to be
+        // empty" must never be a pass.
+        if (string.IsNullOrWhiteSpace(presented))
+            return false;
+
+        if (!TryGetPsk(clusterId, out var expected))
+            return false;
+
+        return CryptographicOperations.FixedTimeEquals(
+            Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
+    }
+
+    /// <inheritdoc />
+    public void SetKnownClusters(IReadOnlyCollection<string> clusterIds)
+    {
+        ArgumentNullException.ThrowIfNull(clusterIds);
+
+        // Copied, so a caller reusing its list cannot widen the gate after the fact; empty collapses
+        // to null so the gate has exactly one "inactive" representation to test for.
+        _knownClusters = clusterIds.Count == 0
+            ? null
+            : clusterIds.ToHashSet(StringComparer.OrdinalIgnoreCase);
+    }
+
    /// <inheritdoc />
    public void Invalidate(string clusterId)
    {
@@ -0,0 +1,200 @@
+using System.Collections.Concurrent;
+using Grpc.Core;
+using Grpc.Net.Client;
+using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.OtOpcUa.Cluster;
+using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
+
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
+
+/// <summary>
+///     Places one central→node command call on one node's <c>NodeCommandService</c>
+///     (ClusterClient→gRPC migration Phase 1A). The seam <c>CentralCommunicationActor</c> fans
+///     across its target set, kept as an interface so the actor's tests never open a socket.
+/// </summary>
+/// <remarks>
+///     One dial, one node, no retry and no fan-out of its own — the fan and its per-node failure
+///     logging belong to the actor, which is the component that knows what the recipient set is.
+/// </remarks>
+public interface IMeshCommandDialer
+{
+    /// <summary>Delivers one command envelope to one node.</summary>
+    /// <param name="nodeEndpoint">The node's gRPC base address, e.g. <c>http://site-a-1:4056</c>.</param>
+    /// <param name="envelope">The wire envelope, already mapped from the domain record.</param>
+    /// <param name="clusterId">
+    ///     The application <c>Cluster</c> the <i>target node</i> belongs to. Selects which pre-shared
+    ///     key is presented, and is echoed to the node in <c>x-otopcua-cluster</c>.
+    /// </param>
+    /// <param name="ct">Cancels the dial; a per-call deadline is applied on top of it.</param>
+    /// <returns>A task completing when the node has accepted the command.</returns>
+    /// <exception cref="RpcException">
+    ///     The node rejected the call, was unreachable, or the deadline expired — and also when no
+    ///     key could be established for <paramref name="clusterId"/>, which is refused locally with
+    ///     <see cref="StatusCode.Unauthenticated"/> rather than dialled unauthenticated.
+    /// </exception>
+    Task DispatchAsync(string nodeEndpoint, CommandEnvelope envelope, string clusterId, CancellationToken ct);
+}
+
+/// <inheritdoc />
+/// <remarks>
+///     <para>
+///         <b>Fail-closed on a missing key, always.</b> There is no arm on which a command leaves
+///         central without the target cluster's bearer. A node's <c>MeshCommandAuthInterceptor</c>
+///         would reject an unauthenticated call anyway, but "the far side would have refused it" is
+///         not a reason to make the call — an unauthenticated dial is a credential-free command
+///         attempt against a node's control surface, and it should not exist even as a rejected one.
+///         The refusal is reported as an <see cref="RpcException"/> so the actor's per-node failure
+///         logging covers it on the same channel as a remote rejection; the operator response is
+///         identical either way (seed the cluster's key), and a second channel would need a second
+///         handler that could drift.
+///     </para>
+///     <para>
+///         <b>One channel per endpoint, cached.</b> Same shape as
+///         <c>GrpcDeploymentArtifactFetcher</c>: a <see cref="GrpcChannel"/> owns a connection pool
+///         and creating one per call leaks sockets. The cache is keyed by the endpoint string
+///         ordinally — endpoints come from <c>ClusterNode.Host</c> + <c>GrpcPort</c>, so two spellings
+///         of one host would merely cost a second channel, never a wrong destination.
+///     </para>
+///     <para>
+///         <b>The deadline is per call, not per fan.</b> A dispatch dials every node in the fleet;
+///         an unbounded call to one unreachable node would hold a task and a socket indefinitely
+///         while the deploy path's own 2-minute apply deadline ticked down.
+///         <see cref="MeshTransportOptions.DispatchTimeoutSeconds"/> bounds it, linked to the
+///         caller's token so actor shutdown still cancels everything in flight.
+///     </para>
+///     <para>
+///         Registered as a singleton and disposed by the container. A dial started after disposal
+///         throws, so no new work repopulates the cache — but a dial already past that check can
+///         still add a channel after the cache was cleared. That leak is bounded to the dials in
+///         flight at shutdown, which is the only time disposal happens, so it is accepted rather
+///         than serialized against.
+///     </para>
+/// </remarks>
+public sealed class MeshGrpcCommandDialer : IMeshCommandDialer, IDisposable
+{
+    /// <summary>
+    ///     Substituted for a non-positive <see cref="MeshTransportOptions.DispatchTimeoutSeconds"/>.
+    ///     Deliberately the option's own default rather than a second value to reconcile.
+    /// </summary>
+    private const int FallbackDispatchTimeoutSeconds = 10;
+
+    /// <summary>
+    ///     Header naming the cluster the presented bearer belongs to. Central's own inbound gate
+    ///     reads the same header on the ack path; sending it outbound keeps one convention and tells
+    ///     a node which key central graded itself against.
+    /// </summary>
+    private const string ClusterHeader = "x-otopcua-cluster";
+
+    private readonly IMeshPskProvider _pskProvider;
+    private readonly ILogger<MeshGrpcCommandDialer> _log;
+    private readonly int _dispatchTimeoutSeconds;
+    private readonly Func<string, NodeCommandService.NodeCommandServiceClient> _clientFactory;
+    private readonly ConcurrentDictionary<string, GrpcChannel> _channels = new(StringComparer.Ordinal);
+
+    private volatile bool _disposed;
+
+    /// <summary>Initializes a new instance of the <see cref="MeshGrpcCommandDialer"/> class.</summary>
+    /// <param name="pskProvider">Supplies the bearer for the target node's cluster.</param>
+    /// <param name="options">Mesh transport options; supplies the per-call deadline.</param>
+    /// <param name="log">Logger for the construction-time deadline check.</param>
+    /// <param name="clientFactory">
+    ///     Seam for tests: builds the generated client for an endpoint. Defaults to a real h2c
+    ///     <see cref="GrpcChannel"/>, one cached per endpoint.
+    /// </param>
+    /// <remarks>
+    ///     A non-positive <c>DispatchTimeoutSeconds</c> is clamped, not honoured. Zero would mean an
+    ///     unbounded call — the exact failure the deadline exists to prevent — and the validator that
+    ///     rejects it is scoped to driver-role nodes, so the one deployment shape that reads this
+    ///     value unvalidated is a pure-admin central. Clamping loudly is better than a fleet-wide fan
+    ///     that can never time out.
+    /// </remarks>
+    public MeshGrpcCommandDialer(
+        IMeshPskProvider pskProvider,
+        IOptions<MeshTransportOptions> options,
+        ILogger<MeshGrpcCommandDialer> log,
+        Func<string, NodeCommandService.NodeCommandServiceClient>? clientFactory = null)
+    {
+        ArgumentNullException.ThrowIfNull(pskProvider);
+        ArgumentNullException.ThrowIfNull(options);
+        ArgumentNullException.ThrowIfNull(log);
+
+        _pskProvider = pskProvider;
+        _log = log;
+
+        var configured = options.Value.DispatchTimeoutSeconds;
+        if (configured <= 0)
+        {
+            _log.LogWarning(
+                "MeshTransport:DispatchTimeoutSeconds is {Configured}, which would leave every mesh "
+                + "command dial unbounded; using {Fallback}s instead. Set a positive value — the "
+                + "boot validator for this key covers driver-role nodes only, so a pure-admin central "
+                + "reaches this line rather than failing at startup.",
+                configured, FallbackDispatchTimeoutSeconds);
+        }
+
+        _dispatchTimeoutSeconds = configured > 0 ? configured : FallbackDispatchTimeoutSeconds;
+        _clientFactory = clientFactory ?? DefaultClientFactory;
+    }
+
+    /// <summary>
+    ///     Gets the number of endpoints holding a cached <see cref="GrpcChannel"/>. Exposed so the
+    ///     cache — the whole defence against a socket per call across a fleet-wide fan — is
+    ///     assertable rather than merely intended.
+    /// </summary>
+    public int CachedChannelCount => _channels.Count;
+
+    /// <inheritdoc />
+    public async Task DispatchAsync(
+        string nodeEndpoint, CommandEnvelope envelope, string clusterId, CancellationToken ct)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(nodeEndpoint);
+        ArgumentNullException.ThrowIfNull(envelope);
+        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
+        ObjectDisposedException.ThrowIf(_disposed, this);
+
+        if (!_pskProvider.TryGetPsk(clusterId, out var psk))
+        {
+            // The cluster id is operator-authored configuration, not a secret, so naming it is what
+            // makes this actionable. The key never appears here or anywhere else in this class.
+            // Kept short on purpose: the actor truncates a status detail at 200 characters because a
+            // REMOTE node authors it, and this locally-authored one has to survive that cap with its
+            // remediation intact. It measures ~160 characters with a six-character cluster id, so an
+            // id would have to exceed ~40 characters to lose the trailing rig hint.
+            throw new RpcException(new Status(
+                StatusCode.Unauthenticated,
+                $"No mesh pre-shared key for cluster '{clusterId}'; dial refused locally. Seed secret "
+                + "OT-MESH-PSK-<clusterId>, or set MeshTransport:ClusterPsks:<clusterId> on a rig."));
+        }
+
+        var headers = new Metadata
+        {
+            { "authorization", $"Bearer {psk}" },
+            { ClusterHeader, clusterId },
+        };
+
+        using var deadlineCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
+        deadlineCts.CancelAfter(TimeSpan.FromSeconds(_dispatchTimeoutSeconds));
+
+        var client = _clientFactory(nodeEndpoint);
+        using var call = client.ExecuteAsync(envelope, headers, cancellationToken: deadlineCts.Token);
+        await call.ResponseAsync.ConfigureAwait(false);
+    }
+
+    /// <inheritdoc />
+    public void Dispose()
+    {
+        _disposed = true;
+
+        foreach (var channel in _channels.Values)
+            channel.Dispose();
+        _channels.Clear();
+    }
+
+    private NodeCommandService.NodeCommandServiceClient DefaultClientFactory(string endpoint)
+    {
+        // h2c: GrpcChannel over an http:// address negotiates prior-knowledge HTTP/2.
+        var channel = _channels.GetOrAdd(endpoint, GrpcChannel.ForAddress);
+        return new NodeCommandService.NodeCommandServiceClient(channel);
+    }
+}
@@ -66,6 +66,21 @@ public static class ServiceCollectionExtensions
            var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
            var options = resolver.GetService<IOptions<MeshTransportOptions>>().Value;

+            // Captured, NOT resolved. Resolving the mesh dialer or PSK provider here made every
+            // central node fail to boot in every transport mode: their graph is dialer →
+            // IMeshPskProvider → ISecretResolver → AddZbSecrets → IAuditWriter → ActorAuditWriter,
+            // whose constructor calls registry.Get<AuditWriterActorKey>() — and WithSingleton
+            // <AuditWriterActorKey> is registered further down THIS builder, so the entry does not
+            // exist while this starter runs. The actor invokes these in PreStart instead, by which
+            // point the host is built. Same shape, same reason, as the coordinator Func below.
+            //
+            // Through IServiceProvider rather than the Akka resolver, deliberately: these two are
+            // OPTIONAL (Grpc mode only, and admin-role only within that), and GetService's
+            // null-for-absent contract is the thing being relied on. The Akka resolver's
+            // GetService<T> makes no such promise, and this same block runs in test harnesses whose
+            // service collections register neither.
+            var serviceProvider = resolver.GetService<IServiceProvider>();
+
            var actor = system.ActorOf(
                CentralCommunicationActor.Props(
                    dbFactory,
@@ -75,7 +90,9 @@ public static class ServiceCollectionExtensions
                    // which Akka.Hosting materialises the singleton proxy relative to this block.
                    () => registry.TryGet<ConfigPublishCoordinatorKey>(out var coordinator)
                        ? coordinator
-                        : null),
+                        : null,
+                    () => serviceProvider.GetService<IMeshCommandDialer>(),
+                    () => serviceProvider.GetService<IMeshPskProvider>()),
                MeshPaths.CentralCommunicationName);

            // Registered unconditionally, in both transport modes. Under Dps nothing dials it, and
@@ -1,5 +1,3 @@
-using System.Security.Cryptography;
-using System.Text;
 using Grpc.Core;
 using Grpc.Core.Interceptors;
 using Microsoft.Extensions.Logging;
@@ -38,25 +36,31 @@ namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
 ///         key this design exists to avoid.
 ///     </para>
 ///     <para>
-///         Comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8 bytes, so a
-///         wrong key cannot be recovered byte-by-byte from response timing. Length differences are
-///         unavoidably observable and are not sensitive.
+///         <b>This interceptor never handles a key.</b> Verification is
+///         <see cref="IMeshPskProvider.Matches"/>, which owns the constant-time comparison, so a
+///         wrong key cannot be recovered byte-by-byte from response timing and no future edit here
+///         can replace that comparison with <c>==</c> and still compile (Phase 1A security register
+///         item 2). Length differences are unavoidably observable and are not sensitive.
 ///     </para>
 ///     <para>
-///         <b>All three denials return the same status text</b>, unlike the sibling interceptors,
-///         which distinguish "no key configured" from "authentication failed". Here that distinction
-///         would be a <i>content</i> oracle: the caller supplies the cluster id, so a reply that said
-///         "no key for that cluster" would let an unauthenticated caller read central's cluster set
-///         straight off the wire. The operator-facing detail goes to the log instead.
+///         <b>All denials return the same status text</b>, unlike the sibling interceptors, which
+///         distinguish "no key configured" from "authentication failed". Here that distinction would
+///         be a <i>content</i> oracle: the caller supplies the cluster id, so a reply that said "no
+///         key for that cluster" would let an unauthenticated caller read central's cluster set
+///         straight off the wire. The operator-facing detail goes to the log instead — and for the
+///         same reason the log no longer separates "no key for that cluster" from "wrong key": the
+///         provider answers both as one <c>false</c>, and collapsing them costs an operator a hint
+///         (restated in the one denial message) rather than costing a diagnosis.
 ///     </para>
 ///     <para>
-///         <b>That is not yet full enumeration resistance, and should not be read as such.</b> A
-///         timing oracle survives it: an unknown cluster misses the provider's cache and pays a store
-///         read on every probe, while a known one is answered from memory after the first. Closing
-///         that needs the lookup to be refused before it reaches the store for any cluster central
-///         does not know — which is the Phase 1A known-cluster-set gate, where central learns its
-///         cluster set from the DB. Until then, uniform text buys content-level indistinguishability
-///         only.
+///         <b>The timing oracle that survived Phase 0 is closed by the known-cluster gate.</b> An
+///         unknown cluster used to miss the provider's cache and pay a secret-store read on every
+///         probe, while a known one was answered from memory after the first — a stable, remotely
+///         measurable distinguisher, and a pre-auth store/audit flood besides.
+///         <c>CentralCommunicationActor</c> now publishes central's cluster set to the provider on
+///         every contact refresh, and a cluster outside it is refused in O(1) with no store touch,
+///         so the two cases cost the same. What remains is a known cluster's first-ever lookup,
+///         which is one read per cluster per process and not attacker-triggerable.
 ///     </para>
 ///     <para>
 ///         <b>Exactly one public constructor.</b> <c>Grpc.AspNetCore</c> silently stops invoking an
@@ -151,13 +155,15 @@ public sealed class CentralCommandAuthInterceptor : Interceptor
                "Central command authentication failed."));
        }

-        if (!_pskProvider.TryGetPsk(clusterId, out var expected))
+        // Checked here rather than deferred to Matches so the log can still separate "the caller
+        // presented nothing" — an operator misconfiguration, and the one case the provider cannot
+        // see — from "the caller presented something that did not verify".
+        var presented = GrpcRequestHeaders.BearerToken(context.RequestHeaders);
+        if (presented is null)
        {
            _logger.LogWarning(
-                "Rejected a central command call to {Method} claiming cluster {ClusterId}: no "
-                + "pre-shared key could be established for it. Seed the secret "
-                + "OT-MESH-PSK-<clusterId> on this node, or set "
-                + "MeshTransport:ClusterPsks:<clusterId> on a rig.",
+                "Rejected a central command call to {Method} claiming cluster {ClusterId}: no bearer "
+                + "token presented.",
                context.Method,
                clusterId);
            throw new RpcException(new Status(
@@ -165,21 +171,20 @@ public sealed class CentralCommandAuthInterceptor : Interceptor
                "Central command authentication failed."));
        }

-        var presented = GrpcRequestHeaders.BearerToken(context.RequestHeaders);
-        if (presented is null || !FixedTimeEquals(presented, expected))
+        if (!_pskProvider.Matches(clusterId, presented))
        {
            _logger.LogWarning(
-                "Rejected a central command call to {Method} claiming cluster {ClusterId}: {Reason}.",
+                "Rejected a central command call to {Method} claiming cluster {ClusterId}: the "
+                + "presented bearer token did not match that cluster's pre-shared key, no key could "
+                + "be established for it, or it is not a cluster this node knows. If the cluster is "
+                + "legitimate, seed the secret OT-MESH-PSK-<clusterId> on this node, or set "
+                + "MeshTransport:ClusterPsks:<clusterId> on a rig, and check it has an enabled, "
+                + "non-maintenance ClusterNode row.",
                context.Method,
-                clusterId,
-                presented is null ? "no bearer token presented" : "bearer token did not match");
+                clusterId);
            throw new RpcException(new Status(
                StatusCode.PermissionDenied,
                "Central command authentication failed."));
        }
    }
-
-    private static bool FixedTimeEquals(string presented, string expected)
-        => CryptographicOperations.FixedTimeEquals(
-            Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
 }
@@ -0,0 +1,110 @@
+using Akka.Actor;
+using Akka.Hosting;
+using Grpc.Core;
+using Microsoft.Extensions.Logging;
+using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
+using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
+using ZB.MOM.WW.OtOpcUa.Runtime;
+// The generated service container is itself named NodeCommandService; alias its nested server base so
+// this impl can keep the natural name without colliding with the generated type.
+using GeneratedServiceBase =
+    ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1.NodeCommandService.NodeCommandServiceBase;
+// Akka.Actor (imported for IActorRef.Tell) declares a Status of its own; only the gRPC one is used here.
+using Status = Grpc.Core.Status;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Grpc;
+
+/// <summary>
+///     Node-side command ingress (ClusterClient→gRPC migration Phase 1). Central dials a driver node
+///     and calls <see cref="Execute"/>; the envelope is decoded back into the same CLR record the
+///     ClusterClient receptionist used to deliver and handed to this node's
+///     <c>NodeCommunicationActor</c>, which re-emits it on the node's <c>EventStream</c> exactly as
+///     before. Nothing downstream of that actor can tell which transport carried the command.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>One-way, verbatim from the transport it replaces.</b> <c>ExecuteAck</c> means "decoded and
+///         handed to the comm actor" — not applied, and not acknowledged by any EventStream subscriber.
+///         The <c>Tell</c> is deliberately not an <c>Ask</c>: no migrated command has a reply, and
+///         waiting for one would invent a semantic central does not have.
+///     </para>
+///     <para>
+///         <b>Decode runs BEFORE the registry lookup, and the two failures are graded differently.</b>
+///         An undecodable envelope is <see cref="StatusCode.InvalidArgument"/> — permanent, and it will
+///         never decode on a retry. A call arriving before <c>AddOtOpcUaRuntime</c> has spawned and
+///         registered the comm actor is <see cref="StatusCode.Unavailable"/> — the house readiness
+///         convention, transient, and correct to retry. Checking readiness first would report a
+///         permanently malformed command as a readiness problem and invite an unbounded retry loop.
+///     </para>
+///     <para>
+///         <b>Every log line carries the envelope's own <c>correlation_id</c>, read as a raw string.</b>
+///         Three of the four commands also carry a domain correlation, but <c>AlarmCommand</c> has no
+///         CLR member for one, so the envelope field is the only id every command shares — and it is
+///         still readable when the inner fields are what failed to parse, which is precisely when an
+///         operator needs it.
+///     </para>
+///     <para>
+///         Inbound auth is <c>MeshCommandAuthInterceptor</c>, scoped to this service's method prefix and
+///         registered at the <c>AddGrpc</c> level; it fail-closes with no <c>MeshTransport:GrpcPsk</c>
+///         configured. Mapping this service is what makes that gate live.
+///     </para>
+/// </remarks>
+public sealed class NodeCommandGrpcService : GeneratedServiceBase
+{
+    private readonly ActorRegistry _registry;
+    private readonly ILogger<NodeCommandGrpcService> _log;
+
+    /// <summary>Initializes a new instance of the <see cref="NodeCommandGrpcService"/> class.</summary>
+    /// <param name="registry">Actor registry holding this node's <c>NodeCommunicationActor</c>.</param>
+    /// <param name="log">Logger.</param>
+    public NodeCommandGrpcService(ActorRegistry registry, ILogger<NodeCommandGrpcService> log)
+    {
+        _registry = registry;
+        _log = log;
+    }
+
+    /// <inheritdoc />
+    public override Task<ExecuteAck> Execute(CommandEnvelope request, ServerCallContext context)
+    {
+        ArgumentNullException.ThrowIfNull(request);
+        ArgumentNullException.ThrowIfNull(context);
+
+        object command;
+        try
+        {
+            command = MeshTransportDtoMapper.FromEnvelope(request);
+        }
+        catch (Exception ex) when (ex is InvalidOperationException or FormatException or ArgumentException)
+        {
+            // The envelope carries no credential material, so logging its correlation id is safe and is
+            // the only handle an operator has on a call whose inner fields did not parse.
+            _log.LogWarning(
+                ex,
+                "Rejected a mesh command (correlationId={CorrelationId}, oneof={CommandCase}): the "
+                + "envelope could not be decoded. The command was NOT delivered.",
+                request.CorrelationId, request.CommandCase);
+            throw new RpcException(new Status(
+                StatusCode.InvalidArgument, "Command envelope could not be decoded."));
+        }
+
+        if (!_registry.TryGet<NodeCommunicationActorKey>(out var comm))
+        {
+            _log.LogWarning(
+                "Rejected a mesh command (correlationId={CorrelationId}, type={CommandType}): this node's "
+                + "communication actor is not registered yet. The node is still starting; the command was "
+                + "NOT delivered and central may retry.",
+                request.CorrelationId, command.GetType().Name);
+            throw new RpcException(new Status(
+                StatusCode.Unavailable, "Node is still starting: the command endpoint is not ready."));
+        }
+
+        comm.Tell(command);
+
+        _log.LogDebug(
+            "Mesh: accepted {CommandType} (correlationId={CorrelationId}) over gRPC; handed to the node "
+            + "communication actor.",
+            command.GetType().Name, request.CorrelationId);
+
+        return Task.FromResult(new ExecuteAck());
+    }
+}
@@ -534,6 +534,14 @@ if (hasAdmin)
    // Depends on ISecretResolver, which AddOtOpcUaSecrets registers unconditionally below (DI
    // registration order is irrelevant; resolution order is not).
    builder.Services.AddSingleton<IMeshPskProvider, MeshPskProvider>();
+    // Central's outbound command dial (ClusterClient→gRPC Phase 1A). Admin-only for the same reason
+    // as the provider above — a node never dials another node — and singleton because it owns a
+    // GrpcChannel per node endpoint; per-call channels would leak sockets across a fleet-wide fan.
+    // Registered in every mode: CentralCommunicationActor only reaches it under Mode=Grpc, and an
+    // idle registration costs nothing, whereas gating it here would mean flipping the mode on a
+    // running fleet needs a central restart before any command lands — the same reasoning that
+    // registers the ClusterClient receptionist unconditionally.
+    builder.Services.AddSingleton<IMeshCommandDialer, MeshGrpcCommandDialer>();
 }

 // Registered unconditionally: driver-role nodes resolve Layer-B DriverConfig secrets and have no auth/DP/AdminUI.
@@ -740,9 +748,18 @@ if (hasAdmin && configServeGrpcPort > 0)

 // Driver-node live-telemetry stream (per-cluster mesh Phase 5). Central dials in; gated by
 // TelemetryStreamAuthInterceptor (fail-closed on empty Telemetry:ApiKey).
+//
+// The node-side command endpoint (ClusterClient→gRPC migration Phase 1) rides the SAME driver-role
+// h2c listener rather than claiming a fourth port: central already dials this node here, both
+// directions are per-cluster-keyed, and a separate port would be one more thing to open on every
+// site firewall for no isolation gain. Mapping it is what makes MeshCommandAuthInterceptor live —
+// it is registered above and is a pass-through until a call matches this service's method prefix.
+// It fail-closes with no MeshTransport:GrpcPsk, so an operator who maps the service without setting
+// a key gets a closed endpoint, not an open one.
 if (hasDriver && telemetryListenPort > 0)
 {
    app.MapGrpcService<TelemetryStreamGrpcService>();
+    app.MapGrpcService<NodeCommandGrpcService>();
 }

 app.MapOtOpcUaHealth();
@@ -1,4 +1,5 @@
 using System.Collections.Immutable;
+using System.Diagnostics.CodeAnalysis;
 using Akka.Actor;
 using Akka.Cluster.Tools.Client;
 using Akka.Cluster.Tools.PublishSubscribe;
@@ -9,6 +10,8 @@ using Xunit;
 using ZB.MOM.WW.OtOpcUa.Cluster;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
+using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
+using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
 using ZB.MOM.WW.OtOpcUa.Commons.Types;
 using ZB.MOM.WW.OtOpcUa.Configuration;
 using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
@@ -56,6 +59,13 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
        ContactRefreshSeconds = refreshSeconds,
    };

+    private static MeshTransportOptions GrpcMode(int refreshSeconds = 60) => new()
+    {
+        Mode = MeshTransportOptions.ModeGrpc,
+        ContactRefreshSeconds = refreshSeconds,
+        DispatchTimeoutSeconds = 10,
+    };
+
    private static void SeedCluster(
        IDbContextFactory<OtOpcUaConfigDbContext> factory,
        string clusterId,
@@ -92,6 +102,46 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
        db.SaveChanges();
    }

+    /// <summary>
+    ///     Seeds a cluster whose nodes carry a <c>GrpcPort</c> — the gRPC command plane's shape.
+    ///     Kept separate from <see cref="SeedCluster"/> so the ClusterClient facts, which have no
+    ///     opinion about the port, stay exactly as they were.
+    /// </summary>
+    private static void SeedGrpcCluster(
+        IDbContextFactory<OtOpcUaConfigDbContext> factory,
+        string clusterId,
+        params (string NodeId, string Host, int? GrpcPort, bool Enabled, bool Maintenance)[] nodes)
+    {
+        using var db = factory.CreateDbContext();
+        db.ServerClusters.Add(new ServerCluster
+        {
+            ClusterId = clusterId,
+            Name = clusterId,
+            Enterprise = "ZB",
+            Site = "Site",
+            RedundancyMode = RedundancyMode.None,
+            NodeCount = 1,
+            CreatedBy = "test",
+        });
+        foreach (var n in nodes)
+        {
+            db.ClusterNodes.Add(new ClusterNode
+            {
+                NodeId = n.NodeId,
+                ClusterId = clusterId,
+                Host = n.Host,
+                AkkaPort = 4053,
+                GrpcPort = n.GrpcPort,
+                ApplicationUri = $"urn:{n.NodeId}",
+                Enabled = n.Enabled,
+                MaintenanceMode = n.Maintenance,
+                CreatedBy = "test",
+            });
+        }
+
+        db.SaveChanges();
+    }
+
    private IActorRef Spawn(
        IDbContextFactory<OtOpcUaConfigDbContext> factory,
        MeshTransportOptions options,
@@ -306,4 +356,628 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
        actor.Tell(new MeshCommand(DeploymentsTopic, "anything"));
        ExpectNoMsg(TimeSpan.FromMilliseconds(200));
    }
+
+    // ------------------------------------------------------------------ Mode = Grpc
+    //
+    // The third transport. The fan-out must reach EXACTLY the recipient set SendToAll reaches today
+    // — every enabled, non-maintenance node in every cluster — because the node side still has no
+    // ClusterId or node filter; scoping happens later, inside the artifact. A narrower fan would
+    // leave nodes silently on their old configuration while the deployment sealed green.
+
+    [Fact]
+    public void Grpc_mode_fans_one_command_to_every_enabled_node_across_all_clusters()
+    {
+        // THE assertion of this task, and the direct counterpart of
+        // A_command_fans_out_a_send_to_all_to_every_cluster_client.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(
+            factory,
+            "C1",
+            ("host-a:4053", "host-a", 4056, true, false),
+            ("host-b:4053", "host-b", 4056, true, false));
+        SeedGrpcCluster(factory, "C2", ("host-c:4053", "host-c", 4056, true, false));
+        var rig = SpawnGrpcRig(factory);
+        AwaitContactsLoaded(rig);
+
+        rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
+
+        AwaitCondition(() => rig.Dialer.Calls.Count == 3, TimeSpan.FromSeconds(5));
+        rig.Dialer.Calls.Select(c => c.Endpoint).ShouldBe(
+            new[] { "http://host-a:4056", "http://host-b:4056", "http://host-c:4056" }, ignoreOrder: true);
+
+        // Each node is dialled with ITS OWN cluster's id — that is what selects the pre-shared key,
+        // so getting it wrong turns every command into a PermissionDenied at the node.
+        rig.Dialer.Calls.Single(c => c.Endpoint.Contains("host-c")).ClusterId.ShouldBe("C2");
+        rig.Dialer.Calls.Where(c => !c.Endpoint.Contains("host-c")).ShouldAllBe(c => c.ClusterId == "C1");
+    }
+
+    [Fact]
+    public void Grpc_mode_creates_no_ClusterClient_and_publishes_nothing_on_pubsub()
+    {
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
+        var clients = new RecordingClientFactory(() => CreateTestProbe());
+        var mediatorProbe = CreateTestProbe();
+        DistributedPubSub.Get(Sys).Mediator.Tell(
+            new Subscribe(DeploymentsTopic, mediatorProbe.Ref), mediatorProbe.Ref);
+        mediatorProbe.ExpectMsg<SubscribeAck>(TimeSpan.FromSeconds(5));
+
+        var rig = SpawnGrpcRig(factory, clientFactory: clients);
+        AwaitContactsLoaded(rig);
+
+        rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
+
+        AwaitCondition(() => rig.Dialer.Calls.Count == 1, TimeSpan.FromSeconds(5));
+        clients.Calls.ShouldBeEmpty();
+        mediatorProbe.ExpectNoMsg(TimeSpan.FromMilliseconds(300));
+    }
+
+    [Fact]
+    public void Grpc_mode_does_not_dial_disabled_or_maintenance_nodes()
+    {
+        // Same recipient filter as the ClusterClient contact load, asserted independently so the two
+        // transports cannot drift on who counts as a node.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(
+            factory,
+            "C1",
+            ("host-a:4053", "host-a", 4056, true, false),
+            ("host-b:4053", "host-b", 4056, false, false),
+            ("host-c:4053", "host-c", 4056, true, true));
+        var rig = SpawnGrpcRig(factory);
+        AwaitContactsLoaded(rig);
+
+        rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
+
+        AwaitCondition(() => rig.Dialer.Calls.Count == 1, TimeSpan.FromSeconds(5));
+        rig.Dialer.Calls[0].Endpoint.ShouldContain("host-a");
+
+        // And no late arrival from the two excluded rows.
+        ExpectNoMsg(TimeSpan.FromMilliseconds(300));
+        rig.Dialer.Calls.Count.ShouldBe(1);
+    }
+
+    [Fact]
+    public void A_node_with_no_GrpcPort_is_skipped_with_a_warning_and_the_others_are_still_dialled()
+    {
+        // Mirrors TelemetryNodeSource: a row with no port exposes no endpoint yet. It must cost that
+        // node its commands, never the whole fan.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(
+            factory,
+            "C1",
+            ("host-a:4053", "host-a", null, true, false),
+            ("host-b:4053", "host-b", 4056, true, false));
+
+        GrpcRig? rig = null;
+        EventFilter.Warning(contains: "has no GrpcPort").ExpectOne(
+            TimeSpan.FromSeconds(5), () => rig = SpawnGrpcRig(factory));
+
+        rig.ShouldNotBeNull();
+        AwaitContactsLoaded(rig);
+        rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
+
+        AwaitCondition(() => rig.Dialer.Calls.Count == 1, TimeSpan.FromSeconds(5));
+        rig.Dialer.Calls[0].Endpoint.ShouldBe("http://host-b:4056");
+    }
+
+    [Fact]
+    public void A_node_that_regains_then_loses_its_GrpcPort_warns_again()
+    {
+        // Warn-once is per continuous skip run, not per process. The suppression exists so a
+        // permanently portless row does not repeat every minute; carried across a repair it would
+        // silence the SECOND time a node loses its endpoint — the occurrence an operator most needs
+        // to see, because it is a regression rather than a known gap.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", null, true, false));
+        var rig = SpawnGrpcRig(factory);
+        AwaitContactsLoaded(rig);
+
+        SetGrpcPort(factory, "host-a:4053", 4056);
+        rig.Actor.Tell(new CentralCommunicationActor.RefreshContacts());
+        AwaitCondition(
+            () => rig.Psk.KnownClusterPublications.Count >= 2, TimeSpan.FromSeconds(5));
+
+        SetGrpcPort(factory, "host-a:4053", null);
+        EventFilter.Warning(contains: "has no GrpcPort").ExpectOne(TimeSpan.FromSeconds(5), () =>
+            rig.Actor.Tell(new CentralCommunicationActor.RefreshContacts()));
+    }
+
+    [Fact]
+    public void One_failing_node_does_not_stop_the_fan_and_is_logged_by_node_id()
+    {
+        // Fire-and-forget: a failure is logged and DROPPED. No retry, no queue, and above all no
+        // abort of the remaining nodes — the deploy path's apply deadline is the backstop.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(
+            factory,
+            "C1",
+            ("host-a:4053", "host-a", 4056, true, false),
+            ("host-b:4053", "host-b", 4056, true, false));
+        var dialer = new RecordingDialer
+        {
+            FailEndpointsContaining = "host-a",
+            Fault = new Grpc.Core.RpcException(
+                new Grpc.Core.Status(Grpc.Core.StatusCode.Unavailable, "node down")),
+        };
+        var rig = SpawnGrpcRig(factory, dialer);
+        AwaitContactsLoaded(rig);
+
+        // The node id, not the endpoint: an operator reading the log is looking at ClusterNode rows.
+        EventFilter.Warning(contains: "host-a:4053").ExpectOne(TimeSpan.FromSeconds(5), () =>
+            rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch())));
+
+        // Both were attempted; the healthy one is unaffected by its neighbour's failure.
+        AwaitCondition(() => rig.Dialer.Calls.Count == 2, TimeSpan.FromSeconds(5));
+
+        // Still serving after a per-node failure — no supervision restart, no state loss.
+        rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
+        AwaitCondition(() => rig.Dialer.Calls.Count == 4, TimeSpan.FromSeconds(5));
+    }
+
+    [Fact]
+    public void A_psk_miss_costs_only_that_node_and_never_becomes_a_fan_level_abort()
+    {
+        // The dialer refuses a keyless cluster locally, so it arrives here as an ordinary per-node
+        // failure. The fan-level assertion is that the other cluster is still dialled.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
+        SeedGrpcCluster(factory, "C2", ("host-b:4053", "host-b", 4056, true, false));
+        var dialer = new RecordingDialer
+        {
+            FailClusters = ["C2"],
+            Fault = new Grpc.Core.RpcException(
+                new Grpc.Core.Status(Grpc.Core.StatusCode.Unauthenticated, "no key")),
+        };
+        var rig = SpawnGrpcRig(factory, dialer);
+        AwaitContactsLoaded(rig);
+
+        EventFilter.Warning(contains: "Unauthenticated").ExpectOne(TimeSpan.FromSeconds(5), () =>
+            rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch())));
+
+        AwaitCondition(() => rig.Dialer.Calls.Count == 2, TimeSpan.FromSeconds(5));
+    }
+
+    [Fact]
+    public void A_command_with_no_grpc_target_is_dropped_with_a_warning()
+    {
+        var factory = NewInMemoryDbFactory();
+        var rig = SpawnGrpcRig(factory);
+        AwaitContactsLoaded(rig);
+
+        EventFilter.Warning(contains: "dropping").ExpectOne(TimeSpan.FromSeconds(5), () =>
+            rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch())));
+
+        rig.Dialer.Calls.ShouldBeEmpty();
+    }
+
+    [Fact]
+    public void A_payload_that_is_not_a_mesh_command_record_is_dropped_with_an_error()
+    {
+        // The mapper refuses to encode an unknown payload rather than shipping something the node
+        // cannot decode. That refusal must cost the command, not the actor.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
+        var rig = SpawnGrpcRig(factory);
+        AwaitContactsLoaded(rig);
+
+        EventFilter.Error(contains: "Cannot encode").ExpectOne(TimeSpan.FromSeconds(5), () =>
+            rig.Actor.Tell(new MeshCommand(DeploymentsTopic, "not a command record")));
+
+        rig.Dialer.Calls.ShouldBeEmpty();
+
+        // Still serving.
+        rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
+        AwaitCondition(() => rig.Dialer.Calls.Count == 1, TimeSpan.FromSeconds(5));
+    }
+
+    [Fact]
+    public void The_envelope_carries_the_commands_own_correlation_and_the_whole_fan_shares_it()
+    {
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(
+            factory,
+            "C1",
+            ("host-a:4053", "host-a", 4056, true, false),
+            ("host-b:4053", "host-b", 4056, true, false));
+        var rig = SpawnGrpcRig(factory);
+        AwaitContactsLoaded(rig);
+
+        var correlationId = new CorrelationId(Guid.NewGuid());
+        rig.Actor.Tell(new MeshCommand(DeploymentsTopic, new DispatchDeployment(
+            new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), correlationId)));
+
+        AwaitCondition(() => rig.Dialer.Calls.Count == 2, TimeSpan.FromSeconds(5));
+        rig.Dialer.Calls.ShouldAllBe(c => c.Envelope.CorrelationId == correlationId.ToString());
+        // One envelope object, encoded once and fanned — not re-encoded per node.
+        rig.Dialer.Calls.Select(c => c.Envelope).Distinct().Count().ShouldBe(1);
+    }
+
+    [Fact]
+    public void An_alarm_command_gets_one_minted_transport_correlation_shared_by_the_whole_fan()
+    {
+        // AlarmCommand is the one record with no correlation member of its own, so the envelope's is
+        // the only handle an operator has on it. Minting per fan rather than per node is what makes
+        // "this alarm command" one traceable event instead of N unrelated ones.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(
+            factory,
+            "C1",
+            ("host-a:4053", "host-a", 4056, true, false),
+            ("host-b:4053", "host-b", 4056, true, false));
+        var rig = SpawnGrpcRig(factory);
+        AwaitContactsLoaded(rig);
+
+        rig.Actor.Tell(new MeshCommand(
+            "alarms", new AlarmCommand("alarm-1", "Acknowledge", "operator", null, null)));
+
+        AwaitCondition(() => rig.Dialer.Calls.Count == 2, TimeSpan.FromSeconds(5));
+        rig.Dialer.Calls.Select(c => c.Envelope.CorrelationId).Distinct().Count().ShouldBe(1);
+        rig.Dialer.Calls[0].Envelope.CorrelationId.ShouldNotBeNullOrWhiteSpace();
+        rig.Dialer.Calls[0].Envelope.AlarmCommand.AlarmId.ShouldBe("alarm-1");
+    }
+
+    [Fact]
+    public void The_contact_refresh_publishes_the_known_cluster_set_and_drops_cached_psks()
+    {
+        // Phase 1A security register item 1, wired: the gate is only a control if something keeps it
+        // current, and the refresh is the moment central's picture of the fleet is up to date. The
+        // InvalidateAll on the same tick is what makes a rotated key take effect within one refresh
+        // instead of requiring a central restart.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
+        SeedGrpcCluster(factory, "C2", ("host-b:4053", "host-b", null, true, false));
+        var rig = SpawnGrpcRig(factory);
+
+        AwaitContactsLoaded(rig);
+
+        // C2 is included even though its only node has no GrpcPort: the gate is about which clusters
+        // central KNOWS, not which it can currently dial. Excluding it would deny that cluster's acks.
+        rig.Psk.KnownClusterPublications[0].ShouldBe(new[] { "C1", "C2" }, ignoreOrder: true);
+        rig.Psk.InvalidateAllCount.ShouldBeGreaterThanOrEqualTo(1);
+    }
+
+    [Fact]
+    public void A_slow_dial_does_not_park_the_actors_mailbox()
+    {
+        // The dialer's first act is a PSK lookup, which blocks on the secret store — bounded, but
+        // seconds, and repeated after every refresh because the refresh drops the cache. An async
+        // method runs synchronously to its first await, so dialing straight from the receive would
+        // stall this actor for that long PER NODE, and deployment acks queue behind that mailbox.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
+        var coordinator = CreateTestProbe();
+        var dialer = new RecordingDialer { BlockBeforeFirstAwait = true };
+        var rig = SpawnGrpcRig(factory, dialer, coordinator: coordinator.Ref);
+        AwaitContactsLoaded(rig);
+
+        try
+        {
+            rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
+
+            // The ack must be handled while the dial is still parked.
+            var ack = new ApplyAck(
+                new DeploymentId(Guid.NewGuid()),
+                new NodeId("host-a:4053"),
+                ApplyAckOutcome.Applied,
+                null,
+                new CorrelationId(Guid.NewGuid()));
+            rig.Actor.Tell(ack);
+
+            coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).ShouldBe(ack);
+        }
+        finally
+        {
+            dialer.Release();
+        }
+    }
+
+    [Fact]
+    public void A_node_added_after_start_joins_the_fan_on_the_next_refresh()
+    {
+        // The refresh is not decoration: operators add and retire nodes, and central must pick them
+        // up without a restart. Under Grpc that is the ONLY thing keeping the dispatch set current —
+        // there is no receptionist to notice a new registration on its own.
+        //
+        // Driven by telling the actor its own RefreshContacts tick rather than by shortening
+        // ContactRefreshSeconds: a one-second timer would run for the whole fact, adding DB load to
+        // every test sharing the runner, and would make this assertion depend on wall-clock timing.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
+        var rig = SpawnGrpcRig(factory);
+        AwaitContactsLoaded(rig);
+
+        using (var db = factory.CreateDbContext())
+        {
+            db.ClusterNodes.Add(new ClusterNode
+            {
+                NodeId = "host-b:4053",
+                ClusterId = "C1",
+                Host = "host-b",
+                AkkaPort = 4053,
+                GrpcPort = 4056,
+                ApplicationUri = "urn:host-b",
+                Enabled = true,
+                CreatedBy = "test",
+            });
+            db.SaveChanges();
+        }
+
+        rig.Actor.Tell(new CentralCommunicationActor.RefreshContacts());
+        AwaitCondition(
+            () => rig.Psk.KnownClusterPublications.Count >= 2, TimeSpan.FromSeconds(5));
+
+        rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
+
+        AwaitCondition(() => rig.Dialer.Calls.Count == 2, TimeSpan.FromSeconds(5));
+        rig.Dialer.Calls.Select(c => c.Endpoint).ShouldBe(
+            new[] { "http://host-a:4056", "http://host-b:4056" }, ignoreOrder: true);
+
+        // Once PER REFRESH, not once at startup. A key can rotate without any membership change, so
+        // an invalidation that only fired when the cluster set moved would serve the pre-rotation
+        // key forever and a rotation would need a central restart.
+        rig.Psk.InvalidateAllCount.ShouldBeGreaterThanOrEqualTo(2);
+    }
+
+    [Theory]
+    [InlineData(MeshTransportOptions.ModeDps)]
+    [InlineData(MeshTransportOptions.ModeClusterClient)]
+    [InlineData(MeshTransportOptions.ModeGrpc)]
+    public void The_known_cluster_gate_is_published_in_every_transport_mode(string mode)
+    {
+        // Replaces an earlier fact asserting the OPPOSITE, whose premise did not survive 1B: the
+        // gate feeds CentralCommandAuthInterceptor, which is registered on every admin node, and the
+        // ack service it protects is mapped by LISTENER PORT — neither consults MeshTransport:Mode.
+        // Gating the publication on Grpc meant a dark-switch rollback to ClusterClient or Dps left
+        // that endpoint serving with a permanently inactive gate, silently restoring both the timing
+        // oracle and the pre-authentication store/audit flood. The set is central's cluster
+        // inventory, not transport state.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
+        var psk = new RecordingPskProvider();
+        var clients = new RecordingClientFactory(() => CreateTestProbe());
+
+        Sys.ActorOf(CentralCommunicationActor.Props(
+            factory,
+            new MeshTransportOptions { Mode = mode, ContactRefreshSeconds = 60 },
+            clients,
+            () => null,
+            () => new RecordingDialer(),
+            () => psk));
+
+        AwaitCondition(
+            () => psk.KnownClusterPublications.Count >= 1, TimeSpan.FromSeconds(5));
+        psk.KnownClusterPublications[0].ShouldBe(new[] { "C1" });
+        psk.InvalidateAllCount.ShouldBeGreaterThanOrEqualTo(1);
+
+        // The one load-bearing line the publication itself cannot witness: with the refresh
+        // timer now running in every mode, only the ClusterClient branch may construct clients.
+        // With rows seeded, deleting the !_useClusterClient guard turns this red for Dps/Grpc.
+        if (mode != MeshTransportOptions.ModeClusterClient)
+            clients.Calls.ShouldBeEmpty();
+    }
+
+    [Fact]
+    public void The_gate_set_includes_a_cluster_the_fan_set_excludes()
+    {
+        // The two sets answer different questions. The FAN asks "who does central talk to" — enabled,
+        // non-maintenance, ported. The GATE asks "is this a cluster central has ever heard of", and
+        // must say yes for a cluster whose every node is disabled or in maintenance: such a cluster
+        // can still send a deployment ack, and refusing it would quietly turn maintenance mode into
+        // an authentication switch.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
+        SeedGrpcCluster(factory, "C-DISABLED", ("host-b:4053", "host-b", 4056, false, false));
+        SeedGrpcCluster(factory, "C-MAINT", ("host-c:4053", "host-c", 4056, true, true));
+        var rig = SpawnGrpcRig(factory);
+        AwaitContactsLoaded(rig);
+
+        rig.Psk.KnownClusterPublications[0].ShouldBe(
+            new[] { "C1", "C-DISABLED", "C-MAINT" }, ignoreOrder: true);
+
+        rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
+
+        AwaitCondition(() => rig.Dialer.Calls.Count == 1, TimeSpan.FromSeconds(5));
+        ExpectNoMsg(TimeSpan.FromMilliseconds(300));
+        rig.Dialer.Calls.Count.ShouldBe(1);
+        rig.Dialer.Calls[0].ClusterId.ShouldBe("C1");
+    }
+
+    [Fact]
+    public void The_gate_is_published_after_the_dispatch_set_is_adopted_and_before_the_cache_is_dropped()
+    {
+        // Two orderings in one handler, both load-bearing. SetKnownClusters must precede
+        // InvalidateAll, or the first lookup after a refresh re-reads the store under the PREVIOUS
+        // membership — briefly resolving a cluster central has just retired. And the dispatch set is
+        // adopted first so a command processed immediately after the publication already fans over
+        // the current targets.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
+        var rig = SpawnGrpcRig(factory);
+        AwaitContactsLoaded(rig);
+
+        rig.Psk.Sequence.ShouldBe(new[] { "SetKnownClusters", "InvalidateAll" });
+
+        // Targets-before-gate, observed the only way it can be from outside the actor: the very next
+        // message finds the dispatch set already populated.
+        rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
+        AwaitCondition(() => rig.Dialer.Calls.Count == 1, TimeSpan.FromSeconds(5));
+    }
+
+    [Fact]
+    public void Grpc_mode_with_no_dialer_wired_drops_the_command_with_an_error()
+    {
+        // Mis-wiring, not misconfiguration: Mode=Grpc validates fine but the DI graph did not supply
+        // a dialer. Silence here would be a fleet-wide command outage with no symptom until a deploy
+        // stalled at its apply deadline two minutes later.
+        var factory = NewInMemoryDbFactory();
+        SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
+        var actor = Sys.ActorOf(CentralCommunicationActor.Props(
+            factory,
+            GrpcMode(),
+            new RecordingClientFactory(() => CreateTestProbe()),
+            () => null,
+            dialer: null,
+            pskProvider: null));
+
+        // Filtered on the DISPATCH message's exact phrase: PreStart logs its own error about the same
+        // mis-wiring, and a looser filter races it into a second match.
+        EventFilter.Error(contains: "no command dialer is wired").ExpectOne(
+            TimeSpan.FromSeconds(5),
+            () => actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch())));
+    }
+
+    private static void SetGrpcPort(
+        IDbContextFactory<OtOpcUaConfigDbContext> factory, string nodeId, int? grpcPort)
+    {
+        using var db = factory.CreateDbContext();
+        var node = db.ClusterNodes.Single(n => n.NodeId == nodeId);
+        node.GrpcPort = grpcPort;
+        db.SaveChanges();
+    }
+
+    private static DispatchDeployment NewDispatch() => new(
+        new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), new CorrelationId(Guid.NewGuid()));
+
+    /// <summary>A spawned gRPC-mode actor plus the two seams the tests observe it through.</summary>
+    private sealed record GrpcRig(IActorRef Actor, RecordingDialer Dialer, RecordingPskProvider Psk);
+
+    private GrpcRig SpawnGrpcRig(
+        IDbContextFactory<OtOpcUaConfigDbContext> factory,
+        RecordingDialer? dialer = null,
+        IMeshClusterClientFactory? clientFactory = null,
+        IActorRef? coordinator = null)
+    {
+        dialer ??= new RecordingDialer();
+        var psk = new RecordingPskProvider();
+        var actor = Sys.ActorOf(CentralCommunicationActor.Props(
+            factory,
+            GrpcMode(),
+            clientFactory ?? new RecordingClientFactory(() => CreateTestProbe()),
+            () => coordinator,
+            () => dialer,
+            () => psk));
+        return new GrpcRig(actor, dialer, psk);
+    }
+
+    /// <summary>
+    ///     Waits until the actor has applied a contact load. The known-cluster publication is the
+    ///     signal because it happens in the same handler, immediately after the target set is
+    ///     adopted — and unlike the target set itself it is observable from outside the actor. A
+    ///     command told before this point would be dropped as "no targets" and never retried, which
+    ///     is correct behaviour and a flaky test.
+    /// </summary>
+    private void AwaitContactsLoaded(GrpcRig rig) =>
+        AwaitCondition(
+            () => rig.Psk.KnownClusterPublications.Count >= 1, TimeSpan.FromSeconds(5));
+
+    /// <summary>Records every dial and answers however the test asks; never opens a socket.</summary>
+    private sealed class RecordingDialer : IMeshCommandDialer
+    {
+        private readonly object _gate = new();
+        private readonly List<Dial> _calls = [];
+        private readonly ManualResetEventSlim _released = new(false);
+
+        /// <summary>
+        ///     When true, every dial blocks its calling thread synchronously until
+        ///     <see cref="Release"/>. Models the dialer's pre-await PSK lookup.
+        /// </summary>
+        public bool BlockBeforeFirstAwait { get; set; }
+
+        /// <summary>Unblocks every dial parked by <see cref="BlockBeforeFirstAwait"/>.</summary>
+        public void Release() => _released.Set();
+
+        /// <summary>Every dial made. Copied under the lock — the fan runs on thread-pool threads.</summary>
+        public IReadOnlyList<Dial> Calls
+        {
+            get { lock (_gate) return _calls.ToList(); }
+        }
+
+        /// <summary>Endpoints whose substring match makes the dial fault.</summary>
+        public string? FailEndpointsContaining { get; set; }
+
+        /// <summary>Cluster ids whose dials fault.</summary>
+        public string[] FailClusters { get; set; } = [];
+
+        /// <summary>
+        ///     The fault raised for a matching dial. Fully qualified because this file's bare
+        ///     <c>Status</c> is Akka's — an existing fact drives the actor's <c>Status.Failure</c>
+        ///     handler, and importing Grpc.Core here would make that reference ambiguous.
+        /// </summary>
+        public Grpc.Core.RpcException? Fault { get; set; }
+
+        public Task DispatchAsync(
+            string nodeEndpoint, CommandEnvelope envelope, string clusterId, CancellationToken ct)
+        {
+            lock (_gate) _calls.Add(new Dial(nodeEndpoint, envelope, clusterId));
+
+            if (BlockBeforeFirstAwait)
+            {
+                // Blocks the CALLING thread, exactly as IMeshPskProvider.TryGetPsk does on a cold
+                // cache: an async method runs synchronously to its first await, so a dial invoked
+                // straight from the actor would park the mailbox here.
+                _released.Wait(TimeSpan.FromSeconds(30));
+            }
+
+            var shouldFail =
+                (FailEndpointsContaining is not null && nodeEndpoint.Contains(FailEndpointsContaining))
+                || FailClusters.Contains(clusterId, StringComparer.OrdinalIgnoreCase);
+
+            return shouldFail && Fault is not null
+                ? Task.FromException(Fault)
+                : Task.CompletedTask;
+        }
+
+        /// <summary>One recorded dial.</summary>
+        public sealed record Dial(string Endpoint, CommandEnvelope Envelope, string ClusterId);
+    }
+
+    /// <summary>Records the gate and invalidation calls the contact refresh is supposed to make.</summary>
+    private sealed class RecordingPskProvider : IMeshPskProvider
+    {
+        private readonly object _gate = new();
+        private readonly List<IReadOnlyList<string>> _publications = [];
+        private readonly List<string> _sequence = [];
+        private int _invalidateAllCount;
+
+        /// <summary>Names of the mutating calls made, in the order they arrived.</summary>
+        public IReadOnlyList<string> Sequence
+        {
+            get { lock (_gate) return _sequence.ToList(); }
+        }
+
+        /// <summary>Every known-cluster set published, in order.</summary>
+        public IReadOnlyList<IReadOnlyList<string>> KnownClusterPublications
+        {
+            get { lock (_gate) return _publications.ToList(); }
+        }
+
+        /// <summary>How many times the whole cache was dropped.</summary>
+        public int InvalidateAllCount => Volatile.Read(ref _invalidateAllCount);
+
+        public bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk)
+        {
+            psk = "psk";
+            return true;
+        }
+
+        public bool Matches(string clusterId, string presented) => true;
+
+        public void SetKnownClusters(IReadOnlyCollection<string> clusterIds)
+        {
+            lock (_gate)
+            {
+                _publications.Add(clusterIds.ToList());
+                _sequence.Add(nameof(SetKnownClusters));
+            }
+        }
+
+        public void Invalidate(string clusterId) { }
+
+        public void InvalidateAll()
+        {
+            Interlocked.Increment(ref _invalidateAllCount);
+            lock (_gate) _sequence.Add(nameof(InvalidateAll));
+        }
+    }
 }
@@ -0,0 +1,249 @@
+using System.Diagnostics.CodeAnalysis;
+using Akka.Actor;
+using Akka.Cluster.Hosting;
+using Akka.Hosting;
+using Microsoft.EntityFrameworkCore;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Options;
+using Shouldly;
+using Xunit;
+using ZB.MOM.WW.Audit;
+using ZB.MOM.WW.OtOpcUa.Cluster;
+using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
+using ZB.MOM.WW.OtOpcUa.Commons.Types;
+using ZB.MOM.WW.OtOpcUa.Configuration;
+using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
+using ZB.MOM.WW.Secrets.Abstractions;
+
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
+
+/// <summary>
+///     Guards the one thing every other test in this project structurally cannot see: whether an
+///     admin node <b>boots</b> with the real registration graph behind the mesh command boundary.
+/// </summary>
+/// <remarks>
+///     <para>
+///         The Phase 1A live gate failed here. <c>WithOtOpcUaControlPlaneSingletons</c>'s first
+///         starter resolved <c>IMeshCommandDialer</c> and <c>IMeshPskProvider</c> eagerly, and their
+///         graph reaches back into the actor registry — dialer → <c>IMeshPskProvider</c> →
+///         <c>ISecretResolver</c> → <c>AddZbSecrets</c> → <c>IAuditWriter</c> →
+///         <c>ActorAuditWriter</c>, whose constructor calls
+///         <c>registry.Get&lt;AuditWriterActorKey&gt;()</c> for a singleton registered further down
+///         the SAME builder. Every central node died on
+///         <c>MissingActorRegistryEntryException</c>, in every transport mode, before serving
+///         anything.
+///     </para>
+///     <para>
+///         <b>Why the existing tests were blind.</b> <c>MeshGrpcCommandDialerTests</c> activates the
+///         dialer from a <c>ServiceCollection</c> that has no secrets registration at all, so its
+///         graph stops one hop in; the <c>CentralCommunicationActorTests</c> facts inject fakes
+///         directly and never touch DI; and the existing whole-host boots register no mesh provider,
+///         so the eager lookups returned <see langword="null"/> and resolved nothing. Every one of
+///         them passed while no central node could start.
+///     </para>
+///     <para>
+///         <b>So the assertion is <c>StartAsync</c> itself.</b> The failure happens while the Akka
+///         builder's starters execute, which is inside <c>StartAsync</c> — building the
+///         <c>ServiceProvider</c> and resolving from it by hand would not reproduce it. The graph
+///         below is the production one from <c>IMeshCommandDialer</c> down to the registry-bound
+///         audit writer; only the secret store itself is a stand-in, because a real one would add a
+///         KEK and a SQLite file to a test whose subject is starter ORDER.
+///     </para>
+/// </remarks>
+public sealed class ControlPlaneStarterOrderTests
+{
+    [Theory]
+    [InlineData(MeshTransportOptions.ModeDps)]
+    [InlineData(MeshTransportOptions.ModeClusterClient)]
+    [InlineData(MeshTransportOptions.ModeGrpc)]
+    public async Task An_admin_node_boots_with_the_real_mesh_dependency_graph(string mode)
+    {
+        // Every mode, because the eager resolve was mode-independent: the starter resolved both
+        // dependencies before anything looked at MeshTransport:Mode, so Dps — the DEFAULT, and the
+        // rollback target — died exactly like Grpc did.
+        using var host = BuildAdminHost(mode);
+
+        await Should.NotThrowAsync(() => host.StartAsync());
+
+        try
+        {
+            var registry = host.Services.GetRequiredService<ActorRegistry>();
+            registry.TryGet<CentralCommunicationActorKey>(out _).ShouldBeTrue(
+                "the command boundary must be registered on a booted admin node");
+            registry.TryGet<AuditWriterActorKey>(out _).ShouldBeTrue(
+                "the audit singleton the eager resolve reached for too early must exist by now");
+        }
+        finally
+        {
+            await host.StopAsync();
+        }
+    }
+
+    [Theory]
+    [InlineData(MeshTransportOptions.ModeDps)]
+    [InlineData(MeshTransportOptions.ModeClusterClient)]
+    [InlineData(MeshTransportOptions.ModeGrpc)]
+    public async Task The_deferred_dependencies_are_resolved_and_used_once_the_host_is_up(string mode)
+    {
+        // Deferring must not become "never resolved" — that would trade a loud boot failure for a
+        // silent one, with the known-cluster gate permanently unpublished and the dialer permanently
+        // absent. So this asserts the far end of the deferral: the actor resolved the PSK provider
+        // FROM THE CONTAINER and drove it, through the ordinary contact-refresh path. A Theory over
+        // all three modes because publication is mode-independent (the gate follows the endpoint).
+        //
+        // The provider is registered as a TYPE with an ISecretResolver constructor, so the resolve
+        // walks the same registry-bound chain the production MeshPskProvider does — which makes
+        // this fact the race detector for the PreStart-vs-starter-#4 window: a lost race that
+        // stayed null forever would never publish, and the TaskCompletionSource below would time
+        // out red. (Memoize-on-first-use in the actor is what heals the lost race at the first
+        // RefreshContacts message.)
+        //
+        // The signal is a TaskCompletionSource rather than a counter, because PreStart runs on the
+        // actor's own thread and the contact load is a piped DB read: there is no instant after
+        // StartAsync at which a synchronous assertion is correct.
+        using var host = BuildAdminHost(mode, useSignallingProvider: true);
+
+        await host.StartAsync();
+
+        try
+        {
+            var signal = (SignallingPskProvider)host.Services.GetRequiredService<IMeshPskProvider>();
+            await signal.Published.WaitAsync(TimeSpan.FromSeconds(20));
+        }
+        finally
+        {
+            await host.StopAsync();
+        }
+    }
+
+    /// <summary>
+    ///     Builds an admin host through the production cluster bootstrap and the production control-plane
+    ///     starter registration, with the mesh dependency graph wired as it is on a real central node.
+    /// </summary>
+    /// <param name="mode">The mesh transport mode to boot under.</param>
+    /// <param name="useSignallingProvider">
+    ///     Substitutes <see cref="SignallingPskProvider"/> for the real <see cref="MeshPskProvider"/>.
+    ///     Only the resolution test sets it; the boot test keeps the production type so the graph it
+    ///     walks is the production graph. The signalling provider is registered as a factory taking
+    ///     <see cref="ISecretResolver"/> from the container, so its resolution walks the same
+    ///     registry-bound chain either way.
+    /// </param>
+    /// <returns>The unstarted host.</returns>
+    private static IHost BuildAdminHost(string mode, bool useSignallingProvider = false)
+    {
+        var clusterOptions = new AkkaClusterOptions
+        {
+            Port = 0,
+            Hostname = "127.0.0.1",
+            PublicHostname = "127.0.0.1",
+            SeedNodes = [],
+            Roles = ["admin"],
+        };
+
+        return Host.CreateDefaultBuilder()
+            .ConfigureServices(services =>
+            {
+                services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(clusterOptions));
+                services.AddSingleton<IOptions<MeshTransportOptions>>(Options.Create(
+                    new MeshTransportOptions { Mode = mode, DispatchTimeoutSeconds = 10 }));
+                services.AddDbContextFactory<OtOpcUaConfigDbContext>(
+                    o => o.UseInMemoryDatabase($"starter-order-{Guid.NewGuid():N}"));
+                services.AddSingleton<IClusterRoleInfo>(new StubClusterRoleInfo());
+
+                // THE GRAPH. Production types from the dialer down to the registry-bound writer; the
+                // order of these registrations is deliberately irrelevant, because the defect was
+                // never about registration order — it was about WHEN the graph is first walked.
+                services.AddSingleton<IAuditWriter, RegistryBoundAuditWriter>();
+                services.AddSingleton<ISecretResolver, AuditingSecretResolver>();
+                if (useSignallingProvider)
+                    services.AddSingleton<IMeshPskProvider>(sp =>
+                        new SignallingPskProvider(sp.GetRequiredService<ISecretResolver>()));
+                else
+                    services.AddSingleton<IMeshPskProvider, MeshPskProvider>();
+                services.AddSingleton<IMeshCommandDialer, MeshGrpcCommandDialer>();
+
+                services.AddAkka("otopcua-starter-order-test", (ab, sp) =>
+                {
+                    ab.WithOtOpcUaClusterBootstrap(sp);
+                    ab.WithOtOpcUaControlPlaneSingletons();
+                });
+            })
+            .Build();
+    }
+
+    /// <summary>
+    ///     Stands in for <c>ActorAuditWriter</c>: an <see cref="IAuditWriter"/> that resolves a
+    ///     cluster singleton from the <see cref="ActorRegistry"/> <b>in its constructor</b>. That
+    ///     constructor is the tripwire — it throws
+    ///     <c>MissingActorRegistryEntryException</c> for exactly as long as the audit singleton has
+    ///     not been registered yet.
+    /// </summary>
+    private sealed class RegistryBoundAuditWriter : IAuditWriter
+    {
+        private readonly IActorRef _proxy;
+
+        public RegistryBoundAuditWriter(ActorRegistry registry) => _proxy = registry.Get<AuditWriterActorKey>();
+
+        public Task WriteAsync(AuditEvent evt, CancellationToken ct = default)
+        {
+            _proxy.Tell(evt);
+            return Task.CompletedTask;
+        }
+    }
+
+    /// <summary>
+    ///     An <see cref="IMeshPskProvider"/> that completes a task the first time the actor publishes
+    ///     the known-cluster set. Takes <see cref="ISecretResolver"/> it never uses, purely so that
+    ///     resolving it still walks the same registry-bound chain the real provider does.
+    /// </summary>
+    private sealed class SignallingPskProvider : IMeshPskProvider
+    {
+        private readonly TaskCompletionSource _published =
+            new(TaskCreationOptions.RunContinuationsAsynchronously);
+
+        public SignallingPskProvider(ISecretResolver resolver) => ArgumentNullException.ThrowIfNull(resolver);
+
+        /// <summary>Completes once <see cref="SetKnownClusters"/> has been called.</summary>
+        public Task Published => _published.Task;
+
+        public bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk)
+        {
+            psk = null;
+            return false;
+        }
+
+        public bool Matches(string clusterId, string presented) => false;
+
+        public void SetKnownClusters(IReadOnlyCollection<string> clusterIds) => _published.TrySetResult();
+
+        public void Invalidate(string clusterId) { }
+
+        public void InvalidateAll() { }
+    }
+
+    /// <summary>
+    ///     Stands in for the shared secret store's resolver. Its only load-bearing property is the
+    ///     one that broke boot: it depends on <see cref="IAuditWriter"/>, exactly as
+    ///     <c>AddZbSecrets</c>' resolver registration does.
+    /// </summary>
+    private sealed class AuditingSecretResolver : ISecretResolver
+    {
+        public AuditingSecretResolver(IAuditWriter auditWriter) => ArgumentNullException.ThrowIfNull(auditWriter);
+
+        public Task<string?> GetAsync(SecretName name, CancellationToken ct) => Task.FromResult<string?>(null);
+    }
+
+    /// <summary>Minimal <see cref="IClusterRoleInfo"/>; the admin singleton factories read it.</summary>
+    private sealed class StubClusterRoleInfo : IClusterRoleInfo
+    {
+        public string? ClusterRole => null;
+        public string? ClusterId => null;
+        public NodeId LocalNode => NodeId.Parse("127.0.0.1:0");
+        public IReadOnlySet<string> LocalRoles => new HashSet<string>();
+        public bool HasRole(string role) => false;
+        public IReadOnlyList<NodeId> MembersWithRole(string role) => [];
+        public NodeId? RoleLeader(string role) => null;
+        public event EventHandler<RoleLeaderChangedEventArgs>? RoleLeaderChanged { add { } remove { } }
+    }
+}
@@ -0,0 +1,340 @@
+using System.Diagnostics.CodeAnalysis;
+using Grpc.Core;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Logging.Abstractions;
+using Microsoft.Extensions.Options;
+using Shouldly;
+using Xunit;
+using ZB.MOM.WW.OtOpcUa.Cluster;
+using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
+using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
+
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
+
+/// <summary>
+///     ClusterClient→gRPC Phase 1A — central's outbound command dial. One node, one call, no retry.
+/// </summary>
+/// <remarks>
+///     The facts that matter are the credential ones: the bearer must be the TARGET node's cluster's
+///     key (not central's, and not some other cluster's), and a cluster with no resolvable key must
+///     never be dialled at all. Everything else is plumbing.
+/// </remarks>
+public sealed class MeshGrpcCommandDialerTests
+{
+    private const string Endpoint = "http://site-a-1:4056";
+
+    private static CommandEnvelope Envelope() => new()
+    {
+        CorrelationId = Guid.NewGuid().ToString("N"),
+        RestartDriver = new RestartDriverCommand
+        {
+            ClusterId = "SITE-A",
+            DriverInstanceId = "drv-1",
+            ActorByUserName = "operator",
+            CorrelationId = Guid.NewGuid().ToString("N"),
+        },
+    };
+
+    private static MeshGrpcCommandDialer CreateDialer(
+        IMeshPskProvider psk,
+        RecordingClient client,
+        int dispatchTimeoutSeconds = 10)
+        => new(
+            psk,
+            Options.Create(new MeshTransportOptions { DispatchTimeoutSeconds = dispatchTimeoutSeconds }),
+            NullLogger<MeshGrpcCommandDialer>.Instance,
+            _ => client);
+
+    [Fact]
+    public async Task Dial_presents_the_target_clusters_key_and_names_that_cluster()
+    {
+        // THE assertion. Central holds N keys; sending the wrong one turns a legitimate command into
+        // a PermissionDenied, and sending a fleet-wide one would be the blast radius this design
+        // exists to avoid.
+        var client = new RecordingClient();
+        var dialer = CreateDialer(
+            new FakePskProvider(("SITE-A", "psk-site-a"), ("SITE-B", "psk-site-b")), client);
+
+        await dialer.DispatchAsync(Endpoint, Envelope(), "SITE-B", CancellationToken.None);
+
+        client.LastHeaders.ShouldNotBeNull();
+        client.LastHeaders!.GetValue("authorization").ShouldBe("Bearer psk-site-b");
+        client.LastHeaders.GetValue("x-otopcua-cluster").ShouldBe("SITE-B");
+    }
+
+    [Fact]
+    public async Task A_cluster_with_no_key_is_refused_locally_and_never_dialled()
+    {
+        // Fail-closed: not "dial and let the node reject it". An unauthenticated command attempt
+        // against a node's control surface should not exist even as a rejected one.
+        var client = new RecordingClient();
+        var dialer = CreateDialer(new FakePskProvider(("SITE-A", "psk-site-a")), client);
+
+        var ex = await Should.ThrowAsync<RpcException>(
+            () => dialer.DispatchAsync(Endpoint, Envelope(), "SITE-Z", CancellationToken.None));
+
+        ex.StatusCode.ShouldBe(StatusCode.Unauthenticated);
+        client.Calls.ShouldBe(0);
+    }
+
+    [Fact]
+    public async Task The_envelope_reaches_the_node_unchanged()
+    {
+        var client = new RecordingClient();
+        var dialer = CreateDialer(new FakePskProvider(("SITE-A", "psk-site-a")), client);
+        var envelope = Envelope();
+
+        await dialer.DispatchAsync(Endpoint, envelope, "SITE-A", CancellationToken.None);
+
+        client.LastRequest.ShouldBe(envelope);
+    }
+
+    [Fact]
+    public async Task A_node_rejection_surfaces_as_its_own_RpcException()
+    {
+        // The actor grades per-node outcomes off the status code, so it must arrive intact rather
+        // than wrapped in an AggregateException or swallowed here.
+        var client = new RecordingClient
+        {
+            Fault = new RpcException(new Status(StatusCode.PermissionDenied, "nope")),
+        };
+        var dialer = CreateDialer(new FakePskProvider(("SITE-A", "psk-site-a")), client);
+
+        var ex = await Should.ThrowAsync<RpcException>(
+            () => dialer.DispatchAsync(Endpoint, Envelope(), "SITE-A", CancellationToken.None));
+
+        ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
+    }
+
+    [Fact]
+    public async Task The_per_call_deadline_bounds_a_node_that_never_answers()
+    {
+        // A dispatch dials every node in the fleet. Without this bound one unreachable node holds a
+        // task and a socket for as long as the process lives, while the deploy path's own apply
+        // deadline expires around it.
+        var client = new RecordingClient { HangUntilCancelled = true };
+        var dialer = CreateDialer(
+            new FakePskProvider(("SITE-A", "psk-site-a")), client, dispatchTimeoutSeconds: 1);
+
+        await Should.ThrowAsync<OperationCanceledException>(
+            () => dialer.DispatchAsync(Endpoint, Envelope(), "SITE-A", CancellationToken.None)
+                .WaitAsync(TimeSpan.FromSeconds(10)));
+    }
+
+    [Fact]
+    public async Task The_callers_token_still_cancels_a_dial_inside_the_deadline()
+    {
+        // Linked, not replaced: actor shutdown must cancel everything in flight even when the
+        // per-call deadline is nowhere near expiring.
+        var client = new RecordingClient { HangUntilCancelled = true };
+        var dialer = CreateDialer(
+            new FakePskProvider(("SITE-A", "psk-site-a")), client, dispatchTimeoutSeconds: 600);
+        using var cts = new CancellationTokenSource();
+
+        var dial = dialer.DispatchAsync(Endpoint, Envelope(), "SITE-A", cts.Token);
+        await cts.CancelAsync();
+
+        await Should.ThrowAsync<OperationCanceledException>(() => dial.WaitAsync(TimeSpan.FromSeconds(10)));
+    }
+
+    [Fact]
+    public async Task A_non_positive_dispatch_timeout_is_clamped_rather_than_honoured()
+    {
+        // Zero would mean an UNBOUNDED call — the exact failure the deadline exists to prevent — and
+        // MeshTransportOptionsValidator's rule for this key is scoped to driver-role nodes, so a
+        // pure-admin central reaches this code with the value unvalidated. Clamp loudly.
+        var client = new RecordingClient { HangUntilCancelled = true };
+        var logger = new RecordingLogger();
+        var dialer = new MeshGrpcCommandDialer(
+            new FakePskProvider(("SITE-A", "psk-site-a")),
+            Options.Create(new MeshTransportOptions { DispatchTimeoutSeconds = 0 }),
+            logger,
+            _ => client);
+
+        logger.Warnings.ShouldContain(w => w.Contains("DispatchTimeoutSeconds"));
+
+        // And the clamp is real, not just logged: the dial must still expire on its own.
+        await Should.ThrowAsync<OperationCanceledException>(
+            () => dialer.DispatchAsync(Endpoint, Envelope(), "SITE-A", CancellationToken.None)
+                .WaitAsync(TimeSpan.FromSeconds(20)));
+    }
+
+    [Fact]
+    public async Task Dialing_after_disposal_throws_rather_than_repopulating_the_channel_cache()
+    {
+        var client = new RecordingClient();
+        var dialer = CreateDialer(new FakePskProvider(("SITE-A", "psk-site-a")), client);
+        dialer.Dispose();
+
+        await Should.ThrowAsync<ObjectDisposedException>(
+            () => dialer.DispatchAsync(Endpoint, Envelope(), "SITE-A", CancellationToken.None));
+    }
+
+    [Fact]
+    public async Task The_real_channel_factory_dials_and_is_disposable()
+    {
+        // The fake client seam covers every other fact here, which means nothing else ever exercises
+        // GrpcChannel.ForAddress or the cache it fills. Dial an endpoint nothing is listening on:
+        // the failure must be an ordinary RpcException the actor can log, not a construction throw,
+        // and Dispose must then clean up the channel the dial created.
+        var dialer = new MeshGrpcCommandDialer(
+            new FakePskProvider(("SITE-A", "psk-site-a")),
+            Options.Create(new MeshTransportOptions { DispatchTimeoutSeconds = 5 }),
+            NullLogger<MeshGrpcCommandDialer>.Instance);
+
+        await Should.ThrowAsync<RpcException>(
+            () => dialer.DispatchAsync(
+                "http://127.0.0.1:1", Envelope(), "SITE-A", CancellationToken.None));
+
+        Should.NotThrow(dialer.Dispose);
+    }
+
+    [Fact]
+    public async Task Two_dials_to_one_endpoint_share_a_single_channel()
+    {
+        // The cache is the whole defence against a socket per call: a deploy dispatch fans across
+        // every node in the fleet, and it repeats on every redeploy. A channel per dial exhausts
+        // ports on central long before anything else notices.
+        var dialer = new MeshGrpcCommandDialer(
+            new FakePskProvider(("SITE-A", "psk-site-a")),
+            // 1 s, not 5: on a host that blackholes loopback instead of refusing, each dial costs
+            // the full deadline — keep the worst case bounded so this never reads as a hang.
+            Options.Create(new MeshTransportOptions { DispatchTimeoutSeconds = 1 }),
+            NullLogger<MeshGrpcCommandDialer>.Instance);
+
+        // Nothing is listening, so both dials fail — irrelevant here: the channel is constructed
+        // before the call is placed, which is exactly what is being counted.
+        for (var i = 0; i < 2; i++)
+        {
+            await Should.ThrowAsync<RpcException>(
+                () => dialer.DispatchAsync(
+                    "http://127.0.0.1:1", Envelope(), "SITE-A", CancellationToken.None));
+        }
+
+        dialer.CachedChannelCount.ShouldBe(1);
+
+        // A second endpoint is a second channel — the cache keys on the endpoint, it does not
+        // collapse every node onto one connection.
+        await Should.ThrowAsync<RpcException>(
+            () => dialer.DispatchAsync(
+                "http://127.0.0.1:2", Envelope(), "SITE-A", CancellationToken.None));
+
+        dialer.CachedChannelCount.ShouldBe(2);
+        Should.NotThrow(dialer.Dispose);
+    }
+
+    [Fact]
+    public void It_activates_from_the_container_despite_the_optional_client_factory_parameter()
+    {
+        // The seam is an optional constructor parameter, which MS.DI honours only because it checks
+        // HasDefaultValue. Get that wrong — by making it non-optional, or by adding a second public
+        // constructor — and the failure is a startup crash on the admin node, invisible to every
+        // other test here because they all construct the dialer by hand.
+        var services = new ServiceCollection();
+        services.AddLogging();
+        services.AddSingleton<IMeshPskProvider>(new FakePskProvider(("SITE-A", "psk-site-a")));
+        services.AddSingleton<IOptions<MeshTransportOptions>>(
+            Options.Create(new MeshTransportOptions { DispatchTimeoutSeconds = 10 }));
+        services.AddSingleton<IMeshCommandDialer, MeshGrpcCommandDialer>();
+
+        using var provider = services.BuildServiceProvider();
+
+        provider.GetRequiredService<IMeshCommandDialer>().ShouldBeOfType<MeshGrpcCommandDialer>();
+    }
+
+    /// <summary>
+    ///     A <see cref="NodeCommandService.NodeCommandServiceClient"/> that records what it was
+    ///     handed and answers however the test asks. The generated client's protected parameterless
+    ///     constructor exists precisely for this.
+    /// </summary>
+    private sealed class RecordingClient : NodeCommandService.NodeCommandServiceClient
+    {
+        /// <summary>Number of calls that reached the client.</summary>
+        public int Calls { get; private set; }
+
+        /// <summary>Headers of the most recent call.</summary>
+        public Metadata? LastHeaders { get; private set; }
+
+        /// <summary>Request of the most recent call.</summary>
+        public CommandEnvelope? LastRequest { get; private set; }
+
+        /// <summary>When set, the call faults with this instead of answering.</summary>
+        public RpcException? Fault { get; set; }
+
+        /// <summary>When true, the call never completes until its token is cancelled.</summary>
+        public bool HangUntilCancelled { get; set; }
+
+        public override AsyncUnaryCall<ExecuteAck> ExecuteAsync(
+            CommandEnvelope request,
+            Metadata? headers = null,
+            DateTime? deadline = null,
+            CancellationToken cancellationToken = default)
+        {
+            Calls++;
+            LastHeaders = headers;
+            LastRequest = request;
+
+            var response = Fault is not null
+                ? Task.FromException<ExecuteAck>(Fault)
+                : HangUntilCancelled
+                    ? HangAsync(cancellationToken)
+                    : Task.FromResult(new ExecuteAck());
+
+            return new AsyncUnaryCall<ExecuteAck>(
+                response,
+                Task.FromResult(new Metadata()),
+                () => Status.DefaultSuccess,
+                () => [],
+                () => { });
+        }
+
+        private static async Task<ExecuteAck> HangAsync(CancellationToken ct)
+        {
+            await Task.Delay(Timeout.InfiniteTimeSpan, ct).ConfigureAwait(false);
+            return new ExecuteAck();
+        }
+    }
+
+    /// <summary>Captures Warning-level messages so a construction-time check is assertable.</summary>
+    private sealed class RecordingLogger : ILogger<MeshGrpcCommandDialer>
+    {
+        /// <summary>Every warning logged, formatted.</summary>
+        public List<string> Warnings { get; } = [];
+
+        public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
+
+        public bool IsEnabled(LogLevel logLevel) => true;
+
+        public void Log<TState>(
+            LogLevel logLevel,
+            EventId eventId,
+            TState state,
+            Exception? exception,
+            Func<TState, Exception?, string> formatter)
+        {
+            if (logLevel == LogLevel.Warning)
+                Warnings.Add(formatter(state, exception));
+        }
+    }
+
+    /// <summary>An <see cref="IMeshPskProvider"/> over a fixed key set; unknown clusters fail closed.</summary>
+    private sealed class FakePskProvider(params (string Cluster, string Psk)[] keys) : IMeshPskProvider
+    {
+        private readonly Dictionary<string, string> _keys =
+            keys.ToDictionary(k => k.Cluster, k => k.Psk, StringComparer.OrdinalIgnoreCase);
+
+        public bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk)
+            => _keys.TryGetValue(clusterId, out psk);
+
+        public bool Matches(string clusterId, string presented)
+            => _keys.TryGetValue(clusterId, out var expected)
+               && string.Equals(expected, presented, StringComparison.Ordinal);
+
+        public void SetKnownClusters(IReadOnlyCollection<string> clusterIds) { }
+
+        public void Invalidate(string clusterId) { }
+
+        public void InvalidateAll() { }
+    }
+}
@@ -367,6 +367,205 @@ public sealed class MeshPskProviderTests
        resolver.CallCount.ShouldBe(0);
    }

+    // ------------------------------------------------------------------ known-cluster gate
+    //
+    // Phase 1A security register item 1. Central rebuilds its cluster set from ClusterNode rows on
+    // every contact refresh; a cluster id outside that set must be refused in O(1) BEFORE anything
+    // touches the secret store. Two things ride on that: an unauthenticated caller can no longer
+    // make central perform a store read (and an audit write) per forged x-otopcua-cluster header,
+    // and the timing difference between a known cluster (cached) and an unknown one (never cached,
+    // because negatives are deliberately not retained) stops being remotely measurable.
+
+    [Fact]
+    public void UnknownCluster_WithTheGatePopulated_IsRefused_WithoutTouchingTheStore()
+    {
+        // THE assertion of the register item. CallCount == 0 is the whole point: the refusal must
+        // cost nothing, or the gate is a comment rather than a control.
+        var resolver = new FakeSecretResolver { Value = "stored-psk" };
+        var provider = CreateProvider(resolver);
+        provider.SetKnownClusters(["SITE-A", "SITE-B"]);
+
+        provider.TryGetPsk("SITE-Z", out var psk).ShouldBeFalse();
+
+        psk.ShouldBeNull();
+        resolver.CallCount.ShouldBe(0);
+    }
+
+    [Fact]
+    public void KnownCluster_WithTheGatePopulated_StillResolves()
+    {
+        var resolver = new FakeSecretResolver { Value = "stored-psk" };
+        var provider = CreateProvider(resolver);
+        provider.SetKnownClusters(["SITE-A"]);
+
+        provider.TryGetPsk("SITE-A", out var psk).ShouldBeTrue();
+
+        psk.ShouldBe("stored-psk");
+        resolver.CallCount.ShouldBe(1);
+    }
+
+    [Fact]
+    public void TheKnownClusterSet_IsCaseInsensitive()
+    {
+        // ClusterId case is whatever the operator typed into the DB row; the header is whatever the
+        // node was configured with. A case-sensitive gate would refuse a legitimate cluster.
+        var resolver = new FakeSecretResolver { Value = "stored-psk" };
+        var provider = CreateProvider(resolver);
+        provider.SetKnownClusters(["SITE-A"]);
+
+        provider.TryGetPsk("site-a", out var psk).ShouldBeTrue();
+
+        psk.ShouldBe("stored-psk");
+    }
+
+    [Fact]
+    public void SwappingTheKnownClusterSet_AddsAndRemovesClusters()
+    {
+        // The set is rebuilt from the DB every refresh, so both directions happen in production: a
+        // new site appears, a retired one stops being accepted.
+        var resolver = new FakeSecretResolver { Value = "stored-psk" };
+        var provider = CreateProvider(resolver);
+
+        provider.SetKnownClusters(["SITE-A"]);
+        provider.TryGetPsk("SITE-A", out _).ShouldBeTrue();
+        provider.TryGetPsk("SITE-B", out _).ShouldBeFalse();
+
+        provider.SetKnownClusters(["SITE-B"]);
+        provider.TryGetPsk("SITE-B", out _).ShouldBeTrue();
+        provider.TryGetPsk("SITE-A", out _).ShouldBeFalse();
+    }
+
+    [Fact]
+    public void AnEmptyKnownClusterSet_LeavesTheGateInactive()
+    {
+        // Fail-OPEN, deliberately and narrowly: central publishes the set from its contact refresh,
+        // and an empty publish means "the DB told me nothing", not "no cluster is legitimate".
+        // Refusing everything there would make a fleet-wide ack outage the failure mode of an empty
+        // or unreadable ClusterNode table.
+        var resolver = new FakeSecretResolver { Value = "stored-psk" };
+        var provider = CreateProvider(resolver);
+        provider.SetKnownClusters(["SITE-A"]);
+        provider.SetKnownClusters([]);
+
+        provider.TryGetPsk("SITE-Z", out var psk).ShouldBeTrue();
+
+        psk.ShouldBe("stored-psk");
+    }
+
+    [Fact]
+    public void AnUnsetKnownClusterSet_LeavesTheGateInactive()
+    {
+        // Admin boot: the interceptor is live from the first request, but the first contact refresh
+        // has not run yet. Gating on a set nobody has published would deny every ack in that window.
+        var resolver = new FakeSecretResolver { Value = "stored-psk" };
+        var provider = CreateProvider(resolver);
+
+        provider.TryGetPsk("SITE-Z", out var psk).ShouldBeTrue();
+
+        psk.ShouldBe("stored-psk");
+    }
+
+    [Fact]
+    public void TheGate_IsConsultedBeforeTheConfigMap()
+    {
+        // Consequence worth pinning rather than discovering: a ClusterPsks entry for a cluster with
+        // no enabled ClusterNode row is refused once the gate is populated. That is correct — central
+        // would neither dial such a cluster nor accept an ack from it — but it is surprising enough
+        // on a rig to deserve a test that names it.
+        var resolver = new FakeSecretResolver();
+        var provider = CreateProvider(
+            resolver, new Dictionary<string, string> { ["SITE-X"] = "rig-psk-site-x" });
+        provider.SetKnownClusters(["SITE-A"]);
+
+        provider.TryGetPsk("SITE-X", out var psk).ShouldBeFalse();
+
+        psk.ShouldBeNull();
+    }
+
+    [Fact]
+    public void SetKnownClusters_TakesACopy_SoALaterMutationOfTheCallersList_CannotWidenTheGate()
+    {
+        var clusters = new List<string> { "SITE-A" };
+        var resolver = new FakeSecretResolver { Value = "stored-psk" };
+        var provider = CreateProvider(resolver);
+        provider.SetKnownClusters(clusters);
+
+        clusters.Add("SITE-Z");
+
+        provider.TryGetPsk("SITE-Z", out _).ShouldBeFalse();
+    }
+
+    // ------------------------------------------------------------------ Matches
+
+    [Fact]
+    public void Matches_AcceptsTheClustersOwnKey()
+    {
+        var resolver = new FakeSecretResolver();
+        var provider = CreateProvider(
+            resolver, new Dictionary<string, string> { ["SITE-A"] = "psk-site-a" });
+
+        provider.Matches("SITE-A", "psk-site-a").ShouldBeTrue();
+    }
+
+    [Fact]
+    public void Matches_RejectsAnotherClustersKey()
+    {
+        // The isolation property, asserted at the provider rather than only at the interceptor:
+        // choosing a cluster only changes which key you are graded against.
+        var resolver = new FakeSecretResolver();
+        var provider = CreateProvider(
+            resolver,
+            new Dictionary<string, string> { ["SITE-A"] = "psk-site-a", ["SITE-B"] = "psk-site-b" });
+
+        provider.Matches("SITE-A", "psk-site-b").ShouldBeFalse();
+    }
+
+    [Fact]
+    public void Matches_IsNotAPrefixMatch()
+    {
+        var resolver = new FakeSecretResolver();
+        var provider = CreateProvider(
+            resolver, new Dictionary<string, string> { ["SITE-A"] = "psk-site-a" });
+
+        provider.Matches("SITE-A", "psk-site-").ShouldBeFalse();
+        provider.Matches("SITE-A", "psk-site-a-and-more").ShouldBeFalse();
+    }
+
+    [Fact]
+    public void Matches_FailsClosed_WhenTheClusterHasNoKey()
+    {
+        var resolver = new FakeSecretResolver { Value = null };
+        var provider = CreateProvider(resolver);
+
+        provider.Matches("SITE-Z", "anything-at-all").ShouldBeFalse();
+    }
+
+    [Theory]
+    [InlineData("")]
+    [InlineData("   ")]
+    public void Matches_RejectsABlankPresentedToken(string presented)
+    {
+        // A blank key never reaches the cache (blank resolves fail closed), but a blank PRESENTED
+        // token must be refused on its own terms rather than relying on that.
+        var resolver = new FakeSecretResolver();
+        var provider = CreateProvider(
+            resolver, new Dictionary<string, string> { ["SITE-A"] = "psk-site-a" });
+
+        provider.Matches("SITE-A", presented).ShouldBeFalse();
+    }
+
+    [Fact]
+    public void Matches_IsRefusedByTheKnownClusterGate_WithoutTouchingTheStore()
+    {
+        var resolver = new FakeSecretResolver { Value = "stored-psk" };
+        var provider = CreateProvider(resolver);
+        provider.SetKnownClusters(["SITE-A"]);
+
+        provider.Matches("SITE-Z", "stored-psk").ShouldBeFalse();
+
+        resolver.CallCount.ShouldBe(0);
+    }
+
    /// <summary>
    ///     An <see cref="ISecretResolver"/> whose answer the test controls between calls, so cache
    ///     staleness and invalidation are observable without a real store.
@@ -2,5 +2,6 @@ using Xunit;

 // The TelemetryStreamGrpcService concurrency counter is process-wide (static). Run this assembly's
 // tests serially so the cap test observes a deterministic active-stream count with no cross-test
-// interference. This project holds only telemetry tests, so serial execution costs nothing.
+// interference. Every other suite here is small and fast (the mesh-command tests spin a TestKit
+// ActorSystem per class), so serial execution costs nothing.
 [assembly: CollectionBehavior(DisableTestParallelization = true)]
@@ -242,14 +242,58 @@ public sealed class CentralCommandAuthInterceptorTests
        publicCtors.Length.ShouldBe(1);
    }

+    [Fact]
+    public async Task Verification_GoesThroughMatches_AndTheInterceptorNeverHandlesTheKeyItself()
+    {
+        // Phase 1A security register item 2. The interceptor must not fetch the raw key and compare
+        // it — that is how a constant-time comparison eventually becomes '=='. Matches owns the
+        // comparison; TryGetPsk survives only for the DIAL side, which has to send the key.
+        // (The production provider's Matches calls TryGetPsk internally; this fake implements the two
+        // independently, so a zero TryGetPsk count here means the INTERCEPTOR never asked for the key.)
+        var provider = new FakeMeshPskProvider(("SITE-A", "psk-site-a"));
+        var interceptor = new CentralCommandAuthInterceptor(
+            provider, NullLogger<CentralCommandAuthInterceptor>.Instance);
+
+        await Invoke(interceptor, CreateContext(ReportApplyMethod, "Bearer psk-site-a", "SITE-A"));
+
+        provider.MatchesCalls.ShouldBe(1);
+        provider.TryGetPskCalls.ShouldBe(0);
+    }
+
    /// <summary>An <see cref="IMeshPskProvider"/> over a fixed key set; unknown clusters fail closed.</summary>
    private sealed class FakeMeshPskProvider(params (string Cluster, string Psk)[] keys) : IMeshPskProvider
    {
        private readonly Dictionary<string, string> _keys =
            keys.ToDictionary(k => k.Cluster, k => k.Psk, StringComparer.OrdinalIgnoreCase);

+        /// <summary>How many times the raw key was asked for — must stay 0 on the verify path.</summary>
+        public int TryGetPskCalls { get; private set; }
+
+        /// <summary>How many times verification went through the provider's comparison.</summary>
+        public int MatchesCalls { get; private set; }
+
        public bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk)
-            => _keys.TryGetValue(clusterId, out psk);
+        {
+            TryGetPskCalls++;
+            return _keys.TryGetValue(clusterId, out psk);
+        }
+
+        /// <summary>
+        ///     Implemented independently of <see cref="TryGetPsk"/> so the two calls are separately
+        ///     countable — the interface declares this with no default implementation, deliberately.
+        ///     Semantics match the production provider's: no key, or a key that is not exactly the
+        ///     presented token.
+        /// </summary>
+        public bool Matches(string clusterId, string presented)
+        {
+            MatchesCalls++;
+            return !string.IsNullOrWhiteSpace(presented)
+                   && _keys.TryGetValue(clusterId, out var expected)
+                   && string.Equals(expected, presented, StringComparison.Ordinal);
+        }
+
+        /// <summary>No-op: the gate is the production provider's behaviour, pinned in its own tests.</summary>
+        public void SetKnownClusters(IReadOnlyCollection<string> clusterIds) { }

        public void Invalidate(string clusterId) { }

@@ -0,0 +1,186 @@
+using Akka.Hosting;
+using Akka.TestKit.Xunit2;
+using Grpc.Core;
+using Microsoft.Extensions.Logging.Abstractions;
+using Shouldly;
+using Xunit;
+using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
+using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
+using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
+using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
+using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
+using ZB.MOM.WW.OtOpcUa.Commons.Types;
+using ZB.MOM.WW.OtOpcUa.Host.Grpc;
+using ZB.MOM.WW.OtOpcUa.Runtime;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Grpc;
+
+/// <summary>
+///     Verifies the node-side command ingress (ClusterClient→gRPC migration Phase 1): each of the four
+///     central→node commands is decoded off its <see cref="CommandEnvelope"/> and handed to the
+///     registered <c>NodeCommunicationActor</c> as the same CLR record the ClusterClient transport
+///     delivered, an undecodable envelope is refused with <see cref="StatusCode.InvalidArgument"/>, and
+///     a call landing in the boot window before the actor is registered is refused with
+///     <see cref="StatusCode.Unavailable"/>.
+/// </summary>
+/// <remarks>
+///     The actor is a <c>TestProbe</c> registered under <see cref="NodeCommunicationActorKey"/>: what
+///     this service owes the rest of the system is the exact message the real actor already knows how to
+///     re-emit, so asserting on what arrives at the registry key is the whole contract. The onward leg —
+///     comm actor to EventStream to the real driver host — is pinned unchanged by
+///     <c>EventStreamCommandDeliveryTests</c> in the Runtime test project.
+/// </remarks>
+public sealed class NodeCommandGrpcServiceTests : TestKit
+{
+    private static readonly CorrelationId AlarmCorrelation = new(Guid.NewGuid());
+
+    private static ServerCallContext Ctx() => new FakeServerCallContext();
+
+    private static NodeCommandGrpcService NewService(ActorRegistry registry) =>
+        new(registry, NullLogger<NodeCommandGrpcService>.Instance);
+
+    [Fact]
+    public async Task Each_of_the_four_commands_reaches_the_comm_actor_as_its_domain_record()
+    {
+        var probe = CreateTestProbe();
+        var registry = new ActorRegistry();
+        registry.Register<NodeCommunicationActorKey>(probe.Ref);
+        var service = NewService(registry);
+
+        var dispatch = new DispatchDeployment(
+            new DeploymentId(Guid.NewGuid()), new RevisionHash("rev-1"), new CorrelationId(Guid.NewGuid()));
+        var restart = new RestartDriver("cluster-a", "drv-1", "alice", Guid.NewGuid());
+        var reconnect = new ReconnectDriver("cluster-a", "drv-2", "bob", Guid.NewGuid());
+        var alarm = new AlarmCommand("alarm-7", "Acknowledge", "carol", "looked at it", null);
+
+        (await service.Execute(MeshTransportDtoMapper.ToEnvelope(dispatch), Ctx())).ShouldNotBeNull();
+        probe.ExpectMsg<DispatchDeployment>().ShouldBe(dispatch);
+
+        await service.Execute(MeshTransportDtoMapper.ToEnvelope(restart), Ctx());
+        probe.ExpectMsg<RestartDriver>().ShouldBe(restart);
+
+        await service.Execute(MeshTransportDtoMapper.ToEnvelope(reconnect), Ctx());
+        probe.ExpectMsg<ReconnectDriver>().ShouldBe(reconnect);
+
+        await service.Execute(MeshTransportDtoMapper.ToEnvelope(alarm, AlarmCorrelation), Ctx());
+        probe.ExpectMsg<AlarmCommand>().ShouldBe(alarm);
+    }
+
+    [Fact]
+    public async Task A_call_in_the_boot_window_before_the_actor_is_registered_is_unavailable()
+    {
+        // Nothing registered: the ActorSystem exists but AddOtOpcUaRuntime has not spawned the comm
+        // actor yet. Unavailable (not Internal) because the condition is transient and retryable —
+        // central's dialer may try the same node again a moment later and succeed.
+        var service = NewService(new ActorRegistry());
+        var dispatch = new DispatchDeployment(
+            new DeploymentId(Guid.NewGuid()), new RevisionHash("rev-1"), new CorrelationId(Guid.NewGuid()));
+
+        var ex = await Should.ThrowAsync<RpcException>(
+            () => service.Execute(MeshTransportDtoMapper.ToEnvelope(dispatch), Ctx()));
+
+        ex.StatusCode.ShouldBe(StatusCode.Unavailable);
+    }
+
+    [Fact]
+    public async Task An_unset_oneof_is_invalid_argument()
+    {
+        var probe = CreateTestProbe();
+        var registry = new ActorRegistry();
+        registry.Register<NodeCommunicationActorKey>(probe.Ref);
+        var service = NewService(registry);
+
+        var ex = await Should.ThrowAsync<RpcException>(() => service.Execute(
+            new CommandEnvelope { CorrelationId = Guid.NewGuid().ToString("N") }, Ctx()));
+
+        ex.StatusCode.ShouldBe(StatusCode.InvalidArgument);
+        probe.ExpectNoMsg(TimeSpan.FromMilliseconds(200));
+    }
+
+    [Theory]
+    [MemberData(nameof(UndecodableEnvelopes))]
+    public async Task An_envelope_whose_fields_do_not_parse_is_invalid_argument(CommandEnvelope envelope)
+    {
+        var probe = CreateTestProbe();
+        var registry = new ActorRegistry();
+        registry.Register<NodeCommunicationActorKey>(probe.Ref);
+        var service = NewService(registry);
+
+        var ex = await Should.ThrowAsync<RpcException>(() => service.Execute(envelope, Ctx()));
+
+        ex.StatusCode.ShouldBe(StatusCode.InvalidArgument);
+        probe.ExpectNoMsg(TimeSpan.FromMilliseconds(200));
+    }
+
+    public static TheoryData<CommandEnvelope> UndecodableEnvelopes() =>
+    [
+        // Deployment id is not a hyphenless GUID -> FormatException out of the mapper.
+        new CommandEnvelope
+        {
+            CorrelationId = Guid.NewGuid().ToString("N"),
+            DispatchDeployment = new DispatchDeploymentCommand
+            {
+                DeploymentId = "not-a-guid",
+                RevisionHash = "rev-1",
+                CorrelationId = Guid.NewGuid().ToString("N"),
+            },
+        },
+        // Revision hash is blank -> ArgumentException out of RevisionHash.Parse.
+        new CommandEnvelope
+        {
+            CorrelationId = Guid.NewGuid().ToString("N"),
+            DispatchDeployment = new DispatchDeploymentCommand
+            {
+                DeploymentId = Guid.NewGuid().ToString("N"),
+                RevisionHash = "",
+                CorrelationId = Guid.NewGuid().ToString("N"),
+            },
+        },
+        // Restart command with an unparseable correlation id -> FormatException.
+        new CommandEnvelope
+        {
+            CorrelationId = Guid.NewGuid().ToString("N"),
+            RestartDriver = new RestartDriverCommand
+            {
+                ClusterId = "cluster-a",
+                DriverInstanceId = "drv-1",
+                ActorByUserName = "alice",
+                CorrelationId = "nope",
+            },
+        },
+    ];
+
+    [Fact]
+    public async Task An_undecodable_envelope_in_the_boot_window_is_invalid_argument_not_unavailable()
+    {
+        // Ordering pin: decode runs BEFORE the registry lookup. Unavailable invites the caller to
+        // retry, and an envelope this build cannot decode will never decode — reporting it as a
+        // readiness problem would turn a permanent fault into an unbounded retry loop.
+        var service = NewService(new ActorRegistry());
+
+        var ex = await Should.ThrowAsync<RpcException>(() => service.Execute(
+            new CommandEnvelope { CorrelationId = Guid.NewGuid().ToString("N") }, Ctx()));
+
+        ex.StatusCode.ShouldBe(StatusCode.InvalidArgument);
+    }
+
+    /// <summary>Minimal <see cref="ServerCallContext"/>; this RPC reads nothing off the context but its method name.</summary>
+    private sealed class FakeServerCallContext : ServerCallContext
+    {
+        protected override string MethodCore => "/mesh_transport.v1.NodeCommandService/Execute";
+        protected override string HostCore => "localhost";
+        protected override string PeerCore => "ipv4:127.0.0.1:0";
+        protected override DateTime DeadlineCore => DateTime.MaxValue;
+        protected override Metadata RequestHeadersCore => [];
+        protected override CancellationToken CancellationTokenCore => CancellationToken.None;
+        protected override Metadata ResponseTrailersCore { get; } = [];
+        protected override Status StatusCore { get; set; }
+        protected override WriteOptions? WriteOptionsCore { get; set; }
+        protected override AuthContext AuthContextCore => new(null, new Dictionary<string, List<AuthProperty>>());
+
+        protected override ContextPropagationToken CreatePropagationTokenCore(ContextPropagationOptions? options) =>
+            throw new NotSupportedException();
+
+        protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders) => Task.CompletedTask;
+    }
+}
@@ -10,6 +10,9 @@
    <ItemGroup>
        <PackageReference Include="xunit"/>
        <PackageReference Include="Shouldly"/>
+        <!-- NodeCommandGrpcServiceTests registers a TestProbe under NodeCommunicationActorKey, so the
+             service is asserted against the same ActorRegistry seam the Host wires in production. -->
+        <PackageReference Include="Akka.TestKit.Xunit2"/>
        <!-- ConfigDbApiKeyStoreTests drives the store against a real EF query, the same in-memory
             provider convention the ControlPlane/Runtime test projects use. No central SQL Server. -->
        <PackageReference Include="Microsoft.EntityFrameworkCore.InMemory"/>
Author	SHA1	Message	Date
Joseph Doherty	a400b41502	docs(plans): record Phase 1A merge SHA `b360239e` v2-ci / build (push) Successful in 4m27s Details v2-ci / unit-tests (push) Failing after 15m14s Details	2026-08-06 08:41:42 -04:00
Joseph Doherty	b360239e5e	merge: ClusterClient→gRPC Phase 1A — central→node command plane over gRPC (NodeCommandService on the telemetry listener, MeshGrpcCommandDialer, third mode branch in CentralCommunicationActor; security-register items 1-2 closed, gate follows the endpoint; live gate PASS 8/8 on re-run after the starter-order boot defect was found live and fixed; #544/#545/#546 filed)	2026-08-06 08:41:30 -04:00
Joseph Doherty	252618182f	docs(plans): Phase 1A close-out — carry-forwards from the gate, #546 filed	2026-08-06 08:39:45 -04:00
Joseph Doherty	e3a0ddbc84	docs(mesh): Phase 1A live gate re-run — PASS Re-ran the 1A gate on the docker-dev rig at `e54883a8` after the defect-4 boot fix. Image rebuilt (otopcua-host:dev -> 1d8452a29018); six host nodes plus both one-shots recreated; run 1's preserved image was stale and deliberately not reused. VERDICT: PASS. Boot: 6/6 nodes healthy INCLUDING the central pair, 0 Hosting failed to start, /health/ready 200 on every node, MAIN mesh 2/2. Check 1 PASS: one DispatchDeployment, one correlation id, accepted by all six nodes; central logs 6 accepted / 0 failed; zero ClusterClient lines on either central node with Akka.Cluster.Tools.Client at Debug. Check 2 PASS: deployment Sealed with all six NodeDeploymentState rows Applied inside ~90 ms, while the four site nodes hold receptionist associations — the only ack path they have in 1A. Mixed transport proven. Check 4 PASS: Ack clicked on the real /alerts page against a live ACTIVATED row; central fanned one AlarmCommand to all six over gRPC and the condition acknowledged 0.54 s later carrying the operator identity. Check 7 PASS: with site-b-2's GrpcPort NULLed, both central nodes warned once and the next dispatch reached exactly five nodes; the deployment then honestly ended TimedOut with site-b-2 alone unapplied. Row restored, 6/6 sealed after. Check 8 PASS and no longer vacuous: 0 denials on all four interceptors, and this time config-serve (28 KB verified artifact fetches from :4055) and telemetry (streams open on all six, live driver health on /hosts) both carried real traffic. Check 3 closed by composition, with the reason now established rather than assumed: <DriverStatusPanel has 0 usages repo-wide, no rendered page carries a Restart/Reconnect control, and the only non-script-analysis POST is /api/deployments. Run 1 drove the node-side leg for both commands; check 4 drives the identical AdminOperationsActor.Dispatch -> DispatchOverGrpc origin path from the AdminUI. Checks 5 + 6 stand from run 1 — nothing in the fix touches the interceptor. New finding 5, recorded because it is load-bearing for the fix's design: the PreStart resolve LOSES its race on real boots — 2 of 3 observed admin boots logged Failed to resolve for both dependencies with the same AuditWriterActorKey cause. The memoize-on-first-use hardening is therefore not theoretical, and its heal was proven live by moving the coordinator singleton onto the node that lost the race (stop central-2) and watching central-1 fan a dispatch to all five live nodes with no dialer error. Residual is cosmetic and carried to Phase 2: the Error text reads as permanent, and it will trip a soak error-grep on a healthy node. Defect 4 marked CLOSED; run 1's record kept in full underneath the combined table. Rig restored: flip reverted, six nodes healthy on all-ClusterClient on the committed image, three mesh ClusterClients back on each central node.	2026-08-06 08:37:48 -04:00
Joseph Doherty	e54883a8e0	fix(mesh): memoize dialer/provider on first use — a lost PreStart race self-heals at the first refresh (review-prescribed)	2026-08-06 08:07:34 -04:00
Joseph Doherty	c0475f2df4	fix(mesh): resolve dialer + PSK provider lazily — central boots again Phase 1A live gate defect 4. Neither central node booted on this branch, in ANY transport mode: Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key ...ControlPlane.AuditWriterActorKey at AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry) at ControlPlane.ServiceCollectionExtensions.<WithOtOpcUaControlPlaneSingletons> WithOtOpcUaControlPlaneSingletons' first WithActors starter resolved IMeshCommandDialer and IMeshPskProvider eagerly for the Props. Their graph reaches back into the registry — dialer -> IMeshPskProvider -> ISecretResolver -> AddZbSecrets -> IAuditWriter -> ActorAuditWriter, whose ctor calls registry.Get<AuditWriterActorKey>() — and WithSingleton<AuditWriterActorKey> is registered ~60 lines LATER in the same builder. Boot died before the entry existed. Mode-independent: the resolve happened before anything consulted MeshTransport:Mode, so Dps (the default, and the rollback target) died exactly like Grpc. Both dependencies now arrive as Func<> and are invoked in PreStart, by which point the host is fully built — the same shape, for the same class of reason, as the coordinator proxy Func one argument earlier. PreStart re-resolves on restart rather than caching across one; both are container singletons, so a restart gets the same instances. A resolution throw is caught and logged rather than propagated: a throw out of PreStart is an actor failure, the supervisor answers it with a restart, and a broken DI graph is not transient — that is an unbounded restart loop that would take the transport-independent ack path down with it. Regression pin: ControlPlaneStarterOrderTests boots a REAL admin host through WithOtOpcUaClusterBootstrap + WithOtOpcUaControlPlaneSingletons in production starter order, with the production graph from IMeshCommandDialer down to a registry-bound IAuditWriter, and asserts StartAsync itself — the failure occurs while the builder's starters execute, so building a ServiceProvider and resolving by hand cannot reproduce it. Verified RED against the eager resolve: 3/3 modes fail with MissingActorRegistryEntryException. A second fact pins that the deferral did not become 'never resolved' — the actor must resolve the provider from the container and drive it through the contact refresh, signalled by a TaskCompletionSource (PreStart runs on the actor thread and the contact load is a piped DB read, so no synchronous assertion after StartAsync is correct). Why nothing caught this: the dialer's container-activation test resolves from a collection with no secrets registration, so its graph stops one hop in; the actor facts inject fakes and never touch DI; and the existing whole-host boots register no mesh provider, so the eager lookups returned null and walked nothing. Two aggravators recorded by the gate, no code here: the failed process did not exit (docker Up while /health/ready 503s — Phase 2 readiness concern, tracked), and this is the third instance of the same eager-resolve-during-AddAkka class after scadaproj#1 and mesh Phase 6's IClusterRoleInfo boot crash. ControlPlane.Tests 236 -> 240, Host.Tests 92, Cluster.Tests 184, 0 warnings in every touched project.	2026-08-06 07:55:31 -04:00
Joseph Doherty	410c2e2925	docs(mesh): Phase 1A live gate record Ran the 1A live gate on the docker-dev rig at `fbf7b708` (image rebuilt from this worktree, all eight containers recreated). VERDICT: FAIL. Both central nodes fail to boot on this branch with MissingActorRegistryEntryException: AuditWriterActorKey — the first WithActors starter eagerly resolves IMeshCommandDialer/IMeshPskProvider, whose graph reaches ActorAuditWriter, whose ctor reads an ActorRegistry key registered ~60 lines later in the same builder. Reproduced on the same image with MeshTransport__Mode=ClusterClient, so it is mode-independent and not caused by the gate's rig-only flip. Recorded as defect 4; no product change made. Checks 5 (auth negative/positive on real Kestrel h2c) and 6 (wrong-key isolation, both directions, with a positive control) PASS. Checks 3 and 4 pass their node-side leg only, driven with real RestartDriver/ReconnectDriver/ AlarmCommand envelopes that restarted and reconnected SITEA-modbus. Checks 1, 2 and 7 are blocked outright — no central, no dispatch, no contact refresh — and check 8 is vacuous for telemetry + config-serve because central dials both and never booted. All five gaps are named in the carry-forward register and must be closed by a 1A gate re-run, not carried past the phase. Issues filed by this task: lmxopcua#544 (ConfigPublishCoordinatorTimeoutTests deflake) and lmxopcua#545 (mesh planes are h2c-only — accept the risk or add a UseTls option); the tracking table's two placeholder rows now carry them. Also recorded: DriverStatusPanel.razor is referenced by no page or component, so the AdminUI Reconnect/Restart controls are unreachable on this branch and on master — pre-existing, out of program scope, carried forward. Rig restored: compose flip reverted, six nodes healthy on all-ClusterClient. The rig runs the PRE-gate image (the 1A build is kept as otopcua-host:grpc-1a); a re-run must rebuild.	2026-08-06 07:38:18 -04:00
Joseph Doherty	fbf7b70847	docs(plans): 1A.2 + 1B.2 close-out — register items 1-2 CLOSED, defect log (3), carry-forwards, decisions	2026-08-06 06:58:34 -04:00
Joseph Doherty	86633f140e	test(mesh): pin the Dps no-ClusterClient guard; bound the channel-share test's blackhole worst case	2026-08-06 06:56:31 -04:00
Joseph Doherty	48dc2120c6	fix(mesh): gate follows the endpoint not the mode + review fixes The known-cluster gate published only under Mode=Grpc, but the interceptor it feeds is registered on every admin node and 1B maps CentralCommandService by LISTENER PORT — neither consults the mode. A dark-switch rollback to ClusterClient or Dps would therefore have left that endpoint serving with the gate permanently inactive, silently restoring both the timing oracle and the pre-auth secret-store/audit flood, as a side effect of a rollback nobody would connect to authentication. The gate is central's cluster inventory, not transport state, so it is now decoupled from the mode entirely: - SetKnownClusters + InvalidateAll run on EVERY successful contact load in EVERY mode, and the contact-refresh timer starts in all three. Under Dps this is a ConfigDb query every ContactRefreshSeconds that did not run before; central always has a ConfigDb, and one indexed read a minute is the cheaper half of the trade. - The GATE set is built from ALL ClusterNode rows. The eligibility filter moved out of the query's WHERE and into the loop, so the FAN set is byte-for-byte what it was (enabled && !maintenance) while the gate widens. This also dissolves the maintenance coupling flagged in 1A.2: a cluster whose nodes are all disabled or in maintenance can still ack, and refusing it would make maintenance mode an authentication switch. Central still refuses clusters it has never heard of. - HandleContactsLoaded keeps a documented order — targets adopted, gate published, cache dropped — now pinned by test. Guards: PreStart mirrors the null-dialer error for a null PSK provider (silent absence is the worst outcome there — nothing malfunctions, so nothing is investigated, while the gate never publishes and keys never rotate); a failed contact load now says out loud that the gate is stale and, before any load has succeeded, inactive; MeshGrpcCommandDialer clamps a non-positive DispatchTimeoutSeconds to 10 with a construction Warning, because the validator for that key is driver-role-scoped and a pure-admin central reads it unvalidated; the dialer's local PSK refusal was shortened so its remediation survives the actor's 200-char truncation of remote-authored status detail; the portless-node warn-once set is rebuilt each refresh, so a node that regains then loses a port warns again — the regression is the occurrence worth seeing. Docs/pins: dialer disposal comment no longer claims one-way; the two interceptor-test comments claiming Matches has a default implementation corrected (it deliberately has none); new pins for per-refresh invalidation, the SetKnownClusters-before-InvalidateAll order, and one channel per endpoint via a new CachedChannelCount. The_known_cluster_set_is_not_published_by_the_other_two_modes is replaced — its premise is obsolete — by three facts: every mode publishes, gate set is a strict superset of the fan set, and the handler ordering. ControlPlane.Tests 230 -> 236, Host.Tests 92, Cluster.Tests 184, 0 warnings.	2026-08-06 06:44:20 -04:00
Joseph Doherty	6a86201105	feat(mesh): gRPC command fan-out behind Mode=Grpc MeshGrpcCommandDialer places one NodeCommandService.Execute call on one node: a GrpcChannel cached per endpoint (the GrpcDeploymentArtifactFetcher shape), the TARGET node's cluster's bearer plus x-otopcua-cluster naming that cluster, and a per-call deadline linked to the caller's token. A cluster with no resolvable key is refused locally with Unauthenticated — there is no arm on which a command leaves central unauthenticated, even one the far side would have rejected. CentralCommunicationActor gains the third mode branch. The contact load now also projects GrpcPort and, from the SAME rows in the SAME pass, builds the dispatch set — which is what makes 'the gRPC fan reaches exactly the SendToAll recipient set' structural rather than two queries that have to keep agreeing. A row with a null GrpcPort is skipped warn-once (the TelemetryNodeSource precedent). Per-node results are piped back for LOGGING ONLY: named node, gRPC status, dropped. No retry, no queue. Every dial is scheduled on the thread pool rather than called from the receive. Found by review and pinned red-first: an async method runs synchronously to its first await, and the dialer's pre-await work is TryGetPsk, which blocks on the secret store — bounded but seconds, and repeated after every refresh since the refresh drops the cache. Dialing inline parked the actor's mailbox, so deployment acks queued behind a slow node; the regression test tells an ApplyAck while a dial is held and fails without the hop. The contact refresh also wires the Phase 1A register: it publishes the loaded cluster ids as the PSK provider's known-cluster set and drops the key cache, so a rotated key takes effect within one ContactRefreshSeconds. Neither happens under Dps or ClusterClient. Both existing mode branches are behaviourally untouched and every existing fact passes unmodified. The one edit to shared text: the Status.Failure warning drops the word 'ClusterClient', since that handler now serves all three transports. ControlPlane.Tests 205 -> 230, Host.Tests 92, Cluster.Tests 184, 0 warnings.	2026-08-06 05:58:21 -04:00
Joseph Doherty	14d21496fe	feat(mesh): known-cluster gate + Matches on the PSK provider Phase 1A security register items 1 and 2, landed ahead of the command fan-out because they are independently testable and the fan-out is the first caller of TryGetPsk. Item 1 — known-cluster gate. MeshPskProvider gains SetKnownClusters, an atomic whole-set swap consulted by TryGetPsk BEFORE the config map and before the secret store. A cluster id outside the set is refused in O(1) with no store read and no audit write, which closes two things at once: the pre-auth store/audit flood an unauthenticated caller could drive with a forged x-otopcua-cluster header, and the timing oracle that defeated CentralCommandAuthInterceptor's uniform denial text (unknown clusters are never cached, so they paid a store read on every probe while known ones answered from memory). Empty or never-published leaves the gate inactive — central publishes the set from its DB contact refresh, and 'the DB told me nothing' must not become a fleet-wide authentication outage during admin boot. Item 2 — bool Matches(clusterId, presented) on IMeshPskProvider, with FixedTimeEquals inside the provider. CentralCommandAuthInterceptor now calls it and never holds key material; its two provider-side denial logs collapse into one (the provider answers both as false) and the operator hint is restated there. No production caller publishes a cluster set yet, so behaviour is unchanged until the fan-out commit wires the contact refresh. ControlPlane.Tests 190 -> 205, Host.Tests 91 -> 92.	2026-08-06 05:31:08 -04:00
Joseph Doherty	2edef72f6f	docs(plans): 1A.1 close-out — reviews ✅, plan FromEnvelope wording fix	2026-08-06 05:19:07 -04:00
Joseph Doherty	64774652b7	feat(mesh): NodeCommandService on the telemetry listener	2026-08-06 05:14:12 -04:00