lmxopcua/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md

ClusterClient → gRPC migration — live gate record

Rig: docker-dev/ (central-1/-2 = admin,driver,cluster-MAIN; site-a-1/-2, site-b-1/-2; sql + traefik + migrator + cluster-seed), rebuilt from the branch under test via docker compose -f docker-dev/docker-compose.yml up -d --build. Recorded check-by-check in the family's live-gate format. Later phases' sections are appended as those phases land; every check row gets a PASS/FAIL verdict plus one quantitative justification clause, with fenced evidence (real commands, real output). A run invalidated by a concurrent rig rebuild is void — note it and re-run (the ScadaBridge void-run rule).

Provenance discipline: every phase section opens with branch + commit SHA + "image rebuilt, all N containers recreated", and Phase 4 additionally verifies the image build timestamp post-dates the deletion merge (a stale image invalidates the boundary checks).

Plan: 2026-08-06-clusterclient-to-grpc.md. Tracking: 2026-08-06-clusterclient-to-grpc-tracking.md.

---

Baseline (run on the PRE-change build, before the Phase 2 flip)

The two gaps this program closes, reproduced rather than assumed:

(a) Unauthenticated Akka remoting crosses the WAN boundary. From the host:

# A live TCP association from a site container to central's Akka port:
docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
# And the receptionist actually accepting cross-mesh traffic (central log after any AdminUI
# driver restart targeting site-a):
docker logs otopcua-dev-central-1-1 2>&1 | grep -i "clusterclient\|receptionist" | tail -5

Expected: OPEN; receptionist/client lines present. Nothing on this path presents any credential.

(b) Silent drop on an unreachable cluster. Stop the site-b pair, dispatch a deployment (POST http://localhost:9200/api/deployments, X-Api-Key: docker-dev-deploy-key), observe: no per-node dispatch error surfaces anywhere; site-b's NodeDeploymentState rows sit Applying until the 2-minute deadline marks the deployment TimedOut/partially failed. The deadline is the only witness. Restart the pair.

• Baseline (a) recorded with evidence — PENDING
• Baseline (b) recorded with evidence — PENDING

---

Phase 0 — contract + auth + config groundwork — PASS

Branch feat/grpc-phase0-contract-auth @ c2bfad4a. Image rebuilt, all six containers recreated (docker compose -f docker-dev/docker-compose.yml up -d --build, run from the worktree at /Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-phase0 so the image carries this branch). The rig was already running (from the main checkout's compose file) before this run; the rebuild recreated all eight containers (sql kept, migrator + cluster-seed one-shots re-ran, central-1/-2 + site-a-1/-2 + site-b-1/-2 + traefik recreated) under the fixed otopcua-dev project name, so this run is not a concurrent-rebuild-invalidated one — nothing else touched the stack afterwards.

#	Check	Result
1	All six nodes boot on ClusterClient with the dark Grpc config present (PSKs, endpoints, ClusterPsks map); no validator fires	PASS — 6/6 nodes restarts=0, docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' reports running restarts=0 on every node; `docker logs
2	Deliberate misconfig: a scratch container with MeshTransport__Mode=Grpc and no GrpcPsk refuses to boot, error names MeshTransport:GrpcPsk + the env form	PASS — Hosting failed to start + OptionsValidationException naming MeshTransport:GrpcPsk and MeshTransport__GrpcPsk, below
3	Same with empty CentralGrpcEndpoints — refuses, names the key	PASS — OptionsValidationException naming MeshTransport:CentralGrpcEndpoints and MeshTransport__CentralGrpcEndpoints__0, below
4	Full suite green, 0 warnings, on the branch	PASS — dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo: 0 failures on a clean run (see evidence below); no .cs files changed on this branch since the last full rebuild, so the pre-existing ~868-warning baseline is unchanged (nothing to newly warn about)

Note: the new gRPC services are not yet mapped in Phase 0, so no wire probes here — the interceptor negative/positive matrix is unit-tested and wire-probed in 1A/1B.

Check 1 evidence — no validator fired on any of the six nodes, all dark Grpc keys present:

$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do
    docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' otopcua-dev-$c-1
  done
running restarts=0    # central-1
running restarts=0    # central-2
running restarts=0    # site-a-1
running restarts=0    # site-a-2
running restarts=0    # site-b-1
running restarts=0    # site-b-2

$ docker logs otopcua-dev-central-1-1 2>&1 | grep -iE "MeshTransport|ClusterClient"
[08:53:26 INF] Mesh ClusterClient for cluster MAIN created with 2 contact point(s)
[08:53:26 INF] Mesh ClusterClient for cluster SITE-A created with 2 contact point(s)
[08:53:26 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s)

$ docker logs otopcua-dev-site-a-1-1 2>&1 | grep -iE "cluster.client|receptionist"
[08:53:26 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]

$ docker inspect otopcua-dev-site-a-1-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep MeshTransport
MeshTransport__CentralGrpcEndpoints__1=http://central-2:4055
MeshTransport__CentralGrpcEndpoints__0=http://central-1:4055
MeshTransport__Mode=ClusterClient
MeshTransport__GrpcPsk=dev-mesh-psk-SITE-A
MeshTransport__CentralContactPoints__1=akka.tcp://otopcua@central-2:4053
MeshTransport__CentralContactPoints__0=akka.tcp://otopcua@central-1:4053

$ curl -sf http://localhost:9200/health/ready
{"status":"Healthy", ..., "akka":{"status":"Healthy","description":"Akka cluster member status: Up",
 "data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","selfRoles":["admin","cluster-MAIN","driver"],
 "memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}}}

Check 2 evidence — scratch container from otopcua-host:dev, site-a-1's real environment with MeshTransport__GrpcPsk removed and Mode forced to Grpc:

$ docker run --rm --name scratch-check2-noPsk --network otopcua-dev_default \
    --env-file check2-env.txt otopcua-host:dev
[08:54:23 ERR] Hosting failed to start
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:GrpcPsk is empty. Under
Grpc this per-cluster pre-shared key is the whole authentication boundary of the command plane
in both directions — without it this node's inbound command service fails closed and every ack
it posts to central is rejected. Supply it via the environment (MeshTransport__GrpcPsk), in
production as ${secret:OT-MESH-PSK-<clusterId>}.
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   ...

Check 3 evidence — same base env, GrpcPsk present, both CentralGrpcEndpoints indices unset:

$ docker run -d --name scratch-check3-noEndpoints --network otopcua-dev_default \
    --env-file check3-env.txt otopcua-host:dev
$ docker logs scratch-check3-noEndpoints 2>&1 | grep -A2 "OptionsValidationException"
[08:55:55 ERR] Hosting failed to start
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:CentralGrpcEndpoints is
empty. Under Grpc a driver node has nowhere to post its deployment ack, and every deployment
would time out naming a node that is running perfectly. Supply them via the environment
(MeshTransport__CentralGrpcEndpoints__0).

Both scratch containers removed after capture (docker rm -f scratch-check2-noPsk scratch-check3-noEndpoints); the rig's six real nodes were never affected (still restarts=0 after both runs).

Check 4 evidence — full suite, clean re-run (a first run hit one race-condition flake in RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed, which the test's own name identifies as exercising a race; re-ran in isolation and it passed, then a full second run passed clean end to end — pre-existing timing sensitivity, not a regression from this branch, which touched only docker-dev/docker-compose.yml and docs/Configuration.md, no .cs files):

$ dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo
...
Passed!  - Failed: 0, Passed: 212, Skipped: 4, Total: 216, Duration: 3m56s - ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.dll (net10.0)
$ echo $?
0

Rig left running on ClusterClient (its normal state) after the gate.

---

Phase 1A — central→node command plane — PASS (on the re-run at e54883a8)

Two runs. Run 1 @ fbf7b708 FAILED at the central boot boundary (defect 4) and could not drive checks 1/2/7 or the central halves of 3/4; its checks 5 + 6 and the node-side halves of 3 + 4 stand as recorded and were not repeated. Run 2 @ e54883a8 — after the boot fix (c0475f2d) and its review-hardening follow-up (e54883a8) — drove every carried check. The result table below is the combined, current verdict; run 1's failure record is preserved underneath it because the defect it caught is the most valuable thing this gate produced.

#	Check	Verdict	Run
—	Boot: all six nodes healthy INCLUDING the central pair, on the branch image	PASS — 6/6 running restarts=0, 0 Hosting failed to start, /health/ready 200 ×6, MAIN mesh memberCount=2 unreachableCount=0. But the Resolve Error line the fix logs DID fire — see finding 5	2
1	Deploy dispatch reaches all six driver nodes over gRPC; zero SendToAll/ClusterClient on central	PASS — 6/6 nodes accepted one DispatchDeployment under a single shared correlationId=cf499b42…; central's own per-node outcome log records 6 accepted / 0 failed; grep -ci clusterclient = 0 on both central nodes	2
2	Acks return over ClusterClient (site mode unchanged) and the deployment seals	PASS — deployment 8DF9AAE0… Status=2 (Sealed) with all six NodeDeploymentState rows Status=1 (Applied) inside ~90 ms; all four site nodes hold live Connected to [akka.tcp://…/system/receptionist] associations, which is their only ack path in 1A	2
3	RestartDriver + ReconnectDriver land on the target site node	PASS (composed) — node-side leg driven live in run 1 (both commands acted on site-a-1). The AdminUI half cannot be driven and this is now proven, not assumed: <DriverStatusPanel has 0 usages repo-wide, seven rendered pages contain no Restart/Reconnect text or button, and the only non-script-analysis HTTP POST in the whole app is /api/deployments. Closed by composition — check 4 proves the same AdminOperationsActor.Dispatch → DispatchOverGrpc origin path live from central for an admin-originated command	1 + 2
4	AlarmCommand from the AdminUI lands and the condition acknowledges	PASS — Ack clicked on the real /alerts page against a live gate481-alarm ACTIVATED row; central fanned one AlarmCommand (correlationId=ccafe375…, minted once) to all six over gRPC, and the condition acknowledged — a new ACKNOWLEDGED row appeared 0.54 s later carrying the operator's identity (multi-role-test)	2
5	Unauthenticated ⇒ PermissionDenied; correct dev PSK ⇒ accepted	PASS — 3/3 probes as designed (run 1; not repeated — nothing in the fix touches the interceptor)	1
6	Wrong-key isolation	PASS — both directions rejected, with a positive control proving the rejection is the key and not reachability (run 1)	1
7	NULL GrpcPort node skipped with the warn-once; fan continues	PASS — with site-b-2:4053 NULLed, both central nodes logged the warn once each and the next dispatch (correlationId=88a9e0a2…) was accepted by exactly 5 nodes, site-b-2 absent; the deployment then honestly ended Status=4 (TimedOut) with site-b-2 alone at Status=0. Row restored; a later dispatch sealed 6/6	2
8	Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures	PASS, and no longer vacuous — 0 denials on all four interceptors across all six nodes, and all three planes carried real traffic this time: site nodes fetched + SHA-verified 28 KB artifacts from http://central-1:4055, Telemetry stream opened on every node with live driver health rendering on /hosts, LocalDb sync active on the site pairs	2

Run 2 provenance

docker compose -f docker-dev/docker-compose.yml up -d --build from the worktree, exit 0; image otopcua-host:dev rebuilt to 1d8452a29018 at 12:15 UTC (run 1's preserved otopcua-host:grpc-1a was stale and deliberately not reused). The six host nodes plus migrator and cluster-seed were recreated (both one-shots exit 0); sql and its volume were kept, and traefik was reused unchanged (its service definition did not change). Nothing else touched the stack during the run, so it is not concurrent-rebuild-invalidated. Same rig-only flip and same three Serilog Debug overrides as run 1; both reverted afterwards, never committed.

$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
    --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
central-1  MeshTransport__Mode=Grpc
central-2  MeshTransport__Mode=Grpc
site-a-1   MeshTransport__Mode=ClusterClient
site-b-1   MeshTransport__Mode=ClusterClient

Run 2 evidence

Boot — the fix works, and the race it was hardened against is real.

$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1  running restarts=0 mode=Grpc hostingFailed=0 resolveErr=2
central-2  running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0
site-a-1   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-a-2   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-b-1   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-b-2   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0

See finding 5 below for the two resolveErr lines on central-1 — they are the PreStart race losing on a real boot, and the memoize retry healing it was verified live rather than argued.

Check 1 — the fan, both ends. One envelope, one correlation id, six recipients:

$ for c in …; do docker logs otopcua-dev-$c-1 | grep "Mesh: accepted DispatchDeployment" | tail -1; done
central-1  [12:16:49 DBG] Mesh: accepted DispatchDeployment (correlationId=cf499b42c6914a578035b2b5bc1020f7) over gRPC; handed to the node communication actor.
central-2  [12:16:49 DBG] …same correlationId…
site-a-1   [12:16:49 DBG] …same correlationId…
site-a-2   [12:16:49 DBG] …same correlationId…
site-b-1   [12:16:49 DBG] …same correlationId…
site-b-2   [12:16:49 DBG] …same correlationId…

$ docker logs otopcua-dev-central-2-1 | grep "accepted by"        # the node hosting the coordinator
[12:16:49 DBG] Mesh: DispatchDeployment (correlationId=cf499b42…) accepted by central-2:4053 over gRPC
[12:16:49 DBG] … accepted by site-b-1:4053 …   [12:16:49 DBG] … accepted by site-a-2:4053 …
[12:16:49 DBG] … accepted by site-a-1:4053 …   [12:16:49 DBG] … accepted by central-1:4053 …
[12:16:49 DBG] … accepted by site-b-2:4053 …
$ docker logs … | grep -c "was NOT delivered to"   → 0 on both central nodes

# Zero ClusterClient on the command side, both central nodes:
central-1  MeshClusterClientCreated=0 clusterclientAnyLine=0
central-2  MeshClusterClientCreated=0 clusterclientAnyLine=0

The clusterclientAnyLine=0 count is over the whole log with Serilog:MinimumLevel:Override:Akka.Cluster.Tools.Client=Debug in force, so it covers Akka's own ClusterClient internals and not merely this repo's creation message.

Check 2 — mixed transport, end to end. Commands out over gRPC (above), acks back over ClusterClient:

$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='8df9aae0-…'
2                                    -- Sealed
$ … SELECT NodeId, Status, AppliedAtUtc FROM dbo.NodeDeploymentState WHERE DeploymentId='8df9aae0-…'
central-1:4053|1|2026-08-06 12:16:49.939     site-a-1:4053|1|2026-08-06 12:16:49.977
central-2:4053|1|2026-08-06 12:16:49.915     site-a-2:4053|1|2026-08-06 12:16:49.950
site-b-1:4053|1|2026-08-06 12:16:49.957      site-b-2:4053|1|2026-08-06 12:16:49.967

$ for c in site-a-1 site-a-2 site-b-1 site-b-2; do docker logs … | grep "receptionist" | tail -1; done
site-a-1   [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
site-a-2   [12:15:20 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
site-b-1   [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
site-b-2   [12:15:25 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
$ grep -c "No central ClusterClient" → 0 on all six      # no ack was dropped for want of a client

A site node has no gRPC ack path in 1A (that is 1B), so the receptionist association above is not merely present, it is the only route those four Applied rows could have taken.

Check 3 — why the AdminUI half is undrivable, established rather than asserted:

$ grep -rn "<DriverStatusPanel" --include="*.razor" --include="*.cs" src/ | wc -l
0                                    # the component holding the buttons is referenced nowhere
$ grep -rn "DriverStatusPanel" --include="*.razor" src/ | grep -v "Shared/Drivers/DriverStatusPanel.razor:"
…5 hits, all of them code COMMENTS in Alerts.razor / Hosts.razor ("mirrors DriverStatusPanel…")

$ node scan-ui.js          # every page a Designer/Administrator/Operator can reach
/            | reconnect/restart text: false | buttons: ["Sign out"]
/hosts       | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
/fleet       | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
/clusters    | reconnect/restart text: false | buttons: ["Sign out"]
/clusters/SITE-A | reconnect/restart text: false | buttons: ["Sign out"]
/alerts      | reconnect/restart text: false | buttons: ["Sign out","Clear"]
/deployments | reconnect/restart text: false | buttons: ["Sign out","Deploy current configuration"]

$ grep -rn "MapPost|MapPut|MapDelete" src/…/AdminUI src/…/Host
…/Api/DeployApiEndpoints.cs:43:  app.MapPost("/api/deployments", …)
…/ScriptAnalysis/ScriptAnalysisEndpoints.cs:19-24:  /diagnostics /completions /hover /signature-help /format /inlay-hints

There is no button and no endpoint. What closes the check is that RestartDriver, ReconnectDriver and AlarmCommand all leave central through the same three lines — AdminOperationsActor.Dispatch → MeshCommand → CentralCommunicationActor.DispatchOverGrpc — and check 4 drove that path live from the AdminUI for AlarmCommand, while run 1 drove the node-side decode-and-act for the two driver commands. The only per-type difference left is MeshTransportDtoMapper.ToEnvelope, which the T0.2 goldens cover member-for-member.

Check 4 — AdminUI ack, fanned by central, acknowledged by the owner. The rig's one scripted alarm sits Normal indefinitely (hr200 > 70000), so a transition was manufactured by editing its predicate to an always-true form and deploying; the /alerts page was already open when it fired, the Ack button was clicked, and the predicate + hash were restored afterwards (verified byte-identical by SELECT) with a further deploy:

page open at 2026-08-06T12:20:59.036Z        initial: empty window
ROWS:  12:21:00.017  gate481-alarm | PressHigh | EQ-6e09bcc2d907 | ACTIVATED | 750 | system | press high
ROW BUTTONS: ["Ack","Shelve","Shelve (timed)","Unshelve"]
clicking Acknowledge at 2026-08-06T12:21:00.362Z
after click:  12:21:00.557 | gate481-alarm | EQ-6e09bcc2d907 | ACKNOWLEDGED | 750 | multi-role-test | press high

$ docker logs otopcua-dev-central-2-1 | grep AlarmCommand
[12:21:00 DBG] Mesh: AlarmCommand (correlationId=ccafe3754aca40cfa8aa0a0602c86cbb) accepted by site-b-2:4053 over gRPC
[12:21:00 DBG] … accepted by central-1:4053 …   [12:21:00 DBG] … accepted by central-2:4053 …
[12:21:00 DBG] … accepted by site-a-1:4053 …    [12:21:00 DBG] … accepted by site-b-1:4053 …
[12:21:00 DBG] … accepted by site-a-2:4053 …
$ for c in …; do grep "Mesh: accepted AlarmCommand" …; done   → all six, same correlationId

One correlation id across all six recipients is the "encode once, before the fan" rule holding for the one command whose CLR record has no correlation member of its own — the case the design called out as the reason encoding is hoisted out of the loop.

Check 7 — the skip, the warn, and the honest deadline.

$ … UPDATE dbo.ClusterNode SET GrpcPort=NULL WHERE NodeId='site-b-2:4053'    # 12:23:14

# one warn per central node, once each, within one ContactRefreshSeconds:
central-2  [12:23:20 WRN] ClusterNode site-b-2:4053 has no GrpcPort; it exposes no command endpoint
           and is skipped in this dispatch set (further skips of this node are silent). Commands
           will not reach it until the row carries a port
central-1  [12:23:19 WRN] …identical…
# (each node also logs the TelemetryNodeSource twin — "exposes no telemetry stream … skipped in
#  this dial refresh" — expected collateral: both target sets read the same column)

# dispatch at 12:23:36 — accepted by five, site-b-2 absent:
central-2  [12:23:37 DBG] … (correlationId=88a9e0a2…) accepted by central-2 / site-a-2 / site-b-1
                          / site-a-1 / central-1 :4053 over gRPC
# cumulative per-node accept counts after this dispatch — site-b-2 one behind everyone:
central-1 4   central-2 4   site-a-1 4   site-a-2 4   site-b-1 4   site-b-2 3

$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='5c8bdc9e-…'   → 4  (TimedOut)
$ … SELECT NodeId, Status FROM dbo.NodeDeploymentState …
central-1|1  central-2|1  site-a-1|1  site-a-2|1  site-b-1|1  site-b-2|0

Row restored to 4056 and a later dispatch sealed 6/6 (080EBB0B…, Status=2, every node Status=1). One operational detail worth carrying: the skip persists for up to a full ContactRefreshSeconds after the row is repaired — a deploy issued 5 s after the restore (DF1C0B74…) was still skipped and still TimedOut, because the dispatch set is only rebuilt on the refresh tick.

Check 8 — denials, and proof the planes were actually used.

$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1  telemetry=0 configserve=0 localdbsync=0 mesh=0
central-2  telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-1   telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-2   telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-1   telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-2   telemetry=0 configserve=0 localdbsync=0 mesh=0

# config-serve carried real, authenticated traffic:
site-a-1  [12:20:59 INF] Fetched + verified artifact for 8d486c6b… from http://central-1:4055 (28313 bytes).
# telemetry likewise, end to end into the UI:
$ for c in …; do grep -c "Telemetry stream opened"; done → 3,4,3,3,3,5
/hosts →  CLUSTER SITE-A … sa-modbus2 SITEA-modbus Modbus HEALTHY 1 host 1 tag  12:30:49 UTC  0 errors

The five telemetry failed … reconnecting warnings in the run are all accounted for: three at 12:15:20 (central dialing nodes that had not finished binding :4056) and two at 12:28:50 / 12:29:30 (the deliberate central-2 stop in finding 5). None is an auth failure.

Phase 1A finding 5 — the PreStart race loses on real boots, and the memoize is what saves it

The hardening commit's premise is not hypothetical. On the run-2 boot, central-1 lost the race on both dependencies:

$ docker logs otopcua-dev-central-1-1 | grep -A3 "Failed to resolve"
[12:15:19 ERR] Failed to resolve IMeshCommandDialer for the mesh command boundary; continuing
               without it. The guards below report what that costs
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
   at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
[12:15:19 ERR] Failed to resolve IMeshPskProvider for the mesh command boundary; …same cause…

It is not rare: across the three admin boots observed in run 2 (central-1 and central-2 at 12:15, central-2 again at 12:29 after the stop/start), two of three lost it. Had the fix stopped at PreStart-only resolution, those nodes would have held a permanent silent null — no dialer, and a never-published known-cluster gate — and the failure would only have surfaced the first time the coordinator singleton moved onto such a node.

The memoize heals it, verified on the node that actually lost the race. Run 2's dispatches were all fanned by central-2 (the singleton host), so central-1's dialer was never exercised by them. Central-2 was therefore stopped to move the singleton, and a dispatch driven while central-1 alone held it:

$ docker compose stop central-2                                   # 12:28:39
$ curl … /health/ready → self akka.tcp://otopcua@central-1:4053, members 1     # 12:28:57
$ curl -X POST /api/deployments  -d '{"createdBy":"1a-gate-central1-heal"}'    # 12:29:01
$ docker logs otopcua-dev-central-1-1 | grep -E "accepted by|no command dialer is wired"
[12:29:02 DBG] Mesh: DispatchDeployment (correlationId=ad97d386…) accepted by site-b-1:4053 over gRPC
[12:29:02 DBG] … accepted by site-a-2 … site-b-2 … site-a-1 … central-1 …      # 5 live nodes
# no "no command dialer is wired" line, and no further "Failed to resolve"
$ docker compose start central-2  → members 2, unreachable 0, status Healthy   # 12:29:51

So the boot-time Error is cosmetically alarming but functionally healed, and that is now an observed property rather than a design claim. Two follow-ups for Phase 2 rather than for 1A: the Error text says "continuing without it", which reads as permanent and should say that the first use retries; and a Failed to resolve line on a healthy node is exactly the kind of signal a soak's PSK-drift/error grep will trip over.

One incidental observation, not a check. The deployment RevisionHash is deterministic (three separate no-change dispatches all produced 642b9d6f79b9…), but restoring the alarm predicate's two text columns did not restore the pre-gate hash 6178a8f302aa… — so the artifact covers a column the UPDATE bumped (Script.RowVersion is the obvious candidate). The rig's configuration content is byte-identical to its pre-gate state; only the hash moved. Not investigated further — out of this program's scope.

Run 1 record — the defect that blocked it

Branch feat/grpc-node-command @ fbf7b708. Image rebuilt from this worktree (/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a) so it carries this branch; docker compose -f docker-dev/docker-compose.yml up -d --build exited 0 and all eight containers were recreated — migrator and cluster-seed re-ran to completion (exit 0), the six host nodes and traefik were recreated, the sql container and its named volume were kept. Nothing else touched the stack afterwards, so this run is not a concurrent-rebuild-invalidated one. Rig-only flip: central-1/-2 MeshTransport__Mode: "Grpc" (hard value); site nodes left on the compose default ClusterClient (coexistence is the point). Three Serilog MinimumLevel:Override Debug scopes were added on all six nodes for this run only (ZB.MOM.WW.OtOpcUa.Host.Grpc, ZB.MOM.WW.OtOpcUa.ControlPlane.Communication, Akka.Cluster.Tools.Client) — the accept/dispatch lines the checks read are Debug and the rig's default Serilog level is Information, so without them checks 1/3 would have had no observable signal. Flip and overrides reverted after the gate, never committed.

Run 1 verdict: FAIL. Neither central node booted, so nothing that originates at central could be driven: checks 1, 2 and 7 were BLOCKED outright, 3 and 4 reached only their node-side legs, and check 8's zero-denial result was vacuous for telemetry and config-serve (central dials both and never booted). Checks 5 and 6 PASSED and are the run-1 evidence carried into the combined table above. Root cause was defect 4 — resolving IMeshCommandDialer / IMeshPskProvider inside the first WithActors starter drags in ISecretResolver → IAuditWriter → ActorAuditWriter, whose constructor reads AuditWriterActorKey out of the ActorRegistry before the singleton that registers it has run. It was mode-independent (reproduced on the same image with MeshTransport__Mode=ClusterClient), so it was a defect of the branch, not of the gate's flip, and it would have failed a ClusterClient rig equally.

Run 1 evidence

Provenance — flip in effect on central only, sites untouched:

$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
    --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
central-1  MeshTransport__Mode=Grpc
central-2  MeshTransport__Mode=Grpc
site-a-1   MeshTransport__Mode=ClusterClient
site-b-1   MeshTransport__Mode=ClusterClient

$ docker images --format '{{.Repository}}:{{.Tag}}|{{.ID}}|{{.CreatedAt}}' | grep otopcua-host
otopcua-host:dev|69360a173f1f|2026-08-06 07:20:14 -0400 EDT     # rebuilt for this run

Defect 4 — the boot failure (checks 1/2/7 blocked by it). Both central nodes, identical:

$ docker logs otopcua-dev-central-1-1 2>&1 | sed -n '25,45p'
[11:21:27 INF] OPC UA server started on opc.tcp://0.0.0.0:4840
[11:21:27 FTL] Unable to start AkkaHostedService - shutting down application
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
   at Akka.Hosting.ActorRegistry.Get[TKey]()
   at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
        in /src/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Audit/ActorAuditWriter.cs:line 24
   ...
   at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_5(IServiceProvider sp)
   ...
   at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_6(IServiceProvider sp)
   ...
   at ZB.MOM.WW.OtOpcUa.ControlPlane.ServiceCollectionExtensions.<>c.<WithOtOpcUaControlPlaneSingletons>b__7_0(
        ActorSystem system, IActorRegistry registry, IDependencyResolver resolver)
        in /src/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs:line 74
   at Akka.Hosting.AkkaConfigurationBuilder.StartAsync(ActorSystem sys)
   at Akka.Hosting.AkkaHostedService.StartAsync(CancellationToken cancellationToken)

Mode-independence, proven rather than argued — same image, central-1's exact environment with MeshTransport__Mode=ClusterClient, own hostname + self-seed so it cannot disturb the rig:

$ docker run -d --name scratch-cc-mode --network otopcua-dev_default --env-file c1-env.txt \
    -e Cluster__PublicHostname=scratch-cc-mode \
    -e Cluster__SeedNodes__0=akka.tcp://otopcua@scratch-cc-mode:4053 otopcua-host:dev
$ docker logs scratch-cc-mode 2>&1 | grep -m4 -E "MissingActorRegistry|Hosting failed|Host starting"
[11:28:45 INF] OtOpcUa.Host starting with roles=[admin,driver,cluster-main] (admin=True, driver=True)
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
[11:28:47 ERR] Hosting failed to start

Scratch container removed after capture (docker rm -f scratch-cc-mode); the six real nodes were untouched by it.

It presents as a healthy container. Worth carrying into Phase 2/4 readings — the failed process does not exit, so docker ps shows Up, RestartCount=0, and only an HTTP probe or the log tells the truth:

$ docker inspect otopcua-dev-central-1-1 -f '{{.State.Status}} restarts={{.RestartCount}}'
running restarts=0
$ docker run --rm --network otopcua-dev_default curlimages/curl -s -o /dev/null \
    -w '%{http_code}\n' --max-time 5 http://central-1:9000/health/ready
000
$ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9200/health/ready
503

Checks 5 + 6 — the wire-probe auth matrix (-import-path onto the in-repo proto; the node has no reflection). An empty CommandEnvelope has no oneof arm set, so FromEnvelope throws and the service answers InvalidArgument — after the interceptor ran, which is exactly why it is the positive signal here: PermissionDenied means the key was refused, InvalidArgument means the key was accepted and only the payload was rejected:

$ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos
$ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \
     -plaintext -import-path /protos -proto mesh_transport.proto"

# 5a  no bearer                        -> site-a-1
Code: PermissionDenied   Message: Node command authentication failed.
# 5b  -H 'authorization: Bearer dev-mesh-psk-SITE-A'   -> site-a-1
Code: InvalidArgument    Message: Command envelope could not be decoded.
# 6a  -H 'authorization: Bearer dev-mesh-psk-SITE-B'   -> site-a-1
Code: PermissionDenied   Message: Node command authentication failed.
# 6b  -H 'authorization: Bearer dev-mesh-psk-SITE-A'   -> site-b-1
Code: PermissionDenied   Message: Node command authentication failed.
# 6c  -H 'authorization: Bearer dev-mesh-psk-SITE-B'   -> site-b-1   (positive control)
Code: InvalidArgument    Message: Command envelope could not be decoded.

Checks 3 + 4 — the node-side leg, driven with real envelopes over the same authenticated h2c path central would use. This is the first live proof that MeshCommandAuthInterceptor is actually invoked on a real Kestrel pipeline (register item 4) and that NodeCommandGrpcService → NodeCommunicationActor → EventStream → DriverHostActor works end to end:

$ $G -H 'authorization: Bearer dev-mesh-psk-SITE-A' -H 'x-otopcua-cluster: SITE-A' \
     -d '{"correlationId":"8461008e...","restartDriver":{"clusterId":"SITE-A",
          "driverInstanceId":"SITEA-modbus","actorByUserName":"1a-gate",
          "correlationId":"8461008e..."}}' \
     site-a-1:4056 mesh_transport.v1.NodeCommandService/Execute
{}

$ docker logs otopcua-dev-site-a-1-1 2>&1 | tail
[11:31:01 DBG] Mesh: accepted RestartDriver (correlationId=8461008e37654724a292219202d78dfa) over gRPC; handed to the node communication actor.
[11:31:01 INF] DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate
[11:31:01 INF] DriverHost site-a-1:4053: spawned Modbus driver SITEA-modbus (stub=False)
[11:31:55 DBG] Mesh: accepted ReconnectDriver (correlationId=29c5dc77283f4eb7869d0c7d6686cadd) over gRPC; handed to the node communication actor.
[11:31:55 INF] DriverHost site-a-1:4053: reconnecting driver SITEA-modbus by request of 1a-gate
[11:31:55 INF] DriverInstance SITEA-modbus: ForceReconnect requested by admin; re-entering Reconnecting
[11:31:56 DBG] Mesh: accepted AlarmCommand (correlationId=39ba451173954229a71b9dd1e00ac1b9) over gRPC; handed to the node communication actor.

Check 8 — interceptor denial counts across every node, whole run:

$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1  telemetry=0 configserve=0 localdbsync=0 mesh=0
central-2  telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-1   telemetry=0 configserve=0 localdbsync=0 mesh=2   # the two deliberate negatives
site-a-2   telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-1   telemetry=0 configserve=0 localdbsync=0 mesh=1   # the one deliberate negative
site-b-2   telemetry=0 configserve=0 localdbsync=0 mesh=0

The TwoNodeClusterHarness I-1 null-provider Error (a known test-harness artifact) did not appear in any rig log — grep -c "IMeshPskProvider" = 0 on all six nodes, as expected for production wiring where every admin node registers the provider.

Phase 1A defect — CLOSED by c0475f2d + e54883a8, re-gated in run 2

Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter. WithOtOpcUaControlPlaneSingletons's WithActors block resolves IMeshCommandDialer and IMeshPskProvider from IServiceProvider to hand to CentralCommunicationActor.Props. MeshGrpcCommandDialer takes IMeshPskProvider, which takes ISecretResolver, which AddZbSecrets builds by resolving IAuditWriter, which on an admin node is ActorAuditWriter — and its constructor does registry.Get<AuditWriterActorKey>(). That singleton is registered ~60 lines later in the same builder, so at this point the key is absent and ActorRegistry.Get throws MissingActorRegistryEntryException, failing AkkaHostedService.StartAsync and the whole host.

Three things make this worth a pinned regression rather than a one-line fix note:

• It is not mode-gated. The two GetService calls run in every transport mode by design (so a mode flip needs no restart), so the eager graph is walked on every admin boot. Proven above with a ClusterClient scratch container on the same image.
• It is invisible to the suite. The 1A.2 container-activation test (It_activates_from_the_container_despite_the_optional_client_factory_parameter) resolves the dialer from a service collection that does not include AddZbSecrets, and TwoNodeClusterHarness registers no IMeshPskProvider at all — so no existing test walks the real graph in starter order. This is the same class as scadaproj#1 ("only visible with the DI graph built in a real host") and as mesh Phase 6's IClusterRoleInfo boot crash.
• The container lies about it. The process does not exit, so docker ps reports Up with restarts=0 while the node serves no HTTP at all. Any Phase 2/4 readiness check that reads container status rather than /health/ready would score this as healthy.

The shape of the fix is the one this file already uses two lines above for the coordinator proxy — hand the actor a lazy accessor (Func<IMeshCommandDialer?> / Func<IMeshPskProvider?>, or resolve on first use) so nothing is constructed while the ActorSystem is still building.

That is what shipped. c0475f2d moved both dependencies behind Func<> invoked in PreStart, pinned by ControlPlaneStarterOrderTests (boots a real admin host through the production starter order and asserts StartAsync; verified RED against the eager resolve in 3/3 modes). e54883a8 then made the resolution memoize-on-first-use rather than PreStart-only, because PreStart runs concurrently with the remaining starters and can still lose the race — which run 2 observed happening on 2 of 3 real admin boots, so that hardening was not theoretical. Both are re-gated above; see finding 5 for the live self-heal proof.

Rig state after run 1 (superseded by the run-2 restoration below)

The compose flip was reverted and the six nodes recreated on the committed all-ClusterClient compose. Because defect 4 was mode-independent, the branch image could not produce a healthy central pair on any mode, so the rig was returned to its pre-gate image rather than left with a dead central pair: the run-1 build was preserved as otopcua-host:grpc-1a (69360a173f1f) and the pre-gate build (8bd81d8aea9b) re-tagged back to otopcua-host:dev.

Rig state after run 2 — restored, six healthy, on the branch image

The flip was reverted (git checkout docker-dev/docker-compose.yml) and the six nodes recreated on the committed all-ClusterClient compose. No image substitution was needed this time: otopcua-host:dev is the run-2 build of the committed source (1d8452a29018), and it boots a healthy central pair, so the rig runs the code under test. The run-1 workaround tag otopcua-host:grpc-1a was deleted to stop anyone reusing a stale image.

central-1  running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
central-2  running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-a-1   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-a-2   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-b-1   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-b-2   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
$ curl … /health/ready → status Healthy, members 2, unreachable 0
$ grep -c "Mesh ClusterClient for cluster"  → 3 on each central node (MAIN + SITE-A + SITE-B)

The three ClusterClients reappearing on the rollback is an incidental but useful result: the dark switch still rolls back cleanly on the fixed branch, which is what Phase 2 will depend on if its cutover has to be reversed.

---

Phase 1B — node→central ack plane — PENDING

Branch feat/grpc-central-ack @ ________ (rebased on merged 1A). Rig-only flip: site-a-1/-2 → Grpc; site-b stays ClusterClient; central back on ClusterClient mode (its command direction) — acks are governed by the node's mode, so this isolates 1B.

#	Check	Result
1	Deploy e2e: site-a acks arrive via ReportApply (central CentralCommandService log lines carry site-a node ids), NodeDeploymentState rows land, deployment seals	PENDING
2	Site-b acks still arrive via ClusterClient in the same deployment — per-pair coexistence proven	PENDING
3	Exactly one ack row per site-a node per dispatch (no double-count from endpoint failover)	PENDING
4	Kill central-1 before the apply completes: site-a acks fail over to central-2's endpoint; deployment still seals	PENDING
5	Unauthenticated / wrong-cluster-key / missing x-otopcua-cluster probes against central :4055 CentralCommandService/ReportApply ⇒ PermissionDenied (missing-header rejection is central-side by design)	PENDING
6	ConfigServe (DeploymentArtifactService) on the same listener unaffected: fetches succeed, 0 auth failures	PENDING

---

Phase 2 — full-rig cutover + soak — PENDING

Rig default flipped and committed: MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}". Branch/SHA ________. Run the Baseline section first on the pre-flip build.

#	Check	Result
1	Full rebuild all-Grpc: six nodes ready, deploy e2e seals, all acks via gRPC	PENDING
2	Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes	PENDING
3	Kill site-a-1 mid-dispatch: per-node Execute failure logged with node id + status (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path)	PENDING
4	Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action	PENDING
5	Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; no duplicate applies after the pair returns (boot-replay path only)	PENDING
6	Zero ClusterClient activity under Grpc: docker logs <every node> | grep -ci "clusterclient" = 0 new lines post-boot	PENDING
7	Soak ≥ 2 h under periodic deploys + command traffic: 0 PermissionDenied/Unauthenticated across all six nodes (PSK drift detector), 0 unexplained Unavailable	PENDING
8	Full suite green on the cutover SHA	PENDING

---

Phase 4 — terminal gate on the post-deletion build — PENDING

master @ ________ (after the Phase 3 deletion merge). Provenance guard: image build timestamp verified LATER than the merge commit time — a pre-deletion image would invalidate checks 6–7.

#	Check	Result
1	PSK negatives on both services (no key / wrong key / wrong cluster's key); unset server key ⇒ all calls rejected (fail-closed proven live)	PENDING
2	Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand)	PENDING
3	Ack matrix: every node acks exactly once, deployment seals	PENDING
4	Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry)	PENDING
5	Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart)	PENDING
6	Boundary check: no Akka association between any site container and central — the Baseline (a) probe now shows no cross-mesh :4053 traffic (pair-internal remoting only)	PENDING
7	Full-rig restart (pairs together): clean convergence; grep -ri "receptionist|clusterclient" over all node logs = 0 lines	PENDING
8	MeshGrpcBoundaryTests + full suite green on the merge SHA	PENDING

Carry-forwards

Anything a phase gate could not drive gets named here with why + what proves it instead, and is re-checked in the next gate (the ScadaBridge proportionate-gate rule).

Item	Deferred from	Why it could not be driven	Proven instead by	Status
Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-GrpcPort skip	1A run 1	Both central nodes failed to boot (defect 4)	—	CLOSED in 1A run 2 — all three PASS live
Check 4 acknowledge half — the condition actually acknowledging	1A run 1	The rig's one ScriptedAlarm sits Normal and its owning central node was down	—	CLOSED in 1A run 2 — acked from the real /alerts page, ACKNOWLEDGED row observed
Check 8 for telemetry + config-serve	1A run 1	Central dials both planes and never booted, so neither carried a call — 0 denials was vacuous	—	CLOSED in 1A run 2 — both planes carried authenticated traffic with 0 denials
Check 3 central half — RestartDriver/ReconnectDriver originating at the AdminUI	1A runs 1 + 2	Structurally undrivable, not a rig problem: <DriverStatusPanel has 0 usages repo-wide (the 5 name hits are comments), no rendered page carries a Restart/Reconnect control, and the only non-script-analysis HTTP POST is /api/deployments. Pre-existing on master, not caused by this program	Composition the controller accepted: run 1's live node-side leg for both commands + run 2's check 4, which drives the identical AdminOperationsActor.Dispatch → DispatchOverGrpc origin path from the AdminUI for another command type; per-type difference is ToEnvelope, covered by the T0.2 goldens	CLOSED by composition. The orphaned panel itself is out of this program's scope — file separately if the control is meant to exist
Failed to resolve Error text says "continuing without it", which reads permanent although first use retries; and it will trip a soak's error grep on a healthy node	1A run 2 (finding 5)	Cosmetic/observability, not behaviour	The self-heal proven live (finding 5)	Phase 2

Defects the gates caught that the suite did not

Recorded per phase as they occur, with root cause + the pinned regression test.

#	Phase	Defect	Root cause	Status
4	1A gate run 1	An admin node cannot boot at all — MissingActorRegistryEntryException: AuditWriterActorKey, both central nodes, every transport mode	The first WithActors starter eagerly resolves IMeshCommandDialer/IMeshPskProvider, whose graph reaches ActorAuditWriter, whose ctor reads an ActorRegistry key registered ~60 lines later in the same builder	CLOSED — c0475f2d (Func<> + PreStart, pinned red-first by ControlPlaneStarterOrderTests) + e54883a8 (memoize-on-first-use). Re-gated in run 2: 6/6 nodes boot, 0 Hosting failed to start
5	1A gate run 2	The PreStart resolve loses its race on real boots — 2 of 3 observed admin boots logged Failed to resolve IMeshCommandDialer / IMeshPskProvider with the same AuditWriterActorKey cause	PreStart runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede AuditWriterActorKey's registration	Not a defect — it is the case e54883a8 was written for, and the memoize demonstrably heals it. Proven live by moving the coordinator singleton onto the node that lost the race (stop central-2) and watching it fan a dispatch to all five live nodes with no dialer error. Residual is cosmetic: the Error text and its effect on soak greps → Phase 2