lmxopcua/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md

ClusterClient → gRPC migration — live gate record

Rig: docker-dev/ (central-1/-2 = admin,driver,cluster-MAIN; site-a-1/-2, site-b-1/-2; sql + traefik + migrator + cluster-seed), rebuilt from the branch under test via docker compose -f docker-dev/docker-compose.yml up -d --build. Recorded check-by-check in the family's live-gate format. Later phases' sections are appended as those phases land; every check row gets a PASS/FAIL verdict plus one quantitative justification clause, with fenced evidence (real commands, real output). A run invalidated by a concurrent rig rebuild is void — note it and re-run (the ScadaBridge void-run rule).

Provenance discipline: every phase section opens with branch + commit SHA + "image rebuilt, all N containers recreated", and Phase 4 additionally verifies the image build timestamp post-dates the deletion merge (a stale image invalidates the boundary checks).

Plan: 2026-08-06-clusterclient-to-grpc.md. Tracking: 2026-08-06-clusterclient-to-grpc-tracking.md.

---

Baseline (run on the PRE-change build, before the Phase 2 flip)

The two gaps this program closes, reproduced rather than assumed:

(a) Unauthenticated Akka remoting crosses the WAN boundary. From the host:

# A live TCP association from a site container to central's Akka port:
docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
# And the receptionist actually accepting cross-mesh traffic (central log after any AdminUI
# driver restart targeting site-a):
docker logs otopcua-dev-central-1-1 2>&1 | grep -i "clusterclient\|receptionist" | tail -5

Expected: OPEN; receptionist/client lines present. Nothing on this path presents any credential.

(b) Silent drop on an unreachable cluster. Stop the site-b pair, dispatch a deployment (POST http://localhost:9200/api/deployments, X-Api-Key: docker-dev-deploy-key), observe: no per-node dispatch error surfaces anywhere; site-b's NodeDeploymentState rows sit Applying until the 2-minute deadline marks the deployment TimedOut/partially failed. The deadline is the only witness. Restart the pair.

• Baseline (a) recorded with evidence — PENDING
• Baseline (b) recorded with evidence — PENDING

---

Phase 0 — contract + auth + config groundwork — PASS

Branch feat/grpc-phase0-contract-auth @ c2bfad4a. Image rebuilt, all six containers recreated (docker compose -f docker-dev/docker-compose.yml up -d --build, run from the worktree at /Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-phase0 so the image carries this branch). The rig was already running (from the main checkout's compose file) before this run; the rebuild recreated all eight containers (sql kept, migrator + cluster-seed one-shots re-ran, central-1/-2 + site-a-1/-2 + site-b-1/-2 + traefik recreated) under the fixed otopcua-dev project name, so this run is not a concurrent-rebuild-invalidated one — nothing else touched the stack afterwards.

#	Check	Result
1	All six nodes boot on ClusterClient with the dark Grpc config present (PSKs, endpoints, ClusterPsks map); no validator fires	PASS — 6/6 nodes restarts=0, docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' reports running restarts=0 on every node; `docker logs
2	Deliberate misconfig: a scratch container with MeshTransport__Mode=Grpc and no GrpcPsk refuses to boot, error names MeshTransport:GrpcPsk + the env form	PASS — Hosting failed to start + OptionsValidationException naming MeshTransport:GrpcPsk and MeshTransport__GrpcPsk, below
3	Same with empty CentralGrpcEndpoints — refuses, names the key	PASS — OptionsValidationException naming MeshTransport:CentralGrpcEndpoints and MeshTransport__CentralGrpcEndpoints__0, below
4	Full suite green, 0 warnings, on the branch	PASS — dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo: 0 failures on a clean run (see evidence below); no .cs files changed on this branch since the last full rebuild, so the pre-existing ~868-warning baseline is unchanged (nothing to newly warn about)

Note: the new gRPC services are not yet mapped in Phase 0, so no wire probes here — the interceptor negative/positive matrix is unit-tested and wire-probed in 1A/1B.

Check 1 evidence — no validator fired on any of the six nodes, all dark Grpc keys present:

$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do
    docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' otopcua-dev-$c-1
  done
running restarts=0    # central-1
running restarts=0    # central-2
running restarts=0    # site-a-1
running restarts=0    # site-a-2
running restarts=0    # site-b-1
running restarts=0    # site-b-2

$ docker logs otopcua-dev-central-1-1 2>&1 | grep -iE "MeshTransport|ClusterClient"
[08:53:26 INF] Mesh ClusterClient for cluster MAIN created with 2 contact point(s)
[08:53:26 INF] Mesh ClusterClient for cluster SITE-A created with 2 contact point(s)
[08:53:26 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s)

$ docker logs otopcua-dev-site-a-1-1 2>&1 | grep -iE "cluster.client|receptionist"
[08:53:26 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]

$ docker inspect otopcua-dev-site-a-1-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep MeshTransport
MeshTransport__CentralGrpcEndpoints__1=http://central-2:4055
MeshTransport__CentralGrpcEndpoints__0=http://central-1:4055
MeshTransport__Mode=ClusterClient
MeshTransport__GrpcPsk=dev-mesh-psk-SITE-A
MeshTransport__CentralContactPoints__1=akka.tcp://otopcua@central-2:4053
MeshTransport__CentralContactPoints__0=akka.tcp://otopcua@central-1:4053

$ curl -sf http://localhost:9200/health/ready
{"status":"Healthy", ..., "akka":{"status":"Healthy","description":"Akka cluster member status: Up",
 "data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","selfRoles":["admin","cluster-MAIN","driver"],
 "memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}}}

Check 2 evidence — scratch container from otopcua-host:dev, site-a-1's real environment with MeshTransport__GrpcPsk removed and Mode forced to Grpc:

$ docker run --rm --name scratch-check2-noPsk --network otopcua-dev_default \
    --env-file check2-env.txt otopcua-host:dev
[08:54:23 ERR] Hosting failed to start
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:GrpcPsk is empty. Under
Grpc this per-cluster pre-shared key is the whole authentication boundary of the command plane
in both directions — without it this node's inbound command service fails closed and every ack
it posts to central is rejected. Supply it via the environment (MeshTransport__GrpcPsk), in
production as ${secret:OT-MESH-PSK-<clusterId>}.
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   ...

Check 3 evidence — same base env, GrpcPsk present, both CentralGrpcEndpoints indices unset:

$ docker run -d --name scratch-check3-noEndpoints --network otopcua-dev_default \
    --env-file check3-env.txt otopcua-host:dev
$ docker logs scratch-check3-noEndpoints 2>&1 | grep -A2 "OptionsValidationException"
[08:55:55 ERR] Hosting failed to start
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:CentralGrpcEndpoints is
empty. Under Grpc a driver node has nowhere to post its deployment ack, and every deployment
would time out naming a node that is running perfectly. Supply them via the environment
(MeshTransport__CentralGrpcEndpoints__0).

Both scratch containers removed after capture (docker rm -f scratch-check2-noPsk scratch-check3-noEndpoints); the rig's six real nodes were never affected (still restarts=0 after both runs).

Check 4 evidence — full suite, clean re-run (a first run hit one race-condition flake in RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed, which the test's own name identifies as exercising a race; re-ran in isolation and it passed, then a full second run passed clean end to end — pre-existing timing sensitivity, not a regression from this branch, which touched only docker-dev/docker-compose.yml and docs/Configuration.md, no .cs files):

$ dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo
...
Passed!  - Failed: 0, Passed: 212, Skipped: 4, Total: 216, Duration: 3m56s - ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.dll (net10.0)
$ echo $?
0

Rig left running on ClusterClient (its normal state) after the gate.

---

Phase 1A — central→node command plane — PENDING

Branch feat/grpc-node-command @ ________. Rig-only flip: central-1/-2 → MeshTransport__Mode=Grpc; site nodes remain ClusterClient (coexistence is the point). Flip reverted after the gate, never committed.

#	Check	Result
1	Deploy dispatch from AdminUI reaches all six driver nodes over gRPC (per-node Execute log lines on every node; zero SendToAll lines on central)	PENDING
2	Acks still return over ClusterClient (site mode unchanged) and the deployment seals — mixed-transport deploy proven end to end	PENDING
3	RestartDriver + ReconnectDriver from AdminUI land on the target site node (EventStream delivery observed via driver restart in logs)	PENDING
4	AlarmCommand (ack a scripted alarm from AdminUI) lands and the condition acknowledges	PENDING
5	Unauthenticated grpcurl to a site node :4056 mesh_transport.v1.NodeCommandService/Execute ⇒ PermissionDenied; correct dev PSK ⇒ accepted	PENDING
6	Wrong-key isolation: site-b's PSK presented to site-a ⇒ PermissionDenied (per-cluster scoping is real, not decorative)	PENDING
7	A node with NULL GrpcPort is skipped with the warn-once, fan continues to the others	PENDING
8	Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures on their interceptors across the run	PENDING

---

Phase 1B — node→central ack plane — PENDING

Branch feat/grpc-central-ack @ ________ (rebased on merged 1A). Rig-only flip: site-a-1/-2 → Grpc; site-b stays ClusterClient; central back on ClusterClient mode (its command direction) — acks are governed by the node's mode, so this isolates 1B.

#	Check	Result
1	Deploy e2e: site-a acks arrive via ReportApply (central CentralCommandService log lines carry site-a node ids), NodeDeploymentState rows land, deployment seals	PENDING
2	Site-b acks still arrive via ClusterClient in the same deployment — per-pair coexistence proven	PENDING
3	Exactly one ack row per site-a node per dispatch (no double-count from endpoint failover)	PENDING
4	Kill central-1 before the apply completes: site-a acks fail over to central-2's endpoint; deployment still seals	PENDING
5	Unauthenticated / wrong-cluster-key / missing x-otopcua-cluster probes against central :4055 CentralCommandService/ReportApply ⇒ PermissionDenied (missing-header rejection is central-side by design)	PENDING
6	ConfigServe (DeploymentArtifactService) on the same listener unaffected: fetches succeed, 0 auth failures	PENDING

---

Phase 2 — full-rig cutover + soak — PENDING

Rig default flipped and committed: MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}". Branch/SHA ________. Run the Baseline section first on the pre-flip build.

#	Check	Result
1	Full rebuild all-Grpc: six nodes ready, deploy e2e seals, all acks via gRPC	PENDING
2	Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes	PENDING
3	Kill site-a-1 mid-dispatch: per-node Execute failure logged with node id + status (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path)	PENDING
4	Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action	PENDING
5	Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; no duplicate applies after the pair returns (boot-replay path only)	PENDING
6	Zero ClusterClient activity under Grpc: docker logs <every node> | grep -ci "clusterclient" = 0 new lines post-boot	PENDING
7	Soak ≥ 2 h under periodic deploys + command traffic: 0 PermissionDenied/Unauthenticated across all six nodes (PSK drift detector), 0 unexplained Unavailable	PENDING
8	Full suite green on the cutover SHA	PENDING

---

Phase 4 — terminal gate on the post-deletion build — PENDING

master @ ________ (after the Phase 3 deletion merge). Provenance guard: image build timestamp verified LATER than the merge commit time — a pre-deletion image would invalidate checks 6–7.

#	Check	Result
1	PSK negatives on both services (no key / wrong key / wrong cluster's key); unset server key ⇒ all calls rejected (fail-closed proven live)	PENDING
2	Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand)	PENDING
3	Ack matrix: every node acks exactly once, deployment seals	PENDING
4	Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry)	PENDING
5	Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart)	PENDING
6	Boundary check: no Akka association between any site container and central — the Baseline (a) probe now shows no cross-mesh :4053 traffic (pair-internal remoting only)	PENDING
7	Full-rig restart (pairs together): clean convergence; grep -ri "receptionist|clusterclient" over all node logs = 0 lines	PENDING
8	MeshGrpcBoundaryTests + full suite green on the merge SHA	PENDING

Carry-forwards

Anything a phase gate could not drive gets named here with why + what proves it instead, and is re-checked in the next gate (the ScadaBridge proportionate-gate rule). Currently: none.

Defects the gates caught that the suite did not

Recorded per phase as they occur, with root cause + the pinned regression test.