lmxopcua/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md

ClusterClient → gRPC migration — live gate record

Rig: docker-dev/ (central-1/-2 = admin,driver,cluster-MAIN; site-a-1/-2, site-b-1/-2; sql + traefik + migrator + cluster-seed), rebuilt from the branch under test via docker compose -f docker-dev/docker-compose.yml up -d --build. Recorded check-by-check in the family's live-gate format. Later phases' sections are appended as those phases land; every check row gets a PASS/FAIL verdict plus one quantitative justification clause, with fenced evidence (real commands, real output). A run invalidated by a concurrent rig rebuild is void — note it and re-run (the ScadaBridge void-run rule).

Provenance discipline: every phase section opens with branch + commit SHA + "image rebuilt, all N containers recreated", and Phase 4 additionally verifies the image build timestamp post-dates the deletion merge (a stale image invalidates the boundary checks).

Plan: 2026-08-06-clusterclient-to-grpc.md. Tracking: 2026-08-06-clusterclient-to-grpc-tracking.md.

---

Baseline (run on the PRE-change build, before the Phase 2 flip)

The two gaps this program closes, reproduced rather than assumed:

(a) Unauthenticated Akka remoting crosses the WAN boundary. From the host:

# A live TCP association from a site container to central's Akka port:
docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
# And the receptionist actually accepting cross-mesh traffic (central log after any AdminUI
# driver restart targeting site-a):
docker logs otopcua-dev-central-1-1 2>&1 | grep -i "clusterclient\|receptionist" | tail -5

Expected: OPEN; receptionist/client lines present. Nothing on this path presents any credential.

(b) Silent drop on an unreachable cluster. Stop the site-b pair, dispatch a deployment (POST http://localhost:9200/api/deployments, X-Api-Key: docker-dev-deploy-key), observe: no per-node dispatch error surfaces anywhere; site-b's NodeDeploymentState rows sit Applying until the 2-minute deadline marks the deployment TimedOut/partially failed. The deadline is the only witness. Restart the pair.

• Baseline (a) recorded with evidence — PENDING
• Baseline (b) recorded with evidence — PENDING

---

Phase 0 — contract + auth + config groundwork — PASS

Branch feat/grpc-phase0-contract-auth @ c2bfad4a. Image rebuilt, all six containers recreated (docker compose -f docker-dev/docker-compose.yml up -d --build, run from the worktree at /Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-phase0 so the image carries this branch). The rig was already running (from the main checkout's compose file) before this run; the rebuild recreated all eight containers (sql kept, migrator + cluster-seed one-shots re-ran, central-1/-2 + site-a-1/-2 + site-b-1/-2 + traefik recreated) under the fixed otopcua-dev project name, so this run is not a concurrent-rebuild-invalidated one — nothing else touched the stack afterwards.

#	Check	Result
1	All six nodes boot on ClusterClient with the dark Grpc config present (PSKs, endpoints, ClusterPsks map); no validator fires	PASS — 6/6 nodes restarts=0, docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' reports running restarts=0 on every node; `docker logs
2	Deliberate misconfig: a scratch container with MeshTransport__Mode=Grpc and no GrpcPsk refuses to boot, error names MeshTransport:GrpcPsk + the env form	PASS — Hosting failed to start + OptionsValidationException naming MeshTransport:GrpcPsk and MeshTransport__GrpcPsk, below
3	Same with empty CentralGrpcEndpoints — refuses, names the key	PASS — OptionsValidationException naming MeshTransport:CentralGrpcEndpoints and MeshTransport__CentralGrpcEndpoints__0, below
4	Full suite green, 0 warnings, on the branch	PASS — dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo: 0 failures on a clean run (see evidence below); no .cs files changed on this branch since the last full rebuild, so the pre-existing ~868-warning baseline is unchanged (nothing to newly warn about)

Note: the new gRPC services are not yet mapped in Phase 0, so no wire probes here — the interceptor negative/positive matrix is unit-tested and wire-probed in 1A/1B.

Check 1 evidence — no validator fired on any of the six nodes, all dark Grpc keys present:

$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do
    docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' otopcua-dev-$c-1
  done
running restarts=0    # central-1
running restarts=0    # central-2
running restarts=0    # site-a-1
running restarts=0    # site-a-2
running restarts=0    # site-b-1
running restarts=0    # site-b-2

$ docker logs otopcua-dev-central-1-1 2>&1 | grep -iE "MeshTransport|ClusterClient"
[08:53:26 INF] Mesh ClusterClient for cluster MAIN created with 2 contact point(s)
[08:53:26 INF] Mesh ClusterClient for cluster SITE-A created with 2 contact point(s)
[08:53:26 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s)

$ docker logs otopcua-dev-site-a-1-1 2>&1 | grep -iE "cluster.client|receptionist"
[08:53:26 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]

$ docker inspect otopcua-dev-site-a-1-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep MeshTransport
MeshTransport__CentralGrpcEndpoints__1=http://central-2:4055
MeshTransport__CentralGrpcEndpoints__0=http://central-1:4055
MeshTransport__Mode=ClusterClient
MeshTransport__GrpcPsk=dev-mesh-psk-SITE-A
MeshTransport__CentralContactPoints__1=akka.tcp://otopcua@central-2:4053
MeshTransport__CentralContactPoints__0=akka.tcp://otopcua@central-1:4053

$ curl -sf http://localhost:9200/health/ready
{"status":"Healthy", ..., "akka":{"status":"Healthy","description":"Akka cluster member status: Up",
 "data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","selfRoles":["admin","cluster-MAIN","driver"],
 "memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}}}

Check 2 evidence — scratch container from otopcua-host:dev, site-a-1's real environment with MeshTransport__GrpcPsk removed and Mode forced to Grpc:

$ docker run --rm --name scratch-check2-noPsk --network otopcua-dev_default \
    --env-file check2-env.txt otopcua-host:dev
[08:54:23 ERR] Hosting failed to start
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:GrpcPsk is empty. Under
Grpc this per-cluster pre-shared key is the whole authentication boundary of the command plane
in both directions — without it this node's inbound command service fails closed and every ack
it posts to central is rejected. Supply it via the environment (MeshTransport__GrpcPsk), in
production as ${secret:OT-MESH-PSK-<clusterId>}.
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   ...

Check 3 evidence — same base env, GrpcPsk present, both CentralGrpcEndpoints indices unset:

$ docker run -d --name scratch-check3-noEndpoints --network otopcua-dev_default \
    --env-file check3-env.txt otopcua-host:dev
$ docker logs scratch-check3-noEndpoints 2>&1 | grep -A2 "OptionsValidationException"
[08:55:55 ERR] Hosting failed to start
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:CentralGrpcEndpoints is
empty. Under Grpc a driver node has nowhere to post its deployment ack, and every deployment
would time out naming a node that is running perfectly. Supply them via the environment
(MeshTransport__CentralGrpcEndpoints__0).

Both scratch containers removed after capture (docker rm -f scratch-check2-noPsk scratch-check3-noEndpoints); the rig's six real nodes were never affected (still restarts=0 after both runs).

Check 4 evidence — full suite, clean re-run (a first run hit one race-condition flake in RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed, which the test's own name identifies as exercising a race; re-ran in isolation and it passed, then a full second run passed clean end to end — pre-existing timing sensitivity, not a regression from this branch, which touched only docker-dev/docker-compose.yml and docs/Configuration.md, no .cs files):

$ dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo
...
Passed!  - Failed: 0, Passed: 212, Skipped: 4, Total: 216, Duration: 3m56s - ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.dll (net10.0)
$ echo $?
0

Rig left running on ClusterClient (its normal state) after the gate.

---

Phase 1A — central→node command plane — FAIL (blocked at the central boot boundary)

Branch feat/grpc-node-command @ fbf7b708. Image rebuilt from this worktree (/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a) so it carries this branch; docker compose -f docker-dev/docker-compose.yml up -d --build exited 0 and all eight containers were recreated — migrator and cluster-seed re-ran to completion (exit 0), the six host nodes and traefik were recreated, the sql container and its named volume were kept. Nothing else touched the stack afterwards, so this run is not a concurrent-rebuild-invalidated one. Rig-only flip: central-1/-2 MeshTransport__Mode: "Grpc" (hard value); site nodes left on the compose default ClusterClient (coexistence is the point). Three Serilog MinimumLevel:Override Debug scopes were added on all six nodes for this run only (ZB.MOM.WW.OtOpcUa.Host.Grpc, ZB.MOM.WW.OtOpcUa.ControlPlane.Communication, Akka.Cluster.Tools.Client) — the accept/dispatch lines the checks read are Debug and the rig's default Serilog level is Information, so without them checks 1/3 would have had no observable signal. Flip and overrides reverted after the gate, never committed.

Verdict: FAIL. Neither central node boots on this branch, so nothing that originates at central could be driven. Root cause is defect 4 below — resolving IMeshCommandDialer / IMeshPskProvider inside the first WithActors starter drags in ISecretResolver → IAuditWriter → ActorAuditWriter, whose constructor reads AuditWriterActorKey out of the ActorRegistry before the singleton that registers it has run. It is mode-independent (reproduced on the same image with MeshTransport__Mode=ClusterClient), so it is a defect of the branch, not of the gate's flip, and it would have failed a ClusterClient rig equally.

#	Check	Result
1	Deploy dispatch from AdminUI reaches all six driver nodes over gRPC (per-node Execute log lines on every node; zero SendToAll lines on central)	BLOCKED — POST /api/deployments is unreachable: traefik answers 503 because neither central node has a live HTTP surface (curl http://central-1:9000/health/ready and central-2 both return 000, connection refused). Nothing dispatched.
2	Acks still return over ClusterClient (site mode unchanged) and the deployment seals — mixed-transport deploy proven end to end	BLOCKED — no deployment could be dispatched (check 1). The most recent dbo.Deployment row is still the pre-gate 03540DA5-… (Status=2, Sealed, 2026-08-06 06:11:22), i.e. nothing was attempted during this run.
3	RestartDriver + ReconnectDriver from AdminUI land on the target site node (EventStream delivery observed via driver restart in logs)	PARTIAL PASS (node-side leg only) — both commands were delivered to site-a-1 over real Kestrel h2c with the correct PSK and both acted: DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate followed by a fresh spawned Modbus driver SITEA-modbus, and reconnecting driver … ForceReconnect requested by admin; re-entering Reconnecting. The AdminUI half was not driven — see the two carry-forwards below (central down; and the UI control is unreachable on this branch anyway).
4	AlarmCommand (ack a scripted alarm from AdminUI) lands and the condition acknowledges	PARTIAL PASS (envelope leg only) — the alarm arm decoded and was delivered node-side (Mesh: accepted AlarmCommand (correlationId=39ba4511…) over gRPC), which is the arm with no CLR correlation member and therefore the one worth wiring live. The acknowledge itself was NOT observed: the rig's single dbo.ScriptedAlarm row (gate481-alarm, equipment EQ-6e09bcc2d907) is not owned by SITE-A, and its owning node is a central one, which is down.
5	Unauthenticated grpcurl to a site node :4056 mesh_transport.v1.NodeCommandService/Execute ⇒ PermissionDenied; correct dev PSK ⇒ accepted	PASS — 3/3 probes graded as designed: no bearer ⇒ PermissionDenied, correct dev-mesh-psk-SITE-A + empty envelope ⇒ InvalidArgument (auth passed, the decode is what failed), and a full RestartDriver envelope with the same key ⇒ {} + observed driver restart.
6	Wrong-key isolation: site-b's PSK presented to site-a ⇒ PermissionDenied (per-cluster scoping is real, not decorative)	PASS — both directions rejected: dev-mesh-psk-SITE-B → site-a-1 ⇒ PermissionDenied, dev-mesh-psk-SITE-A → site-b-1 ⇒ PermissionDenied, while dev-mesh-psk-SITE-B → site-b-1 reaches the decoder (InvalidArgument) — so the rejection is the key, not the reachability.
7	A node with NULL GrpcPort is skipped with the warn-once, fan continues to the others	BLOCKED — the skip is computed by CentralCommunicationActor.AdoptGrpcTargets on a contact refresh; no central node is running to refresh. The rig's ClusterNode rows were left untouched (all six still GrpcPort=4056) rather than mutated for a check that could not be observed.
8	Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures on their interceptors across the run	PARTIAL PASS — 0 Rejected a telemetry stream call / Rejected a config-serve call / Rejected a LocalDb sync call lines across all six nodes, and the only Rejected a mesh command call lines are the 3 deliberate negative probes (2 on site-a-1, 1 on site-b-1). But telemetry and config-serve carried no traffic at all in this run — central dials both and never booted (site-a-1 logs 0 telemetry lines) — so "unaffected" is established for LocalDb sync only.

Phase 1A evidence

Provenance — flip in effect on central only, sites untouched:

$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
    --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
central-1  MeshTransport__Mode=Grpc
central-2  MeshTransport__Mode=Grpc
site-a-1   MeshTransport__Mode=ClusterClient
site-b-1   MeshTransport__Mode=ClusterClient

$ docker images --format '{{.Repository}}:{{.Tag}}|{{.ID}}|{{.CreatedAt}}' | grep otopcua-host
otopcua-host:dev|69360a173f1f|2026-08-06 07:20:14 -0400 EDT     # rebuilt for this run

Defect 4 — the boot failure (checks 1/2/7 blocked by it). Both central nodes, identical:

$ docker logs otopcua-dev-central-1-1 2>&1 | sed -n '25,45p'
[11:21:27 INF] OPC UA server started on opc.tcp://0.0.0.0:4840
[11:21:27 FTL] Unable to start AkkaHostedService - shutting down application
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
   at Akka.Hosting.ActorRegistry.Get[TKey]()
   at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
        in /src/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Audit/ActorAuditWriter.cs:line 24
   ...
   at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_5(IServiceProvider sp)
   ...
   at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_6(IServiceProvider sp)
   ...
   at ZB.MOM.WW.OtOpcUa.ControlPlane.ServiceCollectionExtensions.<>c.<WithOtOpcUaControlPlaneSingletons>b__7_0(
        ActorSystem system, IActorRegistry registry, IDependencyResolver resolver)
        in /src/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs:line 74
   at Akka.Hosting.AkkaConfigurationBuilder.StartAsync(ActorSystem sys)
   at Akka.Hosting.AkkaHostedService.StartAsync(CancellationToken cancellationToken)

Mode-independence, proven rather than argued — same image, central-1's exact environment with MeshTransport__Mode=ClusterClient, own hostname + self-seed so it cannot disturb the rig:

$ docker run -d --name scratch-cc-mode --network otopcua-dev_default --env-file c1-env.txt \
    -e Cluster__PublicHostname=scratch-cc-mode \
    -e Cluster__SeedNodes__0=akka.tcp://otopcua@scratch-cc-mode:4053 otopcua-host:dev
$ docker logs scratch-cc-mode 2>&1 | grep -m4 -E "MissingActorRegistry|Hosting failed|Host starting"
[11:28:45 INF] OtOpcUa.Host starting with roles=[admin,driver,cluster-main] (admin=True, driver=True)
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
[11:28:47 ERR] Hosting failed to start

Scratch container removed after capture (docker rm -f scratch-cc-mode); the six real nodes were untouched by it.

It presents as a healthy container. Worth carrying into Phase 2/4 readings — the failed process does not exit, so docker ps shows Up, RestartCount=0, and only an HTTP probe or the log tells the truth:

$ docker inspect otopcua-dev-central-1-1 -f '{{.State.Status}} restarts={{.RestartCount}}'
running restarts=0
$ docker run --rm --network otopcua-dev_default curlimages/curl -s -o /dev/null \
    -w '%{http_code}\n' --max-time 5 http://central-1:9000/health/ready
000
$ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9200/health/ready
503

Checks 5 + 6 — the wire-probe auth matrix (-import-path onto the in-repo proto; the node has no reflection). An empty CommandEnvelope has no oneof arm set, so FromEnvelope throws and the service answers InvalidArgument — after the interceptor ran, which is exactly why it is the positive signal here: PermissionDenied means the key was refused, InvalidArgument means the key was accepted and only the payload was rejected:

$ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos
$ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \
     -plaintext -import-path /protos -proto mesh_transport.proto"

# 5a  no bearer                        -> site-a-1
Code: PermissionDenied   Message: Node command authentication failed.
# 5b  -H 'authorization: Bearer dev-mesh-psk-SITE-A'   -> site-a-1
Code: InvalidArgument    Message: Command envelope could not be decoded.
# 6a  -H 'authorization: Bearer dev-mesh-psk-SITE-B'   -> site-a-1
Code: PermissionDenied   Message: Node command authentication failed.
# 6b  -H 'authorization: Bearer dev-mesh-psk-SITE-A'   -> site-b-1
Code: PermissionDenied   Message: Node command authentication failed.
# 6c  -H 'authorization: Bearer dev-mesh-psk-SITE-B'   -> site-b-1   (positive control)
Code: InvalidArgument    Message: Command envelope could not be decoded.

Checks 3 + 4 — the node-side leg, driven with real envelopes over the same authenticated h2c path central would use. This is the first live proof that MeshCommandAuthInterceptor is actually invoked on a real Kestrel pipeline (register item 4) and that NodeCommandGrpcService → NodeCommunicationActor → EventStream → DriverHostActor works end to end:

$ $G -H 'authorization: Bearer dev-mesh-psk-SITE-A' -H 'x-otopcua-cluster: SITE-A' \
     -d '{"correlationId":"8461008e...","restartDriver":{"clusterId":"SITE-A",
          "driverInstanceId":"SITEA-modbus","actorByUserName":"1a-gate",
          "correlationId":"8461008e..."}}' \
     site-a-1:4056 mesh_transport.v1.NodeCommandService/Execute
{}

$ docker logs otopcua-dev-site-a-1-1 2>&1 | tail
[11:31:01 DBG] Mesh: accepted RestartDriver (correlationId=8461008e37654724a292219202d78dfa) over gRPC; handed to the node communication actor.
[11:31:01 INF] DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate
[11:31:01 INF] DriverHost site-a-1:4053: spawned Modbus driver SITEA-modbus (stub=False)
[11:31:55 DBG] Mesh: accepted ReconnectDriver (correlationId=29c5dc77283f4eb7869d0c7d6686cadd) over gRPC; handed to the node communication actor.
[11:31:55 INF] DriverHost site-a-1:4053: reconnecting driver SITEA-modbus by request of 1a-gate
[11:31:55 INF] DriverInstance SITEA-modbus: ForceReconnect requested by admin; re-entering Reconnecting
[11:31:56 DBG] Mesh: accepted AlarmCommand (correlationId=39ba451173954229a71b9dd1e00ac1b9) over gRPC; handed to the node communication actor.

Check 8 — interceptor denial counts across every node, whole run:

$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1  telemetry=0 configserve=0 localdbsync=0 mesh=0
central-2  telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-1   telemetry=0 configserve=0 localdbsync=0 mesh=2   # the two deliberate negatives
site-a-2   telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-1   telemetry=0 configserve=0 localdbsync=0 mesh=1   # the one deliberate negative
site-b-2   telemetry=0 configserve=0 localdbsync=0 mesh=0

The TwoNodeClusterHarness I-1 null-provider Error (a known test-harness artifact) did not appear in any rig log — grep -c "IMeshPskProvider" = 0 on all six nodes, as expected for production wiring where every admin node registers the provider.

Phase 1A defect

Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter. WithOtOpcUaControlPlaneSingletons's WithActors block resolves IMeshCommandDialer and IMeshPskProvider from IServiceProvider to hand to CentralCommunicationActor.Props. MeshGrpcCommandDialer takes IMeshPskProvider, which takes ISecretResolver, which AddZbSecrets builds by resolving IAuditWriter, which on an admin node is ActorAuditWriter — and its constructor does registry.Get<AuditWriterActorKey>(). That singleton is registered ~60 lines later in the same builder, so at this point the key is absent and ActorRegistry.Get throws MissingActorRegistryEntryException, failing AkkaHostedService.StartAsync and the whole host.

Three things make this worth a pinned regression rather than a one-line fix note:

• It is not mode-gated. The two GetService calls run in every transport mode by design (so a mode flip needs no restart), so the eager graph is walked on every admin boot. Proven above with a ClusterClient scratch container on the same image.
• It is invisible to the suite. The 1A.2 container-activation test (It_activates_from_the_container_despite_the_optional_client_factory_parameter) resolves the dialer from a service collection that does not include AddZbSecrets, and TwoNodeClusterHarness registers no IMeshPskProvider at all — so no existing test walks the real graph in starter order. This is the same class as scadaproj#1 ("only visible with the DI graph built in a real host") and as mesh Phase 6's IClusterRoleInfo boot crash.
• The container lies about it. The process does not exit, so docker ps reports Up with restarts=0 while the node serves no HTTP at all. Any Phase 2/4 readiness check that reads container status rather than /health/ready would score this as healthy.

The shape of the fix is the one this file already uses two lines above for the coordinator proxy — hand the actor a lazy accessor (Func<IMeshCommandDialer?> / Func<IMeshPskProvider?>, or resolve on first use) so nothing is constructed while the ActorSystem is still building — but the fix and its red-first pin belong to 1A, not to this record.

Rig state after the gate

The compose flip was reverted (git checkout docker-dev/docker-compose.yml) and the six nodes recreated on the committed all-ClusterClient compose. Because defect 4 is mode-independent, the branch image cannot produce a healthy central pair on any mode, so the rig was returned to its pre-gate image rather than left with a dead central pair: the 1A build was preserved as otopcua-host:grpc-1a (69360a173f1f) and the pre-gate build (8bd81d8aea9b, 04:52) re-tagged back to otopcua-host:dev. Anyone re-running this gate must rebuild — the running rig is NOT on the branch under test. Verified after restore: six nodes running restarts=0, zero Hosting failed to start on all six, /health/ready 200 on every node, MAIN mesh memberCount=2 unreachableCount=0, AdminUI 200 on http://localhost:9200.

---

Phase 1B — node→central ack plane — PENDING

Branch feat/grpc-central-ack @ ________ (rebased on merged 1A). Rig-only flip: site-a-1/-2 → Grpc; site-b stays ClusterClient; central back on ClusterClient mode (its command direction) — acks are governed by the node's mode, so this isolates 1B.

#	Check	Result
1	Deploy e2e: site-a acks arrive via ReportApply (central CentralCommandService log lines carry site-a node ids), NodeDeploymentState rows land, deployment seals	PENDING
2	Site-b acks still arrive via ClusterClient in the same deployment — per-pair coexistence proven	PENDING
3	Exactly one ack row per site-a node per dispatch (no double-count from endpoint failover)	PENDING
4	Kill central-1 before the apply completes: site-a acks fail over to central-2's endpoint; deployment still seals	PENDING
5	Unauthenticated / wrong-cluster-key / missing x-otopcua-cluster probes against central :4055 CentralCommandService/ReportApply ⇒ PermissionDenied (missing-header rejection is central-side by design)	PENDING
6	ConfigServe (DeploymentArtifactService) on the same listener unaffected: fetches succeed, 0 auth failures	PENDING

---

Phase 2 — full-rig cutover + soak — PENDING

Rig default flipped and committed: MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}". Branch/SHA ________. Run the Baseline section first on the pre-flip build.

#	Check	Result
1	Full rebuild all-Grpc: six nodes ready, deploy e2e seals, all acks via gRPC	PENDING
2	Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes	PENDING
3	Kill site-a-1 mid-dispatch: per-node Execute failure logged with node id + status (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path)	PENDING
4	Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action	PENDING
5	Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; no duplicate applies after the pair returns (boot-replay path only)	PENDING
6	Zero ClusterClient activity under Grpc: docker logs <every node> | grep -ci "clusterclient" = 0 new lines post-boot	PENDING
7	Soak ≥ 2 h under periodic deploys + command traffic: 0 PermissionDenied/Unauthenticated across all six nodes (PSK drift detector), 0 unexplained Unavailable	PENDING
8	Full suite green on the cutover SHA	PENDING

---

Phase 4 — terminal gate on the post-deletion build — PENDING

master @ ________ (after the Phase 3 deletion merge). Provenance guard: image build timestamp verified LATER than the merge commit time — a pre-deletion image would invalidate checks 6–7.

#	Check	Result
1	PSK negatives on both services (no key / wrong key / wrong cluster's key); unset server key ⇒ all calls rejected (fail-closed proven live)	PENDING
2	Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand)	PENDING
3	Ack matrix: every node acks exactly once, deployment seals	PENDING
4	Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry)	PENDING
5	Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart)	PENDING
6	Boundary check: no Akka association between any site container and central — the Baseline (a) probe now shows no cross-mesh :4053 traffic (pair-internal remoting only)	PENDING
7	Full-rig restart (pairs together): clean convergence; grep -ri "receptionist|clusterclient" over all node logs = 0 lines	PENDING
8	MeshGrpcBoundaryTests + full suite green on the merge SHA	PENDING

Carry-forwards

Anything a phase gate could not drive gets named here with why + what proves it instead, and is re-checked in the next gate (the ScadaBridge proportionate-gate rule).

Item	Deferred from	Why it could not be driven	Proven instead by	Re-check in
Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-GrpcPort skip	1A	Both central nodes fail to boot (defect 4); no AdminUI, no deploy API, no contact refresh	Nothing live. Unit coverage only: the fan Theory, the gate-⊋-fan fact, A_slow_dial_does_not_park_the_actors_mailbox	1A gate re-run after defect 4 is fixed — these are the checks 1A exists to prove and must not be carried past it
Check 3 central half — RestartDriver/ReconnectDriver originating at the AdminUI	1A	(a) central down; (b) independently, DriverStatusPanel.razor — the only component holding the Reconnect/Restart buttons — is referenced by no page or component in the AdminUI, so the control is unreachable in the running UI on this branch and on master	The node-side leg driven live with real envelopes (check 3 evidence); central's encode covered by the T0.2 mapper goldens	1A re-run; the orphaned panel is pre-existing and out of this program's scope — file separately if it is to be fixed
Check 4 acknowledge half — the condition actually acknowledging	1A	The rig's one ScriptedAlarm (gate481-alarm) is not owned by a SITE cluster, and its owning central node is down	The alarm arm decoding + delivering node-side (check 4 evidence)	1A re-run, seeding a SITE-A-owned scripted alarm first
Check 8 for telemetry + config-serve	1A	Central dials both planes and never booted, so neither carried a single call — 0 denials is vacuous for them	LocalDb sync (pair-local) did carry traffic with 0 denials	1A re-run

Defects the gates caught that the suite did not

Recorded per phase as they occur, with root cause + the pinned regression test.

#	Phase	Defect	Root cause	Status
4	1A gate	An admin node cannot boot at all — MissingActorRegistryEntryException: AuditWriterActorKey, both central nodes, every transport mode	The first WithActors starter eagerly resolves IMeshCommandDialer/IMeshPskProvider, whose graph reaches ActorAuditWriter, whose ctor reads an ActorRegistry key registered ~60 lines later in the same builder	OPEN — blocks 1A. Full analysis + evidence in the Phase 1A section above. No product change was made by this gate