lmxopcua/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md

ClusterClient → gRPC migration — live gate record

Rig: docker-dev/ (central-1/-2 = admin,driver,cluster-MAIN; site-a-1/-2, site-b-1/-2; sql + traefik + migrator + cluster-seed), rebuilt from the branch under test via docker compose -f docker-dev/docker-compose.yml up -d --build. Recorded check-by-check in the family's live-gate format. Later phases' sections are appended as those phases land; every check row gets a PASS/FAIL verdict plus one quantitative justification clause, with fenced evidence (real commands, real output). A run invalidated by a concurrent rig rebuild is void — note it and re-run (the ScadaBridge void-run rule).

Provenance discipline: every phase section opens with branch + commit SHA + "image rebuilt, all N containers recreated", and Phase 4 additionally verifies the image build timestamp post-dates the deletion merge (a stale image invalidates the boundary checks).

Plan: 2026-08-06-clusterclient-to-grpc.md. Tracking: 2026-08-06-clusterclient-to-grpc-tracking.md.

---

Baseline (run on the PRE-change build, before the Phase 2 flip)

The two gaps this program closes, reproduced rather than assumed:

(a) Unauthenticated Akka remoting crosses the WAN boundary. From the host:

# A live TCP association from a site container to central's Akka port:
docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
# And the receptionist actually accepting cross-mesh traffic (central log after any AdminUI
# driver restart targeting site-a):
docker logs otopcua-dev-central-1-1 2>&1 | grep -i "clusterclient\|receptionist" | tail -5

Expected: OPEN; receptionist/client lines present. Nothing on this path presents any credential.

(b) Silent drop on an unreachable cluster. Stop the site-b pair, dispatch a deployment (POST http://localhost:9200/api/deployments, X-Api-Key: docker-dev-deploy-key), observe: no per-node dispatch error surfaces anywhere; site-b's NodeDeploymentState rows sit Applying until the 2-minute deadline marks the deployment TimedOut/partially failed. The deadline is the only witness. Restart the pair.

• Baseline (a) recorded with evidence — REPRODUCED 2026-08-06, rig on master @ 46be4c8d (post-1B image, all-ClusterClient default, no mode override; MeshTransport__Mode=ClusterClient confirmed via printenv in the site container)
• Baseline (b) recorded with evidence — REPRODUCED same run

Baseline (a) evidence — unauthenticated Akka remoting crosses the boundary

$ docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
OPEN                                          # no credential presented on this path
$ docker logs otopcua-dev-central-1-1 | grep -i "clusterclient\|receptionist" | tail -5
[13:12:51 INF] Connected to [akka://otopcua/system/receptionist]
[13:12:51 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s)
[13:12:53 INF] Connected to [akka://otopcua/system/receptionist]
[13:12:54 INF] Connected to [akka.tcp://otopcua@site-a-1:4053/system/receptionist]
[13:12:57 INF] Connected to [akka.tcp://otopcua@site-b-1:4053/system/receptionist]

Baseline (b) evidence — silent drop on an unreachable cluster

Site-b pair stopped 13:53:5x UTC; dispatch accepted 13:53:54 (POST /api/deployments → {"outcome":"Accepted","deploymentId":"c6cd0798-…"}, HTTP 202).

# 20 s after dispatch — 4/6 applied within ~15 ms, site-b rows parked, deployment open:
NodeDeploymentState: central-1/-2, site-a-1/-2 → Status 1 @ 13:53:54.9xx
                     site-b-1/-2               → Status 0, AppliedAtUtc NULL
Deployment c6cd0798 → Status 1 (in progress)

# The site-b fan leg is not merely unlogged — it is actively suppressed. The SITE-B
# ClusterClient's SendToAll lands in deadLetters at INF, under Akka's sampled generic
# dead-letter counter, naming no node and no failure:
[13:53:54 INF] Message [DispatchDeployment] wrapped in [$SendToAll] … to [akka://otopcua/deadLetters]
               was not delivered. [467] dead letters encountered, of which 456 were not logged. …
               Message content: DispatchDeployment { DeploymentId = c6cd0798…, … }

# Dispatch-to-deadline window (13:53:50–13:55:53): ambient Akka association-gating noise for
# the downed pair exists (transport chatter, fires with or without a dispatch), but ZERO lines
# correlate site-b with the deployment/dispatch/command:
$ docker logs --since …09:53:50 --until …09:55:53 otopcua-dev-central-1-1 \
    | grep -i "site-b" | grep -icE "deploy|dispatch|command|c6cd0798"
0

# The deadline is the only witness, exactly 2 minutes later:
[13:55:54 WRN] Deployment c6cd0798… timed out after 00:02:00 (4/6 acks landed). No ack from:
               site-b-1:4053, site-b-2:4053. Each is an enabled, non-maintenance ClusterNode
               row — start the node, or set ClusterNode.MaintenanceMode = 1 …
Deployment c6cd0798 → Status 4; site-b rows stay Status 0/NULL (a timed-out deployment is not
retro-acked; boot replay serves the latest artifact without resurrecting the dead record).

Pair restarted after the drill; site-b-1 back healthy. Note for later log reads on this host: docker logs --since/--until parses bare timestamps as HOST-LOCAL time (EDT here, UTC−4) while the app writes UTC — a UTC-shaped --since silently returns nothing (it points 4 h into the future). Convert to local first; earlier in this run that mistake made a window look empty.

---

Phase 0 — contract + auth + config groundwork — PASS

Branch feat/grpc-phase0-contract-auth @ c2bfad4a. Image rebuilt, all six containers recreated (docker compose -f docker-dev/docker-compose.yml up -d --build, run from the worktree at /Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-phase0 so the image carries this branch). The rig was already running (from the main checkout's compose file) before this run; the rebuild recreated all eight containers (sql kept, migrator + cluster-seed one-shots re-ran, central-1/-2 + site-a-1/-2 + site-b-1/-2 + traefik recreated) under the fixed otopcua-dev project name, so this run is not a concurrent-rebuild-invalidated one — nothing else touched the stack afterwards.

#	Check	Result
1	All six nodes boot on ClusterClient with the dark Grpc config present (PSKs, endpoints, ClusterPsks map); no validator fires	PASS — 6/6 nodes restarts=0, docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' reports running restarts=0 on every node; `docker logs
2	Deliberate misconfig: a scratch container with MeshTransport__Mode=Grpc and no GrpcPsk refuses to boot, error names MeshTransport:GrpcPsk + the env form	PASS — Hosting failed to start + OptionsValidationException naming MeshTransport:GrpcPsk and MeshTransport__GrpcPsk, below
3	Same with empty CentralGrpcEndpoints — refuses, names the key	PASS — OptionsValidationException naming MeshTransport:CentralGrpcEndpoints and MeshTransport__CentralGrpcEndpoints__0, below
4	Full suite green, 0 warnings, on the branch	PASS — dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo: 0 failures on a clean run (see evidence below); no .cs files changed on this branch since the last full rebuild, so the pre-existing ~868-warning baseline is unchanged (nothing to newly warn about)

Note: the new gRPC services are not yet mapped in Phase 0, so no wire probes here — the interceptor negative/positive matrix is unit-tested and wire-probed in 1A/1B.

Check 1 evidence — no validator fired on any of the six nodes, all dark Grpc keys present:

$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do
    docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' otopcua-dev-$c-1
  done
running restarts=0    # central-1
running restarts=0    # central-2
running restarts=0    # site-a-1
running restarts=0    # site-a-2
running restarts=0    # site-b-1
running restarts=0    # site-b-2

$ docker logs otopcua-dev-central-1-1 2>&1 | grep -iE "MeshTransport|ClusterClient"
[08:53:26 INF] Mesh ClusterClient for cluster MAIN created with 2 contact point(s)
[08:53:26 INF] Mesh ClusterClient for cluster SITE-A created with 2 contact point(s)
[08:53:26 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s)

$ docker logs otopcua-dev-site-a-1-1 2>&1 | grep -iE "cluster.client|receptionist"
[08:53:26 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]

$ docker inspect otopcua-dev-site-a-1-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep MeshTransport
MeshTransport__CentralGrpcEndpoints__1=http://central-2:4055
MeshTransport__CentralGrpcEndpoints__0=http://central-1:4055
MeshTransport__Mode=ClusterClient
MeshTransport__GrpcPsk=dev-mesh-psk-SITE-A
MeshTransport__CentralContactPoints__1=akka.tcp://otopcua@central-2:4053
MeshTransport__CentralContactPoints__0=akka.tcp://otopcua@central-1:4053

$ curl -sf http://localhost:9200/health/ready
{"status":"Healthy", ..., "akka":{"status":"Healthy","description":"Akka cluster member status: Up",
 "data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","selfRoles":["admin","cluster-MAIN","driver"],
 "memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}}}

Check 2 evidence — scratch container from otopcua-host:dev, site-a-1's real environment with MeshTransport__GrpcPsk removed and Mode forced to Grpc:

$ docker run --rm --name scratch-check2-noPsk --network otopcua-dev_default \
    --env-file check2-env.txt otopcua-host:dev
[08:54:23 ERR] Hosting failed to start
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:GrpcPsk is empty. Under
Grpc this per-cluster pre-shared key is the whole authentication boundary of the command plane
in both directions — without it this node's inbound command service fails closed and every ack
it posts to central is rejected. Supply it via the environment (MeshTransport__GrpcPsk), in
production as ${secret:OT-MESH-PSK-<clusterId>}.
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   ...

Check 3 evidence — same base env, GrpcPsk present, both CentralGrpcEndpoints indices unset:

$ docker run -d --name scratch-check3-noEndpoints --network otopcua-dev_default \
    --env-file check3-env.txt otopcua-host:dev
$ docker logs scratch-check3-noEndpoints 2>&1 | grep -A2 "OptionsValidationException"
[08:55:55 ERR] Hosting failed to start
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:CentralGrpcEndpoints is
empty. Under Grpc a driver node has nowhere to post its deployment ack, and every deployment
would time out naming a node that is running perfectly. Supply them via the environment
(MeshTransport__CentralGrpcEndpoints__0).

Both scratch containers removed after capture (docker rm -f scratch-check2-noPsk scratch-check3-noEndpoints); the rig's six real nodes were never affected (still restarts=0 after both runs).

Check 4 evidence — full suite, clean re-run (a first run hit one race-condition flake in RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed, which the test's own name identifies as exercising a race; re-ran in isolation and it passed, then a full second run passed clean end to end — pre-existing timing sensitivity, not a regression from this branch, which touched only docker-dev/docker-compose.yml and docs/Configuration.md, no .cs files):

$ dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo
...
Passed!  - Failed: 0, Passed: 212, Skipped: 4, Total: 216, Duration: 3m56s - ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.dll (net10.0)
$ echo $?
0

Rig left running on ClusterClient (its normal state) after the gate.

---

Phase 1A — central→node command plane — PASS (on the re-run at e54883a8)

Two runs. Run 1 @ fbf7b708 FAILED at the central boot boundary (defect 4) and could not drive checks 1/2/7 or the central halves of 3/4; its checks 5 + 6 and the node-side halves of 3 + 4 stand as recorded and were not repeated. Run 2 @ e54883a8 — after the boot fix (c0475f2d) and its review-hardening follow-up (e54883a8) — drove every carried check. The result table below is the combined, current verdict; run 1's failure record is preserved underneath it because the defect it caught is the most valuable thing this gate produced.

#	Check	Verdict	Run
—	Boot: all six nodes healthy INCLUDING the central pair, on the branch image	PASS — 6/6 running restarts=0, 0 Hosting failed to start, /health/ready 200 ×6, MAIN mesh memberCount=2 unreachableCount=0. But the Resolve Error line the fix logs DID fire — see finding 5	2
1	Deploy dispatch reaches all six driver nodes over gRPC; zero SendToAll/ClusterClient on central	PASS — 6/6 nodes accepted one DispatchDeployment under a single shared correlationId=cf499b42…; central's own per-node outcome log records 6 accepted / 0 failed; grep -ci clusterclient = 0 on both central nodes	2
2	Acks return over ClusterClient (site mode unchanged) and the deployment seals	PASS — deployment 8DF9AAE0… Status=2 (Sealed) with all six NodeDeploymentState rows Status=1 (Applied) inside ~90 ms; all four site nodes hold live Connected to [akka.tcp://…/system/receptionist] associations, which is their only ack path in 1A	2
3	RestartDriver + ReconnectDriver land on the target site node	PASS (composed) — node-side leg driven live in run 1 (both commands acted on site-a-1). The AdminUI half cannot be driven and this is now proven, not assumed: <DriverStatusPanel has 0 usages repo-wide, seven rendered pages contain no Restart/Reconnect text or button, and the only non-script-analysis HTTP POST in the whole app is /api/deployments. Closed by composition — check 4 proves the same AdminOperationsActor.Dispatch → DispatchOverGrpc origin path live from central for an admin-originated command	1 + 2
4	AlarmCommand from the AdminUI lands and the condition acknowledges	PASS — Ack clicked on the real /alerts page against a live gate481-alarm ACTIVATED row; central fanned one AlarmCommand (correlationId=ccafe375…, minted once) to all six over gRPC, and the condition acknowledged — a new ACKNOWLEDGED row appeared 0.54 s later carrying the operator's identity (multi-role-test)	2
5	Unauthenticated ⇒ PermissionDenied; correct dev PSK ⇒ accepted	PASS — 3/3 probes as designed (run 1; not repeated — nothing in the fix touches the interceptor)	1
6	Wrong-key isolation	PASS — both directions rejected, with a positive control proving the rejection is the key and not reachability (run 1)	1
7	NULL GrpcPort node skipped with the warn-once; fan continues	PASS — with site-b-2:4053 NULLed, both central nodes logged the warn once each and the next dispatch (correlationId=88a9e0a2…) was accepted by exactly 5 nodes, site-b-2 absent; the deployment then honestly ended Status=4 (TimedOut) with site-b-2 alone at Status=0. Row restored; a later dispatch sealed 6/6	2
8	Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures	PASS, and no longer vacuous — 0 denials on all four interceptors across all six nodes, and all three planes carried real traffic this time: site nodes fetched + SHA-verified 28 KB artifacts from http://central-1:4055, Telemetry stream opened on every node with live driver health rendering on /hosts, LocalDb sync active on the site pairs	2

Run 2 provenance

docker compose -f docker-dev/docker-compose.yml up -d --build from the worktree, exit 0; image otopcua-host:dev rebuilt to 1d8452a29018 at 12:15 UTC (run 1's preserved otopcua-host:grpc-1a was stale and deliberately not reused). The six host nodes plus migrator and cluster-seed were recreated (both one-shots exit 0); sql and its volume were kept, and traefik was reused unchanged (its service definition did not change). Nothing else touched the stack during the run, so it is not concurrent-rebuild-invalidated. Same rig-only flip and same three Serilog Debug overrides as run 1; both reverted afterwards, never committed.

$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
    --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
central-1  MeshTransport__Mode=Grpc
central-2  MeshTransport__Mode=Grpc
site-a-1   MeshTransport__Mode=ClusterClient
site-b-1   MeshTransport__Mode=ClusterClient

Run 2 evidence

Boot — the fix works, and the race it was hardened against is real.

$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1  running restarts=0 mode=Grpc hostingFailed=0 resolveErr=2
central-2  running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0
site-a-1   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-a-2   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-b-1   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-b-2   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0

See finding 5 below for the two resolveErr lines on central-1 — they are the PreStart race losing on a real boot, and the memoize retry healing it was verified live rather than argued.

Check 1 — the fan, both ends. One envelope, one correlation id, six recipients:

$ for c in …; do docker logs otopcua-dev-$c-1 | grep "Mesh: accepted DispatchDeployment" | tail -1; done
central-1  [12:16:49 DBG] Mesh: accepted DispatchDeployment (correlationId=cf499b42c6914a578035b2b5bc1020f7) over gRPC; handed to the node communication actor.
central-2  [12:16:49 DBG] …same correlationId…
site-a-1   [12:16:49 DBG] …same correlationId…
site-a-2   [12:16:49 DBG] …same correlationId…
site-b-1   [12:16:49 DBG] …same correlationId…
site-b-2   [12:16:49 DBG] …same correlationId…

$ docker logs otopcua-dev-central-2-1 | grep "accepted by"        # the node hosting the coordinator
[12:16:49 DBG] Mesh: DispatchDeployment (correlationId=cf499b42…) accepted by central-2:4053 over gRPC
[12:16:49 DBG] … accepted by site-b-1:4053 …   [12:16:49 DBG] … accepted by site-a-2:4053 …
[12:16:49 DBG] … accepted by site-a-1:4053 …   [12:16:49 DBG] … accepted by central-1:4053 …
[12:16:49 DBG] … accepted by site-b-2:4053 …
$ docker logs … | grep -c "was NOT delivered to"   → 0 on both central nodes

# Zero ClusterClient on the command side, both central nodes:
central-1  MeshClusterClientCreated=0 clusterclientAnyLine=0
central-2  MeshClusterClientCreated=0 clusterclientAnyLine=0

The clusterclientAnyLine=0 count is over the whole log with Serilog:MinimumLevel:Override:Akka.Cluster.Tools.Client=Debug in force, so it covers Akka's own ClusterClient internals and not merely this repo's creation message.

Check 2 — mixed transport, end to end. Commands out over gRPC (above), acks back over ClusterClient:

$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='8df9aae0-…'
2                                    -- Sealed
$ … SELECT NodeId, Status, AppliedAtUtc FROM dbo.NodeDeploymentState WHERE DeploymentId='8df9aae0-…'
central-1:4053|1|2026-08-06 12:16:49.939     site-a-1:4053|1|2026-08-06 12:16:49.977
central-2:4053|1|2026-08-06 12:16:49.915     site-a-2:4053|1|2026-08-06 12:16:49.950
site-b-1:4053|1|2026-08-06 12:16:49.957      site-b-2:4053|1|2026-08-06 12:16:49.967

$ for c in site-a-1 site-a-2 site-b-1 site-b-2; do docker logs … | grep "receptionist" | tail -1; done
site-a-1   [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
site-a-2   [12:15:20 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
site-b-1   [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
site-b-2   [12:15:25 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
$ grep -c "No central ClusterClient" → 0 on all six      # no ack was dropped for want of a client

A site node has no gRPC ack path in 1A (that is 1B), so the receptionist association above is not merely present, it is the only route those four Applied rows could have taken.

Check 3 — why the AdminUI half is undrivable, established rather than asserted:

$ grep -rn "<DriverStatusPanel" --include="*.razor" --include="*.cs" src/ | wc -l
0                                    # the component holding the buttons is referenced nowhere
$ grep -rn "DriverStatusPanel" --include="*.razor" src/ | grep -v "Shared/Drivers/DriverStatusPanel.razor:"
…5 hits, all of them code COMMENTS in Alerts.razor / Hosts.razor ("mirrors DriverStatusPanel…")

$ node scan-ui.js          # every page a Designer/Administrator/Operator can reach
/            | reconnect/restart text: false | buttons: ["Sign out"]
/hosts       | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
/fleet       | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
/clusters    | reconnect/restart text: false | buttons: ["Sign out"]
/clusters/SITE-A | reconnect/restart text: false | buttons: ["Sign out"]
/alerts      | reconnect/restart text: false | buttons: ["Sign out","Clear"]
/deployments | reconnect/restart text: false | buttons: ["Sign out","Deploy current configuration"]

$ grep -rn "MapPost|MapPut|MapDelete" src/…/AdminUI src/…/Host
…/Api/DeployApiEndpoints.cs:43:  app.MapPost("/api/deployments", …)
…/ScriptAnalysis/ScriptAnalysisEndpoints.cs:19-24:  /diagnostics /completions /hover /signature-help /format /inlay-hints

There is no button and no endpoint. What closes the check is that RestartDriver, ReconnectDriver and AlarmCommand all leave central through the same three lines — AdminOperationsActor.Dispatch → MeshCommand → CentralCommunicationActor.DispatchOverGrpc — and check 4 drove that path live from the AdminUI for AlarmCommand, while run 1 drove the node-side decode-and-act for the two driver commands. The only per-type difference left is MeshTransportDtoMapper.ToEnvelope, which the T0.2 goldens cover member-for-member.

Check 4 — AdminUI ack, fanned by central, acknowledged by the owner. The rig's one scripted alarm sits Normal indefinitely (hr200 > 70000), so a transition was manufactured by editing its predicate to an always-true form and deploying; the /alerts page was already open when it fired, the Ack button was clicked, and the predicate + hash were restored afterwards (verified byte-identical by SELECT) with a further deploy:

page open at 2026-08-06T12:20:59.036Z        initial: empty window
ROWS:  12:21:00.017  gate481-alarm | PressHigh | EQ-6e09bcc2d907 | ACTIVATED | 750 | system | press high
ROW BUTTONS: ["Ack","Shelve","Shelve (timed)","Unshelve"]
clicking Acknowledge at 2026-08-06T12:21:00.362Z
after click:  12:21:00.557 | gate481-alarm | EQ-6e09bcc2d907 | ACKNOWLEDGED | 750 | multi-role-test | press high

$ docker logs otopcua-dev-central-2-1 | grep AlarmCommand
[12:21:00 DBG] Mesh: AlarmCommand (correlationId=ccafe3754aca40cfa8aa0a0602c86cbb) accepted by site-b-2:4053 over gRPC
[12:21:00 DBG] … accepted by central-1:4053 …   [12:21:00 DBG] … accepted by central-2:4053 …
[12:21:00 DBG] … accepted by site-a-1:4053 …    [12:21:00 DBG] … accepted by site-b-1:4053 …
[12:21:00 DBG] … accepted by site-a-2:4053 …
$ for c in …; do grep "Mesh: accepted AlarmCommand" …; done   → all six, same correlationId

One correlation id across all six recipients is the "encode once, before the fan" rule holding for the one command whose CLR record has no correlation member of its own — the case the design called out as the reason encoding is hoisted out of the loop.

Check 7 — the skip, the warn, and the honest deadline.

$ … UPDATE dbo.ClusterNode SET GrpcPort=NULL WHERE NodeId='site-b-2:4053'    # 12:23:14

# one warn per central node, once each, within one ContactRefreshSeconds:
central-2  [12:23:20 WRN] ClusterNode site-b-2:4053 has no GrpcPort; it exposes no command endpoint
           and is skipped in this dispatch set (further skips of this node are silent). Commands
           will not reach it until the row carries a port
central-1  [12:23:19 WRN] …identical…
# (each node also logs the TelemetryNodeSource twin — "exposes no telemetry stream … skipped in
#  this dial refresh" — expected collateral: both target sets read the same column)

# dispatch at 12:23:36 — accepted by five, site-b-2 absent:
central-2  [12:23:37 DBG] … (correlationId=88a9e0a2…) accepted by central-2 / site-a-2 / site-b-1
                          / site-a-1 / central-1 :4053 over gRPC
# cumulative per-node accept counts after this dispatch — site-b-2 one behind everyone:
central-1 4   central-2 4   site-a-1 4   site-a-2 4   site-b-1 4   site-b-2 3

$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='5c8bdc9e-…'   → 4  (TimedOut)
$ … SELECT NodeId, Status FROM dbo.NodeDeploymentState …
central-1|1  central-2|1  site-a-1|1  site-a-2|1  site-b-1|1  site-b-2|0

Row restored to 4056 and a later dispatch sealed 6/6 (080EBB0B…, Status=2, every node Status=1). One operational detail worth carrying: the skip persists for up to a full ContactRefreshSeconds after the row is repaired — a deploy issued 5 s after the restore (DF1C0B74…) was still skipped and still TimedOut, because the dispatch set is only rebuilt on the refresh tick.

Check 8 — denials, and proof the planes were actually used.

$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1  telemetry=0 configserve=0 localdbsync=0 mesh=0
central-2  telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-1   telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-2   telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-1   telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-2   telemetry=0 configserve=0 localdbsync=0 mesh=0

# config-serve carried real, authenticated traffic:
site-a-1  [12:20:59 INF] Fetched + verified artifact for 8d486c6b… from http://central-1:4055 (28313 bytes).
# telemetry likewise, end to end into the UI:
$ for c in …; do grep -c "Telemetry stream opened"; done → 3,4,3,3,3,5
/hosts →  CLUSTER SITE-A … sa-modbus2 SITEA-modbus Modbus HEALTHY 1 host 1 tag  12:30:49 UTC  0 errors

The five telemetry failed … reconnecting warnings in the run are all accounted for: three at 12:15:20 (central dialing nodes that had not finished binding :4056) and two at 12:28:50 / 12:29:30 (the deliberate central-2 stop in finding 5). None is an auth failure.

Phase 1A finding 5 — the PreStart race loses on real boots, and the memoize is what saves it

The hardening commit's premise is not hypothetical. On the run-2 boot, central-1 lost the race on both dependencies:

$ docker logs otopcua-dev-central-1-1 | grep -A3 "Failed to resolve"
[12:15:19 ERR] Failed to resolve IMeshCommandDialer for the mesh command boundary; continuing
               without it. The guards below report what that costs
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
   at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
[12:15:19 ERR] Failed to resolve IMeshPskProvider for the mesh command boundary; …same cause…

It is not rare: across the three admin boots observed in run 2 (central-1 and central-2 at 12:15, central-2 again at 12:29 after the stop/start), two of three lost it. Had the fix stopped at PreStart-only resolution, those nodes would have held a permanent silent null — no dialer, and a never-published known-cluster gate — and the failure would only have surfaced the first time the coordinator singleton moved onto such a node.

The memoize heals it, verified on the node that actually lost the race. Run 2's dispatches were all fanned by central-2 (the singleton host), so central-1's dialer was never exercised by them. Central-2 was therefore stopped to move the singleton, and a dispatch driven while central-1 alone held it:

$ docker compose stop central-2                                   # 12:28:39
$ curl … /health/ready → self akka.tcp://otopcua@central-1:4053, members 1     # 12:28:57
$ curl -X POST /api/deployments  -d '{"createdBy":"1a-gate-central1-heal"}'    # 12:29:01
$ docker logs otopcua-dev-central-1-1 | grep -E "accepted by|no command dialer is wired"
[12:29:02 DBG] Mesh: DispatchDeployment (correlationId=ad97d386…) accepted by site-b-1:4053 over gRPC
[12:29:02 DBG] … accepted by site-a-2 … site-b-2 … site-a-1 … central-1 …      # 5 live nodes
# no "no command dialer is wired" line, and no further "Failed to resolve"
$ docker compose start central-2  → members 2, unreachable 0, status Healthy   # 12:29:51

So the boot-time Error is cosmetically alarming but functionally healed, and that is now an observed property rather than a design claim. Two follow-ups for Phase 2 rather than for 1A: the Error text says "continuing without it", which reads as permanent and should say that the first use retries; and a Failed to resolve line on a healthy node is exactly the kind of signal a soak's PSK-drift/error grep will trip over.

One incidental observation, not a check. The deployment RevisionHash is deterministic (three separate no-change dispatches all produced 642b9d6f79b9…), but restoring the alarm predicate's two text columns did not restore the pre-gate hash 6178a8f302aa… — so the artifact covers a column the UPDATE bumped (Script.RowVersion is the obvious candidate). The rig's configuration content is byte-identical to its pre-gate state; only the hash moved. Not investigated further — out of this program's scope.

Run 1 record — the defect that blocked it

Branch feat/grpc-node-command @ fbf7b708. Image rebuilt from this worktree (/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a) so it carries this branch; docker compose -f docker-dev/docker-compose.yml up -d --build exited 0 and all eight containers were recreated — migrator and cluster-seed re-ran to completion (exit 0), the six host nodes and traefik were recreated, the sql container and its named volume were kept. Nothing else touched the stack afterwards, so this run is not a concurrent-rebuild-invalidated one. Rig-only flip: central-1/-2 MeshTransport__Mode: "Grpc" (hard value); site nodes left on the compose default ClusterClient (coexistence is the point). Three Serilog MinimumLevel:Override Debug scopes were added on all six nodes for this run only (ZB.MOM.WW.OtOpcUa.Host.Grpc, ZB.MOM.WW.OtOpcUa.ControlPlane.Communication, Akka.Cluster.Tools.Client) — the accept/dispatch lines the checks read are Debug and the rig's default Serilog level is Information, so without them checks 1/3 would have had no observable signal. Flip and overrides reverted after the gate, never committed.

Run 1 verdict: FAIL. Neither central node booted, so nothing that originates at central could be driven: checks 1, 2 and 7 were BLOCKED outright, 3 and 4 reached only their node-side legs, and check 8's zero-denial result was vacuous for telemetry and config-serve (central dials both and never booted). Checks 5 and 6 PASSED and are the run-1 evidence carried into the combined table above. Root cause was defect 4 — resolving IMeshCommandDialer / IMeshPskProvider inside the first WithActors starter drags in ISecretResolver → IAuditWriter → ActorAuditWriter, whose constructor reads AuditWriterActorKey out of the ActorRegistry before the singleton that registers it has run. It was mode-independent (reproduced on the same image with MeshTransport__Mode=ClusterClient), so it was a defect of the branch, not of the gate's flip, and it would have failed a ClusterClient rig equally.

Run 1 evidence

Provenance — flip in effect on central only, sites untouched:

$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
    --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
central-1  MeshTransport__Mode=Grpc
central-2  MeshTransport__Mode=Grpc
site-a-1   MeshTransport__Mode=ClusterClient
site-b-1   MeshTransport__Mode=ClusterClient

$ docker images --format '{{.Repository}}:{{.Tag}}|{{.ID}}|{{.CreatedAt}}' | grep otopcua-host
otopcua-host:dev|69360a173f1f|2026-08-06 07:20:14 -0400 EDT     # rebuilt for this run

Defect 4 — the boot failure (checks 1/2/7 blocked by it). Both central nodes, identical:

$ docker logs otopcua-dev-central-1-1 2>&1 | sed -n '25,45p'
[11:21:27 INF] OPC UA server started on opc.tcp://0.0.0.0:4840
[11:21:27 FTL] Unable to start AkkaHostedService - shutting down application
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
   at Akka.Hosting.ActorRegistry.Get[TKey]()
   at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
        in /src/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Audit/ActorAuditWriter.cs:line 24
   ...
   at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_5(IServiceProvider sp)
   ...
   at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_6(IServiceProvider sp)
   ...
   at ZB.MOM.WW.OtOpcUa.ControlPlane.ServiceCollectionExtensions.<>c.<WithOtOpcUaControlPlaneSingletons>b__7_0(
        ActorSystem system, IActorRegistry registry, IDependencyResolver resolver)
        in /src/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs:line 74
   at Akka.Hosting.AkkaConfigurationBuilder.StartAsync(ActorSystem sys)
   at Akka.Hosting.AkkaHostedService.StartAsync(CancellationToken cancellationToken)

Mode-independence, proven rather than argued — same image, central-1's exact environment with MeshTransport__Mode=ClusterClient, own hostname + self-seed so it cannot disturb the rig:

$ docker run -d --name scratch-cc-mode --network otopcua-dev_default --env-file c1-env.txt \
    -e Cluster__PublicHostname=scratch-cc-mode \
    -e Cluster__SeedNodes__0=akka.tcp://otopcua@scratch-cc-mode:4053 otopcua-host:dev
$ docker logs scratch-cc-mode 2>&1 | grep -m4 -E "MissingActorRegistry|Hosting failed|Host starting"
[11:28:45 INF] OtOpcUa.Host starting with roles=[admin,driver,cluster-main] (admin=True, driver=True)
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
[11:28:47 ERR] Hosting failed to start

Scratch container removed after capture (docker rm -f scratch-cc-mode); the six real nodes were untouched by it.

It presents as a healthy container. Worth carrying into Phase 2/4 readings — the failed process does not exit, so docker ps shows Up, RestartCount=0, and only an HTTP probe or the log tells the truth:

$ docker inspect otopcua-dev-central-1-1 -f '{{.State.Status}} restarts={{.RestartCount}}'
running restarts=0
$ docker run --rm --network otopcua-dev_default curlimages/curl -s -o /dev/null \
    -w '%{http_code}\n' --max-time 5 http://central-1:9000/health/ready
000
$ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9200/health/ready
503

Checks 5 + 6 — the wire-probe auth matrix (-import-path onto the in-repo proto; the node has no reflection). An empty CommandEnvelope has no oneof arm set, so FromEnvelope throws and the service answers InvalidArgument — after the interceptor ran, which is exactly why it is the positive signal here: PermissionDenied means the key was refused, InvalidArgument means the key was accepted and only the payload was rejected:

$ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos
$ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \
     -plaintext -import-path /protos -proto mesh_transport.proto"

# 5a  no bearer                        -> site-a-1
Code: PermissionDenied   Message: Node command authentication failed.
# 5b  -H 'authorization: Bearer dev-mesh-psk-SITE-A'   -> site-a-1
Code: InvalidArgument    Message: Command envelope could not be decoded.
# 6a  -H 'authorization: Bearer dev-mesh-psk-SITE-B'   -> site-a-1
Code: PermissionDenied   Message: Node command authentication failed.
# 6b  -H 'authorization: Bearer dev-mesh-psk-SITE-A'   -> site-b-1
Code: PermissionDenied   Message: Node command authentication failed.
# 6c  -H 'authorization: Bearer dev-mesh-psk-SITE-B'   -> site-b-1   (positive control)
Code: InvalidArgument    Message: Command envelope could not be decoded.

Checks 3 + 4 — the node-side leg, driven with real envelopes over the same authenticated h2c path central would use. This is the first live proof that MeshCommandAuthInterceptor is actually invoked on a real Kestrel pipeline (register item 4) and that NodeCommandGrpcService → NodeCommunicationActor → EventStream → DriverHostActor works end to end:

$ $G -H 'authorization: Bearer dev-mesh-psk-SITE-A' -H 'x-otopcua-cluster: SITE-A' \
     -d '{"correlationId":"8461008e...","restartDriver":{"clusterId":"SITE-A",
          "driverInstanceId":"SITEA-modbus","actorByUserName":"1a-gate",
          "correlationId":"8461008e..."}}' \
     site-a-1:4056 mesh_transport.v1.NodeCommandService/Execute
{}

$ docker logs otopcua-dev-site-a-1-1 2>&1 | tail
[11:31:01 DBG] Mesh: accepted RestartDriver (correlationId=8461008e37654724a292219202d78dfa) over gRPC; handed to the node communication actor.
[11:31:01 INF] DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate
[11:31:01 INF] DriverHost site-a-1:4053: spawned Modbus driver SITEA-modbus (stub=False)
[11:31:55 DBG] Mesh: accepted ReconnectDriver (correlationId=29c5dc77283f4eb7869d0c7d6686cadd) over gRPC; handed to the node communication actor.
[11:31:55 INF] DriverHost site-a-1:4053: reconnecting driver SITEA-modbus by request of 1a-gate
[11:31:55 INF] DriverInstance SITEA-modbus: ForceReconnect requested by admin; re-entering Reconnecting
[11:31:56 DBG] Mesh: accepted AlarmCommand (correlationId=39ba451173954229a71b9dd1e00ac1b9) over gRPC; handed to the node communication actor.

Check 8 — interceptor denial counts across every node, whole run:

$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1  telemetry=0 configserve=0 localdbsync=0 mesh=0
central-2  telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-1   telemetry=0 configserve=0 localdbsync=0 mesh=2   # the two deliberate negatives
site-a-2   telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-1   telemetry=0 configserve=0 localdbsync=0 mesh=1   # the one deliberate negative
site-b-2   telemetry=0 configserve=0 localdbsync=0 mesh=0

The TwoNodeClusterHarness I-1 null-provider Error (a known test-harness artifact) did not appear in any rig log — grep -c "IMeshPskProvider" = 0 on all six nodes, as expected for production wiring where every admin node registers the provider.

Phase 1A defect — CLOSED by c0475f2d + e54883a8, re-gated in run 2

Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter. WithOtOpcUaControlPlaneSingletons's WithActors block resolves IMeshCommandDialer and IMeshPskProvider from IServiceProvider to hand to CentralCommunicationActor.Props. MeshGrpcCommandDialer takes IMeshPskProvider, which takes ISecretResolver, which AddZbSecrets builds by resolving IAuditWriter, which on an admin node is ActorAuditWriter — and its constructor does registry.Get<AuditWriterActorKey>(). That singleton is registered ~60 lines later in the same builder, so at this point the key is absent and ActorRegistry.Get throws MissingActorRegistryEntryException, failing AkkaHostedService.StartAsync and the whole host.

Three things make this worth a pinned regression rather than a one-line fix note:

• It is not mode-gated. The two GetService calls run in every transport mode by design (so a mode flip needs no restart), so the eager graph is walked on every admin boot. Proven above with a ClusterClient scratch container on the same image.
• It is invisible to the suite. The 1A.2 container-activation test (It_activates_from_the_container_despite_the_optional_client_factory_parameter) resolves the dialer from a service collection that does not include AddZbSecrets, and TwoNodeClusterHarness registers no IMeshPskProvider at all — so no existing test walks the real graph in starter order. This is the same class as scadaproj#1 ("only visible with the DI graph built in a real host") and as mesh Phase 6's IClusterRoleInfo boot crash.
• The container lies about it. The process does not exit, so docker ps reports Up with restarts=0 while the node serves no HTTP at all. Any Phase 2/4 readiness check that reads container status rather than /health/ready would score this as healthy.

The shape of the fix is the one this file already uses two lines above for the coordinator proxy — hand the actor a lazy accessor (Func<IMeshCommandDialer?> / Func<IMeshPskProvider?>, or resolve on first use) so nothing is constructed while the ActorSystem is still building.

That is what shipped. c0475f2d moved both dependencies behind Func<> invoked in PreStart, pinned by ControlPlaneStarterOrderTests (boots a real admin host through the production starter order and asserts StartAsync; verified RED against the eager resolve in 3/3 modes). e54883a8 then made the resolution memoize-on-first-use rather than PreStart-only, because PreStart runs concurrently with the remaining starters and can still lose the race — which run 2 observed happening on 2 of 3 real admin boots, so that hardening was not theoretical. Both are re-gated above; see finding 5 for the live self-heal proof.

Rig state after run 1 (superseded by the run-2 restoration below)

The compose flip was reverted and the six nodes recreated on the committed all-ClusterClient compose. Because defect 4 was mode-independent, the branch image could not produce a healthy central pair on any mode, so the rig was returned to its pre-gate image rather than left with a dead central pair: the run-1 build was preserved as otopcua-host:grpc-1a (69360a173f1f) and the pre-gate build (8bd81d8aea9b) re-tagged back to otopcua-host:dev.

Rig state after run 2 — restored, six healthy, on the branch image

The flip was reverted (git checkout docker-dev/docker-compose.yml) and the six nodes recreated on the committed all-ClusterClient compose. No image substitution was needed this time: otopcua-host:dev is the run-2 build of the committed source (1d8452a29018), and it boots a healthy central pair, so the rig runs the code under test. The run-1 workaround tag otopcua-host:grpc-1a was deleted to stop anyone reusing a stale image.

central-1  running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
central-2  running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-a-1   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-a-2   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-b-1   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-b-2   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
$ curl … /health/ready → status Healthy, members 2, unreachable 0
$ grep -c "Mesh ClusterClient for cluster"  → 3 on each central node (MAIN + SITE-A + SITE-B)

The three ClusterClients reappearing on the rollback is an incidental but useful result: the dark switch still rolls back cleanly on the fixed branch, which is what Phase 2 will depend on if its cutover has to be reversed.

---

Phase 1B — node→central ack plane — PASS

Branch feat/grpc-central-ack @ 4a2a3d3a (rebased on master, which carries the merged 1A). Rig-only flip: site-a-1/-2 → Grpc; site-b stays ClusterClient; central stays ClusterClient (its command direction) — acks are governed by the node's mode, so this isolates 1B.

#	Check	Result
1	Deploy e2e: site-a acks arrive via ReportApply, NodeDeploymentState rows land, deployment seals	PASS — deployment 53BD2626… Status=2 (Sealed) with all six rows Status=1 (Applied) inside ~50 ms; both site-a nodes logged Posted the ApplyAck … to http://central-1:4055 and central-1 logged the matching ApplyAck … from site-a-1:4053 / site-a-2:4053 … handed to the central communication actor. 10/10 acks delivered across the whole run, 0 drops on any node
2	Site-b acks still arrive via ClusterClient in the SAME deployment — per-pair coexistence	PASS — in 53BD2626… site-b-1/-2 rows are Applied while 0 ReportApply lines name a site-b node on either central node, and site-b-1/-2 each hold a live Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]; conversely site-a-1/-2 hold 0 receptionist associations and created 0 ClusterClients. Each pair's only possible ack route is the one its mode selects
3	Exactly one ack row per site-a node per dispatch (no double-count from endpoint failover)	PASS — across 10 dispatches, acks=1 for every (site-a node, deployment) pair on the happy path and on the docker stop failover. The one exception is the 4b pause drill (458b8629…, acks=2 per node) and it is the documented at-least-once case, not a defect: the frozen server had in fact received the ack, and the coordinator absorbed it without inflating the count — it reported 2/6 acks landed, not 4/6. NodeDeploymentState holds exactly one row per node per deployment throughout
4	Kill central-1 (docker stop) before the apply: site-a acks fail over to central-2; restart, verify convergence	PASS — with central-1 stopped, both site-a nodes logged the Information failover line … to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable) and central-2 accepted both within the same second; the deployment honestly ended Status=4 (TimedOut) with central-1 alone at Status=0 — sealing 6/6 is structurally impossible while one of the six expected nodes is down. After start central-1 the pair reconverged (members 2, unreachable 0) and the next dispatch sealed 6/6
4b	Hang drill (docker pause central-1): the per-endpoint FRESH deadline is what lets central-2 be reached at all	PASS — with central-1 paused (hung, not refusing), both site-a nodes logged … to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => DeadlineExceeded) and the ack landed on central-2. Observed failover latency ≈ 10 s (dispatch accepted 13:03:51 → ack at 13:04:01), i.e. exactly one AckTimeoutSeconds. A single hoisted deadline would have entered the central-2 attempt already expired and dropped the ack; this is the live proof of the fix in 667639e6
5	Auth probes against central :4055 CentralCommandService/ReportApply	PASS — 7/7 as designed: no bearer ⇒ PermissionDenied; SITE-B's key claiming SITE-A ⇒ PermissionDenied; no x-otopcua-cluster ⇒ PermissionDenied; unknown cluster SITE-Z with a valid key ⇒ PermissionDenied; unauthenticated against central-2 ⇒ PermissionDenied; correct key+header with a fake deployment id ⇒ {} (auth passed, coordinator ignored it — 0 rows written for that id); malformed id ⇒ InvalidArgument (auth passed, payload refused). Positive control: SITE-B key + SITE-B header also reaches the decoder
6	ConfigServe (DeploymentArtifactService) on the same listener unaffected	PASS — all four site nodes Fetched + verified a 28 296-byte artifact from http://central-1:4055 on a real revision change, and a direct authenticated Fetch returned artifact bytes; unauthenticated Fetch is refused by its own interceptor (Unauthenticated / "Config-serve authentication failed."), never by the mesh one — which is the method-path scoping holding. 1 config-serve denial in the whole run: the deliberate probe

Provenance

docker compose -f docker-dev/docker-compose.yml up -d --build from the worktree /Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1b, exit 0; image otopcua-host:dev rebuilt to 4105cf631003 at 12:51:47 UTC. All six host nodes plus migrator, cluster-seed and traefik were recreated (both one-shots exit 0); sql and its volume were kept. Nothing else touched the stack during the run, so it is not a concurrent-rebuild-invalidated one. Four rig-only Serilog MinimumLevel:Override Debug scopes were added on all six nodes for this run only — ZB.MOM.WW.OtOpcUa.Runtime.Communication (the sender's Posted the ApplyAck line), ZB.MOM.WW.OtOpcUa.AdminUI.Grpc (central's accept line), ZB.MOM.WW.OtOpcUa.ControlPlane.Communication and Akka.Cluster.Tools.Client — because the rig's default Serilog level is Information and checks 1–3 would otherwise have had no signal. Flip and overrides reverted after the gate, never committed.

$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1  running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=2
central-2  running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=2
site-a-1   running restarts=0 mode=Grpc          hostingFailed=0 resolveErr=0
site-a-2   running restarts=0 mode=Grpc          hostingFailed=0 resolveErr=0
site-b-1   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-b-2   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0

$ curl … /health/ready      → Healthy, members 2, unreachable 0  (MAIN)
$ … http://<site>:8080/health/ready → 200 on all four site nodes

Checks 1 + 2 evidence — the two transports, side by side in one deployment

$ curl -X POST http://localhost:9200/api/deployments -H 'X-Api-Key: docker-dev-deploy-key' \
       -d '{"createdBy":"1b-gate-check1"}'
{"outcome":"Accepted","deploymentId":"53bd2626-ce9f-4255-a964-24e4cf3c7ccb", …}   HTTP 202

$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='53bd2626-…'
2                                            -- Sealed
$ … SELECT NodeId, Status, AppliedAtUtc FROM dbo.NodeDeploymentState WHERE DeploymentId='53bd2626-…'
central-1:4053|1|2026-08-06 12:55:53.909     site-a-1:4053|1|2026-08-06 12:55:53.930
central-2:4053|1|2026-08-06 12:55:53.890     site-a-2:4053|1|2026-08-06 12:55:53.936
site-b-1:4053|1|2026-08-06 12:55:53.915      site-b-2:4053|1|2026-08-06 12:55:53.924

# node side — the gRPC pair:
site-a-1  [12:55:53 DBG] Posted the ApplyAck for 53bd2626… (Applied) to http://central-1:4055.
site-a-2  [12:55:53 DBG] Posted the ApplyAck for 53bd2626… (Applied) to http://central-1:4055.
# central side — same two acks, named:
central-1 [12:55:53 DBG] ApplyAck for 53bd2626… from site-a-1:4053 (Applied) handed to the central communication actor
central-1 [12:55:53 DBG] ApplyAck for 53bd2626… from site-a-2:4053 (Applied) handed to the central communication actor
central-2 (none)

# per-node ReportApply accept counts, BOTH central nodes, this deployment:
central-1  site-a-1=1  site-a-2=1  site-b-1=0  site-b-2=0  central-1=0  central-2=0
central-2  site-a-1=0  site-a-2=0  site-b-1=0  site-b-2=0  central-1=0  central-2=0

The site-b half is proven by exclusion plus its live association, which is the same shape 1A used for the site pairs:

$ for c in site-b-1 site-b-2; do docker logs … | grep -c "Posted the ApplyAck"; done   → 0, 0
site-b-1  [12:53:02 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
site-b-2  [12:53:07 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]

$ for c in site-a-1 site-a-2; do docker logs … | grep -c "Connected to .*receptionist"; done → 0, 0
$ for c in site-a-1 site-a-2 site-b-1 site-b-2; do grep -c "Mesh ClusterClient for cluster"; done → 0,0,0,0

Site-b rows are Applied while nothing on either central node logged a ReportApply for them, and site-a holds no receptionist association at all — so neither pair could have used the other's transport. Coexistence is per pair, in one deployment, from one dispatch.

Check 3 evidence — one ack per dispatch, and the duplicate that proved the absorber

Across all ten dispatches, counting ReportApply accepts on both central nodes:

site-a-1 dep=53bd2626… acks=1   site-a-2 dep=53bd2626… acks=1
site-a-1 dep=d0869c9e… acks=1   site-a-2 dep=d0869c9e… acks=1
site-a-1 dep=17507001… acks=1   site-a-2 dep=17507001… acks=1     # the stop-failover dispatch
site-a-1 dep=b1e837a2… acks=1   site-a-2 dep=b1e837a2… acks=1
site-a-1 dep=a394eec6… acks=1   site-a-2 dep=a394eec6… acks=1
site-a-1 dep=9bb520a4… acks=1   site-a-2 dep=9bb520a4… acks=1
site-a-1 dep=458b8629… acks=2   site-a-2 dep=458b8629… acks=2     # the 4b PAUSE dispatch
site-a-1 dep=fc2b7b64… acks=1   site-a-2 dep=fc2b7b64… acks=1
site-a-1 dep=41e4935d… acks=1   site-a-2 dep=41e4935d… acks=1
site-a-1 dep=4126636e… acks=1   site-a-2 dep=4126636e… acks=1
$ … SELECT NodeId, COUNT(*) FROM dbo.NodeDeploymentState WHERE DeploymentId='53bd2626-…' GROUP BY NodeId
central-1:4053|1  central-2:4053|1  site-a-1:4053|1  site-a-2:4053|1  site-b-1:4053|1  site-b-2:4053|1

A refused endpoint (docker stop) cannot produce a duplicate — the server never saw the call — which is why 17507001… is 1 per node despite failing over. The pause drill is the one shape that can, and it did:

central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed …   # failover target
central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed …
central-1 [13:04:33 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed …   # at `docker unpause`
central-1 [13:04:33 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed …
central-1 [13:05:03 INF] Coordinator recovered in-flight deployment 458b8629… (2/6 acks landed)

2/6, not 4/6. The frozen central had in fact received both acks and replayed them on unpause, and ConfigPublishCoordinator.HandleAck's NodeId-keyed absorption made the duplicate free — the exact claim GrpcApplyAckSender's "at-least-once walk" remark makes, observed rather than argued.

Check 4 evidence — refusal failover

$ docker compose stop central-1                                                   # 12:57:11
$ curl -X POST … -d '{"createdBy":"1b-gate-check4-failover"}'                      # 12:57:30
site-a-1 [12:57:30 INF] Posted the ApplyAck for 17507001… (Applied) to http://central-2:4055
                        after 1 endpoint(s) failed (http://central-1:4055 => Unavailable).
site-a-2 [12:57:30 INF] …identical…
central-2 [12:57:30 DBG] ApplyAck for 17507001… from site-a-1:4053 (Applied) handed …
central-2 [12:57:30 DBG] ApplyAck for 17507001… from site-a-2:4053 (Applied) handed …

$ … SELECT NodeId, Status FROM dbo.NodeDeploymentState WHERE DeploymentId='17507001-…'
central-1:4053|0   central-2:4053|1   site-a-1:4053|1   site-a-2:4053|1
site-b-1:4053|1    site-b-2:4053|1
$ … SELECT Status FROM dbo.Deployment …   → 4 (TimedOut) after the 2-minute deadline

$ docker compose start central-1                                                  # 13:00:10
$ curl … /health/ready → Healthy, members 2, unreachable 0
$ curl -X POST … -d '{"createdBy":"1b-gate-check4-converge"}'  → Status=2, all six Status=1

One operational detail worth carrying. After central-1 returned, site-a-1's cached GrpcChannel to http://central-1:4055 stayed in transient failure for roughly two minutes while site-a-2 reached the same restarted endpoint immediately — three dispatches apart:

13:01:25  site-a-1 → central-2 after (central-1 => Unavailable)   |  site-a-2 → central-1 OK
13:01:48  site-a-1 → central-2 after (central-1 => Unavailable)   |  site-a-2 → central-1 OK
13:02:49  site-a-1 → central-1 OK                                 |  site-a-2 → central-1 OK
$ docker exec otopcua-dev-site-a-1-1 bash -c '</dev/tcp/central-1/4055 && echo OPEN'  → OPEN
$ central-1 mesh denials in that window → 0     # the call never reached the interceptor

TCP was open and central-1 logged nothing, so this is the client channel's own reconnect backoff, not a server or auth problem. No ack was lost — the ordered walk absorbed it every time and said so in the log. The Phase 2 reading: a central restart shifts that node's ack load onto the surviving endpoint for a minute or two afterwards, which a soak's failover-line count will show as a burst rather than a fault.

Check 4b evidence — the hang, and why the fresh deadline is the whole check

$ docker pause otopcua-dev-central-1-1                                     # 13:03:35
$ curl … /health/ready → self akka.tcp://otopcua@central-2:4053            # traefik dropped central-1
$ curl -X POST … -d '{"createdBy":"1b-gate-check4b-hang"}'                 # 13:03:51  (202)
site-a-1 [13:04:01 INF] Posted the ApplyAck for 458b8629… (Applied) to http://central-2:4055
                        after 1 endpoint(s) failed (http://central-1:4055 => DeadlineExceeded).
site-a-2 [13:04:01 INF] …identical…
central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed …
central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed …
$ … SELECT NodeId, Status, AppliedAtUtc … WHERE DeploymentId='458b8629-…'
site-a-1:4053|1|2026-08-06 13:04:01.837     site-a-2:4053|1|2026-08-06 13:04:01.774
central-1:4053|0|NULL  central-2:4053|0|NULL  site-b-1:4053|0|NULL  site-b-2:4053|0|NULL
$ docker unpause otopcua-dev-central-1-1                                   # 13:04:33

DeadlineExceeded — not Unavailable — is the discriminator: the first endpoint burned a full AckTimeoutSeconds (10 s, the default; 13:03:51 → 13:04:01) and central-2 was still dialled with a live deadline. Under the hoisted deadline this branch shipped before 667639e6, that second attempt would have started already expired and the ack would have been dropped with a Warning. Observed failover latency: ≈10 s, one whole AckTimeoutSeconds.

Collateral the drill produced, recorded because it is the honest picture of a pause. Pausing a cluster member is a heavier disturbance than stopping one: central-2 marked central-1 UNREACHABLE at 13:03:46 and auto-downed it at 13:04:01, and after unpause the two central nodes each held a one-member view until central-1 was restarted (13:06:36). In that window the two gRPC acks landed and none of the four ClusterClient acks did — central-2 and both site-b nodes all logged received DispatchDeployment at 13:03:51 yet produced no ack the coordinator saw, and the deployment ended TimedOut at 2/6 naming exactly those four:

central-1 [13:05:51 WRN] Deployment 458b8629… timed out after 00:02:00 (2/6 acks landed).
          No ack from: central-1:4053, central-2:4053, site-b-1:4053, site-b-2:4053
central-1 [13:06:36 WRN] DriverHost central-1:4053: found orphan Applying row for deployment
          458b8629-…; replaying
central-1 [13:06:37 INF] DriverHost central-1:4053: applied deployment 458b8629… (children=10)

Stated at the strength the evidence supports: the four ClusterClient nodes received the dispatch and none of their acks reached the coordinator, while both gRPC nodes' acks did. The site nodes emit no apply-completion line, so whether each of the four actually finished applying is not determinable from their logs — what is determinable is that the ack did not arrive and the deadline said so honestly. The boot-replay backstop then repaired central-1. Not a 1B finding either way (no gRPC ack was involved), but it is the Phase 2 auto-down behaviour showing up early, and the split view needed a manual restart central-1 to heal.

Check 5 evidence — the auth matrix on the ack endpoint

-import-path onto the in-repo proto (the node serves no reflection). PermissionDenied means the interceptor refused; anything past it ({} or InvalidArgument) means auth passed and only the payload was graded:

$ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos
$ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \
     -plaintext -import-path /protos -proto mesh_transport.proto"
$ BODY='{"deploymentId":"ffffffffffffffffffffffffffffffff","nodeId":"site-a-1:4053",
         "outcome":"APPLY_OUTCOME_APPLIED","correlationId":"eeeeeeee…"}'

# 5a  no bearer, cluster header present            -> central-1
Code: PermissionDenied   Message: Central command authentication failed.
# 5b  Bearer dev-mesh-psk-SITE-B + cluster SITE-A  -> central-1
Code: PermissionDenied   Message: Central command authentication failed.
# 5c  Bearer dev-mesh-psk-SITE-A, NO cluster header-> central-1
Code: PermissionDenied   Message: Central command authentication failed.
# 5d  Bearer dev-mesh-psk-SITE-A + cluster SITE-A, FAKE deployment id
{}
# 5e  same, MALFORMED deployment id "not-a-guid"
Code: InvalidArgument    Message: undecodable ApplyAck
# 5f  Bearer dev-mesh-psk-SITE-B + cluster SITE-B   (positive control)
Code: InvalidArgument    Message: undecodable ApplyAck
# 5g  Bearer dev-mesh-psk-SITE-A + cluster SITE-Z   (unknown cluster)
Code: PermissionDenied   Message: Central command authentication failed.
# 5h  no bearer                                     -> central-2
Code: PermissionDenied   Message: Central command authentication failed.

The accepted fake ack changed nothing — the coordinator ignored a deployment it is not tracking:

$ … SELECT COUNT(*) FROM dbo.NodeDeploymentState WHERE DeploymentId='ffffffff-…'   → 0
$ … SELECT COUNT(*) FROM dbo.Deployment           WHERE DeploymentId='ffffffff-…'   → 0
central-1 [13:08:08 DBG] ApplyAck for ffffffff… from site-a-1:4053 (Applied) handed to the
          central communication actor

Central's denial log separates the three operator-facing causes even though the wire status is deliberately uniform:

[13:08:07 WRN] Rejected a central command call to /mesh_transport.v1.CentralCommandService/ReportApply
               claiming cluster SITE-A: no bearer token presented.
[13:08:07 WRN] … claiming cluster SITE-A: the presented bearer token did not match that cluster's
               pre-shared key, no key could be established for it, or it is not a cluster this node knows…
[13:08:07 WRN] Rejected a central command call to …/ReportApply: no x-otopcua-cluster header, so there
               is no cluster whose pre-shared key the caller could be graded against.
[13:08:25 WRN] … claiming cluster SITE-Z: … or it is not a cluster this node knows…

5g is the known-cluster gate answering live: a valid SITE-A key presented for SITE-Z is refused in O(1), which also confirms the gate had in fact been published — see observation (b).

Check 6 evidence — ConfigServe on the same listener

A real node-driven fetch needed a real revision change, so ServerCluster.Notes for MAIN was edited, deployed, and restored (the revision hash returned to its pre-gate 642b9d6f79b9…, verified by a further dispatch):

$ … UPDATE dbo.ServerCluster SET Notes = '<original> | 1b-gate-fetch-probe' WHERE ClusterId='MAIN'
$ curl -X POST … → revisionHash 20af950fb211…   (was 642b9d6f79b9…)   Status=2
site-a-1 [13:09:31 INF] Fetched + verified artifact for 41e4935d… from http://central-1:4055 (28296 bytes).
site-a-2 [13:09:31 INF] …same…      site-b-1 [13:09:32 INF] …same…      site-b-2 [13:09:32 INF] …same…
$ … UPDATE dbo.ServerCluster SET Notes = '<original>' …
$ curl -X POST … → revisionHash 642b9d6f79b9…   Status=2      # restored, byte-identical

Both interceptors on the one listener, each answering for its own service:

# authenticated Fetch, config-serve key:
$ $G -H 'authorization: Bearer configserve-docker-dev-key' -d '{"deploymentId":"FC2B7B64-…"}' \
     central-1:4055 deployment_artifact.v1.DeploymentArtifactService/Fetch
{ "data": "eyJDbHVzdGVycyI6W3siQ2x1c3RlcklkIjoiTUFJTiIsIk5hbWUiOiJNYWluIGNsdXN0ZXIiL…" }
# unauthenticated Fetch — refused by the CONFIG-SERVE interceptor, not the mesh one:
Code: Unauthenticated    Message: Config-serve authentication failed.

Unauthenticated here versus PermissionDenied on the mesh service is the method-path scoping in CentralCommandAuthInterceptor holding: the two services share one AddGrpc pipeline and neither interceptor grades the other's calls.

Observations carried forward from the reviews

(a) The I-4 shape is not exercisable on this rig, and that is a property of the rig, not a skipped check. MeshPskProvider.TryGetPsk consults MeshTransport:ClusterPsks before the secret store and caches only store-resolved values, and docker-dev supplies MeshTransport__ClusterPsks__{MAIN,SITE-A,SITE-B} on both central nodes. The InvalidateAll() a contact-refresh tick performs (ContactRefreshSeconds = 60) therefore drops an always-empty cache, and an ack arriving just after a tick pays no store read. No latency delta was measurable and none should be: every ack in the run landed inside the same second it was posted, refresh tick or not. A real deployment resolving OT-MESH-PSK-<clusterId> from the store is where I-4 becomes observable — Phase 2 on a store-backed rig, not here.

(b) The Failed to resolve Error fired on 4 of 4 admin boots — and healed every time. Worse odds than 1A's 2 of 3, and both dependencies lost the race on every boot:

central-1  [12:53:00 ERR] Failed to resolve IMeshCommandDialer … / IMeshPskProvider …   # boot 1
central-1  [13:00:29 ERR] …both again…                                                  # boot 2 (check 4 restart)
central-1  [13:06:36 ERR] …both again…                                                  # boot 3 (4b restart)
central-2  [12:53:02 ERR] …both…                                                        # its single boot
central-1  [12:53:00 ERR] No IMeshPskProvider was supplied to this actor; the known-cluster gate
           will NEVER be published and cached per-cluster keys will NEVER be dropped…

The second line is the one that reads worst, and 1B disproves it directly rather than by argument: every site-a ack central-1 accepted required that provider, including the ones after its third boot (13:07:36, 13:09:31, 13:09:59), and probe 5g's SITE-Z refusal is only possible if SetKnownClusters had in fact been published. The memoize heals it; the Error text does not say so. This is the 1A finding-5 carry-forward, now with a second, independent live confirmation and a stronger case for the wording fix — the message contains the word "NEVER" twice about something that self-corrects within one deploy.

(c) Zero mesh interceptor denials outside the deliberate probes. Whole-run counts on every node and every interceptor:

central-1  telemetry=0 configserve=1 localdbsync=0 meshCentralCmd=4 nodeCmd=0
central-2  telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=1 nodeCmd=0
site-a-1   telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0
site-a-2   telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0
site-b-1   telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0
site-b-2   telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0

meshCentralCmd = 4 on central-1 (probes 5a/5b/5c/5g) + 1 on central-2 (5h) = exactly the five deliberate negatives; configserve = 1 = the deliberate 6b. Nothing else was refused anywhere.

(d) Pre-existing rig noise, so a Phase 2 error grep is not surprised by it. Site nodes log Unavailable … Name or service not known from the historian client — the rig points ServerHistorian__Endpoint at http://histgw-absent.invalid:5222 on purpose (45 occurrences on site-a-1, 1 on site-b-1, 4 on central-1 in this run). Nothing to do with the ack plane.

Rig state after the gate — restored, six healthy, all-ClusterClient, on the branch image

The flip and the four Serilog overrides were reverted (git checkout docker-dev/docker-compose.yml, grep -c "GATE-1B ONLY" → 0) and the nodes recreated on the committed compose. The simultaneous recreate cold-started both central nodes at once and they self-formed one-member views — the known simultaneous-cold-start split-brain the (default-OFF) Cluster:BootstrapGuard addresses — healed with a single restart central-2:

central-1  running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
central-2  running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
site-a-1   running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
site-a-2   running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
site-b-1   running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
site-b-2   running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0

$ for h in central-1 central-2; do … /health/ready; done
central-1 Healthy members 2 unreachable 0 leader akka.tcp://otopcua@central-1:4053
central-2 Healthy members 2 unreachable 0 leader akka.tcp://otopcua@central-1:4053
$ … http://<site>:8080/health/ready  → 200 ×4

$ curl -X POST … -d '{"createdBy":"1b-gate-rig-restore-verify"}'
CDDA1D80-…|2      central-1|1 central-2|1 site-a-1|1 site-a-2|1 site-b-1|1 site-b-2|1
$ for c in site-a-1 site-a-2 site-b-1 site-b-2; do grep -c "Posted the ApplyAck"; done → 0,0,0,0
$ for c in central-1 central-2; do grep -c "Mesh ClusterClient for cluster"; done      → 3 per boot

Zero gRPC acks after the rollback and three ClusterClients per central boot: the dark switch rolls back cleanly in the ack direction too, which is what Phase 2 depends on if its cutover has to be reversed. Rig configuration is byte-identical to its pre-gate state (ServerCluster.Notes restored; revision hash back to 642b9d6f79b9…).

Phase 1B defects

None. No product code was changed during or after this gate.

---

Phase 2 — full-rig cutover + soak — PASS 8/8 + baseline + PSK negatives

Rig default flipped and committed: MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}" (caf2eec6, all six node entries). Rig rebuilt from master @ caf2eec6 — which also carries Task 2.0 (a94eb8a3+d7c9ead6, honest PreStart-race logging, review-approved) and the DI role-scoping pin (2e06dfe5+027f0403, review-approved) — image rebuilt, node containers recreated, no OTOPCUA_MESH_MODE override in the environment. Baseline was run on the pre-flip build first (see Baseline section, 1bb3a937).

#	Check	Result
1	Full rebuild all-Grpc: six nodes ready, deploy e2e seals, all acks via gRPC	PASS — deploy 949d2cc7 sealed (Status 2) 6/6 within ~1 s; every node logged accepted DispatchDeployment … over gRPC and Posted the ApplyAck … to http://central-1:4055, central logged each ApplyAck … handed to the central communication actor; wire-level: central holds established TCP to every node's :4056, node TIME_WAIT to central :4055 observed. An earlier identical deploy (8809464a, pre-Debug-override) also sealed 6/6 in ~150 ms
2	Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes	PASS — via authenticated grpcurl envelopes (the 1A shape): Restart SITEA-modbus → site-a-1 {} + restarting driver … by request of p2-gate + spawned + connected; Reconnect + AlarmCommand accepted + handed (correlation ids c2b1…/c2c1…); site-b-1 delivery proven (c2d1… accepted over gRPC). AdminUI origin composed per the 1A carry-forward (no UI control exists; same DispatchOverGrpc path driven by check 1)
3	Kill site-a-1 mid-dispatch: per-node Execute failure logged with node id + status (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path)	PASS — deploy 069b57a4 1 s after docker kill: central-2 WRN DispatchDeployment … was NOT delivered to site-a-1:4053 (Unavailable: Error connecting to subchannel.). It is dropped and will NOT be retried… — exactly 1 such line (no retry storm); 5/6 acked; timed out after 00:02:00 (5/6 acks landed). No ack from: site-a-1:4053 at exactly +2:00; restart → FetchAndCache boot-replay restored last-known-good, timed-out deployment NOT retro-acked (ack count stayed 5)
4	Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action	PASS — deploy ffc75eae with central-1 dead: site nodes logged Posted the ApplyAck … to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable) (the ordered walk, per node), all live acks landed on central-2; deadline honest at +2:00 naming central-1, site-a-1 (site-a-1 = the documented ~2-min GrpcChannel backoff carry-forward from its own earlier kill — loud, single-attempt). First post-restart probe caught central-1 still booting (5/6, honest TimedOut); second probe 594ecda3 sealed 6/6 — full convergence, no manual action beyond docker start
5	Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; no duplicate applies after the pair returns (boot-replay path only)	PASS — deploy e5a14c5f: BOTH NOT delivered to site-b-{1,2}:4053 (Unavailable…) WRN lines in the SAME SECOND as dispatch (baseline (b): zero correlated lines, sampled-deadLetters INF); timed out after 00:02:00 (4/6…). No ack from: site-b-1:4053, site-b-2:4053; pair restart → boot-replay only (FetchAndCache boot — restored served state), rows stay 0/0, ack count stays 4 — no dupes, no resurrection
6	Zero ClusterClient activity under Grpc: docker logs <every node> | grep -ci "clusterclient" = 0 new lines post-boot	PASS — final sweep after the full soak: grep -ic "clusterclient|receptionist" = 0 on all six nodes, covering checks 1–5, every kill/restart cycle, and the whole 135-min soak
7	Soak ≥ 2 h under periodic deploys + command traffic: 0 PermissionDenied/Unauthenticated across all six nodes (PSK drift detector), 0 unexplained Unavailable	PASS — 135 min (15:03:44Z→17:18Z), 14/14 deploy cycles sealed 6/6 (most in 0–1 s) + Restart/AlarmCommand traffic every cycle, 0 traffic failures. Grep since soak start: PermissionDenied=0, Unauthenticated=0 on all six (PSK drift detector clean). Review items, both explained: (a) site-a-2 Unavailable×3807 — ALL at DBG/INF (its WRN/ERR count was 0), the alarm-historian SendEvent retry loop failing DNS on the HistorianGateway host, which does not exist on this rig — pre-existing rig-shape noise, not mesh; (b) central-2 WRN×28 = exactly 14 deploys × 2 known per-deploy StartDeployment authoring warnings (TAG-gatelong ModbusRegion + script-compile RSS advisory) — pre-existing on the old transport. The predicted post-central-restart failover-line burst did not occur during the soak because no node was killed inside the soak window (kills all preceded it)
8	Full suite green on the cutover SHA	PASS — dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo exit 0, zero failures across every assembly, run in parallel with the live rig + soak traffic. Spot totals match every baseline: ControlPlane 245 (Task 2.0 set), Host 95 (role-scoping pins), Runtime 558+3skip, AdminUI 1109, Commons 346, Cluster 193, Host.IntegrationTests 212+4skip

Extra (plan-text requirement): PSK negatives on RUNNING containers, both services — PASS: NodeCommandService/Execute @ site-a-1: no bearer → PermissionDenied: Node command authentication failed.; SITE-B key → same uniform denial; SITE-A key + empty envelope (positive control) → InvalidArgument: Command envelope could not be decoded. (interceptor ran, payload rejected — auth and decode are distinct). CentralCommandService/ReportApply @ central-1:4055: no bearer / wrong key / SITE-A key claiming x-otopcua-cluster: SITE-Z → all PermissionDenied: Central command authentication failed. (uniform text, unknown-cluster claim refused).

Gate observability note: after check 1's first pass, the six node containers were recreated once with an UNCOMMITTED scratchpad compose override raising Serilog default to Debug (the mesh delivery lines are DBG; the base rig logs at Information and the 1A/1B gates had used the same temporary raise). Same image, same SHA — no rebuild, so no void-run. All kill drills ran after that recreation.

---

Phase 4 — terminal gate on the post-deletion build — PASS 8/8 — PROGRAM CLOSED

Run 2026-08-06 (log times UTC, 23:53–00:27 spanning midnight into 08-07). master @ 0b194003 (= merge 9420f8ef + one docs-only commit; the Phase 3 deletion merge is 2fa880b6). Provenance guard PASS: image otopcua-host:dev = d2a290fc5711, built 2026-08-06 17:55:10 EDT — later than both the deletion merge (16:57:05 EDT) and the last code merge 9420f8ef (17:28:48 EDT), so the boundary checks ran on the post-deletion binary. No rebuild during the gate. The six node containers were recreated twice before any drill (once for the gate's Debug scopes, once to widen them to the ack plane — the "Posted the ApplyAck" sender logs in Runtime.Communication and the central receiver in AdminUI.Grpc, neither covered by the 1A/2 scope set); same image both times, so no void-run. Check 1 ran before the second recreation (interceptor probes — log-reset does not affect them); checks 2–7 after it, so the final greps cover every drill. Override reverted after the gate, never committed.

#	Check	Result
1	PSK negatives on both services (no key / wrong key / wrong cluster's key); unset server key ⇒ all calls rejected (fail-closed proven live)	PASS — 7/7: Execute @ site-a-1 no-bearer → PermissionDenied: Node command authentication failed., SITE-B key → same uniform text, SITE-A key + empty envelope → InvalidArgument (interceptor ran, payload rejected); ReportApply @ central-1 no-bearer / garbage key / SITE-A key claiming x-otopcua-cluster: SITE-Z → all PermissionDenied: Central command authentication failed. (uniform, unknown-cluster refused = no key configured for that cluster ⇒ every call rejected), SITE-A key + SITE-A claim → InvalidArgument: undecodable ApplyAck (positive control). Unset-key fail-closed at the earliest gate: scratch container on the rig image, Grpc mode, GrpcPsk removed → Hosting failed to start, OptionsValidationException naming MeshTransport:GrpcPsk + the env form (scratch removed after capture)
2	Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand)	PASS — deploy 22dfec4b fanned as ONE envelope: all six nodes logged accepted DispatchDeployment (correlationId=c532b8f7…) in the same second. Restart + Reconnect driven from the real /hosts AdminUI panel for the first time in any gate (the #546 wire-in closed the 1A "structurally undrivable" carry-forward): each fanned to all six under a single correlation id (41157cfb… / cb81bc5e…), and BOTH SITE-A nodes acted (restarting driver SITEA-modbus by request of multi-role-test + spawned, ForceReconnect requested by admin) — actor = the logged-in operator, i.e. the full AdminOperationsActor.Dispatch → DispatchOverGrpc origin path live. AlarmCommand: authenticated envelopes accepted + handed on site-b-1 and central-1
3	Ack matrix: every node acks exactly once, deployment seals	PASS — for 22dfec4b: every node logged exactly ONE Posted the ApplyAck … (Applied) to http://central-1:4055, central-1 served exactly 6 ReportApply 200s and handed 6 acks to the central communication actor, Deployment.Status=2 with 6/6 NodeDeploymentState Applied inside a ~30 ms spread
4	Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry)	PASS — kill central-1 → deploy 3edf043d: all five live nodes posted to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable) (the ordered walk, one loud INF per node), deadline honest at exactly +2:00 naming only central-1 (5/6). Restart → first re-deploy hit the documented ~2-min GrpcChannel backoff (ONE was NOT delivered … will NOT be retried WRN, honest TimedOut), next dispatch sealed 6/6 — no manual action beyond docker start. Kill site-a-1 + dispatch in the same second → exactly 1 NOT delivered to site-a-1 (Unavailable…) line (no retry storm), 5/6 acks, deadline at exactly +2:00 naming only site-a-1; restart → FetchAndCache boot — restored served state and the timed-out deployment NOT retro-acked (count stayed 5)
5	Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart)	PASS — deploy f311cf5b dispatched and the coordinator host (central-2) killed in the same second with the deployment open; central-1's recovered coordinator fired the deadline at exactly +2:00 (timed out … (5/6 acks landed). No ack from: site-b-2:4053 — the deliberately-stopped node), with central-2's own ack — posted before its death — preserved in the 5. After restarting both: per-node Posted the ApplyAck count for f311cf5b exactly 1 on the five that acked / 0 on site-b-2, DB count stayed 5 (no retro-ack, no dupes), site-b-2 boot-replayed last-known-good, and reconverge deploy 12a46546 sealed 6/6
6	Boundary check: no Akka association between any site container and central	PASS — netstat in every node's netns: exactly THREE established :4053 associations rig-wide, each strictly pair-internal (central-1↔central-2, site-a-1↔site-a-2, site-b-1↔site-b-2 by container IP); zero site↔central :4053 connections. Baseline (a)'s unauthenticated cross-mesh association is gone at the TCP level
7	Full-rig restart (pairs together): clean convergence; grep -ri "receptionist|clusterclient" over all node logs = 0 lines	PASS, with one non-mesh observation — all six restarted in one compose restart; both SITE pairs re-formed cleanly within 7 s; the CENTRAL pair hit the documented simultaneous-cold-start split-brain (same-second mutual InitJoinNack → both self-formed — the exact scenario the Phase 7 bootstrap guard d1dac87f ships for, opt-in and default-OFF, not enabled on this rig; pre-existing pair-formation behavior, zero mesh-transport involvement) and was healed by the standard operator action, a single-node restart of central-2, which rejoined as Secondary (memberCount=2 unreachableCount=0, leader central-1 — the Phase 7 rejoin drill behavior holding). Transport verdict clean both ways: `grep -icE "receptionist
8	MeshGrpcBoundaryTests + full suite green on the merge SHA	PASS — targeted MeshGrpcBoundaryTests 3/3; terminal full suite on master @ 0b194003: dotnet test direct exit-code capture (no pipe) RC=0, 50/50 suites, 9,333 passed / 0 failed — identical totals to the Phase-3 DoD baseline, run in parallel with the live rig per the Phase 2 precedent

Phase 4 carry-forward closures

Item	Disposition
1A check-3 AdminUI origin (closed by composition in 1A/2)	Now closed by direct evidence — #546 wired DriverStatusPanel into /hosts, and this gate drove Restart + Reconnect from the real page (fan + act, operator identity carried)
Failed to resolve Error-text cosmetics (1A finding 5 → Phase 2)	Closed in Phase 2 Task 2.0 (a94eb8a3+d7c9ead6, honest PreStart-race logging)
~2-min GrpcChannel backoff after a peer dies (1B → Phase 2 soak note)	Behavior confirmed three more times this gate (post-central-1-restart re-deploy; central-2→site-a-1 dispatch skip at 00:10:21 after the drill-B kill). Always loud (one WRN, single attempt), never lossy beyond the honest deadline, always healed by the next dispatch after the window. Stands as an operator note in docs/Redundancy.md territory — expect one TimedOut deploy within ~2 min of any node kill/restart
GrpcPort NULL→repair skip persists ≤ ContactRefreshSeconds (1A check 7)	Operational note stands (dispatch set rebuilds on the refresh tick); no Phase 4 re-check needed — the mechanism is unchanged since 1A proved it live
I-4 cold-PSK-cache latency after a refresh tick (1B → Phase 2 → Phase 4)	Re-carried out of the program with rationale, unchanged: still structurally unmeasurable on docker-dev — the rig configures MeshTransport:ClusterPsks, which answers ahead of the store, so the cache never holds a store-resolved value and InvalidateAll() clears nothing. Every ack in every phase landed within the second it was posted. To be measured once on the first store-backed deployment (a real ${secret:OT-MESH-PSK-…} fleet); until then it is an unquantified-latency note, not a correctness gap — the fail-closed behavior on a cold cache is pinned by unit tests

No new defects. The central-pair cold-start split-brain in check 7 is pre-existing, documented (Phase 7 bootstrap guard, deliberately opt-in), reproduced here only because a same-second pair restart is exactly its trigger — recorded as the reason an operator restarting a central pair together should either stagger the two nodes or enable Cluster:BootstrapGuard.

Program CLOSED. Both baseline gaps are shut on the wire: (a) zero cross-mesh Akka associations and zero receptionist/ClusterClient log lines on the post-deletion binary; (b) an unreachable node now costs one loud per-node WRN in the dispatch second plus an honest deadline, instead of a sampled dead-letter counter and silence.

Carry-forwards

Anything a phase gate could not drive gets named here with why + what proves it instead, and is re-checked in the next gate (the ScadaBridge proportionate-gate rule).

Item	Deferred from	Why it could not be driven	Proven instead by	Status
Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-GrpcPort skip	1A run 1	Both central nodes failed to boot (defect 4)	—	CLOSED in 1A run 2 — all three PASS live
Check 4 acknowledge half — the condition actually acknowledging	1A run 1	The rig's one ScriptedAlarm sits Normal and its owning central node was down	—	CLOSED in 1A run 2 — acked from the real /alerts page, ACKNOWLEDGED row observed
Check 8 for telemetry + config-serve	1A run 1	Central dials both planes and never booted, so neither carried a call — 0 denials was vacuous	—	CLOSED in 1A run 2 — both planes carried authenticated traffic with 0 denials
Check 3 central half — RestartDriver/ReconnectDriver originating at the AdminUI	1A runs 1 + 2	Structurally undrivable, not a rig problem: <DriverStatusPanel has 0 usages repo-wide (the 5 name hits are comments), no rendered page carries a Restart/Reconnect control, and the only non-script-analysis HTTP POST is /api/deployments. Pre-existing on master, not caused by this program	Composition the controller accepted: run 1's live node-side leg for both commands + run 2's check 4, which drives the identical AdminOperationsActor.Dispatch → DispatchOverGrpc origin path from the AdminUI for another command type; per-type difference is ToEnvelope, covered by the T0.2 goldens	CLOSED by composition. The orphaned panel itself is out of this program's scope — file separately if the control is meant to exist
Failed to resolve Error text says "continuing without it", which reads permanent although first use retries; and it will trip a soak's error grep on a healthy node	1A run 2 (finding 5)	Cosmetic/observability, not behaviour	The self-heal proven live (finding 5), and re-confirmed independently in 1B: 4 of 4 admin boots lost the race, yet every site-a ack central-1 accepted and probe 5g's SITE-Z refusal both require the provider it claimed would "NEVER" be published	Phase 2
I-4 cold-PSK-cache latency after a contact-refresh tick	1B	Not exercisable on docker-dev, structurally: TryGetPsk reads MeshTransport:ClusterPsks ahead of the store and caches only store-resolved values, and the rig configures that map — so InvalidateAll() clears an always-empty cache and no ack pays a store read	Every ack in the run landed inside the second it was posted, tick or no tick (1B observation (a))	Phase 2, on a store-backed rig
A restarted central's cached GrpcChannel can stay in transient failure for ~2 min on the node that saw it die, while a sibling node reaches the same endpoint at once	1B check 4	Not a defect — the ordered walk absorbed it and logged the failover each time; no ack lost. TCP was open and central logged nothing, so it is the client channel's backoff	3 dispatches across the window, 2 failed over and the 3rd went direct (1B check 4 evidence)	Phase 2 soak — expect a burst of failover lines after any central restart, not a fault

Defects the gates caught that the suite did not

Recorded per phase as they occur, with root cause + the pinned regression test.

#	Phase	Defect	Root cause	Status
4	1A gate run 1	An admin node cannot boot at all — MissingActorRegistryEntryException: AuditWriterActorKey, both central nodes, every transport mode	The first WithActors starter eagerly resolves IMeshCommandDialer/IMeshPskProvider, whose graph reaches ActorAuditWriter, whose ctor reads an ActorRegistry key registered ~60 lines later in the same builder	CLOSED — c0475f2d (Func<> + PreStart, pinned red-first by ControlPlaneStarterOrderTests) + e54883a8 (memoize-on-first-use). Re-gated in run 2: 6/6 nodes boot, 0 Hosting failed to start
5	1A gate run 2	The PreStart resolve loses its race on real boots — 2 of 3 observed admin boots logged Failed to resolve IMeshCommandDialer / IMeshPskProvider with the same AuditWriterActorKey cause	PreStart runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede AuditWriterActorKey's registration	Not a defect — it is the case e54883a8 was written for, and the memoize demonstrably heals it. Proven live by moving the coordinator singleton onto the node that lost the race (stop central-2) and watching it fan a dispatch to all five live nodes with no dialer error. Residual is cosmetic: the Error text and its effect on soak greps → Phase 2