lmxopcua/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md

# ClusterClient → gRPC migration — live gate record

Rig: `docker-dev/` (central-1/-2 = `admin,driver,cluster-MAIN`; site-a-1/-2, site-b-1/-2;
sql + traefik + migrator + cluster-seed), rebuilt from the branch under test via
`docker compose -f docker-dev/docker-compose.yml up -d --build`. Recorded check-by-check in
the family's live-gate format. Later phases' sections are appended as those phases land; every
check row gets a **PASS**/**FAIL** verdict plus one quantitative justification clause, with
fenced evidence (real commands, real output). A run invalidated by a concurrent rig rebuild is
**void** — note it and re-run (the ScadaBridge void-run rule).

Provenance discipline: every phase section opens with branch + commit SHA + "image rebuilt,
all N containers recreated", and Phase 4 additionally verifies the image build timestamp
post-dates the deletion merge (a stale image invalidates the boundary checks).

Plan: `2026-08-06-clusterclient-to-grpc.md`. Tracking:
`2026-08-06-clusterclient-to-grpc-tracking.md`.

---

## Baseline (run on the PRE-change build, before the Phase 2 flip)

The two gaps this program closes, **reproduced rather than assumed**:

**(a) Unauthenticated Akka remoting crosses the WAN boundary.** From the host:

```bash
# A live TCP association from a site container to central's Akka port:
docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
# And the receptionist actually accepting cross-mesh traffic (central log after any AdminUI
# driver restart targeting site-a):
docker logs otopcua-dev-central-1-1 2>&1 | grep -i "clusterclient\|receptionist" | tail -5
```

Expected: `OPEN`; receptionist/client lines present. Nothing on this path presents any
credential.

**(b) Silent drop on an unreachable cluster.** Stop the site-b pair, dispatch a deployment
(`POST http://localhost:9200/api/deployments`, `X-Api-Key: docker-dev-deploy-key`), observe:
no per-node dispatch error surfaces anywhere; site-b's `NodeDeploymentState` rows sit
`Applying` until the 2-minute deadline marks the deployment `TimedOut`/partially failed. The
deadline is the only witness. Restart the pair.

- [ ] Baseline (a) recorded with evidence — **PENDING**
- [ ] Baseline (b) recorded with evidence — **PENDING**

---

## Phase 0 — contract + auth + config groundwork — **PASS**

Branch `feat/grpc-phase0-contract-auth` @ `c2bfad4a`. Image rebuilt, all six containers
recreated (`docker compose -f docker-dev/docker-compose.yml up -d --build`, run from the
worktree at `/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-phase0` so the image carries this
branch). The rig was already running (from the main checkout's compose file) before this run;
the rebuild recreated all eight containers (sql kept, migrator + cluster-seed one-shots re-ran,
central-1/-2 + site-a-1/-2 + site-b-1/-2 + traefik recreated) under the fixed `otopcua-dev`
project name, so this run is not a concurrent-rebuild-invalidated one — nothing else touched
the stack afterwards.

| # | Check | Result |
|---|---|---|
| 1 | All six nodes boot on `ClusterClient` with the dark `Grpc` config present (PSKs, endpoints, ClusterPsks map); no validator fires | **PASS** — 6/6 nodes `restarts=0`, `docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}'` reports `running restarts=0` on every node; `docker logs <node> | grep -iE "OptionsValidationException\|MeshTransport.*(is empty\|refuses to start)"` = 0 lines on all six |
| 2 | Deliberate misconfig: a scratch container with `MeshTransport__Mode=Grpc` and **no** `GrpcPsk` refuses to boot, error names `MeshTransport:GrpcPsk` + the env form | **PASS** — `Hosting failed to start` + `OptionsValidationException` naming `MeshTransport:GrpcPsk` and `MeshTransport__GrpcPsk`, below |
| 3 | Same with empty `CentralGrpcEndpoints` — refuses, names the key | **PASS** — `OptionsValidationException` naming `MeshTransport:CentralGrpcEndpoints` and `MeshTransport__CentralGrpcEndpoints__0`, below |
| 4 | Full suite green, 0 warnings, on the branch | **PASS** — `dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo`: 0 failures on a clean run (see evidence below); no `.cs` files changed on this branch since the last full rebuild, so the pre-existing ~868-warning baseline is unchanged (nothing to newly warn about) |

Note: the new gRPC services are not yet mapped in Phase 0, so no wire probes here — the
interceptor negative/positive matrix is unit-tested and wire-probed in 1A/1B.

**Check 1 evidence** — no validator fired on any of the six nodes, all dark `Grpc` keys present:

```
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do
    docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' otopcua-dev-$c-1
  done
running restarts=0    # central-1
running restarts=0    # central-2
running restarts=0    # site-a-1
running restarts=0    # site-a-2
running restarts=0    # site-b-1
running restarts=0    # site-b-2

$ docker logs otopcua-dev-central-1-1 2>&1 | grep -iE "MeshTransport|ClusterClient"
[08:53:26 INF] Mesh ClusterClient for cluster MAIN created with 2 contact point(s)
[08:53:26 INF] Mesh ClusterClient for cluster SITE-A created with 2 contact point(s)
[08:53:26 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s)

$ docker logs otopcua-dev-site-a-1-1 2>&1 | grep -iE "cluster.client|receptionist"
[08:53:26 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]

$ docker inspect otopcua-dev-site-a-1-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep MeshTransport
MeshTransport__CentralGrpcEndpoints__1=http://central-2:4055
MeshTransport__CentralGrpcEndpoints__0=http://central-1:4055
MeshTransport__Mode=ClusterClient
MeshTransport__GrpcPsk=dev-mesh-psk-SITE-A
MeshTransport__CentralContactPoints__1=akka.tcp://otopcua@central-2:4053
MeshTransport__CentralContactPoints__0=akka.tcp://otopcua@central-1:4053

$ curl -sf http://localhost:9200/health/ready
{"status":"Healthy", ..., "akka":{"status":"Healthy","description":"Akka cluster member status: Up",
 "data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","selfRoles":["admin","cluster-MAIN","driver"],
 "memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}}}
```

**Check 2 evidence** — scratch container from `otopcua-host:dev`, site-a-1's real environment
with `MeshTransport__GrpcPsk` removed and `Mode` forced to `Grpc`:

```
$ docker run --rm --name scratch-check2-noPsk --network otopcua-dev_default \
    --env-file check2-env.txt otopcua-host:dev
[08:54:23 ERR] Hosting failed to start
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:GrpcPsk is empty. Under
Grpc this per-cluster pre-shared key is the whole authentication boundary of the command plane
in both directions — without it this node's inbound command service fails closed and every ack
it posts to central is rejected. Supply it via the environment (MeshTransport__GrpcPsk), in
production as ${secret:OT-MESH-PSK-<clusterId>}.
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   ...
```

**Check 3 evidence** — same base env, `GrpcPsk` present, both `CentralGrpcEndpoints` indices
unset:

```
$ docker run -d --name scratch-check3-noEndpoints --network otopcua-dev_default \
    --env-file check3-env.txt otopcua-host:dev
$ docker logs scratch-check3-noEndpoints 2>&1 | grep -A2 "OptionsValidationException"
[08:55:55 ERR] Hosting failed to start
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:CentralGrpcEndpoints is
empty. Under Grpc a driver node has nowhere to post its deployment ack, and every deployment
would time out naming a node that is running perfectly. Supply them via the environment
(MeshTransport__CentralGrpcEndpoints__0).
```

Both scratch containers removed after capture (`docker rm -f scratch-check2-noPsk
scratch-check3-noEndpoints`); the rig's six real nodes were never affected (still `restarts=0`
after both runs).

**Check 4 evidence** — full suite, clean re-run (a first run hit one race-condition flake in
`RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed`,
which the test's own name identifies as exercising a race; re-ran in isolation and it passed,
then a full second run passed clean end to end — pre-existing timing sensitivity, not a
regression from this branch, which touched only `docker-dev/docker-compose.yml` and
`docs/Configuration.md`, no `.cs` files):

```
$ dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo
...
Passed!  - Failed: 0, Passed: 212, Skipped: 4, Total: 216, Duration: 3m56s - ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.dll (net10.0)
$ echo $?
0
```

Rig left running on `ClusterClient` (its normal state) after the gate.

---

## Phase 1A — central→node command plane — **FAIL (blocked at the central boot boundary)**

Branch `feat/grpc-node-command` @ `fbf7b708`. Image rebuilt from this worktree
(`/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a`) so it carries this branch;
`docker compose -f docker-dev/docker-compose.yml up -d --build` exited 0 and **all eight
containers were recreated** — `migrator` and `cluster-seed` re-ran to completion (exit 0), the
six host nodes and traefik were recreated, the `sql` container and its named volume were kept.
Nothing else touched the stack afterwards, so this run is not a concurrent-rebuild-invalidated
one. Rig-only flip: central-1/-2 `MeshTransport__Mode: "Grpc"` (hard value); **site nodes left
on the compose default `ClusterClient`** (coexistence is the point). Three Serilog
`MinimumLevel:Override` Debug scopes were added on all six nodes for this run only
(`ZB.MOM.WW.OtOpcUa.Host.Grpc`, `ZB.MOM.WW.OtOpcUa.ControlPlane.Communication`,
`Akka.Cluster.Tools.Client`) — the accept/dispatch lines the checks read are `Debug` and the
rig's default Serilog level is `Information`, so without them checks 1/3 would have had no
observable signal. Flip and overrides reverted after the gate, never committed.

**Verdict: FAIL.** Neither central node boots on this branch, so nothing that originates at
central could be driven. Root cause is defect 4 below — resolving `IMeshCommandDialer` /
`IMeshPskProvider` inside the first `WithActors` starter drags in `ISecretResolver` →
`IAuditWriter` → `ActorAuditWriter`, whose constructor reads `AuditWriterActorKey` out of the
`ActorRegistry` before the singleton that registers it has run. It is **mode-independent**
(reproduced on the same image with `MeshTransport__Mode=ClusterClient`), so it is a defect of
the branch, not of the gate's flip, and it would have failed a `ClusterClient` rig equally.

| # | Check | Result |
|---|---|---|
| 1 | Deploy dispatch from AdminUI reaches **all six** driver nodes over gRPC (per-node `Execute` log lines on every node; zero `SendToAll` lines on central) | **BLOCKED** — `POST /api/deployments` is unreachable: traefik answers `503` because neither central node has a live HTTP surface (`curl http://central-1:9000/health/ready` and `central-2` both return `000`, connection refused). Nothing dispatched. |
| 2 | Acks still return over ClusterClient (site mode unchanged) and the deployment **seals** — mixed-transport deploy proven end to end | **BLOCKED** — no deployment could be dispatched (check 1). The most recent `dbo.Deployment` row is still the pre-gate `03540DA5-…` (`Status=2`, Sealed, `2026-08-06 06:11:22`), i.e. nothing was attempted during this run. |
| 3 | `RestartDriver` + `ReconnectDriver` from AdminUI land on the target site node (EventStream delivery observed via driver restart in logs) | **PARTIAL PASS (node-side leg only)** — both commands were delivered to site-a-1 over real Kestrel h2c with the correct PSK and both acted: `DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate` followed by a fresh `spawned Modbus driver SITEA-modbus`, and `reconnecting driver … ForceReconnect requested by admin; re-entering Reconnecting`. The **AdminUI half was not driven** — see the two carry-forwards below (central down; and the UI control is unreachable on this branch anyway). |
| 4 | `AlarmCommand` (ack a scripted alarm from AdminUI) lands and the condition acknowledges | **PARTIAL PASS (envelope leg only)** — the alarm arm decoded and was delivered node-side (`Mesh: accepted AlarmCommand (correlationId=39ba4511…) over gRPC`), which is the arm with no CLR correlation member and therefore the one worth wiring live. **The acknowledge itself was NOT observed**: the rig's single `dbo.ScriptedAlarm` row (`gate481-alarm`, equipment `EQ-6e09bcc2d907`) is not owned by SITE-A, and its owning node is a central one, which is down. |
| 5 | Unauthenticated `grpcurl` to a site node `:4056` `mesh_transport.v1.NodeCommandService/Execute` ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | **PASS** — 3/3 probes graded as designed: no bearer ⇒ `PermissionDenied`, correct `dev-mesh-psk-SITE-A` + empty envelope ⇒ `InvalidArgument` (**auth passed**, the decode is what failed), and a full `RestartDriver` envelope with the same key ⇒ `{}` + observed driver restart. |
| 6 | **Wrong-key isolation:** site-b's PSK presented to site-a ⇒ `PermissionDenied` (per-cluster scoping is real, not decorative) | **PASS** — both directions rejected: `dev-mesh-psk-SITE-B` → site-a-1 ⇒ `PermissionDenied`, `dev-mesh-psk-SITE-A` → site-b-1 ⇒ `PermissionDenied`, while `dev-mesh-psk-SITE-B` → site-b-1 reaches the decoder (`InvalidArgument`) — so the rejection is the key, not the reachability. |
| 7 | A node with NULL `GrpcPort` is skipped with the warn-once, fan continues to the others | **BLOCKED** — the skip is computed by `CentralCommunicationActor.AdoptGrpcTargets` on a contact refresh; no central node is running to refresh. The rig's `ClusterNode` rows were left untouched (all six still `GrpcPort=4056`) rather than mutated for a check that could not be observed. |
| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures on their interceptors across the run | **PARTIAL PASS** — 0 `Rejected a telemetry stream call` / `Rejected a config-serve call` / `Rejected a LocalDb sync call` lines across all six nodes, and the only `Rejected a mesh command call` lines are the 3 deliberate negative probes (2 on site-a-1, 1 on site-b-1). **But telemetry and config-serve carried no traffic at all** in this run — central dials both and never booted (site-a-1 logs 0 telemetry lines) — so "unaffected" is established for LocalDb sync only. |

### Phase 1A evidence

**Provenance** — flip in effect on central only, sites untouched:

```
$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
    --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
central-1  MeshTransport__Mode=Grpc
central-2  MeshTransport__Mode=Grpc
site-a-1   MeshTransport__Mode=ClusterClient
site-b-1   MeshTransport__Mode=ClusterClient

$ docker images --format '{{.Repository}}:{{.Tag}}|{{.ID}}|{{.CreatedAt}}' | grep otopcua-host
otopcua-host:dev|69360a173f1f|2026-08-06 07:20:14 -0400 EDT     # rebuilt for this run
```

**Defect 4 — the boot failure (checks 1/2/7 blocked by it).** Both central nodes, identical:

```
$ docker logs otopcua-dev-central-1-1 2>&1 | sed -n '25,45p'
[11:21:27 INF] OPC UA server started on opc.tcp://0.0.0.0:4840
[11:21:27 FTL] Unable to start AkkaHostedService - shutting down application
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
   at Akka.Hosting.ActorRegistry.Get[TKey]()
   at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
        in /src/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Audit/ActorAuditWriter.cs:line 24
   ...
   at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_5(IServiceProvider sp)
   ...
   at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_6(IServiceProvider sp)
   ...
   at ZB.MOM.WW.OtOpcUa.ControlPlane.ServiceCollectionExtensions.<>c.<WithOtOpcUaControlPlaneSingletons>b__7_0(
        ActorSystem system, IActorRegistry registry, IDependencyResolver resolver)
        in /src/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs:line 74
   at Akka.Hosting.AkkaConfigurationBuilder.StartAsync(ActorSystem sys)
   at Akka.Hosting.AkkaHostedService.StartAsync(CancellationToken cancellationToken)
```

**Mode-independence, proven rather than argued** — same image, central-1's exact environment
with `MeshTransport__Mode=ClusterClient`, own hostname + self-seed so it cannot disturb the rig:

```
$ docker run -d --name scratch-cc-mode --network otopcua-dev_default --env-file c1-env.txt \
    -e Cluster__PublicHostname=scratch-cc-mode \
    -e Cluster__SeedNodes__0=akka.tcp://otopcua@scratch-cc-mode:4053 otopcua-host:dev
$ docker logs scratch-cc-mode 2>&1 | grep -m4 -E "MissingActorRegistry|Hosting failed|Host starting"
[11:28:45 INF] OtOpcUa.Host starting with roles=[admin,driver,cluster-main] (admin=True, driver=True)
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
[11:28:47 ERR] Hosting failed to start
```

Scratch container removed after capture (`docker rm -f scratch-cc-mode`); the six real nodes
were untouched by it.

**It presents as a healthy container.** Worth carrying into Phase 2/4 readings — the failed
process does not exit, so `docker ps` shows `Up`, `RestartCount=0`, and only an HTTP probe or
the log tells the truth:

```
$ docker inspect otopcua-dev-central-1-1 -f '{{.State.Status}} restarts={{.RestartCount}}'
running restarts=0
$ docker run --rm --network otopcua-dev_default curlimages/curl -s -o /dev/null \
    -w '%{http_code}\n' --max-time 5 http://central-1:9000/health/ready
000
$ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9200/health/ready
503
```

**Checks 5 + 6 — the wire-probe auth matrix** (`-import-path` onto the in-repo proto; the node
has no reflection). An empty `CommandEnvelope` has no `oneof` arm set, so `FromEnvelope` throws
and the service answers `InvalidArgument` — **after** the interceptor ran, which is exactly why
it is the positive signal here: `PermissionDenied` means the key was refused, `InvalidArgument`
means the key was accepted and only the payload was rejected:

```
$ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos
$ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \
     -plaintext -import-path /protos -proto mesh_transport.proto"

# 5a  no bearer                        -> site-a-1
Code: PermissionDenied   Message: Node command authentication failed.
# 5b  -H 'authorization: Bearer dev-mesh-psk-SITE-A'   -> site-a-1
Code: InvalidArgument    Message: Command envelope could not be decoded.
# 6a  -H 'authorization: Bearer dev-mesh-psk-SITE-B'   -> site-a-1
Code: PermissionDenied   Message: Node command authentication failed.
# 6b  -H 'authorization: Bearer dev-mesh-psk-SITE-A'   -> site-b-1
Code: PermissionDenied   Message: Node command authentication failed.
# 6c  -H 'authorization: Bearer dev-mesh-psk-SITE-B'   -> site-b-1   (positive control)
Code: InvalidArgument    Message: Command envelope could not be decoded.
```

**Checks 3 + 4 — the node-side leg, driven with real envelopes** over the same authenticated
h2c path central would use. This is the first live proof that `MeshCommandAuthInterceptor` is
actually invoked on a real Kestrel pipeline (register item 4) and that
`NodeCommandGrpcService` → `NodeCommunicationActor` → EventStream → `DriverHostActor` works
end to end:

```
$ $G -H 'authorization: Bearer dev-mesh-psk-SITE-A' -H 'x-otopcua-cluster: SITE-A' \
     -d '{"correlationId":"8461008e...","restartDriver":{"clusterId":"SITE-A",
          "driverInstanceId":"SITEA-modbus","actorByUserName":"1a-gate",
          "correlationId":"8461008e..."}}' \
     site-a-1:4056 mesh_transport.v1.NodeCommandService/Execute
{}

$ docker logs otopcua-dev-site-a-1-1 2>&1 | tail
[11:31:01 DBG] Mesh: accepted RestartDriver (correlationId=8461008e37654724a292219202d78dfa) over gRPC; handed to the node communication actor.
[11:31:01 INF] DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate
[11:31:01 INF] DriverHost site-a-1:4053: spawned Modbus driver SITEA-modbus (stub=False)
[11:31:55 DBG] Mesh: accepted ReconnectDriver (correlationId=29c5dc77283f4eb7869d0c7d6686cadd) over gRPC; handed to the node communication actor.
[11:31:55 INF] DriverHost site-a-1:4053: reconnecting driver SITEA-modbus by request of 1a-gate
[11:31:55 INF] DriverInstance SITEA-modbus: ForceReconnect requested by admin; re-entering Reconnecting
[11:31:56 DBG] Mesh: accepted AlarmCommand (correlationId=39ba451173954229a71b9dd1e00ac1b9) over gRPC; handed to the node communication actor.
```

**Check 8 — interceptor denial counts across every node, whole run:**

```
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1  telemetry=0 configserve=0 localdbsync=0 mesh=0
central-2  telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-1   telemetry=0 configserve=0 localdbsync=0 mesh=2   # the two deliberate negatives
site-a-2   telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-1   telemetry=0 configserve=0 localdbsync=0 mesh=1   # the one deliberate negative
site-b-2   telemetry=0 configserve=0 localdbsync=0 mesh=0
```

The `TwoNodeClusterHarness` I-1 null-provider Error (a known test-harness artifact) did **not**
appear in any rig log — `grep -c "IMeshPskProvider"` = 0 on all six nodes, as expected for
production wiring where every admin node registers the provider.

### Phase 1A defect

**Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter.**
`WithOtOpcUaControlPlaneSingletons`'s `WithActors` block resolves `IMeshCommandDialer` and
`IMeshPskProvider` from `IServiceProvider` to hand to `CentralCommunicationActor.Props`.
`MeshGrpcCommandDialer` takes `IMeshPskProvider`, which takes `ISecretResolver`, which
`AddZbSecrets` builds by resolving `IAuditWriter`, which on an admin node is
`ActorAuditWriter` — and its constructor does `registry.Get<AuditWriterActorKey>()`. That
singleton is registered ~60 lines later in the same builder, so at this point the key is
absent and `ActorRegistry.Get` throws `MissingActorRegistryEntryException`, failing
`AkkaHostedService.StartAsync` and the whole host.

Three things make this worth a pinned regression rather than a one-line fix note:

- **It is not mode-gated.** The two `GetService` calls run in every transport mode by design
  (so a mode flip needs no restart), so the eager graph is walked on every admin boot. Proven
  above with a `ClusterClient` scratch container on the same image.
- **It is invisible to the suite.** The 1A.2 container-activation test
  (`It_activates_from_the_container_despite_the_optional_client_factory_parameter`) resolves
  the dialer from a service collection that does not include `AddZbSecrets`, and
  `TwoNodeClusterHarness` registers no `IMeshPskProvider` at all — so no existing test walks
  the real graph in starter order. This is the same class as scadaproj#1 ("only visible with
  the DI graph built in a real host") and as mesh Phase 6's `IClusterRoleInfo` boot crash.
- **The container lies about it.** The process does not exit, so `docker ps` reports `Up` with
  `restarts=0` while the node serves no HTTP at all. Any Phase 2/4 readiness check that reads
  container status rather than `/health/ready` would score this as healthy.

The shape of the fix is the one this file already uses two lines above for the coordinator
proxy — hand the actor a lazy accessor (`Func<IMeshCommandDialer?>` / `Func<IMeshPskProvider?>`,
or resolve on first use) so nothing is constructed while the `ActorSystem` is still building —
but the fix and its red-first pin belong to 1A, not to this record.

### Rig state after the gate

The compose flip was reverted (`git checkout docker-dev/docker-compose.yml`) and the six nodes
recreated on the committed all-`ClusterClient` compose. Because defect 4 is mode-independent,
the branch image cannot produce a healthy central pair on *any* mode, so the rig was returned
to its pre-gate image rather than left with a dead central pair: the 1A build was preserved as
`otopcua-host:grpc-1a` (`69360a173f1f`) and the pre-gate build (`8bd81d8aea9b`, 04:52) re-tagged
back to `otopcua-host:dev`. **Anyone re-running this gate must rebuild — the running rig is
NOT on the branch under test.** Verified after restore: six nodes `running restarts=0`, zero
`Hosting failed to start` on all six, `/health/ready` 200 on every node, MAIN mesh
`memberCount=2 unreachableCount=0`, AdminUI 200 on `http://localhost:9200`.

---

## Phase 1B — node→central ack plane — **PENDING**

Branch `feat/grpc-central-ack` @ `________` (rebased on merged 1A). Rig-only flip:
site-a-1/-2 → `Grpc`; site-b stays `ClusterClient`; central back on `ClusterClient` mode
(its command direction) — acks are governed by the **node's** mode, so this isolates 1B.

| # | Check | Result |
|---|---|---|
| 1 | Deploy e2e: site-a acks arrive via `ReportApply` (central `CentralCommandService` log lines carry site-a node ids), `NodeDeploymentState` rows land, deployment seals | PENDING |
| 2 | Site-b acks still arrive via ClusterClient in the same deployment — per-pair coexistence proven | PENDING |
| 3 | Exactly **one** ack row per site-a node per dispatch (no double-count from endpoint failover) | PENDING |
| 4 | Kill central-1 before the apply completes: site-a acks fail over to central-2's endpoint; deployment still seals | PENDING |
| 5 | Unauthenticated / wrong-cluster-key / missing `x-otopcua-cluster` probes against central `:4055` `CentralCommandService/ReportApply` ⇒ `PermissionDenied` (missing-header rejection is central-side by design) | PENDING |
| 6 | ConfigServe (`DeploymentArtifactService`) on the same listener unaffected: fetches succeed, 0 auth failures | PENDING |

---

## Phase 2 — full-rig cutover + soak — **PENDING**

Rig default flipped and committed: `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"`.
Branch/SHA `________`. **Run the Baseline section first on the pre-flip build.**

| # | Check | Result |
|---|---|---|
| 1 | Full rebuild all-`Grpc`: six nodes ready, deploy e2e seals, all acks via gRPC | PENDING |
| 2 | Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes | PENDING |
| 3 | Kill site-a-1 mid-dispatch: per-node `Execute` failure **logged with node id + status** (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path) | PENDING |
| 4 | Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action | PENDING |
| 5 | Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; **no duplicate applies** after the pair returns (boot-replay path only) | PENDING |
| 6 | **Zero ClusterClient activity under `Grpc`:** `docker logs <every node> \| grep -ci "clusterclient"` = 0 new lines post-boot | PENDING |
| 7 | Soak ≥ 2 h under periodic deploys + command traffic: 0 `PermissionDenied`/`Unauthenticated` across all six nodes (PSK drift detector), 0 unexplained `Unavailable` | PENDING |
| 8 | Full suite green on the cutover SHA | PENDING |

---

## Phase 4 — terminal gate on the post-deletion build — **PENDING**

`master` @ `________` (after the Phase 3 deletion merge). **Provenance guard:** image build
timestamp verified LATER than the merge commit time — a pre-deletion image would invalidate
checks 6–7.

| # | Check | Result |
|---|---|---|
| 1 | PSK negatives on both services (no key / wrong key / wrong cluster's key); **unset server key ⇒ all calls rejected** (fail-closed proven live) | PENDING |
| 2 | Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand) | PENDING |
| 3 | Ack matrix: every node acks exactly once, deployment seals | PENDING |
| 4 | Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry) | PENDING |
| 5 | Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart) | PENDING |
| 6 | **Boundary check: no Akka association between any site container and central** — the Baseline (a) probe now shows no cross-mesh `:4053` traffic (pair-internal remoting only) | PENDING |
| 7 | Full-rig restart (pairs together): clean convergence; `grep -ri "receptionist\|clusterclient"` over all node logs = 0 lines | PENDING |
| 8 | `MeshGrpcBoundaryTests` + full suite green on the merge SHA | PENDING |

### Carry-forwards

Anything a phase gate could not drive gets named here with why + what proves it instead, and
is re-checked in the next gate (the ScadaBridge proportionate-gate rule).

| Item | Deferred from | Why it could not be driven | Proven instead by | Re-check in |
|---|---|---|---|---|
| Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-`GrpcPort` skip | 1A | Both central nodes fail to boot (defect 4); no AdminUI, no deploy API, no contact refresh | Nothing live. Unit coverage only: the fan Theory, the gate-⊋-fan fact, `A_slow_dial_does_not_park_the_actors_mailbox` | **1A gate re-run after defect 4 is fixed** — these are the checks 1A exists to prove and must not be carried past it |
| Check 3 central half — `RestartDriver`/`ReconnectDriver` **originating at the AdminUI** | 1A | (a) central down; (b) independently, `DriverStatusPanel.razor` — the only component holding the Reconnect/Restart buttons — is referenced by **no page or component in the AdminUI**, so the control is unreachable in the running UI on this branch and on `master` | The node-side leg driven live with real envelopes (check 3 evidence); central's encode covered by the T0.2 mapper goldens | 1A re-run; the orphaned panel is pre-existing and out of this program's scope — file separately if it is to be fixed |
| Check 4 acknowledge half — the condition actually acknowledging | 1A | The rig's one `ScriptedAlarm` (`gate481-alarm`) is not owned by a SITE cluster, and its owning central node is down | The alarm arm decoding + delivering node-side (check 4 evidence) | 1A re-run, seeding a SITE-A-owned scripted alarm first |
| Check 8 for telemetry + config-serve | 1A | Central dials both planes and never booted, so neither carried a single call — 0 denials is vacuous for them | LocalDb sync (pair-local) did carry traffic with 0 denials | 1A re-run |

### Defects the gates caught that the suite did not

Recorded per phase as they occur, with root cause + the pinned regression test.

| # | Phase | Defect | Root cause | Status |
|---|---|---|---|---|
| 4 | 1A gate | **An admin node cannot boot at all** — `MissingActorRegistryEntryException: AuditWriterActorKey`, both central nodes, every transport mode | The first `WithActors` starter eagerly resolves `IMeshCommandDialer`/`IMeshPskProvider`, whose graph reaches `ActorAuditWriter`, whose ctor reads an `ActorRegistry` key registered ~60 lines later in the same builder | **OPEN — blocks 1A.** Full analysis + evidence in the Phase 1A section above. No product change was made by this gate |