lmxopcua/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md

# ClusterClient → gRPC migration — live gate record

Rig: `docker-dev/` (central-1/-2 = `admin,driver,cluster-MAIN`; site-a-1/-2, site-b-1/-2;
sql + traefik + migrator + cluster-seed), rebuilt from the branch under test via
`docker compose -f docker-dev/docker-compose.yml up -d --build`. Recorded check-by-check in
the family's live-gate format. Later phases' sections are appended as those phases land; every
check row gets a **PASS**/**FAIL** verdict plus one quantitative justification clause, with
fenced evidence (real commands, real output). A run invalidated by a concurrent rig rebuild is
**void** — note it and re-run (the ScadaBridge void-run rule).

Provenance discipline: every phase section opens with branch + commit SHA + "image rebuilt,
all N containers recreated", and Phase 4 additionally verifies the image build timestamp
post-dates the deletion merge (a stale image invalidates the boundary checks).

Plan: `2026-08-06-clusterclient-to-grpc.md`. Tracking:
`2026-08-06-clusterclient-to-grpc-tracking.md`.

---

## Baseline (run on the PRE-change build, before the Phase 2 flip)

The two gaps this program closes, **reproduced rather than assumed**:

**(a) Unauthenticated Akka remoting crosses the WAN boundary.** From the host:

```bash
# A live TCP association from a site container to central's Akka port:
docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
# And the receptionist actually accepting cross-mesh traffic (central log after any AdminUI
# driver restart targeting site-a):
docker logs otopcua-dev-central-1-1 2>&1 | grep -i "clusterclient\|receptionist" | tail -5
```

Expected: `OPEN`; receptionist/client lines present. Nothing on this path presents any
credential.

**(b) Silent drop on an unreachable cluster.** Stop the site-b pair, dispatch a deployment
(`POST http://localhost:9200/api/deployments`, `X-Api-Key: docker-dev-deploy-key`), observe:
no per-node dispatch error surfaces anywhere; site-b's `NodeDeploymentState` rows sit
`Applying` until the 2-minute deadline marks the deployment `TimedOut`/partially failed. The
deadline is the only witness. Restart the pair.

- [ ] Baseline (a) recorded with evidence — **PENDING**
- [ ] Baseline (b) recorded with evidence — **PENDING**

---

## Phase 0 — contract + auth + config groundwork — **PASS**

Branch `feat/grpc-phase0-contract-auth` @ `c2bfad4a`. Image rebuilt, all six containers
recreated (`docker compose -f docker-dev/docker-compose.yml up -d --build`, run from the
worktree at `/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-phase0` so the image carries this
branch). The rig was already running (from the main checkout's compose file) before this run;
the rebuild recreated all eight containers (sql kept, migrator + cluster-seed one-shots re-ran,
central-1/-2 + site-a-1/-2 + site-b-1/-2 + traefik recreated) under the fixed `otopcua-dev`
project name, so this run is not a concurrent-rebuild-invalidated one — nothing else touched
the stack afterwards.

| # | Check | Result |
|---|---|---|
| 1 | All six nodes boot on `ClusterClient` with the dark `Grpc` config present (PSKs, endpoints, ClusterPsks map); no validator fires | **PASS** — 6/6 nodes `restarts=0`, `docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}'` reports `running restarts=0` on every node; `docker logs <node> | grep -iE "OptionsValidationException\|MeshTransport.*(is empty\|refuses to start)"` = 0 lines on all six |
| 2 | Deliberate misconfig: a scratch container with `MeshTransport__Mode=Grpc` and **no** `GrpcPsk` refuses to boot, error names `MeshTransport:GrpcPsk` + the env form | **PASS** — `Hosting failed to start` + `OptionsValidationException` naming `MeshTransport:GrpcPsk` and `MeshTransport__GrpcPsk`, below |
| 3 | Same with empty `CentralGrpcEndpoints` — refuses, names the key | **PASS** — `OptionsValidationException` naming `MeshTransport:CentralGrpcEndpoints` and `MeshTransport__CentralGrpcEndpoints__0`, below |
| 4 | Full suite green, 0 warnings, on the branch | **PASS** — `dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo`: 0 failures on a clean run (see evidence below); no `.cs` files changed on this branch since the last full rebuild, so the pre-existing ~868-warning baseline is unchanged (nothing to newly warn about) |

Note: the new gRPC services are not yet mapped in Phase 0, so no wire probes here — the
interceptor negative/positive matrix is unit-tested and wire-probed in 1A/1B.

**Check 1 evidence** — no validator fired on any of the six nodes, all dark `Grpc` keys present:

```
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do
    docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' otopcua-dev-$c-1
  done
running restarts=0    # central-1
running restarts=0    # central-2
running restarts=0    # site-a-1
running restarts=0    # site-a-2
running restarts=0    # site-b-1
running restarts=0    # site-b-2

$ docker logs otopcua-dev-central-1-1 2>&1 | grep -iE "MeshTransport|ClusterClient"
[08:53:26 INF] Mesh ClusterClient for cluster MAIN created with 2 contact point(s)
[08:53:26 INF] Mesh ClusterClient for cluster SITE-A created with 2 contact point(s)
[08:53:26 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s)

$ docker logs otopcua-dev-site-a-1-1 2>&1 | grep -iE "cluster.client|receptionist"
[08:53:26 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]

$ docker inspect otopcua-dev-site-a-1-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep MeshTransport
MeshTransport__CentralGrpcEndpoints__1=http://central-2:4055
MeshTransport__CentralGrpcEndpoints__0=http://central-1:4055
MeshTransport__Mode=ClusterClient
MeshTransport__GrpcPsk=dev-mesh-psk-SITE-A
MeshTransport__CentralContactPoints__1=akka.tcp://otopcua@central-2:4053
MeshTransport__CentralContactPoints__0=akka.tcp://otopcua@central-1:4053

$ curl -sf http://localhost:9200/health/ready
{"status":"Healthy", ..., "akka":{"status":"Healthy","description":"Akka cluster member status: Up",
 "data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","selfRoles":["admin","cluster-MAIN","driver"],
 "memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}}}
```

**Check 2 evidence** — scratch container from `otopcua-host:dev`, site-a-1's real environment
with `MeshTransport__GrpcPsk` removed and `Mode` forced to `Grpc`:

```
$ docker run --rm --name scratch-check2-noPsk --network otopcua-dev_default \
    --env-file check2-env.txt otopcua-host:dev
[08:54:23 ERR] Hosting failed to start
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:GrpcPsk is empty. Under
Grpc this per-cluster pre-shared key is the whole authentication boundary of the command plane
in both directions — without it this node's inbound command service fails closed and every ack
it posts to central is rejected. Supply it via the environment (MeshTransport__GrpcPsk), in
production as ${secret:OT-MESH-PSK-<clusterId>}.
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   ...
```

**Check 3 evidence** — same base env, `GrpcPsk` present, both `CentralGrpcEndpoints` indices
unset:

```
$ docker run -d --name scratch-check3-noEndpoints --network otopcua-dev_default \
    --env-file check3-env.txt otopcua-host:dev
$ docker logs scratch-check3-noEndpoints 2>&1 | grep -A2 "OptionsValidationException"
[08:55:55 ERR] Hosting failed to start
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:CentralGrpcEndpoints is
empty. Under Grpc a driver node has nowhere to post its deployment ack, and every deployment
would time out naming a node that is running perfectly. Supply them via the environment
(MeshTransport__CentralGrpcEndpoints__0).
```

Both scratch containers removed after capture (`docker rm -f scratch-check2-noPsk
scratch-check3-noEndpoints`); the rig's six real nodes were never affected (still `restarts=0`
after both runs).

**Check 4 evidence** — full suite, clean re-run (a first run hit one race-condition flake in
`RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed`,
which the test's own name identifies as exercising a race; re-ran in isolation and it passed,
then a full second run passed clean end to end — pre-existing timing sensitivity, not a
regression from this branch, which touched only `docker-dev/docker-compose.yml` and
`docs/Configuration.md`, no `.cs` files):

```
$ dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo
...
Passed!  - Failed: 0, Passed: 212, Skipped: 4, Total: 216, Duration: 3m56s - ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.dll (net10.0)
$ echo $?
0
```

Rig left running on `ClusterClient` (its normal state) after the gate.

---

## Phase 1A — central→node command plane — **PASS (on the re-run at `e54883a8`)**

Two runs. **Run 1 @ `fbf7b708` FAILED** at the central boot boundary (defect 4) and could not
drive checks 1/2/7 or the central halves of 3/4; its checks 5 + 6 and the node-side halves of
3 + 4 stand as recorded and were not repeated. **Run 2 @ `e54883a8`** — after the boot fix
(`c0475f2d`) and its review-hardening follow-up (`e54883a8`) — drove every carried check. The
result table below is the **combined, current** verdict; run 1's failure record is preserved
underneath it because the defect it caught is the most valuable thing this gate produced.

| # | Check | Verdict | Run |
|---|---|---|---|
| — | **Boot:** all six nodes healthy INCLUDING the central pair, on the branch image | **PASS** — 6/6 `running restarts=0`, 0 `Hosting failed to start`, `/health/ready` 200 ×6, MAIN mesh `memberCount=2 unreachableCount=0`. **But the `Resolve` Error line the fix logs DID fire — see finding 5** | 2 |
| 1 | Deploy dispatch reaches **all six** driver nodes over gRPC; zero `SendToAll`/ClusterClient on central | **PASS** — 6/6 nodes accepted one `DispatchDeployment` under a single shared `correlationId=cf499b42…`; central's own per-node outcome log records 6 accepted / 0 failed; `grep -ci clusterclient` = **0** on both central nodes | 2 |
| 2 | Acks return over ClusterClient (site mode unchanged) and the deployment **seals** | **PASS** — deployment `8DF9AAE0…` `Status=2` (Sealed) with all six `NodeDeploymentState` rows `Status=1` (Applied) inside ~90 ms; all four site nodes hold live `Connected to [akka.tcp://…/system/receptionist]` associations, which is their only ack path in 1A | 2 |
| 3 | `RestartDriver` + `ReconnectDriver` land on the target site node | **PASS (composed)** — node-side leg driven live in run 1 (both commands acted on site-a-1). The AdminUI half **cannot be driven and this is now proven, not assumed**: `<DriverStatusPanel` has **0** usages repo-wide, seven rendered pages contain no Restart/Reconnect text or button, and the only non-script-analysis HTTP POST in the whole app is `/api/deployments`. Closed by composition — check 4 proves the *same* `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path live from central for an admin-originated command | 1 + 2 |
| 4 | `AlarmCommand` from the AdminUI lands and the condition acknowledges | **PASS** — Ack clicked on the real `/alerts` page against a live `gate481-alarm` ACTIVATED row; central fanned **one** `AlarmCommand` (`correlationId=ccafe375…`, minted once) to all six over gRPC, and the condition acknowledged — a new `ACKNOWLEDGED` row appeared 0.54 s later carrying the operator's identity (`multi-role-test`) | 2 |
| 5 | Unauthenticated ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | **PASS** — 3/3 probes as designed (run 1; not repeated — nothing in the fix touches the interceptor) | 1 |
| 6 | **Wrong-key isolation** | **PASS** — both directions rejected, with a positive control proving the rejection is the key and not reachability (run 1) | 1 |
| 7 | NULL `GrpcPort` node skipped with the warn-once; fan continues | **PASS** — with `site-b-2:4053` NULLed, both central nodes logged the warn **once** each and the next dispatch (`correlationId=88a9e0a2…`) was accepted by exactly **5** nodes, site-b-2 absent; the deployment then honestly ended `Status=4` (TimedOut) with site-b-2 alone at `Status=0`. Row restored; a later dispatch sealed 6/6 | 2 |
| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures | **PASS, and no longer vacuous** — 0 denials on all four interceptors across all six nodes, **and all three planes carried real traffic this time**: site nodes fetched + SHA-verified 28 KB artifacts from `http://central-1:4055`, `Telemetry stream opened` on every node with live driver health rendering on `/hosts`, LocalDb sync active on the site pairs | 2 |

### Run 2 provenance

`docker compose -f docker-dev/docker-compose.yml up -d --build` from the worktree, **exit 0**;
image `otopcua-host:dev` rebuilt to `1d8452a29018` at 12:15 UTC (run 1's preserved
`otopcua-host:grpc-1a` was stale and deliberately **not** reused). The six host nodes plus
`migrator` and `cluster-seed` were recreated (both one-shots exit 0); `sql` and its volume were
kept, and `traefik` was reused unchanged (its service definition did not change). Nothing else
touched the stack during the run, so it is not concurrent-rebuild-invalidated. Same rig-only
flip and same three Serilog Debug overrides as run 1; both reverted afterwards, never committed.

```
$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
    --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
central-1  MeshTransport__Mode=Grpc
central-2  MeshTransport__Mode=Grpc
site-a-1   MeshTransport__Mode=ClusterClient
site-b-1   MeshTransport__Mode=ClusterClient
```

### Run 2 evidence

**Boot — the fix works, and the race it was hardened against is real.**

```
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1  running restarts=0 mode=Grpc hostingFailed=0 resolveErr=2
central-2  running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0
site-a-1   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-a-2   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-b-1   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-b-2   running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
```

See **finding 5** below for the two `resolveErr` lines on central-1 — they are the PreStart
race losing on a real boot, and the memoize retry healing it was verified live rather than
argued.

**Check 1 — the fan, both ends.** One envelope, one correlation id, six recipients:

```
$ for c in …; do docker logs otopcua-dev-$c-1 | grep "Mesh: accepted DispatchDeployment" | tail -1; done
central-1  [12:16:49 DBG] Mesh: accepted DispatchDeployment (correlationId=cf499b42c6914a578035b2b5bc1020f7) over gRPC; handed to the node communication actor.
central-2  [12:16:49 DBG] …same correlationId…
site-a-1   [12:16:49 DBG] …same correlationId…
site-a-2   [12:16:49 DBG] …same correlationId…
site-b-1   [12:16:49 DBG] …same correlationId…
site-b-2   [12:16:49 DBG] …same correlationId…

$ docker logs otopcua-dev-central-2-1 | grep "accepted by"        # the node hosting the coordinator
[12:16:49 DBG] Mesh: DispatchDeployment (correlationId=cf499b42…) accepted by central-2:4053 over gRPC
[12:16:49 DBG] … accepted by site-b-1:4053 …   [12:16:49 DBG] … accepted by site-a-2:4053 …
[12:16:49 DBG] … accepted by site-a-1:4053 …   [12:16:49 DBG] … accepted by central-1:4053 …
[12:16:49 DBG] … accepted by site-b-2:4053 …
$ docker logs … | grep -c "was NOT delivered to"   → 0 on both central nodes

# Zero ClusterClient on the command side, both central nodes:
central-1  MeshClusterClientCreated=0 clusterclientAnyLine=0
central-2  MeshClusterClientCreated=0 clusterclientAnyLine=0
```

The `clusterclientAnyLine=0` count is over the whole log with
`Serilog:MinimumLevel:Override:Akka.Cluster.Tools.Client=Debug` in force, so it covers Akka's
own ClusterClient internals and not merely this repo's creation message.

**Check 2 — mixed transport, end to end.** Commands out over gRPC (above), acks back over
ClusterClient:

```
$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='8df9aae0-…'
2                                    -- Sealed
$ … SELECT NodeId, Status, AppliedAtUtc FROM dbo.NodeDeploymentState WHERE DeploymentId='8df9aae0-…'
central-1:4053|1|2026-08-06 12:16:49.939     site-a-1:4053|1|2026-08-06 12:16:49.977
central-2:4053|1|2026-08-06 12:16:49.915     site-a-2:4053|1|2026-08-06 12:16:49.950
site-b-1:4053|1|2026-08-06 12:16:49.957      site-b-2:4053|1|2026-08-06 12:16:49.967

$ for c in site-a-1 site-a-2 site-b-1 site-b-2; do docker logs … | grep "receptionist" | tail -1; done
site-a-1   [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
site-a-2   [12:15:20 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
site-b-1   [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
site-b-2   [12:15:25 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
$ grep -c "No central ClusterClient" → 0 on all six      # no ack was dropped for want of a client
```

A site node has **no** gRPC ack path in 1A (that is 1B), so the receptionist association above
is not merely present, it is the only route those four Applied rows could have taken.

**Check 3 — why the AdminUI half is undrivable, established rather than asserted:**

```
$ grep -rn "<DriverStatusPanel" --include="*.razor" --include="*.cs" src/ | wc -l
0                                    # the component holding the buttons is referenced nowhere
$ grep -rn "DriverStatusPanel" --include="*.razor" src/ | grep -v "Shared/Drivers/DriverStatusPanel.razor:"
…5 hits, all of them code COMMENTS in Alerts.razor / Hosts.razor ("mirrors DriverStatusPanel…")

$ node scan-ui.js          # every page a Designer/Administrator/Operator can reach
/            | reconnect/restart text: false | buttons: ["Sign out"]
/hosts       | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
/fleet       | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
/clusters    | reconnect/restart text: false | buttons: ["Sign out"]
/clusters/SITE-A | reconnect/restart text: false | buttons: ["Sign out"]
/alerts      | reconnect/restart text: false | buttons: ["Sign out","Clear"]
/deployments | reconnect/restart text: false | buttons: ["Sign out","Deploy current configuration"]

$ grep -rn "MapPost|MapPut|MapDelete" src/…/AdminUI src/…/Host
…/Api/DeployApiEndpoints.cs:43:  app.MapPost("/api/deployments", …)
…/ScriptAnalysis/ScriptAnalysisEndpoints.cs:19-24:  /diagnostics /completions /hover /signature-help /format /inlay-hints
```

There is no button and no endpoint. What closes the check is that `RestartDriver`,
`ReconnectDriver` and `AlarmCommand` all leave central through the **same three lines** —
`AdminOperationsActor.Dispatch` → `MeshCommand` → `CentralCommunicationActor.DispatchOverGrpc`
— and check 4 drove that path live from the AdminUI for `AlarmCommand`, while run 1 drove the
node-side decode-and-act for the two driver commands. The only per-type difference left is
`MeshTransportDtoMapper.ToEnvelope`, which the T0.2 goldens cover member-for-member.

**Check 4 — AdminUI ack, fanned by central, acknowledged by the owner.** The rig's one
scripted alarm sits Normal indefinitely (`hr200 > 70000`), so a transition was manufactured by
editing its predicate to an always-true form and deploying; the `/alerts` page was already open
when it fired, the Ack button was clicked, and the predicate + hash were restored afterwards
(verified byte-identical by `SELECT`) with a further deploy:

```
page open at 2026-08-06T12:20:59.036Z        initial: empty window
ROWS:  12:21:00.017  gate481-alarm | PressHigh | EQ-6e09bcc2d907 | ACTIVATED | 750 | system | press high
ROW BUTTONS: ["Ack","Shelve","Shelve (timed)","Unshelve"]
clicking Acknowledge at 2026-08-06T12:21:00.362Z
after click:  12:21:00.557 | gate481-alarm | EQ-6e09bcc2d907 | ACKNOWLEDGED | 750 | multi-role-test | press high

$ docker logs otopcua-dev-central-2-1 | grep AlarmCommand
[12:21:00 DBG] Mesh: AlarmCommand (correlationId=ccafe3754aca40cfa8aa0a0602c86cbb) accepted by site-b-2:4053 over gRPC
[12:21:00 DBG] … accepted by central-1:4053 …   [12:21:00 DBG] … accepted by central-2:4053 …
[12:21:00 DBG] … accepted by site-a-1:4053 …    [12:21:00 DBG] … accepted by site-b-1:4053 …
[12:21:00 DBG] … accepted by site-a-2:4053 …
$ for c in …; do grep "Mesh: accepted AlarmCommand" …; done   → all six, same correlationId
```

One correlation id across all six recipients is the "encode once, before the fan" rule holding
for the one command whose CLR record has no correlation member of its own — the case the design
called out as the reason encoding is hoisted out of the loop.

**Check 7 — the skip, the warn, and the honest deadline.**

```
$ … UPDATE dbo.ClusterNode SET GrpcPort=NULL WHERE NodeId='site-b-2:4053'    # 12:23:14

# one warn per central node, once each, within one ContactRefreshSeconds:
central-2  [12:23:20 WRN] ClusterNode site-b-2:4053 has no GrpcPort; it exposes no command endpoint
           and is skipped in this dispatch set (further skips of this node are silent). Commands
           will not reach it until the row carries a port
central-1  [12:23:19 WRN] …identical…
# (each node also logs the TelemetryNodeSource twin — "exposes no telemetry stream … skipped in
#  this dial refresh" — expected collateral: both target sets read the same column)

# dispatch at 12:23:36 — accepted by five, site-b-2 absent:
central-2  [12:23:37 DBG] … (correlationId=88a9e0a2…) accepted by central-2 / site-a-2 / site-b-1
                          / site-a-1 / central-1 :4053 over gRPC
# cumulative per-node accept counts after this dispatch — site-b-2 one behind everyone:
central-1 4   central-2 4   site-a-1 4   site-a-2 4   site-b-1 4   site-b-2 3

$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='5c8bdc9e-…'   → 4  (TimedOut)
$ … SELECT NodeId, Status FROM dbo.NodeDeploymentState …
central-1|1  central-2|1  site-a-1|1  site-a-2|1  site-b-1|1  site-b-2|0
```

Row restored to `4056` and a later dispatch sealed 6/6 (`080EBB0B…`, `Status=2`, every node
`Status=1`). One operational detail worth carrying: **the skip persists for up to a full
`ContactRefreshSeconds` after the row is repaired** — a deploy issued 5 s after the restore
(`DF1C0B74…`) was still skipped and still TimedOut, because the dispatch set is only rebuilt on
the refresh tick.

**Check 8 — denials, and proof the planes were actually used.**

```
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1  telemetry=0 configserve=0 localdbsync=0 mesh=0
central-2  telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-1   telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-2   telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-1   telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-2   telemetry=0 configserve=0 localdbsync=0 mesh=0

# config-serve carried real, authenticated traffic:
site-a-1  [12:20:59 INF] Fetched + verified artifact for 8d486c6b… from http://central-1:4055 (28313 bytes).
# telemetry likewise, end to end into the UI:
$ for c in …; do grep -c "Telemetry stream opened"; done → 3,4,3,3,3,5
/hosts →  CLUSTER SITE-A … sa-modbus2 SITEA-modbus Modbus HEALTHY 1 host 1 tag  12:30:49 UTC  0 errors
```

The five telemetry `failed … reconnecting` warnings in the run are all accounted for: three at
12:15:20 (central dialing nodes that had not finished binding `:4056`) and two at 12:28:50 /
12:29:30 (the deliberate central-2 stop in finding 5). None is an auth failure.

### Phase 1A finding 5 — the PreStart race loses on real boots, and the memoize is what saves it

The hardening commit's premise is not hypothetical. On the run-2 boot, **central-1 lost the
race on both dependencies**:

```
$ docker logs otopcua-dev-central-1-1 | grep -A3 "Failed to resolve"
[12:15:19 ERR] Failed to resolve IMeshCommandDialer for the mesh command boundary; continuing
               without it. The guards below report what that costs
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
   at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
[12:15:19 ERR] Failed to resolve IMeshPskProvider for the mesh command boundary; …same cause…
```

It is not rare: across the three admin boots observed in run 2 (central-1 and central-2 at
12:15, central-2 again at 12:29 after the stop/start), **two of three lost it**. Had the fix
stopped at PreStart-only resolution, those nodes would have held a permanent silent `null` —
no dialer, and a never-published known-cluster gate — and the failure would only have surfaced
the first time the coordinator singleton moved onto such a node.

**The memoize heals it, verified on the node that actually lost the race.** Run 2's dispatches
were all fanned by central-2 (the singleton host), so central-1's dialer was never exercised
by them. Central-2 was therefore stopped to move the singleton, and a dispatch driven while
central-1 alone held it:

```
$ docker compose stop central-2                                   # 12:28:39
$ curl … /health/ready → self akka.tcp://otopcua@central-1:4053, members 1     # 12:28:57
$ curl -X POST /api/deployments  -d '{"createdBy":"1a-gate-central1-heal"}'    # 12:29:01
$ docker logs otopcua-dev-central-1-1 | grep -E "accepted by|no command dialer is wired"
[12:29:02 DBG] Mesh: DispatchDeployment (correlationId=ad97d386…) accepted by site-b-1:4053 over gRPC
[12:29:02 DBG] … accepted by site-a-2 … site-b-2 … site-a-1 … central-1 …      # 5 live nodes
# no "no command dialer is wired" line, and no further "Failed to resolve"
$ docker compose start central-2  → members 2, unreachable 0, status Healthy   # 12:29:51
```

So the boot-time Error is **cosmetically alarming but functionally healed**, and that is now an
observed property rather than a design claim. Two follow-ups for Phase 2 rather than for 1A:
the Error text says "continuing without it", which reads as permanent and should say that the
first use retries; and a `Failed to resolve` line on a healthy node is exactly the kind of
signal a soak's PSK-drift/error grep will trip over.

**One incidental observation, not a check.** The deployment `RevisionHash` is deterministic
(three separate no-change dispatches all produced `642b9d6f79b9…`), but restoring the alarm
predicate's two text columns did **not** restore the pre-gate hash `6178a8f302aa…` — so the
artifact covers a column the `UPDATE` bumped (`Script.RowVersion` is the obvious candidate).
The rig's configuration content is byte-identical to its pre-gate state; only the hash moved.
Not investigated further — out of this program's scope.

### Run 1 record — the defect that blocked it

Branch `feat/grpc-node-command` @ `fbf7b708`. Image rebuilt from this worktree
(`/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a`) so it carries this branch;
`docker compose -f docker-dev/docker-compose.yml up -d --build` exited 0 and **all eight
containers were recreated** — `migrator` and `cluster-seed` re-ran to completion (exit 0), the
six host nodes and traefik were recreated, the `sql` container and its named volume were kept.
Nothing else touched the stack afterwards, so this run is not a concurrent-rebuild-invalidated
one. Rig-only flip: central-1/-2 `MeshTransport__Mode: "Grpc"` (hard value); **site nodes left
on the compose default `ClusterClient`** (coexistence is the point). Three Serilog
`MinimumLevel:Override` Debug scopes were added on all six nodes for this run only
(`ZB.MOM.WW.OtOpcUa.Host.Grpc`, `ZB.MOM.WW.OtOpcUa.ControlPlane.Communication`,
`Akka.Cluster.Tools.Client`) — the accept/dispatch lines the checks read are `Debug` and the
rig's default Serilog level is `Information`, so without them checks 1/3 would have had no
observable signal. Flip and overrides reverted after the gate, never committed.

**Run 1 verdict: FAIL.** Neither central node booted, so nothing that originates at central
could be driven: checks 1, 2 and 7 were **BLOCKED** outright, 3 and 4 reached only their
node-side legs, and check 8's zero-denial result was vacuous for telemetry and config-serve
(central dials both and never booted). Checks 5 and 6 **PASSED** and are the run-1 evidence
carried into the combined table above. Root cause was defect 4 — resolving `IMeshCommandDialer`
/ `IMeshPskProvider` inside the first `WithActors` starter drags in `ISecretResolver` →
`IAuditWriter` → `ActorAuditWriter`, whose constructor reads `AuditWriterActorKey` out of the
`ActorRegistry` before the singleton that registers it has run. It was **mode-independent**
(reproduced on the same image with `MeshTransport__Mode=ClusterClient`), so it was a defect of
the branch, not of the gate's flip, and it would have failed a `ClusterClient` rig equally.

### Run 1 evidence

**Provenance** — flip in effect on central only, sites untouched:

```
$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
    --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
central-1  MeshTransport__Mode=Grpc
central-2  MeshTransport__Mode=Grpc
site-a-1   MeshTransport__Mode=ClusterClient
site-b-1   MeshTransport__Mode=ClusterClient

$ docker images --format '{{.Repository}}:{{.Tag}}|{{.ID}}|{{.CreatedAt}}' | grep otopcua-host
otopcua-host:dev|69360a173f1f|2026-08-06 07:20:14 -0400 EDT     # rebuilt for this run
```

**Defect 4 — the boot failure (checks 1/2/7 blocked by it).** Both central nodes, identical:

```
$ docker logs otopcua-dev-central-1-1 2>&1 | sed -n '25,45p'
[11:21:27 INF] OPC UA server started on opc.tcp://0.0.0.0:4840
[11:21:27 FTL] Unable to start AkkaHostedService - shutting down application
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
   at Akka.Hosting.ActorRegistry.Get[TKey]()
   at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
        in /src/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Audit/ActorAuditWriter.cs:line 24
   ...
   at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_5(IServiceProvider sp)
   ...
   at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_6(IServiceProvider sp)
   ...
   at ZB.MOM.WW.OtOpcUa.ControlPlane.ServiceCollectionExtensions.<>c.<WithOtOpcUaControlPlaneSingletons>b__7_0(
        ActorSystem system, IActorRegistry registry, IDependencyResolver resolver)
        in /src/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs:line 74
   at Akka.Hosting.AkkaConfigurationBuilder.StartAsync(ActorSystem sys)
   at Akka.Hosting.AkkaHostedService.StartAsync(CancellationToken cancellationToken)
```

**Mode-independence, proven rather than argued** — same image, central-1's exact environment
with `MeshTransport__Mode=ClusterClient`, own hostname + self-seed so it cannot disturb the rig:

```
$ docker run -d --name scratch-cc-mode --network otopcua-dev_default --env-file c1-env.txt \
    -e Cluster__PublicHostname=scratch-cc-mode \
    -e Cluster__SeedNodes__0=akka.tcp://otopcua@scratch-cc-mode:4053 otopcua-host:dev
$ docker logs scratch-cc-mode 2>&1 | grep -m4 -E "MissingActorRegistry|Hosting failed|Host starting"
[11:28:45 INF] OtOpcUa.Host starting with roles=[admin,driver,cluster-main] (admin=True, driver=True)
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
    ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
[11:28:47 ERR] Hosting failed to start
```

Scratch container removed after capture (`docker rm -f scratch-cc-mode`); the six real nodes
were untouched by it.

**It presents as a healthy container.** Worth carrying into Phase 2/4 readings — the failed
process does not exit, so `docker ps` shows `Up`, `RestartCount=0`, and only an HTTP probe or
the log tells the truth:

```
$ docker inspect otopcua-dev-central-1-1 -f '{{.State.Status}} restarts={{.RestartCount}}'
running restarts=0
$ docker run --rm --network otopcua-dev_default curlimages/curl -s -o /dev/null \
    -w '%{http_code}\n' --max-time 5 http://central-1:9000/health/ready
000
$ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9200/health/ready
503
```

**Checks 5 + 6 — the wire-probe auth matrix** (`-import-path` onto the in-repo proto; the node
has no reflection). An empty `CommandEnvelope` has no `oneof` arm set, so `FromEnvelope` throws
and the service answers `InvalidArgument` — **after** the interceptor ran, which is exactly why
it is the positive signal here: `PermissionDenied` means the key was refused, `InvalidArgument`
means the key was accepted and only the payload was rejected:

```
$ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos
$ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \
     -plaintext -import-path /protos -proto mesh_transport.proto"

# 5a  no bearer                        -> site-a-1
Code: PermissionDenied   Message: Node command authentication failed.
# 5b  -H 'authorization: Bearer dev-mesh-psk-SITE-A'   -> site-a-1
Code: InvalidArgument    Message: Command envelope could not be decoded.
# 6a  -H 'authorization: Bearer dev-mesh-psk-SITE-B'   -> site-a-1
Code: PermissionDenied   Message: Node command authentication failed.
# 6b  -H 'authorization: Bearer dev-mesh-psk-SITE-A'   -> site-b-1
Code: PermissionDenied   Message: Node command authentication failed.
# 6c  -H 'authorization: Bearer dev-mesh-psk-SITE-B'   -> site-b-1   (positive control)
Code: InvalidArgument    Message: Command envelope could not be decoded.
```

**Checks 3 + 4 — the node-side leg, driven with real envelopes** over the same authenticated
h2c path central would use. This is the first live proof that `MeshCommandAuthInterceptor` is
actually invoked on a real Kestrel pipeline (register item 4) and that
`NodeCommandGrpcService` → `NodeCommunicationActor` → EventStream → `DriverHostActor` works
end to end:

```
$ $G -H 'authorization: Bearer dev-mesh-psk-SITE-A' -H 'x-otopcua-cluster: SITE-A' \
     -d '{"correlationId":"8461008e...","restartDriver":{"clusterId":"SITE-A",
          "driverInstanceId":"SITEA-modbus","actorByUserName":"1a-gate",
          "correlationId":"8461008e..."}}' \
     site-a-1:4056 mesh_transport.v1.NodeCommandService/Execute
{}

$ docker logs otopcua-dev-site-a-1-1 2>&1 | tail
[11:31:01 DBG] Mesh: accepted RestartDriver (correlationId=8461008e37654724a292219202d78dfa) over gRPC; handed to the node communication actor.
[11:31:01 INF] DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate
[11:31:01 INF] DriverHost site-a-1:4053: spawned Modbus driver SITEA-modbus (stub=False)
[11:31:55 DBG] Mesh: accepted ReconnectDriver (correlationId=29c5dc77283f4eb7869d0c7d6686cadd) over gRPC; handed to the node communication actor.
[11:31:55 INF] DriverHost site-a-1:4053: reconnecting driver SITEA-modbus by request of 1a-gate
[11:31:55 INF] DriverInstance SITEA-modbus: ForceReconnect requested by admin; re-entering Reconnecting
[11:31:56 DBG] Mesh: accepted AlarmCommand (correlationId=39ba451173954229a71b9dd1e00ac1b9) over gRPC; handed to the node communication actor.
```

**Check 8 — interceptor denial counts across every node, whole run:**

```
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1  telemetry=0 configserve=0 localdbsync=0 mesh=0
central-2  telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-1   telemetry=0 configserve=0 localdbsync=0 mesh=2   # the two deliberate negatives
site-a-2   telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-1   telemetry=0 configserve=0 localdbsync=0 mesh=1   # the one deliberate negative
site-b-2   telemetry=0 configserve=0 localdbsync=0 mesh=0
```

The `TwoNodeClusterHarness` I-1 null-provider Error (a known test-harness artifact) did **not**
appear in any rig log — `grep -c "IMeshPskProvider"` = 0 on all six nodes, as expected for
production wiring where every admin node registers the provider.

### Phase 1A defect — **CLOSED by `c0475f2d` + `e54883a8`, re-gated in run 2**

**Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter.**
`WithOtOpcUaControlPlaneSingletons`'s `WithActors` block resolves `IMeshCommandDialer` and
`IMeshPskProvider` from `IServiceProvider` to hand to `CentralCommunicationActor.Props`.
`MeshGrpcCommandDialer` takes `IMeshPskProvider`, which takes `ISecretResolver`, which
`AddZbSecrets` builds by resolving `IAuditWriter`, which on an admin node is
`ActorAuditWriter` — and its constructor does `registry.Get<AuditWriterActorKey>()`. That
singleton is registered ~60 lines later in the same builder, so at this point the key is
absent and `ActorRegistry.Get` throws `MissingActorRegistryEntryException`, failing
`AkkaHostedService.StartAsync` and the whole host.

Three things make this worth a pinned regression rather than a one-line fix note:

- **It is not mode-gated.** The two `GetService` calls run in every transport mode by design
  (so a mode flip needs no restart), so the eager graph is walked on every admin boot. Proven
  above with a `ClusterClient` scratch container on the same image.
- **It is invisible to the suite.** The 1A.2 container-activation test
  (`It_activates_from_the_container_despite_the_optional_client_factory_parameter`) resolves
  the dialer from a service collection that does not include `AddZbSecrets`, and
  `TwoNodeClusterHarness` registers no `IMeshPskProvider` at all — so no existing test walks
  the real graph in starter order. This is the same class as scadaproj#1 ("only visible with
  the DI graph built in a real host") and as mesh Phase 6's `IClusterRoleInfo` boot crash.
- **The container lies about it.** The process does not exit, so `docker ps` reports `Up` with
  `restarts=0` while the node serves no HTTP at all. Any Phase 2/4 readiness check that reads
  container status rather than `/health/ready` would score this as healthy.

The shape of the fix is the one this file already uses two lines above for the coordinator
proxy — hand the actor a lazy accessor (`Func<IMeshCommandDialer?>` / `Func<IMeshPskProvider?>`,
or resolve on first use) so nothing is constructed while the `ActorSystem` is still building.

**That is what shipped.** `c0475f2d` moved both dependencies behind `Func<>` invoked in
`PreStart`, pinned by `ControlPlaneStarterOrderTests` (boots a real admin host through the
production starter order and asserts `StartAsync`; verified RED against the eager resolve in
3/3 modes). `e54883a8` then made the resolution memoize-on-first-use rather than PreStart-only,
because `PreStart` runs concurrently with the remaining starters and can still lose the race —
which **run 2 observed happening on 2 of 3 real admin boots**, so that hardening was not
theoretical. Both are re-gated above; see finding 5 for the live self-heal proof.

### Rig state after run 1 (superseded by the run-2 restoration below)

The compose flip was reverted and the six nodes recreated on the committed all-`ClusterClient`
compose. Because defect 4 was mode-independent, the branch image could not produce a healthy
central pair on *any* mode, so the rig was returned to its pre-gate image rather than left with
a dead central pair: the run-1 build was preserved as `otopcua-host:grpc-1a` (`69360a173f1f`)
and the pre-gate build (`8bd81d8aea9b`) re-tagged back to `otopcua-host:dev`.

### Rig state after run 2 — **restored, six healthy, on the branch image**

The flip was reverted (`git checkout docker-dev/docker-compose.yml`) and the six nodes
recreated on the committed all-`ClusterClient` compose. No image substitution was needed this
time: `otopcua-host:dev` is the run-2 build of the committed source (`1d8452a29018`), and it
boots a healthy central pair, so the rig runs the code under test. The run-1 workaround tag
`otopcua-host:grpc-1a` was deleted to stop anyone reusing a stale image.

```
central-1  running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
central-2  running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-a-1   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-a-2   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-b-1   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-b-2   running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
$ curl … /health/ready → status Healthy, members 2, unreachable 0
$ grep -c "Mesh ClusterClient for cluster"  → 3 on each central node (MAIN + SITE-A + SITE-B)
```

The three ClusterClients reappearing on the rollback is an incidental but useful result: the
dark switch still rolls back cleanly on the fixed branch, which is what Phase 2 will depend on
if its cutover has to be reversed.

---

## Phase 1B — node→central ack plane — **PENDING**

Branch `feat/grpc-central-ack` @ `________` (rebased on merged 1A). Rig-only flip:
site-a-1/-2 → `Grpc`; site-b stays `ClusterClient`; central back on `ClusterClient` mode
(its command direction) — acks are governed by the **node's** mode, so this isolates 1B.

| # | Check | Result |
|---|---|---|
| 1 | Deploy e2e: site-a acks arrive via `ReportApply` (central `CentralCommandService` log lines carry site-a node ids), `NodeDeploymentState` rows land, deployment seals | PENDING |
| 2 | Site-b acks still arrive via ClusterClient in the same deployment — per-pair coexistence proven | PENDING |
| 3 | Exactly **one** ack row per site-a node per dispatch (no double-count from endpoint failover) | PENDING |
| 4 | Kill central-1 before the apply completes: site-a acks fail over to central-2's endpoint; deployment still seals | PENDING |
| 5 | Unauthenticated / wrong-cluster-key / missing `x-otopcua-cluster` probes against central `:4055` `CentralCommandService/ReportApply` ⇒ `PermissionDenied` (missing-header rejection is central-side by design) | PENDING |
| 6 | ConfigServe (`DeploymentArtifactService`) on the same listener unaffected: fetches succeed, 0 auth failures | PENDING |

---

## Phase 2 — full-rig cutover + soak — **PENDING**

Rig default flipped and committed: `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"`.
Branch/SHA `________`. **Run the Baseline section first on the pre-flip build.**

| # | Check | Result |
|---|---|---|
| 1 | Full rebuild all-`Grpc`: six nodes ready, deploy e2e seals, all acks via gRPC | PENDING |
| 2 | Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes | PENDING |
| 3 | Kill site-a-1 mid-dispatch: per-node `Execute` failure **logged with node id + status** (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path) | PENDING |
| 4 | Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action | PENDING |
| 5 | Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; **no duplicate applies** after the pair returns (boot-replay path only) | PENDING |
| 6 | **Zero ClusterClient activity under `Grpc`:** `docker logs <every node> \| grep -ci "clusterclient"` = 0 new lines post-boot | PENDING |
| 7 | Soak ≥ 2 h under periodic deploys + command traffic: 0 `PermissionDenied`/`Unauthenticated` across all six nodes (PSK drift detector), 0 unexplained `Unavailable` | PENDING |
| 8 | Full suite green on the cutover SHA | PENDING |

---

## Phase 4 — terminal gate on the post-deletion build — **PENDING**

`master` @ `________` (after the Phase 3 deletion merge). **Provenance guard:** image build
timestamp verified LATER than the merge commit time — a pre-deletion image would invalidate
checks 6–7.

| # | Check | Result |
|---|---|---|
| 1 | PSK negatives on both services (no key / wrong key / wrong cluster's key); **unset server key ⇒ all calls rejected** (fail-closed proven live) | PENDING |
| 2 | Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand) | PENDING |
| 3 | Ack matrix: every node acks exactly once, deployment seals | PENDING |
| 4 | Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry) | PENDING |
| 5 | Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart) | PENDING |
| 6 | **Boundary check: no Akka association between any site container and central** — the Baseline (a) probe now shows no cross-mesh `:4053` traffic (pair-internal remoting only) | PENDING |
| 7 | Full-rig restart (pairs together): clean convergence; `grep -ri "receptionist\|clusterclient"` over all node logs = 0 lines | PENDING |
| 8 | `MeshGrpcBoundaryTests` + full suite green on the merge SHA | PENDING |

### Carry-forwards

Anything a phase gate could not drive gets named here with why + what proves it instead, and
is re-checked in the next gate (the ScadaBridge proportionate-gate rule).

| Item | Deferred from | Why it could not be driven | Proven instead by | Status |
|---|---|---|---|---|
| Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-`GrpcPort` skip | 1A run 1 | Both central nodes failed to boot (defect 4) | — | **CLOSED in 1A run 2** — all three PASS live |
| Check 4 acknowledge half — the condition actually acknowledging | 1A run 1 | The rig's one `ScriptedAlarm` sits Normal and its owning central node was down | — | **CLOSED in 1A run 2** — acked from the real `/alerts` page, `ACKNOWLEDGED` row observed |
| Check 8 for telemetry + config-serve | 1A run 1 | Central dials both planes and never booted, so neither carried a call — 0 denials was vacuous | — | **CLOSED in 1A run 2** — both planes carried authenticated traffic with 0 denials |
| Check 3 central half — `RestartDriver`/`ReconnectDriver` **originating at the AdminUI** | 1A runs 1 + 2 | **Structurally undrivable, not a rig problem:** `<DriverStatusPanel` has 0 usages repo-wide (the 5 name hits are comments), no rendered page carries a Restart/Reconnect control, and the only non-script-analysis HTTP POST is `/api/deployments`. Pre-existing on `master`, not caused by this program | Composition the controller accepted: run 1's live node-side leg for both commands + run 2's check 4, which drives the identical `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path from the AdminUI for another command type; per-type difference is `ToEnvelope`, covered by the T0.2 goldens | **CLOSED by composition.** The orphaned panel itself is out of this program's scope — file separately if the control is meant to exist |
| `Failed to resolve` Error text says "continuing without it", which reads permanent although first use retries; and it will trip a soak's error grep on a healthy node | 1A run 2 (finding 5) | Cosmetic/observability, not behaviour | The self-heal proven live (finding 5) | Phase 2 |

### Defects the gates caught that the suite did not

Recorded per phase as they occur, with root cause + the pinned regression test.

| # | Phase | Defect | Root cause | Status |
|---|---|---|---|---|
| 4 | 1A gate run 1 | **An admin node cannot boot at all** — `MissingActorRegistryEntryException: AuditWriterActorKey`, both central nodes, every transport mode | The first `WithActors` starter eagerly resolves `IMeshCommandDialer`/`IMeshPskProvider`, whose graph reaches `ActorAuditWriter`, whose ctor reads an `ActorRegistry` key registered ~60 lines later in the same builder | **CLOSED** — `c0475f2d` (`Func<>` + `PreStart`, pinned red-first by `ControlPlaneStarterOrderTests`) + `e54883a8` (memoize-on-first-use). Re-gated in run 2: 6/6 nodes boot, 0 `Hosting failed to start` |
| 5 | 1A gate run 2 | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider` with the same `AuditWriterActorKey` cause | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede `AuditWriterActorKey`'s registration | **Not a defect — it is the case `e54883a8` was written for, and the memoize demonstrably heals it.** Proven live by moving the coordinator singleton onto the node that lost the race (stop central-2) and watching it fan a dispatch to all five live nodes with no dialer error. Residual is cosmetic: the Error text and its effect on soak greps → Phase 2 |