dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d7c9ead64e8ba01eed3fdd4a693de04b8b5d2a21
lmxopcua/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md
T

1147 lines
77 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# ClusterClient → gRPC migration — live gate record
Rig: `docker-dev/` (central-1/-2 = `admin,driver,cluster-MAIN`; site-a-1/-2, site-b-1/-2;
sql + traefik + migrator + cluster-seed), rebuilt from the branch under test via
`docker compose -f docker-dev/docker-compose.yml up -d --build`. Recorded check-by-check in
the family's live-gate format. Later phases' sections are appended as those phases land; every
check row gets a **PASS**/**FAIL** verdict plus one quantitative justification clause, with
fenced evidence (real commands, real output). A run invalidated by a concurrent rig rebuild is
**void** — note it and re-run (the ScadaBridge void-run rule).
Provenance discipline: every phase section opens with branch + commit SHA + "image rebuilt,
all N containers recreated", and Phase 4 additionally verifies the image build timestamp
post-dates the deletion merge (a stale image invalidates the boundary checks).
Plan: `2026-08-06-clusterclient-to-grpc.md`. Tracking:
`2026-08-06-clusterclient-to-grpc-tracking.md`.
---
## Baseline (run on the PRE-change build, before the Phase 2 flip)
The two gaps this program closes, **reproduced rather than assumed**:
**(a) Unauthenticated Akka remoting crosses the WAN boundary.** From the host:
```bash
# A live TCP association from a site container to central's Akka port:
docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
# And the receptionist actually accepting cross-mesh traffic (central log after any AdminUI
# driver restart targeting site-a):
docker logs otopcua-dev-central-1-1 2>&1 | grep -i "clusterclient\|receptionist" | tail -5
```
Expected: `OPEN`; receptionist/client lines present. Nothing on this path presents any
credential.
**(b) Silent drop on an unreachable cluster.** Stop the site-b pair, dispatch a deployment
(`POST http://localhost:9200/api/deployments`, `X-Api-Key: docker-dev-deploy-key`), observe:
no per-node dispatch error surfaces anywhere; site-b's `NodeDeploymentState` rows sit
`Applying` until the 2-minute deadline marks the deployment `TimedOut`/partially failed. The
deadline is the only witness. Restart the pair.
- [x] Baseline (a) recorded with evidence — **REPRODUCED** 2026-08-06, rig on `master @ 46be4c8d`
(post-1B image, all-ClusterClient default, no mode override; `MeshTransport__Mode=ClusterClient`
confirmed via `printenv` in the site container)
- [x] Baseline (b) recorded with evidence — **REPRODUCED** same run
### Baseline (a) evidence — unauthenticated Akka remoting crosses the boundary
```
$ docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
OPEN # no credential presented on this path
$ docker logs otopcua-dev-central-1-1 | grep -i "clusterclient\|receptionist" | tail -5
[13:12:51 INF] Connected to [akka://otopcua/system/receptionist]
[13:12:51 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s)
[13:12:53 INF] Connected to [akka://otopcua/system/receptionist]
[13:12:54 INF] Connected to [akka.tcp://otopcua@site-a-1:4053/system/receptionist]
[13:12:57 INF] Connected to [akka.tcp://otopcua@site-b-1:4053/system/receptionist]
```
### Baseline (b) evidence — silent drop on an unreachable cluster
Site-b pair stopped 13:53:5x UTC; dispatch accepted 13:53:54
(`POST /api/deployments` → `{"outcome":"Accepted","deploymentId":"c6cd0798-…"}`, HTTP 202).
```
# 20 s after dispatch — 4/6 applied within ~15 ms, site-b rows parked, deployment open:
NodeDeploymentState: central-1/-2, site-a-1/-2 → Status 1 @ 13:53:54.9xx
site-b-1/-2 → Status 0, AppliedAtUtc NULL
Deployment c6cd0798 → Status 1 (in progress)
# The site-b fan leg is not merely unlogged — it is actively suppressed. The SITE-B
# ClusterClient's SendToAll lands in deadLetters at INF, under Akka's sampled generic
# dead-letter counter, naming no node and no failure:
[13:53:54 INF] Message [DispatchDeployment] wrapped in [$SendToAll] … to [akka://otopcua/deadLetters]
was not delivered. [467] dead letters encountered, of which 456 were not logged. …
Message content: DispatchDeployment { DeploymentId = c6cd0798…, … }
# Dispatch-to-deadline window (13:53:50–13:55:53): ambient Akka association-gating noise for
# the downed pair exists (transport chatter, fires with or without a dispatch), but ZERO lines
# correlate site-b with the deployment/dispatch/command:
$ docker logs --since …09:53:50 --until …09:55:53 otopcua-dev-central-1-1 \
| grep -i "site-b" | grep -icE "deploy|dispatch|command|c6cd0798"
0
# The deadline is the only witness, exactly 2 minutes later:
[13:55:54 WRN] Deployment c6cd0798… timed out after 00:02:00 (4/6 acks landed). No ack from:
site-b-1:4053, site-b-2:4053. Each is an enabled, non-maintenance ClusterNode
row — start the node, or set ClusterNode.MaintenanceMode = 1 …
Deployment c6cd0798 → Status 4; site-b rows stay Status 0/NULL (a timed-out deployment is not
retro-acked; boot replay serves the latest artifact without resurrecting the dead record).
```
Pair restarted after the drill; site-b-1 back healthy. Note for later log reads on this host:
`docker logs --since/--until` parses bare timestamps as HOST-LOCAL time (EDT here, UTC−4) while
the app writes UTC — a UTC-shaped `--since` silently returns nothing (it points 4 h into the
future). Convert to local first; earlier in this run that mistake made a window look empty.
---
## Phase 0 — contract + auth + config groundwork — **PASS**
Branch `feat/grpc-phase0-contract-auth` @ `c2bfad4a`. Image rebuilt, all six containers
recreated (`docker compose -f docker-dev/docker-compose.yml up -d --build`, run from the
worktree at `/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-phase0` so the image carries this
branch). The rig was already running (from the main checkout's compose file) before this run;
the rebuild recreated all eight containers (sql kept, migrator + cluster-seed one-shots re-ran,
central-1/-2 + site-a-1/-2 + site-b-1/-2 + traefik recreated) under the fixed `otopcua-dev`
project name, so this run is not a concurrent-rebuild-invalidated one — nothing else touched
the stack afterwards.
| # | Check | Result |
|---|---|---|
| 1 | All six nodes boot on `ClusterClient` with the dark `Grpc` config present (PSKs, endpoints, ClusterPsks map); no validator fires | **PASS** — 6/6 nodes `restarts=0`, `docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}'` reports `running restarts=0` on every node; `docker logs <node> | grep -iE "OptionsValidationException\|MeshTransport.*(is empty\|refuses to start)"` = 0 lines on all six |
| 2 | Deliberate misconfig: a scratch container with `MeshTransport__Mode=Grpc` and **no** `GrpcPsk` refuses to boot, error names `MeshTransport:GrpcPsk` + the env form | **PASS** — `Hosting failed to start` + `OptionsValidationException` naming `MeshTransport:GrpcPsk` and `MeshTransport__GrpcPsk`, below |
| 3 | Same with empty `CentralGrpcEndpoints` — refuses, names the key | **PASS** — `OptionsValidationException` naming `MeshTransport:CentralGrpcEndpoints` and `MeshTransport__CentralGrpcEndpoints__0`, below |
| 4 | Full suite green, 0 warnings, on the branch | **PASS** — `dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo`: 0 failures on a clean run (see evidence below); no `.cs` files changed on this branch since the last full rebuild, so the pre-existing ~868-warning baseline is unchanged (nothing to newly warn about) |
Note: the new gRPC services are not yet mapped in Phase 0, so no wire probes here — the
interceptor negative/positive matrix is unit-tested and wire-probed in 1A/1B.
**Check 1 evidence** — no validator fired on any of the six nodes, all dark `Grpc` keys present:
```
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do
docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' otopcua-dev-$c-1
done
running restarts=0 # central-1
running restarts=0 # central-2
running restarts=0 # site-a-1
running restarts=0 # site-a-2
running restarts=0 # site-b-1
running restarts=0 # site-b-2
$ docker logs otopcua-dev-central-1-1 2>&1 | grep -iE "MeshTransport|ClusterClient"
[08:53:26 INF] Mesh ClusterClient for cluster MAIN created with 2 contact point(s)
[08:53:26 INF] Mesh ClusterClient for cluster SITE-A created with 2 contact point(s)
[08:53:26 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s)
$ docker logs otopcua-dev-site-a-1-1 2>&1 | grep -iE "cluster.client|receptionist"
[08:53:26 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
$ docker inspect otopcua-dev-site-a-1-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep MeshTransport
MeshTransport__CentralGrpcEndpoints__1=http://central-2:4055
MeshTransport__CentralGrpcEndpoints__0=http://central-1:4055
MeshTransport__Mode=ClusterClient
MeshTransport__GrpcPsk=dev-mesh-psk-SITE-A
MeshTransport__CentralContactPoints__1=akka.tcp://otopcua@central-2:4053
MeshTransport__CentralContactPoints__0=akka.tcp://otopcua@central-1:4053
$ curl -sf http://localhost:9200/health/ready
{"status":"Healthy", ..., "akka":{"status":"Healthy","description":"Akka cluster member status: Up",
"data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","selfRoles":["admin","cluster-MAIN","driver"],
"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}}}
```
**Check 2 evidence** — scratch container from `otopcua-host:dev`, site-a-1's real environment
with `MeshTransport__GrpcPsk` removed and `Mode` forced to `Grpc`:
```
$ docker run --rm --name scratch-check2-noPsk --network otopcua-dev_default \
--env-file check2-env.txt otopcua-host:dev
[08:54:23 ERR] Hosting failed to start
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:GrpcPsk is empty. Under
Grpc this per-cluster pre-shared key is the whole authentication boundary of the command plane
in both directions — without it this node's inbound command service fails closed and every ack
it posts to central is rejected. Supply it via the environment (MeshTransport__GrpcPsk), in
production as ${secret:OT-MESH-PSK-<clusterId>}.
at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
...
```
**Check 3 evidence** — same base env, `GrpcPsk` present, both `CentralGrpcEndpoints` indices
unset:
```
$ docker run -d --name scratch-check3-noEndpoints --network otopcua-dev_default \
--env-file check3-env.txt otopcua-host:dev
$ docker logs scratch-check3-noEndpoints 2>&1 | grep -A2 "OptionsValidationException"
[08:55:55 ERR] Hosting failed to start
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:CentralGrpcEndpoints is
empty. Under Grpc a driver node has nowhere to post its deployment ack, and every deployment
would time out naming a node that is running perfectly. Supply them via the environment
(MeshTransport__CentralGrpcEndpoints__0).
```
Both scratch containers removed after capture (`docker rm -f scratch-check2-noPsk
scratch-check3-noEndpoints`); the rig's six real nodes were never affected (still `restarts=0`
after both runs).
**Check 4 evidence** — full suite, clean re-run (a first run hit one race-condition flake in
`RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed`,
which the test's own name identifies as exercising a race; re-ran in isolation and it passed,
then a full second run passed clean end to end — pre-existing timing sensitivity, not a
regression from this branch, which touched only `docker-dev/docker-compose.yml` and
`docs/Configuration.md`, no `.cs` files):
```
$ dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo
...
Passed! - Failed: 0, Passed: 212, Skipped: 4, Total: 216, Duration: 3m56s - ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.dll (net10.0)
$ echo $?
0
```
Rig left running on `ClusterClient` (its normal state) after the gate.
---
## Phase 1A — central→node command plane — **PASS (on the re-run at `e54883a8`)**
Two runs. **Run 1 @ `fbf7b708` FAILED** at the central boot boundary (defect 4) and could not
drive checks 1/2/7 or the central halves of 3/4; its checks 5 + 6 and the node-side halves of
3 + 4 stand as recorded and were not repeated. **Run 2 @ `e54883a8`** — after the boot fix
(`c0475f2d`) and its review-hardening follow-up (`e54883a8`) — drove every carried check. The
result table below is the **combined, current** verdict; run 1's failure record is preserved
underneath it because the defect it caught is the most valuable thing this gate produced.
| # | Check | Verdict | Run |
|---|---|---|---|
| — | **Boot:** all six nodes healthy INCLUDING the central pair, on the branch image | **PASS** — 6/6 `running restarts=0`, 0 `Hosting failed to start`, `/health/ready` 200 ×6, MAIN mesh `memberCount=2 unreachableCount=0`. **But the `Resolve` Error line the fix logs DID fire — see finding 5** | 2 |
| 1 | Deploy dispatch reaches **all six** driver nodes over gRPC; zero `SendToAll`/ClusterClient on central | **PASS** — 6/6 nodes accepted one `DispatchDeployment` under a single shared `correlationId=cf499b42…`; central's own per-node outcome log records 6 accepted / 0 failed; `grep -ci clusterclient` = **0** on both central nodes | 2 |
| 2 | Acks return over ClusterClient (site mode unchanged) and the deployment **seals** | **PASS** — deployment `8DF9AAE0…` `Status=2` (Sealed) with all six `NodeDeploymentState` rows `Status=1` (Applied) inside ~90 ms; all four site nodes hold live `Connected to [akka.tcp://…/system/receptionist]` associations, which is their only ack path in 1A | 2 |
| 3 | `RestartDriver` + `ReconnectDriver` land on the target site node | **PASS (composed)** — node-side leg driven live in run 1 (both commands acted on site-a-1). The AdminUI half **cannot be driven and this is now proven, not assumed**: `<DriverStatusPanel` has **0** usages repo-wide, seven rendered pages contain no Restart/Reconnect text or button, and the only non-script-analysis HTTP POST in the whole app is `/api/deployments`. Closed by composition — check 4 proves the *same* `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path live from central for an admin-originated command | 1 + 2 |
| 4 | `AlarmCommand` from the AdminUI lands and the condition acknowledges | **PASS** — Ack clicked on the real `/alerts` page against a live `gate481-alarm` ACTIVATED row; central fanned **one** `AlarmCommand` (`correlationId=ccafe375…`, minted once) to all six over gRPC, and the condition acknowledged — a new `ACKNOWLEDGED` row appeared 0.54 s later carrying the operator's identity (`multi-role-test`) | 2 |
| 5 | Unauthenticated ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | **PASS** — 3/3 probes as designed (run 1; not repeated — nothing in the fix touches the interceptor) | 1 |
| 6 | **Wrong-key isolation** | **PASS** — both directions rejected, with a positive control proving the rejection is the key and not reachability (run 1) | 1 |
| 7 | NULL `GrpcPort` node skipped with the warn-once; fan continues | **PASS** — with `site-b-2:4053` NULLed, both central nodes logged the warn **once** each and the next dispatch (`correlationId=88a9e0a2…`) was accepted by exactly **5** nodes, site-b-2 absent; the deployment then honestly ended `Status=4` (TimedOut) with site-b-2 alone at `Status=0`. Row restored; a later dispatch sealed 6/6 | 2 |
| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures | **PASS, and no longer vacuous** — 0 denials on all four interceptors across all six nodes, **and all three planes carried real traffic this time**: site nodes fetched + SHA-verified 28 KB artifacts from `http://central-1:4055`, `Telemetry stream opened` on every node with live driver health rendering on `/hosts`, LocalDb sync active on the site pairs | 2 |
### Run 2 provenance
`docker compose -f docker-dev/docker-compose.yml up -d --build` from the worktree, **exit 0**;
image `otopcua-host:dev` rebuilt to `1d8452a29018` at 12:15 UTC (run 1's preserved
`otopcua-host:grpc-1a` was stale and deliberately **not** reused). The six host nodes plus
`migrator` and `cluster-seed` were recreated (both one-shots exit 0); `sql` and its volume were
kept, and `traefik` was reused unchanged (its service definition did not change). Nothing else
touched the stack during the run, so it is not concurrent-rebuild-invalidated. Same rig-only
flip and same three Serilog Debug overrides as run 1; both reverted afterwards, never committed.
```
$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
--format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
central-1 MeshTransport__Mode=Grpc
central-2 MeshTransport__Mode=Grpc
site-a-1 MeshTransport__Mode=ClusterClient
site-b-1 MeshTransport__Mode=ClusterClient
```
### Run 2 evidence
**Boot — the fix works, and the race it was hardened against is real.**
```
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=2
central-2 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0
site-a-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-a-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-b-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-b-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
```
See **finding 5** below for the two `resolveErr` lines on central-1 — they are the PreStart
race losing on a real boot, and the memoize retry healing it was verified live rather than
argued.
**Check 1 — the fan, both ends.** One envelope, one correlation id, six recipients:
```
$ for c in …; do docker logs otopcua-dev-$c-1 | grep "Mesh: accepted DispatchDeployment" | tail -1; done
central-1 [12:16:49 DBG] Mesh: accepted DispatchDeployment (correlationId=cf499b42c6914a578035b2b5bc1020f7) over gRPC; handed to the node communication actor.
central-2 [12:16:49 DBG] …same correlationId…
site-a-1 [12:16:49 DBG] …same correlationId…
site-a-2 [12:16:49 DBG] …same correlationId…
site-b-1 [12:16:49 DBG] …same correlationId…
site-b-2 [12:16:49 DBG] …same correlationId…
$ docker logs otopcua-dev-central-2-1 | grep "accepted by" # the node hosting the coordinator
[12:16:49 DBG] Mesh: DispatchDeployment (correlationId=cf499b42…) accepted by central-2:4053 over gRPC
[12:16:49 DBG] … accepted by site-b-1:4053 … [12:16:49 DBG] … accepted by site-a-2:4053 …
[12:16:49 DBG] … accepted by site-a-1:4053 … [12:16:49 DBG] … accepted by central-1:4053 …
[12:16:49 DBG] … accepted by site-b-2:4053 …
$ docker logs … | grep -c "was NOT delivered to" → 0 on both central nodes
# Zero ClusterClient on the command side, both central nodes:
central-1 MeshClusterClientCreated=0 clusterclientAnyLine=0
central-2 MeshClusterClientCreated=0 clusterclientAnyLine=0
```
The `clusterclientAnyLine=0` count is over the whole log with
`Serilog:MinimumLevel:Override:Akka.Cluster.Tools.Client=Debug` in force, so it covers Akka's
own ClusterClient internals and not merely this repo's creation message.
**Check 2 — mixed transport, end to end.** Commands out over gRPC (above), acks back over
ClusterClient:
```
$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='8df9aae0-…'
2 -- Sealed
$ … SELECT NodeId, Status, AppliedAtUtc FROM dbo.NodeDeploymentState WHERE DeploymentId='8df9aae0-…'
central-1:4053|1|2026-08-06 12:16:49.939 site-a-1:4053|1|2026-08-06 12:16:49.977
central-2:4053|1|2026-08-06 12:16:49.915 site-a-2:4053|1|2026-08-06 12:16:49.950
site-b-1:4053|1|2026-08-06 12:16:49.957 site-b-2:4053|1|2026-08-06 12:16:49.967
$ for c in site-a-1 site-a-2 site-b-1 site-b-2; do docker logs … | grep "receptionist" | tail -1; done
site-a-1 [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
site-a-2 [12:15:20 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
site-b-1 [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
site-b-2 [12:15:25 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
$ grep -c "No central ClusterClient" → 0 on all six # no ack was dropped for want of a client
```
A site node has **no** gRPC ack path in 1A (that is 1B), so the receptionist association above
is not merely present, it is the only route those four Applied rows could have taken.
**Check 3 — why the AdminUI half is undrivable, established rather than asserted:**
```
$ grep -rn "<DriverStatusPanel" --include="*.razor" --include="*.cs" src/ | wc -l
0 # the component holding the buttons is referenced nowhere
$ grep -rn "DriverStatusPanel" --include="*.razor" src/ | grep -v "Shared/Drivers/DriverStatusPanel.razor:"
…5 hits, all of them code COMMENTS in Alerts.razor / Hosts.razor ("mirrors DriverStatusPanel…")
$ node scan-ui.js # every page a Designer/Administrator/Operator can reach
/ | reconnect/restart text: false | buttons: ["Sign out"]
/hosts | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
/fleet | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
/clusters | reconnect/restart text: false | buttons: ["Sign out"]
/clusters/SITE-A | reconnect/restart text: false | buttons: ["Sign out"]
/alerts | reconnect/restart text: false | buttons: ["Sign out","Clear"]
/deployments | reconnect/restart text: false | buttons: ["Sign out","Deploy current configuration"]
$ grep -rn "MapPost|MapPut|MapDelete" src/…/AdminUI src/…/Host
…/Api/DeployApiEndpoints.cs:43: app.MapPost("/api/deployments", …)
…/ScriptAnalysis/ScriptAnalysisEndpoints.cs:19-24: /diagnostics /completions /hover /signature-help /format /inlay-hints
```
There is no button and no endpoint. What closes the check is that `RestartDriver`,
`ReconnectDriver` and `AlarmCommand` all leave central through the **same three lines** —
`AdminOperationsActor.Dispatch` → `MeshCommand` → `CentralCommunicationActor.DispatchOverGrpc`
— and check 4 drove that path live from the AdminUI for `AlarmCommand`, while run 1 drove the
node-side decode-and-act for the two driver commands. The only per-type difference left is
`MeshTransportDtoMapper.ToEnvelope`, which the T0.2 goldens cover member-for-member.
**Check 4 — AdminUI ack, fanned by central, acknowledged by the owner.** The rig's one
scripted alarm sits Normal indefinitely (`hr200 > 70000`), so a transition was manufactured by
editing its predicate to an always-true form and deploying; the `/alerts` page was already open
when it fired, the Ack button was clicked, and the predicate + hash were restored afterwards
(verified byte-identical by `SELECT`) with a further deploy:
```
page open at 2026-08-06T12:20:59.036Z initial: empty window
ROWS: 12:21:00.017 gate481-alarm | PressHigh | EQ-6e09bcc2d907 | ACTIVATED | 750 | system | press high
ROW BUTTONS: ["Ack","Shelve","Shelve (timed)","Unshelve"]
clicking Acknowledge at 2026-08-06T12:21:00.362Z
after click: 12:21:00.557 | gate481-alarm | EQ-6e09bcc2d907 | ACKNOWLEDGED | 750 | multi-role-test | press high
$ docker logs otopcua-dev-central-2-1 | grep AlarmCommand
[12:21:00 DBG] Mesh: AlarmCommand (correlationId=ccafe3754aca40cfa8aa0a0602c86cbb) accepted by site-b-2:4053 over gRPC
[12:21:00 DBG] … accepted by central-1:4053 … [12:21:00 DBG] … accepted by central-2:4053 …
[12:21:00 DBG] … accepted by site-a-1:4053 … [12:21:00 DBG] … accepted by site-b-1:4053 …
[12:21:00 DBG] … accepted by site-a-2:4053 …
$ for c in …; do grep "Mesh: accepted AlarmCommand" …; done → all six, same correlationId
```
One correlation id across all six recipients is the "encode once, before the fan" rule holding
for the one command whose CLR record has no correlation member of its own — the case the design
called out as the reason encoding is hoisted out of the loop.
**Check 7 — the skip, the warn, and the honest deadline.**
```
$ … UPDATE dbo.ClusterNode SET GrpcPort=NULL WHERE NodeId='site-b-2:4053' # 12:23:14
# one warn per central node, once each, within one ContactRefreshSeconds:
central-2 [12:23:20 WRN] ClusterNode site-b-2:4053 has no GrpcPort; it exposes no command endpoint
and is skipped in this dispatch set (further skips of this node are silent). Commands
will not reach it until the row carries a port
central-1 [12:23:19 WRN] …identical…
# (each node also logs the TelemetryNodeSource twin — "exposes no telemetry stream … skipped in
# this dial refresh" — expected collateral: both target sets read the same column)
# dispatch at 12:23:36 — accepted by five, site-b-2 absent:
central-2 [12:23:37 DBG] … (correlationId=88a9e0a2…) accepted by central-2 / site-a-2 / site-b-1
/ site-a-1 / central-1 :4053 over gRPC
# cumulative per-node accept counts after this dispatch — site-b-2 one behind everyone:
central-1 4 central-2 4 site-a-1 4 site-a-2 4 site-b-1 4 site-b-2 3
$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='5c8bdc9e-…' → 4 (TimedOut)
$ … SELECT NodeId, Status FROM dbo.NodeDeploymentState …
central-1|1 central-2|1 site-a-1|1 site-a-2|1 site-b-1|1 site-b-2|0
```
Row restored to `4056` and a later dispatch sealed 6/6 (`080EBB0B…`, `Status=2`, every node
`Status=1`). One operational detail worth carrying: **the skip persists for up to a full
`ContactRefreshSeconds` after the row is repaired** — a deploy issued 5 s after the restore
(`DF1C0B74…`) was still skipped and still TimedOut, because the dispatch set is only rebuilt on
the refresh tick.
**Check 8 — denials, and proof the planes were actually used.**
```
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
central-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
# config-serve carried real, authenticated traffic:
site-a-1 [12:20:59 INF] Fetched + verified artifact for 8d486c6b… from http://central-1:4055 (28313 bytes).
# telemetry likewise, end to end into the UI:
$ for c in …; do grep -c "Telemetry stream opened"; done → 3,4,3,3,3,5
/hosts → CLUSTER SITE-A … sa-modbus2 SITEA-modbus Modbus HEALTHY 1 host 1 tag 12:30:49 UTC 0 errors
```
The five telemetry `failed … reconnecting` warnings in the run are all accounted for: three at
12:15:20 (central dialing nodes that had not finished binding `:4056`) and two at 12:28:50 /
12:29:30 (the deliberate central-2 stop in finding 5). None is an auth failure.
### Phase 1A finding 5 — the PreStart race loses on real boots, and the memoize is what saves it
The hardening commit's premise is not hypothetical. On the run-2 boot, **central-1 lost the
race on both dependencies**:
```
$ docker logs otopcua-dev-central-1-1 | grep -A3 "Failed to resolve"
[12:15:19 ERR] Failed to resolve IMeshCommandDialer for the mesh command boundary; continuing
without it. The guards below report what that costs
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
[12:15:19 ERR] Failed to resolve IMeshPskProvider for the mesh command boundary; …same cause…
```
It is not rare: across the three admin boots observed in run 2 (central-1 and central-2 at
12:15, central-2 again at 12:29 after the stop/start), **two of three lost it**. Had the fix
stopped at PreStart-only resolution, those nodes would have held a permanent silent `null` —
no dialer, and a never-published known-cluster gate — and the failure would only have surfaced
the first time the coordinator singleton moved onto such a node.
**The memoize heals it, verified on the node that actually lost the race.** Run 2's dispatches
were all fanned by central-2 (the singleton host), so central-1's dialer was never exercised
by them. Central-2 was therefore stopped to move the singleton, and a dispatch driven while
central-1 alone held it:
```
$ docker compose stop central-2 # 12:28:39
$ curl … /health/ready → self akka.tcp://otopcua@central-1:4053, members 1 # 12:28:57
$ curl -X POST /api/deployments -d '{"createdBy":"1a-gate-central1-heal"}' # 12:29:01
$ docker logs otopcua-dev-central-1-1 | grep -E "accepted by|no command dialer is wired"
[12:29:02 DBG] Mesh: DispatchDeployment (correlationId=ad97d386…) accepted by site-b-1:4053 over gRPC
[12:29:02 DBG] … accepted by site-a-2 … site-b-2 … site-a-1 … central-1 … # 5 live nodes
# no "no command dialer is wired" line, and no further "Failed to resolve"
$ docker compose start central-2 → members 2, unreachable 0, status Healthy # 12:29:51
```
So the boot-time Error is **cosmetically alarming but functionally healed**, and that is now an
observed property rather than a design claim. Two follow-ups for Phase 2 rather than for 1A:
the Error text says "continuing without it", which reads as permanent and should say that the
first use retries; and a `Failed to resolve` line on a healthy node is exactly the kind of
signal a soak's PSK-drift/error grep will trip over.
**One incidental observation, not a check.** The deployment `RevisionHash` is deterministic
(three separate no-change dispatches all produced `642b9d6f79b9…`), but restoring the alarm
predicate's two text columns did **not** restore the pre-gate hash `6178a8f302aa…` — so the
artifact covers a column the `UPDATE` bumped (`Script.RowVersion` is the obvious candidate).
The rig's configuration content is byte-identical to its pre-gate state; only the hash moved.
Not investigated further — out of this program's scope.
### Run 1 record — the defect that blocked it
Branch `feat/grpc-node-command` @ `fbf7b708`. Image rebuilt from this worktree
(`/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a`) so it carries this branch;
`docker compose -f docker-dev/docker-compose.yml up -d --build` exited 0 and **all eight
containers were recreated** — `migrator` and `cluster-seed` re-ran to completion (exit 0), the
six host nodes and traefik were recreated, the `sql` container and its named volume were kept.
Nothing else touched the stack afterwards, so this run is not a concurrent-rebuild-invalidated
one. Rig-only flip: central-1/-2 `MeshTransport__Mode: "Grpc"` (hard value); **site nodes left
on the compose default `ClusterClient`** (coexistence is the point). Three Serilog
`MinimumLevel:Override` Debug scopes were added on all six nodes for this run only
(`ZB.MOM.WW.OtOpcUa.Host.Grpc`, `ZB.MOM.WW.OtOpcUa.ControlPlane.Communication`,
`Akka.Cluster.Tools.Client`) — the accept/dispatch lines the checks read are `Debug` and the
rig's default Serilog level is `Information`, so without them checks 1/3 would have had no
observable signal. Flip and overrides reverted after the gate, never committed.
**Run 1 verdict: FAIL.** Neither central node booted, so nothing that originates at central
could be driven: checks 1, 2 and 7 were **BLOCKED** outright, 3 and 4 reached only their
node-side legs, and check 8's zero-denial result was vacuous for telemetry and config-serve
(central dials both and never booted). Checks 5 and 6 **PASSED** and are the run-1 evidence
carried into the combined table above. Root cause was defect 4 — resolving `IMeshCommandDialer`
/ `IMeshPskProvider` inside the first `WithActors` starter drags in `ISecretResolver` →
`IAuditWriter` → `ActorAuditWriter`, whose constructor reads `AuditWriterActorKey` out of the
`ActorRegistry` before the singleton that registers it has run. It was **mode-independent**
(reproduced on the same image with `MeshTransport__Mode=ClusterClient`), so it was a defect of
the branch, not of the gate's flip, and it would have failed a `ClusterClient` rig equally.
### Run 1 evidence
**Provenance** — flip in effect on central only, sites untouched:
```
$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
--format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
central-1 MeshTransport__Mode=Grpc
central-2 MeshTransport__Mode=Grpc
site-a-1 MeshTransport__Mode=ClusterClient
site-b-1 MeshTransport__Mode=ClusterClient
$ docker images --format '{{.Repository}}:{{.Tag}}|{{.ID}}|{{.CreatedAt}}' | grep otopcua-host
otopcua-host:dev|69360a173f1f|2026-08-06 07:20:14 -0400 EDT # rebuilt for this run
```
**Defect 4 — the boot failure (checks 1/2/7 blocked by it).** Both central nodes, identical:
```
$ docker logs otopcua-dev-central-1-1 2>&1 | sed -n '25,45p'
[11:21:27 INF] OPC UA server started on opc.tcp://0.0.0.0:4840
[11:21:27 FTL] Unable to start AkkaHostedService - shutting down application
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
at Akka.Hosting.ActorRegistry.Get[TKey]()
at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
in /src/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Audit/ActorAuditWriter.cs:line 24
...
at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_5(IServiceProvider sp)
...
at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_6(IServiceProvider sp)
...
at ZB.MOM.WW.OtOpcUa.ControlPlane.ServiceCollectionExtensions.<>c.<WithOtOpcUaControlPlaneSingletons>b__7_0(
ActorSystem system, IActorRegistry registry, IDependencyResolver resolver)
in /src/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs:line 74
at Akka.Hosting.AkkaConfigurationBuilder.StartAsync(ActorSystem sys)
at Akka.Hosting.AkkaHostedService.StartAsync(CancellationToken cancellationToken)
```
**Mode-independence, proven rather than argued** — same image, central-1's exact environment
with `MeshTransport__Mode=ClusterClient`, own hostname + self-seed so it cannot disturb the rig:
```
$ docker run -d --name scratch-cc-mode --network otopcua-dev_default --env-file c1-env.txt \
-e Cluster__PublicHostname=scratch-cc-mode \
-e Cluster__SeedNodes__0=akka.tcp://otopcua@scratch-cc-mode:4053 otopcua-host:dev
$ docker logs scratch-cc-mode 2>&1 | grep -m4 -E "MissingActorRegistry|Hosting failed|Host starting"
[11:28:45 INF] OtOpcUa.Host starting with roles=[admin,driver,cluster-main] (admin=True, driver=True)
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
[11:28:47 ERR] Hosting failed to start
```
Scratch container removed after capture (`docker rm -f scratch-cc-mode`); the six real nodes
were untouched by it.
**It presents as a healthy container.** Worth carrying into Phase 2/4 readings — the failed
process does not exit, so `docker ps` shows `Up`, `RestartCount=0`, and only an HTTP probe or
the log tells the truth:
```
$ docker inspect otopcua-dev-central-1-1 -f '{{.State.Status}} restarts={{.RestartCount}}'
running restarts=0
$ docker run --rm --network otopcua-dev_default curlimages/curl -s -o /dev/null \
-w '%{http_code}\n' --max-time 5 http://central-1:9000/health/ready
000
$ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9200/health/ready
503
```
**Checks 5 + 6 — the wire-probe auth matrix** (`-import-path` onto the in-repo proto; the node
has no reflection). An empty `CommandEnvelope` has no `oneof` arm set, so `FromEnvelope` throws
and the service answers `InvalidArgument` — **after** the interceptor ran, which is exactly why
it is the positive signal here: `PermissionDenied` means the key was refused, `InvalidArgument`
means the key was accepted and only the payload was rejected:
```
$ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos
$ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \
-plaintext -import-path /protos -proto mesh_transport.proto"
# 5a no bearer -> site-a-1
Code: PermissionDenied Message: Node command authentication failed.
# 5b -H 'authorization: Bearer dev-mesh-psk-SITE-A' -> site-a-1
Code: InvalidArgument Message: Command envelope could not be decoded.
# 6a -H 'authorization: Bearer dev-mesh-psk-SITE-B' -> site-a-1
Code: PermissionDenied Message: Node command authentication failed.
# 6b -H 'authorization: Bearer dev-mesh-psk-SITE-A' -> site-b-1
Code: PermissionDenied Message: Node command authentication failed.
# 6c -H 'authorization: Bearer dev-mesh-psk-SITE-B' -> site-b-1 (positive control)
Code: InvalidArgument Message: Command envelope could not be decoded.
```
**Checks 3 + 4 — the node-side leg, driven with real envelopes** over the same authenticated
h2c path central would use. This is the first live proof that `MeshCommandAuthInterceptor` is
actually invoked on a real Kestrel pipeline (register item 4) and that
`NodeCommandGrpcService` → `NodeCommunicationActor` → EventStream → `DriverHostActor` works
end to end:
```
$ $G -H 'authorization: Bearer dev-mesh-psk-SITE-A' -H 'x-otopcua-cluster: SITE-A' \
-d '{"correlationId":"8461008e...","restartDriver":{"clusterId":"SITE-A",
"driverInstanceId":"SITEA-modbus","actorByUserName":"1a-gate",
"correlationId":"8461008e..."}}' \
site-a-1:4056 mesh_transport.v1.NodeCommandService/Execute
{}
$ docker logs otopcua-dev-site-a-1-1 2>&1 | tail
[11:31:01 DBG] Mesh: accepted RestartDriver (correlationId=8461008e37654724a292219202d78dfa) over gRPC; handed to the node communication actor.
[11:31:01 INF] DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate
[11:31:01 INF] DriverHost site-a-1:4053: spawned Modbus driver SITEA-modbus (stub=False)
[11:31:55 DBG] Mesh: accepted ReconnectDriver (correlationId=29c5dc77283f4eb7869d0c7d6686cadd) over gRPC; handed to the node communication actor.
[11:31:55 INF] DriverHost site-a-1:4053: reconnecting driver SITEA-modbus by request of 1a-gate
[11:31:55 INF] DriverInstance SITEA-modbus: ForceReconnect requested by admin; re-entering Reconnecting
[11:31:56 DBG] Mesh: accepted AlarmCommand (correlationId=39ba451173954229a71b9dd1e00ac1b9) over gRPC; handed to the node communication actor.
```
**Check 8 — interceptor denial counts across every node, whole run:**
```
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
central-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-1 telemetry=0 configserve=0 localdbsync=0 mesh=2 # the two deliberate negatives
site-a-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-1 telemetry=0 configserve=0 localdbsync=0 mesh=1 # the one deliberate negative
site-b-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
```
The `TwoNodeClusterHarness` I-1 null-provider Error (a known test-harness artifact) did **not**
appear in any rig log — `grep -c "IMeshPskProvider"` = 0 on all six nodes, as expected for
production wiring where every admin node registers the provider.
### Phase 1A defect — **CLOSED by `c0475f2d` + `e54883a8`, re-gated in run 2**
**Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter.**
`WithOtOpcUaControlPlaneSingletons`'s `WithActors` block resolves `IMeshCommandDialer` and
`IMeshPskProvider` from `IServiceProvider` to hand to `CentralCommunicationActor.Props`.
`MeshGrpcCommandDialer` takes `IMeshPskProvider`, which takes `ISecretResolver`, which
`AddZbSecrets` builds by resolving `IAuditWriter`, which on an admin node is
`ActorAuditWriter` — and its constructor does `registry.Get<AuditWriterActorKey>()`. That
singleton is registered ~60 lines later in the same builder, so at this point the key is
absent and `ActorRegistry.Get` throws `MissingActorRegistryEntryException`, failing
`AkkaHostedService.StartAsync` and the whole host.
Three things make this worth a pinned regression rather than a one-line fix note:
- **It is not mode-gated.** The two `GetService` calls run in every transport mode by design
(so a mode flip needs no restart), so the eager graph is walked on every admin boot. Proven
above with a `ClusterClient` scratch container on the same image.
- **It is invisible to the suite.** The 1A.2 container-activation test
(`It_activates_from_the_container_despite_the_optional_client_factory_parameter`) resolves
the dialer from a service collection that does not include `AddZbSecrets`, and
`TwoNodeClusterHarness` registers no `IMeshPskProvider` at all — so no existing test walks
the real graph in starter order. This is the same class as scadaproj#1 ("only visible with
the DI graph built in a real host") and as mesh Phase 6's `IClusterRoleInfo` boot crash.
- **The container lies about it.** The process does not exit, so `docker ps` reports `Up` with
`restarts=0` while the node serves no HTTP at all. Any Phase 2/4 readiness check that reads
container status rather than `/health/ready` would score this as healthy.
The shape of the fix is the one this file already uses two lines above for the coordinator
proxy — hand the actor a lazy accessor (`Func<IMeshCommandDialer?>` / `Func<IMeshPskProvider?>`,
or resolve on first use) so nothing is constructed while the `ActorSystem` is still building.
**That is what shipped.** `c0475f2d` moved both dependencies behind `Func<>` invoked in
`PreStart`, pinned by `ControlPlaneStarterOrderTests` (boots a real admin host through the
production starter order and asserts `StartAsync`; verified RED against the eager resolve in
3/3 modes). `e54883a8` then made the resolution memoize-on-first-use rather than PreStart-only,
because `PreStart` runs concurrently with the remaining starters and can still lose the race —
which **run 2 observed happening on 2 of 3 real admin boots**, so that hardening was not
theoretical. Both are re-gated above; see finding 5 for the live self-heal proof.
### Rig state after run 1 (superseded by the run-2 restoration below)
The compose flip was reverted and the six nodes recreated on the committed all-`ClusterClient`
compose. Because defect 4 was mode-independent, the branch image could not produce a healthy
central pair on *any* mode, so the rig was returned to its pre-gate image rather than left with
a dead central pair: the run-1 build was preserved as `otopcua-host:grpc-1a` (`69360a173f1f`)
and the pre-gate build (`8bd81d8aea9b`) re-tagged back to `otopcua-host:dev`.
### Rig state after run 2 — **restored, six healthy, on the branch image**
The flip was reverted (`git checkout docker-dev/docker-compose.yml`) and the six nodes
recreated on the committed all-`ClusterClient` compose. No image substitution was needed this
time: `otopcua-host:dev` is the run-2 build of the committed source (`1d8452a29018`), and it
boots a healthy central pair, so the rig runs the code under test. The run-1 workaround tag
`otopcua-host:grpc-1a` was deleted to stop anyone reusing a stale image.
```
central-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
central-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-a-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-a-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-b-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-b-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
$ curl … /health/ready → status Healthy, members 2, unreachable 0
$ grep -c "Mesh ClusterClient for cluster" → 3 on each central node (MAIN + SITE-A + SITE-B)
```
The three ClusterClients reappearing on the rollback is an incidental but useful result: the
dark switch still rolls back cleanly on the fixed branch, which is what Phase 2 will depend on
if its cutover has to be reversed.
---
## Phase 1B — node→central ack plane — **PASS**
Branch `feat/grpc-central-ack` @ `4a2a3d3a` (rebased on `master`, which carries the merged 1A).
Rig-only flip: site-a-1/-2 → `Grpc`; site-b stays `ClusterClient`; central stays `ClusterClient`
(its command direction) — acks are governed by the **node's** mode, so this isolates 1B.
| # | Check | Result |
|---|---|---|
| 1 | Deploy e2e: site-a acks arrive via `ReportApply`, `NodeDeploymentState` rows land, deployment seals | **PASS** — deployment `53BD2626…` `Status=2` (Sealed) with all six rows `Status=1` (Applied) inside ~50 ms; both site-a nodes logged `Posted the ApplyAck … to http://central-1:4055` and central-1 logged the matching `ApplyAck … from site-a-1:4053 / site-a-2:4053 … handed to the central communication actor`. **10/10 acks delivered across the whole run**, 0 drops on any node |
| 2 | Site-b acks still arrive via ClusterClient in the SAME deployment — per-pair coexistence | **PASS** — in `53BD2626…` site-b-1/-2 rows are `Applied` while **0** `ReportApply` lines name a site-b node on either central node, and site-b-1/-2 each hold a live `Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]`; conversely site-a-1/-2 hold **0** receptionist associations and created **0** ClusterClients. Each pair's only possible ack route is the one its mode selects |
| 3 | Exactly **one** ack row per site-a node per dispatch (no double-count from endpoint failover) | **PASS** — across 10 dispatches, `acks=1` for every (site-a node, deployment) pair on the happy path and on the `docker stop` failover. The **one** exception is the 4b pause drill (`458b8629…`, `acks=2` per node) and it is the documented at-least-once case, not a defect: the frozen server had in fact received the ack, and **the coordinator absorbed it without inflating the count — it reported `2/6 acks landed`, not `4/6`**. `NodeDeploymentState` holds exactly one row per node per deployment throughout |
| 4 | Kill central-1 (`docker stop`) before the apply: site-a acks fail over to central-2; restart, verify convergence | **PASS** — with central-1 stopped, both site-a nodes logged the Information failover line `… to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable)` and central-2 accepted both **within the same second**; the deployment honestly ended `Status=4` (TimedOut) with central-1 alone at `Status=0` — sealing 6/6 is structurally impossible while one of the six expected nodes is down. After `start central-1` the pair reconverged (`members 2, unreachable 0`) and the next dispatch sealed **6/6** |
| **4b** | **Hang drill** (`docker pause central-1`): the per-endpoint FRESH deadline is what lets central-2 be reached at all | **PASS** — with central-1 **paused** (hung, not refusing), both site-a nodes logged `… to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => DeadlineExceeded)` and the ack landed on central-2. **Observed failover latency ≈ 10 s** (dispatch accepted 13:03:51 → ack at 13:04:01), i.e. exactly one `AckTimeoutSeconds`. A single hoisted deadline would have entered the central-2 attempt already expired and dropped the ack; this is the live proof of the fix in `667639e6` |
| 5 | Auth probes against central `:4055` `CentralCommandService/ReportApply` | **PASS** — 7/7 as designed: no bearer ⇒ `PermissionDenied`; SITE-B's key claiming SITE-A ⇒ `PermissionDenied`; no `x-otopcua-cluster` ⇒ `PermissionDenied`; unknown cluster `SITE-Z` with a valid key ⇒ `PermissionDenied`; unauthenticated against central-**2** ⇒ `PermissionDenied`; correct key+header with a **fake** deployment id ⇒ `{}` (auth passed, coordinator ignored it — **0 rows written** for that id); malformed id ⇒ `InvalidArgument` (auth passed, payload refused). Positive control: SITE-B key + SITE-B header also reaches the decoder |
| 6 | ConfigServe (`DeploymentArtifactService`) on the same listener unaffected | **PASS** — all four site nodes **Fetched + verified** a 28 296-byte artifact from `http://central-1:4055` on a real revision change, and a direct authenticated `Fetch` returned artifact bytes; unauthenticated `Fetch` is refused by its **own** interceptor (`Unauthenticated` / "Config-serve authentication failed."), never by the mesh one — which is the method-path scoping holding. **1** config-serve denial in the whole run: the deliberate probe |
### Provenance
`docker compose -f docker-dev/docker-compose.yml up -d --build` from the worktree
`/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1b`, **exit 0**; image `otopcua-host:dev`
rebuilt to `4105cf631003` at 12:51:47 UTC. **All six host nodes plus `migrator`,
`cluster-seed` and `traefik` were recreated** (both one-shots exit 0); `sql` and its volume
were kept. Nothing else touched the stack during the run, so it is not a
concurrent-rebuild-invalidated one. Four rig-only Serilog `MinimumLevel:Override` Debug scopes
were added on all six nodes for this run only — `ZB.MOM.WW.OtOpcUa.Runtime.Communication` (the
sender's `Posted the ApplyAck` line), `ZB.MOM.WW.OtOpcUa.AdminUI.Grpc` (central's accept line),
`ZB.MOM.WW.OtOpcUa.ControlPlane.Communication` and `Akka.Cluster.Tools.Client` — because the
rig's default Serilog level is Information and checks 1–3 would otherwise have had no signal.
Flip and overrides reverted after the gate, never committed.
```
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=2
central-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=2
site-a-1 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0
site-a-2 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0
site-b-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-b-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
$ curl … /health/ready → Healthy, members 2, unreachable 0 (MAIN)
$ … http://<site>:8080/health/ready → 200 on all four site nodes
```
### Checks 1 + 2 evidence — the two transports, side by side in one deployment
```
$ curl -X POST http://localhost:9200/api/deployments -H 'X-Api-Key: docker-dev-deploy-key' \
-d '{"createdBy":"1b-gate-check1"}'
{"outcome":"Accepted","deploymentId":"53bd2626-ce9f-4255-a964-24e4cf3c7ccb", …} HTTP 202
$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='53bd2626-…'
2 -- Sealed
$ … SELECT NodeId, Status, AppliedAtUtc FROM dbo.NodeDeploymentState WHERE DeploymentId='53bd2626-…'
central-1:4053|1|2026-08-06 12:55:53.909 site-a-1:4053|1|2026-08-06 12:55:53.930
central-2:4053|1|2026-08-06 12:55:53.890 site-a-2:4053|1|2026-08-06 12:55:53.936
site-b-1:4053|1|2026-08-06 12:55:53.915 site-b-2:4053|1|2026-08-06 12:55:53.924
# node side — the gRPC pair:
site-a-1 [12:55:53 DBG] Posted the ApplyAck for 53bd2626… (Applied) to http://central-1:4055.
site-a-2 [12:55:53 DBG] Posted the ApplyAck for 53bd2626… (Applied) to http://central-1:4055.
# central side — same two acks, named:
central-1 [12:55:53 DBG] ApplyAck for 53bd2626… from site-a-1:4053 (Applied) handed to the central communication actor
central-1 [12:55:53 DBG] ApplyAck for 53bd2626… from site-a-2:4053 (Applied) handed to the central communication actor
central-2 (none)
# per-node ReportApply accept counts, BOTH central nodes, this deployment:
central-1 site-a-1=1 site-a-2=1 site-b-1=0 site-b-2=0 central-1=0 central-2=0
central-2 site-a-1=0 site-a-2=0 site-b-1=0 site-b-2=0 central-1=0 central-2=0
```
The site-b half is proven by exclusion plus its live association, which is the same shape 1A
used for the site pairs:
```
$ for c in site-b-1 site-b-2; do docker logs … | grep -c "Posted the ApplyAck"; done → 0, 0
site-b-1 [12:53:02 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
site-b-2 [12:53:07 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
$ for c in site-a-1 site-a-2; do docker logs … | grep -c "Connected to .*receptionist"; done → 0, 0
$ for c in site-a-1 site-a-2 site-b-1 site-b-2; do grep -c "Mesh ClusterClient for cluster"; done → 0,0,0,0
```
Site-b rows are `Applied` while nothing on either central node logged a `ReportApply` for them,
and site-a holds no receptionist association at all — so neither pair could have used the
other's transport. Coexistence is per pair, in one deployment, from one dispatch.
### Check 3 evidence — one ack per dispatch, and the duplicate that proved the absorber
Across all ten dispatches, counting `ReportApply` accepts on **both** central nodes:
```
site-a-1 dep=53bd2626… acks=1 site-a-2 dep=53bd2626… acks=1
site-a-1 dep=d0869c9e… acks=1 site-a-2 dep=d0869c9e… acks=1
site-a-1 dep=17507001… acks=1 site-a-2 dep=17507001… acks=1 # the stop-failover dispatch
site-a-1 dep=b1e837a2… acks=1 site-a-2 dep=b1e837a2… acks=1
site-a-1 dep=a394eec6… acks=1 site-a-2 dep=a394eec6… acks=1
site-a-1 dep=9bb520a4… acks=1 site-a-2 dep=9bb520a4… acks=1
site-a-1 dep=458b8629… acks=2 site-a-2 dep=458b8629… acks=2 # the 4b PAUSE dispatch
site-a-1 dep=fc2b7b64… acks=1 site-a-2 dep=fc2b7b64… acks=1
site-a-1 dep=41e4935d… acks=1 site-a-2 dep=41e4935d… acks=1
site-a-1 dep=4126636e… acks=1 site-a-2 dep=4126636e… acks=1
$ … SELECT NodeId, COUNT(*) FROM dbo.NodeDeploymentState WHERE DeploymentId='53bd2626-…' GROUP BY NodeId
central-1:4053|1 central-2:4053|1 site-a-1:4053|1 site-a-2:4053|1 site-b-1:4053|1 site-b-2:4053|1
```
A refused endpoint (`docker stop`) cannot produce a duplicate — the server never saw the call —
which is why `17507001…` is 1 per node despite failing over. The pause drill is the one shape
that can, and it did:
```
central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed … # failover target
central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed …
central-1 [13:04:33 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed … # at `docker unpause`
central-1 [13:04:33 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed …
central-1 [13:05:03 INF] Coordinator recovered in-flight deployment 458b8629… (2/6 acks landed)
```
`2/6`, not `4/6`. The frozen central had in fact received both acks and replayed them on
unpause, and `ConfigPublishCoordinator.HandleAck`'s NodeId-keyed absorption made the duplicate
free — the exact claim `GrpcApplyAckSender`'s "at-least-once walk" remark makes, observed
rather than argued.
### Check 4 evidence — refusal failover
```
$ docker compose stop central-1 # 12:57:11
$ curl -X POST … -d '{"createdBy":"1b-gate-check4-failover"}' # 12:57:30
site-a-1 [12:57:30 INF] Posted the ApplyAck for 17507001… (Applied) to http://central-2:4055
after 1 endpoint(s) failed (http://central-1:4055 => Unavailable).
site-a-2 [12:57:30 INF] …identical…
central-2 [12:57:30 DBG] ApplyAck for 17507001… from site-a-1:4053 (Applied) handed …
central-2 [12:57:30 DBG] ApplyAck for 17507001… from site-a-2:4053 (Applied) handed …
$ … SELECT NodeId, Status FROM dbo.NodeDeploymentState WHERE DeploymentId='17507001-…'
central-1:4053|0 central-2:4053|1 site-a-1:4053|1 site-a-2:4053|1
site-b-1:4053|1 site-b-2:4053|1
$ … SELECT Status FROM dbo.Deployment … → 4 (TimedOut) after the 2-minute deadline
$ docker compose start central-1 # 13:00:10
$ curl … /health/ready → Healthy, members 2, unreachable 0
$ curl -X POST … -d '{"createdBy":"1b-gate-check4-converge"}' → Status=2, all six Status=1
```
**One operational detail worth carrying.** After central-1 returned, site-a-**1**'s cached
`GrpcChannel` to `http://central-1:4055` stayed in transient failure for roughly two minutes
while site-a-**2** reached the same restarted endpoint immediately — three dispatches apart:
```
13:01:25 site-a-1 → central-2 after (central-1 => Unavailable) | site-a-2 → central-1 OK
13:01:48 site-a-1 → central-2 after (central-1 => Unavailable) | site-a-2 → central-1 OK
13:02:49 site-a-1 → central-1 OK | site-a-2 → central-1 OK
$ docker exec otopcua-dev-site-a-1-1 bash -c '</dev/tcp/central-1/4055 && echo OPEN' → OPEN
$ central-1 mesh denials in that window → 0 # the call never reached the interceptor
```
TCP was open and central-1 logged nothing, so this is the client channel's own reconnect
backoff, not a server or auth problem. **No ack was lost** — the ordered walk absorbed it every
time and said so in the log. The Phase 2 reading: a central restart shifts that node's ack load
onto the surviving endpoint for a minute or two afterwards, which a soak's failover-line count
will show as a burst rather than a fault.
### Check 4b evidence — the hang, and why the fresh deadline is the whole check
```
$ docker pause otopcua-dev-central-1-1 # 13:03:35
$ curl … /health/ready → self akka.tcp://otopcua@central-2:4053 # traefik dropped central-1
$ curl -X POST … -d '{"createdBy":"1b-gate-check4b-hang"}' # 13:03:51 (202)
site-a-1 [13:04:01 INF] Posted the ApplyAck for 458b8629… (Applied) to http://central-2:4055
after 1 endpoint(s) failed (http://central-1:4055 => DeadlineExceeded).
site-a-2 [13:04:01 INF] …identical…
central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed …
central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed …
$ … SELECT NodeId, Status, AppliedAtUtc … WHERE DeploymentId='458b8629-…'
site-a-1:4053|1|2026-08-06 13:04:01.837 site-a-2:4053|1|2026-08-06 13:04:01.774
central-1:4053|0|NULL central-2:4053|0|NULL site-b-1:4053|0|NULL site-b-2:4053|0|NULL
$ docker unpause otopcua-dev-central-1-1 # 13:04:33
```
`DeadlineExceeded` — not `Unavailable` — is the discriminator: the first endpoint burned a full
`AckTimeoutSeconds` (10 s, the default; `13:03:51 → 13:04:01`) and central-2 was **still**
dialled with a live deadline. Under the hoisted deadline this branch shipped before
`667639e6`, that second attempt would have started already expired and the ack would have been
dropped with a Warning. Observed failover latency: **≈10 s, one whole `AckTimeoutSeconds`**.
**Collateral the drill produced, recorded because it is the honest picture of a `pause`.**
Pausing a cluster member is a heavier disturbance than stopping one: central-2 marked central-1
UNREACHABLE at 13:03:46 and auto-downed it at 13:04:01, and after `unpause` the two central
nodes each held a one-member view until central-1 was restarted (13:06:36). In that window the
**two gRPC acks landed and none of the four ClusterClient acks did** — central-2 and both
site-b nodes all logged `received DispatchDeployment` at 13:03:51 yet produced no ack the
coordinator saw, and the deployment ended `TimedOut` at `2/6` naming exactly those four:
```
central-1 [13:05:51 WRN] Deployment 458b8629… timed out after 00:02:00 (2/6 acks landed).
No ack from: central-1:4053, central-2:4053, site-b-1:4053, site-b-2:4053
central-1 [13:06:36 WRN] DriverHost central-1:4053: found orphan Applying row for deployment
458b8629-…; replaying
central-1 [13:06:37 INF] DriverHost central-1:4053: applied deployment 458b8629… (children=10)
```
Stated at the strength the evidence supports: the four ClusterClient nodes received the
dispatch and none of their acks reached the coordinator, while both gRPC nodes' acks did. The
site nodes emit no apply-completion line, so whether each of the four actually finished
applying is not determinable from their logs — what is determinable is that the ack did not
arrive and the deadline said so honestly. The boot-replay backstop then repaired central-1.
Not a 1B finding either way (no gRPC ack was involved), but it is the Phase 2 auto-down
behaviour showing up early, and the split view needed a manual `restart central-1` to heal.
### Check 5 evidence — the auth matrix on the ack endpoint
`-import-path` onto the in-repo proto (the node serves no reflection). `PermissionDenied` means
the interceptor refused; anything past it (`{}` or `InvalidArgument`) means auth **passed** and
only the payload was graded:
```
$ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos
$ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \
-plaintext -import-path /protos -proto mesh_transport.proto"
$ BODY='{"deploymentId":"ffffffffffffffffffffffffffffffff","nodeId":"site-a-1:4053",
"outcome":"APPLY_OUTCOME_APPLIED","correlationId":"eeeeeeee…"}'
# 5a no bearer, cluster header present -> central-1
Code: PermissionDenied Message: Central command authentication failed.
# 5b Bearer dev-mesh-psk-SITE-B + cluster SITE-A -> central-1
Code: PermissionDenied Message: Central command authentication failed.
# 5c Bearer dev-mesh-psk-SITE-A, NO cluster header-> central-1
Code: PermissionDenied Message: Central command authentication failed.
# 5d Bearer dev-mesh-psk-SITE-A + cluster SITE-A, FAKE deployment id
{}
# 5e same, MALFORMED deployment id "not-a-guid"
Code: InvalidArgument Message: undecodable ApplyAck
# 5f Bearer dev-mesh-psk-SITE-B + cluster SITE-B (positive control)
Code: InvalidArgument Message: undecodable ApplyAck
# 5g Bearer dev-mesh-psk-SITE-A + cluster SITE-Z (unknown cluster)
Code: PermissionDenied Message: Central command authentication failed.
# 5h no bearer -> central-2
Code: PermissionDenied Message: Central command authentication failed.
```
The accepted fake ack changed nothing — the coordinator ignored a deployment it is not tracking:
```
$ … SELECT COUNT(*) FROM dbo.NodeDeploymentState WHERE DeploymentId='ffffffff-…' → 0
$ … SELECT COUNT(*) FROM dbo.Deployment WHERE DeploymentId='ffffffff-…' → 0
central-1 [13:08:08 DBG] ApplyAck for ffffffff… from site-a-1:4053 (Applied) handed to the
central communication actor
```
Central's denial log separates the three operator-facing causes even though the wire status is
deliberately uniform:
```
[13:08:07 WRN] Rejected a central command call to /mesh_transport.v1.CentralCommandService/ReportApply
claiming cluster SITE-A: no bearer token presented.
[13:08:07 WRN] … claiming cluster SITE-A: the presented bearer token did not match that cluster's
pre-shared key, no key could be established for it, or it is not a cluster this node knows…
[13:08:07 WRN] Rejected a central command call to …/ReportApply: no x-otopcua-cluster header, so there
is no cluster whose pre-shared key the caller could be graded against.
[13:08:25 WRN] … claiming cluster SITE-Z: … or it is not a cluster this node knows…
```
5g is the **known-cluster gate answering live**: a valid SITE-A key presented for `SITE-Z` is
refused in O(1), which also confirms the gate had in fact been published — see observation (b).
### Check 6 evidence — ConfigServe on the same listener
A real node-driven fetch needed a real revision change, so `ServerCluster.Notes` for `MAIN` was
edited, deployed, and **restored** (the revision hash returned to its pre-gate
`642b9d6f79b9…`, verified by a further dispatch):
```
$ … UPDATE dbo.ServerCluster SET Notes = '<original> | 1b-gate-fetch-probe' WHERE ClusterId='MAIN'
$ curl -X POST … → revisionHash 20af950fb211… (was 642b9d6f79b9…) Status=2
site-a-1 [13:09:31 INF] Fetched + verified artifact for 41e4935d… from http://central-1:4055 (28296 bytes).
site-a-2 [13:09:31 INF] …same… site-b-1 [13:09:32 INF] …same… site-b-2 [13:09:32 INF] …same…
$ … UPDATE dbo.ServerCluster SET Notes = '<original>' …
$ curl -X POST … → revisionHash 642b9d6f79b9… Status=2 # restored, byte-identical
```
Both interceptors on the one listener, each answering for its own service:
```
# authenticated Fetch, config-serve key:
$ $G -H 'authorization: Bearer configserve-docker-dev-key' -d '{"deploymentId":"FC2B7B64-…"}' \
central-1:4055 deployment_artifact.v1.DeploymentArtifactService/Fetch
{ "data": "eyJDbHVzdGVycyI6W3siQ2x1c3RlcklkIjoiTUFJTiIsIk5hbWUiOiJNYWluIGNsdXN0ZXIiL…" }
# unauthenticated Fetch — refused by the CONFIG-SERVE interceptor, not the mesh one:
Code: Unauthenticated Message: Config-serve authentication failed.
```
`Unauthenticated` here versus `PermissionDenied` on the mesh service is the method-path scoping
in `CentralCommandAuthInterceptor` holding: the two services share one `AddGrpc` pipeline and
neither interceptor grades the other's calls.
### Observations carried forward from the reviews
**(a) The I-4 shape is not exercisable on this rig, and that is a property of the rig, not a
skipped check.** `MeshPskProvider.TryGetPsk` consults `MeshTransport:ClusterPsks` **before** the
secret store and caches only store-resolved values, and docker-dev supplies
`MeshTransport__ClusterPsks__{MAIN,SITE-A,SITE-B}` on both central nodes. The
`InvalidateAll()` a contact-refresh tick performs (`ContactRefreshSeconds` = 60) therefore drops
an always-empty cache, and an ack arriving just after a tick pays no store read. No latency
delta was measurable and none should be: every ack in the run landed inside the same second it
was posted, refresh tick or not. A real deployment resolving `OT-MESH-PSK-<clusterId>` from the
store is where I-4 becomes observable — Phase 2 on a store-backed rig, not here.
**(b) The `Failed to resolve` Error fired on 4 of 4 admin boots — and healed every time.** Worse
odds than 1A's 2 of 3, and both dependencies lost the race on every boot:
```
central-1 [12:53:00 ERR] Failed to resolve IMeshCommandDialer … / IMeshPskProvider … # boot 1
central-1 [13:00:29 ERR] …both again… # boot 2 (check 4 restart)
central-1 [13:06:36 ERR] …both again… # boot 3 (4b restart)
central-2 [12:53:02 ERR] …both… # its single boot
central-1 [12:53:00 ERR] No IMeshPskProvider was supplied to this actor; the known-cluster gate
will NEVER be published and cached per-cluster keys will NEVER be dropped…
```
The second line is the one that reads worst, and 1B disproves it directly rather than by
argument: **every site-a ack central-1 accepted required that provider**, including the ones
after its third boot (13:07:36, 13:09:31, 13:09:59), and probe 5g's `SITE-Z` refusal is only
possible if `SetKnownClusters` had in fact been published. The memoize heals it; the Error text
does not say so. This is the 1A finding-5 carry-forward, now with a second, independent live
confirmation and a stronger case for the wording fix — the message contains the word "NEVER"
twice about something that self-corrects within one deploy.
**(c) Zero mesh interceptor denials outside the deliberate probes.** Whole-run counts on every
node and every interceptor:
```
central-1 telemetry=0 configserve=1 localdbsync=0 meshCentralCmd=4 nodeCmd=0
central-2 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=1 nodeCmd=0
site-a-1 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0
site-a-2 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0
site-b-1 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0
site-b-2 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0
```
`meshCentralCmd` = 4 on central-1 (probes 5a/5b/5c/5g) + 1 on central-2 (5h) = exactly the five
deliberate negatives; `configserve` = 1 = the deliberate 6b. Nothing else was refused anywhere.
**(d) Pre-existing rig noise, so a Phase 2 error grep is not surprised by it.** Site nodes log
`Unavailable … Name or service not known` from the historian client — the rig points
`ServerHistorian__Endpoint` at `http://histgw-absent.invalid:5222` on purpose (45 occurrences on
site-a-1, 1 on site-b-1, 4 on central-1 in this run). Nothing to do with the ack plane.
### Rig state after the gate — **restored, six healthy, all-ClusterClient, on the branch image**
The flip and the four Serilog overrides were reverted (`git checkout docker-dev/docker-compose.yml`,
`grep -c "GATE-1B ONLY"` → 0) and the nodes recreated on the committed compose. The simultaneous
recreate cold-started both central nodes at once and they self-formed one-member views — the
known simultaneous-cold-start split-brain the (default-OFF) `Cluster:BootstrapGuard` addresses —
healed with a single `restart central-2`:
```
central-1 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
central-2 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
site-a-1 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
site-a-2 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
site-b-1 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
site-b-2 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
$ for h in central-1 central-2; do … /health/ready; done
central-1 Healthy members 2 unreachable 0 leader akka.tcp://otopcua@central-1:4053
central-2 Healthy members 2 unreachable 0 leader akka.tcp://otopcua@central-1:4053
$ … http://<site>:8080/health/ready → 200 ×4
$ curl -X POST … -d '{"createdBy":"1b-gate-rig-restore-verify"}'
CDDA1D80-…|2 central-1|1 central-2|1 site-a-1|1 site-a-2|1 site-b-1|1 site-b-2|1
$ for c in site-a-1 site-a-2 site-b-1 site-b-2; do grep -c "Posted the ApplyAck"; done → 0,0,0,0
$ for c in central-1 central-2; do grep -c "Mesh ClusterClient for cluster"; done → 3 per boot
```
Zero gRPC acks after the rollback and three ClusterClients per central boot: the dark switch
rolls back cleanly in the ack direction too, which is what Phase 2 depends on if its cutover has
to be reversed. Rig configuration is byte-identical to its pre-gate state (`ServerCluster.Notes`
restored; revision hash back to `642b9d6f79b9…`).
### Phase 1B defects
**None.** No product code was changed during or after this gate.
---
## Phase 2 — full-rig cutover + soak — **PENDING**
Rig default flipped and committed: `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"`.
Branch/SHA `________`. **Run the Baseline section first on the pre-flip build.**
| # | Check | Result |
|---|---|---|
| 1 | Full rebuild all-`Grpc`: six nodes ready, deploy e2e seals, all acks via gRPC | PENDING |
| 2 | Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes | PENDING |
| 3 | Kill site-a-1 mid-dispatch: per-node `Execute` failure **logged with node id + status** (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path) | PENDING |
| 4 | Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action | PENDING |
| 5 | Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; **no duplicate applies** after the pair returns (boot-replay path only) | PENDING |
| 6 | **Zero ClusterClient activity under `Grpc`:** `docker logs <every node> \| grep -ci "clusterclient"` = 0 new lines post-boot | PENDING |
| 7 | Soak ≥ 2 h under periodic deploys + command traffic: 0 `PermissionDenied`/`Unauthenticated` across all six nodes (PSK drift detector), 0 unexplained `Unavailable` | PENDING |
| 8 | Full suite green on the cutover SHA | PENDING |
---
## Phase 4 — terminal gate on the post-deletion build — **PENDING**
`master` @ `________` (after the Phase 3 deletion merge). **Provenance guard:** image build
timestamp verified LATER than the merge commit time — a pre-deletion image would invalidate
checks 6–7.
| # | Check | Result |
|---|---|---|
| 1 | PSK negatives on both services (no key / wrong key / wrong cluster's key); **unset server key ⇒ all calls rejected** (fail-closed proven live) | PENDING |
| 2 | Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand) | PENDING |
| 3 | Ack matrix: every node acks exactly once, deployment seals | PENDING |
| 4 | Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry) | PENDING |
| 5 | Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart) | PENDING |
| 6 | **Boundary check: no Akka association between any site container and central** — the Baseline (a) probe now shows no cross-mesh `:4053` traffic (pair-internal remoting only) | PENDING |
| 7 | Full-rig restart (pairs together): clean convergence; `grep -ri "receptionist\|clusterclient"` over all node logs = 0 lines | PENDING |
| 8 | `MeshGrpcBoundaryTests` + full suite green on the merge SHA | PENDING |
### Carry-forwards
Anything a phase gate could not drive gets named here with why + what proves it instead, and
is re-checked in the next gate (the ScadaBridge proportionate-gate rule).
| Item | Deferred from | Why it could not be driven | Proven instead by | Status |
|---|---|---|---|---|
| Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-`GrpcPort` skip | 1A run 1 | Both central nodes failed to boot (defect 4) | — | **CLOSED in 1A run 2** — all three PASS live |
| Check 4 acknowledge half — the condition actually acknowledging | 1A run 1 | The rig's one `ScriptedAlarm` sits Normal and its owning central node was down | — | **CLOSED in 1A run 2** — acked from the real `/alerts` page, `ACKNOWLEDGED` row observed |
| Check 8 for telemetry + config-serve | 1A run 1 | Central dials both planes and never booted, so neither carried a call — 0 denials was vacuous | — | **CLOSED in 1A run 2** — both planes carried authenticated traffic with 0 denials |
| Check 3 central half — `RestartDriver`/`ReconnectDriver` **originating at the AdminUI** | 1A runs 1 + 2 | **Structurally undrivable, not a rig problem:** `<DriverStatusPanel` has 0 usages repo-wide (the 5 name hits are comments), no rendered page carries a Restart/Reconnect control, and the only non-script-analysis HTTP POST is `/api/deployments`. Pre-existing on `master`, not caused by this program | Composition the controller accepted: run 1's live node-side leg for both commands + run 2's check 4, which drives the identical `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path from the AdminUI for another command type; per-type difference is `ToEnvelope`, covered by the T0.2 goldens | **CLOSED by composition.** The orphaned panel itself is out of this program's scope — file separately if the control is meant to exist |
| `Failed to resolve` Error text says "continuing without it", which reads permanent although first use retries; and it will trip a soak's error grep on a healthy node | 1A run 2 (finding 5) | Cosmetic/observability, not behaviour | The self-heal proven live (finding 5), and **re-confirmed independently in 1B**: 4 of 4 admin boots lost the race, yet every site-a ack central-1 accepted and probe 5g's `SITE-Z` refusal both require the provider it claimed would "NEVER" be published | Phase 2 |
| I-4 cold-PSK-cache latency after a contact-refresh tick | 1B | **Not exercisable on docker-dev, structurally:** `TryGetPsk` reads `MeshTransport:ClusterPsks` ahead of the store and caches only store-resolved values, and the rig configures that map — so `InvalidateAll()` clears an always-empty cache and no ack pays a store read | Every ack in the run landed inside the second it was posted, tick or no tick (1B observation (a)) | Phase 2, on a store-backed rig |
| A restarted central's cached `GrpcChannel` can stay in transient failure for ~2 min on the node that saw it die, while a sibling node reaches the same endpoint at once | 1B check 4 | Not a defect — the ordered walk absorbed it and logged the failover each time; no ack lost. TCP was open and central logged nothing, so it is the client channel's backoff | 3 dispatches across the window, 2 failed over and the 3rd went direct (1B check 4 evidence) | Phase 2 soak — expect a burst of failover lines after any central restart, not a fault |
### Defects the gates caught that the suite did not
Recorded per phase as they occur, with root cause + the pinned regression test.
| # | Phase | Defect | Root cause | Status |
|---|---|---|---|---|
| 4 | 1A gate run 1 | **An admin node cannot boot at all** — `MissingActorRegistryEntryException: AuditWriterActorKey`, both central nodes, every transport mode | The first `WithActors` starter eagerly resolves `IMeshCommandDialer`/`IMeshPskProvider`, whose graph reaches `ActorAuditWriter`, whose ctor reads an `ActorRegistry` key registered ~60 lines later in the same builder | **CLOSED** — `c0475f2d` (`Func<>` + `PreStart`, pinned red-first by `ControlPlaneStarterOrderTests`) + `e54883a8` (memoize-on-first-use). Re-gated in run 2: 6/6 nodes boot, 0 `Hosting failed to start` |
| 5 | 1A gate run 2 | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider` with the same `AuditWriterActorKey` cause | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede `AuditWriterActorKey`'s registration | **Not a defect — it is the case `e54883a8` was written for, and the memoize demonstrably heals it.** Proven live by moving the coordinator singleton onto the node that lost the race (stop central-2) and watching it fan a dispatch to all five live nodes with no dialer error. Residual is cosmetic: the Error text and its effect on soak greps → Phase 2 |
Reference in New Issue View Git Blame Copy Permalink