Joseph Doherty
d0ac6fc58c
Ran on master @ 0b194003, post-deletion image d2a290fc5711 (provenance-guarded).
PSK matrix 7/7 incl. unset-key boot refusal; command matrix with Restart/Reconnect
driven from the real /hosts panel (#546 closed the 1A composition carry-forward by
direct evidence); exactly-once ack matrix sealed 6/6; central-kill + site-kill drills
honest (one loud WRN, +2:00 deadlines, boot-replay, no retro-ack); mid-deploy
coordinator kill: recovered coordinator fired the deadline at exactly +2:00, all
landed acks preserved, zero dupes; boundary: 3 established :4053 associations, all
pair-internal, zero site↔central; full-rig restart transport grep = 0 on all six
(one non-mesh observation: central pair reproduced the documented guard-off
cold-start split-brain, healed by single-node rejoin); MeshGrpcBoundaryTests 3/3;
terminal full suite RC=0 direct capture, 50/50 suites, 9,333/0.
I-4 cold-PSK-cache re-carried out of the program (unmeasurable on docker-dev).
No new defects.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
1194 lines
90 KiB
Markdown
1194 lines
90 KiB
Markdown
# ClusterClient → gRPC migration — live gate record
|
||
|
||
Rig: `docker-dev/` (central-1/-2 = `admin,driver,cluster-MAIN`; site-a-1/-2, site-b-1/-2;
|
||
sql + traefik + migrator + cluster-seed), rebuilt from the branch under test via
|
||
`docker compose -f docker-dev/docker-compose.yml up -d --build`. Recorded check-by-check in
|
||
the family's live-gate format. Later phases' sections are appended as those phases land; every
|
||
check row gets a **PASS**/**FAIL** verdict plus one quantitative justification clause, with
|
||
fenced evidence (real commands, real output). A run invalidated by a concurrent rig rebuild is
|
||
**void** — note it and re-run (the ScadaBridge void-run rule).
|
||
|
||
Provenance discipline: every phase section opens with branch + commit SHA + "image rebuilt,
|
||
all N containers recreated", and Phase 4 additionally verifies the image build timestamp
|
||
post-dates the deletion merge (a stale image invalidates the boundary checks).
|
||
|
||
Plan: `2026-08-06-clusterclient-to-grpc.md`. Tracking:
|
||
`2026-08-06-clusterclient-to-grpc-tracking.md`.
|
||
|
||
---
|
||
|
||
## Baseline (run on the PRE-change build, before the Phase 2 flip)
|
||
|
||
The two gaps this program closes, **reproduced rather than assumed**:
|
||
|
||
**(a) Unauthenticated Akka remoting crosses the WAN boundary.** From the host:
|
||
|
||
```bash
|
||
# A live TCP association from a site container to central's Akka port:
|
||
docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
|
||
# And the receptionist actually accepting cross-mesh traffic (central log after any AdminUI
|
||
# driver restart targeting site-a):
|
||
docker logs otopcua-dev-central-1-1 2>&1 | grep -i "clusterclient\|receptionist" | tail -5
|
||
```
|
||
|
||
Expected: `OPEN`; receptionist/client lines present. Nothing on this path presents any
|
||
credential.
|
||
|
||
**(b) Silent drop on an unreachable cluster.** Stop the site-b pair, dispatch a deployment
|
||
(`POST http://localhost:9200/api/deployments`, `X-Api-Key: docker-dev-deploy-key`), observe:
|
||
no per-node dispatch error surfaces anywhere; site-b's `NodeDeploymentState` rows sit
|
||
`Applying` until the 2-minute deadline marks the deployment `TimedOut`/partially failed. The
|
||
deadline is the only witness. Restart the pair.
|
||
|
||
- [x] Baseline (a) recorded with evidence — **REPRODUCED** 2026-08-06, rig on `master @ 46be4c8d`
|
||
(post-1B image, all-ClusterClient default, no mode override; `MeshTransport__Mode=ClusterClient`
|
||
confirmed via `printenv` in the site container)
|
||
- [x] Baseline (b) recorded with evidence — **REPRODUCED** same run
|
||
|
||
### Baseline (a) evidence — unauthenticated Akka remoting crosses the boundary
|
||
|
||
```
|
||
$ docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
|
||
OPEN # no credential presented on this path
|
||
$ docker logs otopcua-dev-central-1-1 | grep -i "clusterclient\|receptionist" | tail -5
|
||
[13:12:51 INF] Connected to [akka://otopcua/system/receptionist]
|
||
[13:12:51 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s)
|
||
[13:12:53 INF] Connected to [akka://otopcua/system/receptionist]
|
||
[13:12:54 INF] Connected to [akka.tcp://otopcua@site-a-1:4053/system/receptionist]
|
||
[13:12:57 INF] Connected to [akka.tcp://otopcua@site-b-1:4053/system/receptionist]
|
||
```
|
||
|
||
### Baseline (b) evidence — silent drop on an unreachable cluster
|
||
|
||
Site-b pair stopped 13:53:5x UTC; dispatch accepted 13:53:54
|
||
(`POST /api/deployments` → `{"outcome":"Accepted","deploymentId":"c6cd0798-…"}`, HTTP 202).
|
||
|
||
```
|
||
# 20 s after dispatch — 4/6 applied within ~15 ms, site-b rows parked, deployment open:
|
||
NodeDeploymentState: central-1/-2, site-a-1/-2 → Status 1 @ 13:53:54.9xx
|
||
site-b-1/-2 → Status 0, AppliedAtUtc NULL
|
||
Deployment c6cd0798 → Status 1 (in progress)
|
||
|
||
# The site-b fan leg is not merely unlogged — it is actively suppressed. The SITE-B
|
||
# ClusterClient's SendToAll lands in deadLetters at INF, under Akka's sampled generic
|
||
# dead-letter counter, naming no node and no failure:
|
||
[13:53:54 INF] Message [DispatchDeployment] wrapped in [$SendToAll] … to [akka://otopcua/deadLetters]
|
||
was not delivered. [467] dead letters encountered, of which 456 were not logged. …
|
||
Message content: DispatchDeployment { DeploymentId = c6cd0798…, … }
|
||
|
||
# Dispatch-to-deadline window (13:53:50–13:55:53): ambient Akka association-gating noise for
|
||
# the downed pair exists (transport chatter, fires with or without a dispatch), but ZERO lines
|
||
# correlate site-b with the deployment/dispatch/command:
|
||
$ docker logs --since …09:53:50 --until …09:55:53 otopcua-dev-central-1-1 \
|
||
| grep -i "site-b" | grep -icE "deploy|dispatch|command|c6cd0798"
|
||
0
|
||
|
||
# The deadline is the only witness, exactly 2 minutes later:
|
||
[13:55:54 WRN] Deployment c6cd0798… timed out after 00:02:00 (4/6 acks landed). No ack from:
|
||
site-b-1:4053, site-b-2:4053. Each is an enabled, non-maintenance ClusterNode
|
||
row — start the node, or set ClusterNode.MaintenanceMode = 1 …
|
||
Deployment c6cd0798 → Status 4; site-b rows stay Status 0/NULL (a timed-out deployment is not
|
||
retro-acked; boot replay serves the latest artifact without resurrecting the dead record).
|
||
```
|
||
|
||
Pair restarted after the drill; site-b-1 back healthy. Note for later log reads on this host:
|
||
`docker logs --since/--until` parses bare timestamps as HOST-LOCAL time (EDT here, UTC−4) while
|
||
the app writes UTC — a UTC-shaped `--since` silently returns nothing (it points 4 h into the
|
||
future). Convert to local first; earlier in this run that mistake made a window look empty.
|
||
|
||
---
|
||
|
||
## Phase 0 — contract + auth + config groundwork — **PASS**
|
||
|
||
Branch `feat/grpc-phase0-contract-auth` @ `c2bfad4a`. Image rebuilt, all six containers
|
||
recreated (`docker compose -f docker-dev/docker-compose.yml up -d --build`, run from the
|
||
worktree at `/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-phase0` so the image carries this
|
||
branch). The rig was already running (from the main checkout's compose file) before this run;
|
||
the rebuild recreated all eight containers (sql kept, migrator + cluster-seed one-shots re-ran,
|
||
central-1/-2 + site-a-1/-2 + site-b-1/-2 + traefik recreated) under the fixed `otopcua-dev`
|
||
project name, so this run is not a concurrent-rebuild-invalidated one — nothing else touched
|
||
the stack afterwards.
|
||
|
||
| # | Check | Result |
|
||
|---|---|---|
|
||
| 1 | All six nodes boot on `ClusterClient` with the dark `Grpc` config present (PSKs, endpoints, ClusterPsks map); no validator fires | **PASS** — 6/6 nodes `restarts=0`, `docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}'` reports `running restarts=0` on every node; `docker logs <node> | grep -iE "OptionsValidationException\|MeshTransport.*(is empty\|refuses to start)"` = 0 lines on all six |
|
||
| 2 | Deliberate misconfig: a scratch container with `MeshTransport__Mode=Grpc` and **no** `GrpcPsk` refuses to boot, error names `MeshTransport:GrpcPsk` + the env form | **PASS** — `Hosting failed to start` + `OptionsValidationException` naming `MeshTransport:GrpcPsk` and `MeshTransport__GrpcPsk`, below |
|
||
| 3 | Same with empty `CentralGrpcEndpoints` — refuses, names the key | **PASS** — `OptionsValidationException` naming `MeshTransport:CentralGrpcEndpoints` and `MeshTransport__CentralGrpcEndpoints__0`, below |
|
||
| 4 | Full suite green, 0 warnings, on the branch | **PASS** — `dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo`: 0 failures on a clean run (see evidence below); no `.cs` files changed on this branch since the last full rebuild, so the pre-existing ~868-warning baseline is unchanged (nothing to newly warn about) |
|
||
|
||
Note: the new gRPC services are not yet mapped in Phase 0, so no wire probes here — the
|
||
interceptor negative/positive matrix is unit-tested and wire-probed in 1A/1B.
|
||
|
||
**Check 1 evidence** — no validator fired on any of the six nodes, all dark `Grpc` keys present:
|
||
|
||
```
|
||
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do
|
||
docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' otopcua-dev-$c-1
|
||
done
|
||
running restarts=0 # central-1
|
||
running restarts=0 # central-2
|
||
running restarts=0 # site-a-1
|
||
running restarts=0 # site-a-2
|
||
running restarts=0 # site-b-1
|
||
running restarts=0 # site-b-2
|
||
|
||
$ docker logs otopcua-dev-central-1-1 2>&1 | grep -iE "MeshTransport|ClusterClient"
|
||
[08:53:26 INF] Mesh ClusterClient for cluster MAIN created with 2 contact point(s)
|
||
[08:53:26 INF] Mesh ClusterClient for cluster SITE-A created with 2 contact point(s)
|
||
[08:53:26 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s)
|
||
|
||
$ docker logs otopcua-dev-site-a-1-1 2>&1 | grep -iE "cluster.client|receptionist"
|
||
[08:53:26 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
|
||
|
||
$ docker inspect otopcua-dev-site-a-1-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep MeshTransport
|
||
MeshTransport__CentralGrpcEndpoints__1=http://central-2:4055
|
||
MeshTransport__CentralGrpcEndpoints__0=http://central-1:4055
|
||
MeshTransport__Mode=ClusterClient
|
||
MeshTransport__GrpcPsk=dev-mesh-psk-SITE-A
|
||
MeshTransport__CentralContactPoints__1=akka.tcp://otopcua@central-2:4053
|
||
MeshTransport__CentralContactPoints__0=akka.tcp://otopcua@central-1:4053
|
||
|
||
$ curl -sf http://localhost:9200/health/ready
|
||
{"status":"Healthy", ..., "akka":{"status":"Healthy","description":"Akka cluster member status: Up",
|
||
"data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","selfRoles":["admin","cluster-MAIN","driver"],
|
||
"memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}}}
|
||
```
|
||
|
||
**Check 2 evidence** — scratch container from `otopcua-host:dev`, site-a-1's real environment
|
||
with `MeshTransport__GrpcPsk` removed and `Mode` forced to `Grpc`:
|
||
|
||
```
|
||
$ docker run --rm --name scratch-check2-noPsk --network otopcua-dev_default \
|
||
--env-file check2-env.txt otopcua-host:dev
|
||
[08:54:23 ERR] Hosting failed to start
|
||
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:GrpcPsk is empty. Under
|
||
Grpc this per-cluster pre-shared key is the whole authentication boundary of the command plane
|
||
in both directions — without it this node's inbound command service fails closed and every ack
|
||
it posts to central is rejected. Supply it via the environment (MeshTransport__GrpcPsk), in
|
||
production as ${secret:OT-MESH-PSK-<clusterId>}.
|
||
at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
|
||
...
|
||
```
|
||
|
||
**Check 3 evidence** — same base env, `GrpcPsk` present, both `CentralGrpcEndpoints` indices
|
||
unset:
|
||
|
||
```
|
||
$ docker run -d --name scratch-check3-noEndpoints --network otopcua-dev_default \
|
||
--env-file check3-env.txt otopcua-host:dev
|
||
$ docker logs scratch-check3-noEndpoints 2>&1 | grep -A2 "OptionsValidationException"
|
||
[08:55:55 ERR] Hosting failed to start
|
||
Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:CentralGrpcEndpoints is
|
||
empty. Under Grpc a driver node has nowhere to post its deployment ack, and every deployment
|
||
would time out naming a node that is running perfectly. Supply them via the environment
|
||
(MeshTransport__CentralGrpcEndpoints__0).
|
||
```
|
||
|
||
Both scratch containers removed after capture (`docker rm -f scratch-check2-noPsk
|
||
scratch-check3-noEndpoints`); the rig's six real nodes were never affected (still `restarts=0`
|
||
after both runs).
|
||
|
||
**Check 4 evidence** — full suite, clean re-run (a first run hit one race-condition flake in
|
||
`RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed`,
|
||
which the test's own name identifies as exercising a race; re-ran in isolation and it passed,
|
||
then a full second run passed clean end to end — pre-existing timing sensitivity, not a
|
||
regression from this branch, which touched only `docker-dev/docker-compose.yml` and
|
||
`docs/Configuration.md`, no `.cs` files):
|
||
|
||
```
|
||
$ dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo
|
||
...
|
||
Passed! - Failed: 0, Passed: 212, Skipped: 4, Total: 216, Duration: 3m56s - ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.dll (net10.0)
|
||
$ echo $?
|
||
0
|
||
```
|
||
|
||
Rig left running on `ClusterClient` (its normal state) after the gate.
|
||
|
||
---
|
||
|
||
## Phase 1A — central→node command plane — **PASS (on the re-run at `e54883a8`)**
|
||
|
||
Two runs. **Run 1 @ `fbf7b708` FAILED** at the central boot boundary (defect 4) and could not
|
||
drive checks 1/2/7 or the central halves of 3/4; its checks 5 + 6 and the node-side halves of
|
||
3 + 4 stand as recorded and were not repeated. **Run 2 @ `e54883a8`** — after the boot fix
|
||
(`c0475f2d`) and its review-hardening follow-up (`e54883a8`) — drove every carried check. The
|
||
result table below is the **combined, current** verdict; run 1's failure record is preserved
|
||
underneath it because the defect it caught is the most valuable thing this gate produced.
|
||
|
||
| # | Check | Verdict | Run |
|
||
|---|---|---|---|
|
||
| — | **Boot:** all six nodes healthy INCLUDING the central pair, on the branch image | **PASS** — 6/6 `running restarts=0`, 0 `Hosting failed to start`, `/health/ready` 200 ×6, MAIN mesh `memberCount=2 unreachableCount=0`. **But the `Resolve` Error line the fix logs DID fire — see finding 5** | 2 |
|
||
| 1 | Deploy dispatch reaches **all six** driver nodes over gRPC; zero `SendToAll`/ClusterClient on central | **PASS** — 6/6 nodes accepted one `DispatchDeployment` under a single shared `correlationId=cf499b42…`; central's own per-node outcome log records 6 accepted / 0 failed; `grep -ci clusterclient` = **0** on both central nodes | 2 |
|
||
| 2 | Acks return over ClusterClient (site mode unchanged) and the deployment **seals** | **PASS** — deployment `8DF9AAE0…` `Status=2` (Sealed) with all six `NodeDeploymentState` rows `Status=1` (Applied) inside ~90 ms; all four site nodes hold live `Connected to [akka.tcp://…/system/receptionist]` associations, which is their only ack path in 1A | 2 |
|
||
| 3 | `RestartDriver` + `ReconnectDriver` land on the target site node | **PASS (composed)** — node-side leg driven live in run 1 (both commands acted on site-a-1). The AdminUI half **cannot be driven and this is now proven, not assumed**: `<DriverStatusPanel` has **0** usages repo-wide, seven rendered pages contain no Restart/Reconnect text or button, and the only non-script-analysis HTTP POST in the whole app is `/api/deployments`. Closed by composition — check 4 proves the *same* `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path live from central for an admin-originated command | 1 + 2 |
|
||
| 4 | `AlarmCommand` from the AdminUI lands and the condition acknowledges | **PASS** — Ack clicked on the real `/alerts` page against a live `gate481-alarm` ACTIVATED row; central fanned **one** `AlarmCommand` (`correlationId=ccafe375…`, minted once) to all six over gRPC, and the condition acknowledged — a new `ACKNOWLEDGED` row appeared 0.54 s later carrying the operator's identity (`multi-role-test`) | 2 |
|
||
| 5 | Unauthenticated ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | **PASS** — 3/3 probes as designed (run 1; not repeated — nothing in the fix touches the interceptor) | 1 |
|
||
| 6 | **Wrong-key isolation** | **PASS** — both directions rejected, with a positive control proving the rejection is the key and not reachability (run 1) | 1 |
|
||
| 7 | NULL `GrpcPort` node skipped with the warn-once; fan continues | **PASS** — with `site-b-2:4053` NULLed, both central nodes logged the warn **once** each and the next dispatch (`correlationId=88a9e0a2…`) was accepted by exactly **5** nodes, site-b-2 absent; the deployment then honestly ended `Status=4` (TimedOut) with site-b-2 alone at `Status=0`. Row restored; a later dispatch sealed 6/6 | 2 |
|
||
| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures | **PASS, and no longer vacuous** — 0 denials on all four interceptors across all six nodes, **and all three planes carried real traffic this time**: site nodes fetched + SHA-verified 28 KB artifacts from `http://central-1:4055`, `Telemetry stream opened` on every node with live driver health rendering on `/hosts`, LocalDb sync active on the site pairs | 2 |
|
||
|
||
### Run 2 provenance
|
||
|
||
`docker compose -f docker-dev/docker-compose.yml up -d --build` from the worktree, **exit 0**;
|
||
image `otopcua-host:dev` rebuilt to `1d8452a29018` at 12:15 UTC (run 1's preserved
|
||
`otopcua-host:grpc-1a` was stale and deliberately **not** reused). The six host nodes plus
|
||
`migrator` and `cluster-seed` were recreated (both one-shots exit 0); `sql` and its volume were
|
||
kept, and `traefik` was reused unchanged (its service definition did not change). Nothing else
|
||
touched the stack during the run, so it is not concurrent-rebuild-invalidated. Same rig-only
|
||
flip and same three Serilog Debug overrides as run 1; both reverted afterwards, never committed.
|
||
|
||
```
|
||
$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
|
||
--format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
|
||
central-1 MeshTransport__Mode=Grpc
|
||
central-2 MeshTransport__Mode=Grpc
|
||
site-a-1 MeshTransport__Mode=ClusterClient
|
||
site-b-1 MeshTransport__Mode=ClusterClient
|
||
```
|
||
|
||
### Run 2 evidence
|
||
|
||
**Boot — the fix works, and the race it was hardened against is real.**
|
||
|
||
```
|
||
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
|
||
central-1 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=2
|
||
central-2 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0
|
||
site-a-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
|
||
site-a-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
|
||
site-b-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
|
||
site-b-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
|
||
```
|
||
|
||
See **finding 5** below for the two `resolveErr` lines on central-1 — they are the PreStart
|
||
race losing on a real boot, and the memoize retry healing it was verified live rather than
|
||
argued.
|
||
|
||
**Check 1 — the fan, both ends.** One envelope, one correlation id, six recipients:
|
||
|
||
```
|
||
$ for c in …; do docker logs otopcua-dev-$c-1 | grep "Mesh: accepted DispatchDeployment" | tail -1; done
|
||
central-1 [12:16:49 DBG] Mesh: accepted DispatchDeployment (correlationId=cf499b42c6914a578035b2b5bc1020f7) over gRPC; handed to the node communication actor.
|
||
central-2 [12:16:49 DBG] …same correlationId…
|
||
site-a-1 [12:16:49 DBG] …same correlationId…
|
||
site-a-2 [12:16:49 DBG] …same correlationId…
|
||
site-b-1 [12:16:49 DBG] …same correlationId…
|
||
site-b-2 [12:16:49 DBG] …same correlationId…
|
||
|
||
$ docker logs otopcua-dev-central-2-1 | grep "accepted by" # the node hosting the coordinator
|
||
[12:16:49 DBG] Mesh: DispatchDeployment (correlationId=cf499b42…) accepted by central-2:4053 over gRPC
|
||
[12:16:49 DBG] … accepted by site-b-1:4053 … [12:16:49 DBG] … accepted by site-a-2:4053 …
|
||
[12:16:49 DBG] … accepted by site-a-1:4053 … [12:16:49 DBG] … accepted by central-1:4053 …
|
||
[12:16:49 DBG] … accepted by site-b-2:4053 …
|
||
$ docker logs … | grep -c "was NOT delivered to" → 0 on both central nodes
|
||
|
||
# Zero ClusterClient on the command side, both central nodes:
|
||
central-1 MeshClusterClientCreated=0 clusterclientAnyLine=0
|
||
central-2 MeshClusterClientCreated=0 clusterclientAnyLine=0
|
||
```
|
||
|
||
The `clusterclientAnyLine=0` count is over the whole log with
|
||
`Serilog:MinimumLevel:Override:Akka.Cluster.Tools.Client=Debug` in force, so it covers Akka's
|
||
own ClusterClient internals and not merely this repo's creation message.
|
||
|
||
**Check 2 — mixed transport, end to end.** Commands out over gRPC (above), acks back over
|
||
ClusterClient:
|
||
|
||
```
|
||
$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='8df9aae0-…'
|
||
2 -- Sealed
|
||
$ … SELECT NodeId, Status, AppliedAtUtc FROM dbo.NodeDeploymentState WHERE DeploymentId='8df9aae0-…'
|
||
central-1:4053|1|2026-08-06 12:16:49.939 site-a-1:4053|1|2026-08-06 12:16:49.977
|
||
central-2:4053|1|2026-08-06 12:16:49.915 site-a-2:4053|1|2026-08-06 12:16:49.950
|
||
site-b-1:4053|1|2026-08-06 12:16:49.957 site-b-2:4053|1|2026-08-06 12:16:49.967
|
||
|
||
$ for c in site-a-1 site-a-2 site-b-1 site-b-2; do docker logs … | grep "receptionist" | tail -1; done
|
||
site-a-1 [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
|
||
site-a-2 [12:15:20 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
|
||
site-b-1 [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
|
||
site-b-2 [12:15:25 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
|
||
$ grep -c "No central ClusterClient" → 0 on all six # no ack was dropped for want of a client
|
||
```
|
||
|
||
A site node has **no** gRPC ack path in 1A (that is 1B), so the receptionist association above
|
||
is not merely present, it is the only route those four Applied rows could have taken.
|
||
|
||
**Check 3 — why the AdminUI half is undrivable, established rather than asserted:**
|
||
|
||
```
|
||
$ grep -rn "<DriverStatusPanel" --include="*.razor" --include="*.cs" src/ | wc -l
|
||
0 # the component holding the buttons is referenced nowhere
|
||
$ grep -rn "DriverStatusPanel" --include="*.razor" src/ | grep -v "Shared/Drivers/DriverStatusPanel.razor:"
|
||
…5 hits, all of them code COMMENTS in Alerts.razor / Hosts.razor ("mirrors DriverStatusPanel…")
|
||
|
||
$ node scan-ui.js # every page a Designer/Administrator/Operator can reach
|
||
/ | reconnect/restart text: false | buttons: ["Sign out"]
|
||
/hosts | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
|
||
/fleet | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
|
||
/clusters | reconnect/restart text: false | buttons: ["Sign out"]
|
||
/clusters/SITE-A | reconnect/restart text: false | buttons: ["Sign out"]
|
||
/alerts | reconnect/restart text: false | buttons: ["Sign out","Clear"]
|
||
/deployments | reconnect/restart text: false | buttons: ["Sign out","Deploy current configuration"]
|
||
|
||
$ grep -rn "MapPost|MapPut|MapDelete" src/…/AdminUI src/…/Host
|
||
…/Api/DeployApiEndpoints.cs:43: app.MapPost("/api/deployments", …)
|
||
…/ScriptAnalysis/ScriptAnalysisEndpoints.cs:19-24: /diagnostics /completions /hover /signature-help /format /inlay-hints
|
||
```
|
||
|
||
There is no button and no endpoint. What closes the check is that `RestartDriver`,
|
||
`ReconnectDriver` and `AlarmCommand` all leave central through the **same three lines** —
|
||
`AdminOperationsActor.Dispatch` → `MeshCommand` → `CentralCommunicationActor.DispatchOverGrpc`
|
||
— and check 4 drove that path live from the AdminUI for `AlarmCommand`, while run 1 drove the
|
||
node-side decode-and-act for the two driver commands. The only per-type difference left is
|
||
`MeshTransportDtoMapper.ToEnvelope`, which the T0.2 goldens cover member-for-member.
|
||
|
||
**Check 4 — AdminUI ack, fanned by central, acknowledged by the owner.** The rig's one
|
||
scripted alarm sits Normal indefinitely (`hr200 > 70000`), so a transition was manufactured by
|
||
editing its predicate to an always-true form and deploying; the `/alerts` page was already open
|
||
when it fired, the Ack button was clicked, and the predicate + hash were restored afterwards
|
||
(verified byte-identical by `SELECT`) with a further deploy:
|
||
|
||
```
|
||
page open at 2026-08-06T12:20:59.036Z initial: empty window
|
||
ROWS: 12:21:00.017 gate481-alarm | PressHigh | EQ-6e09bcc2d907 | ACTIVATED | 750 | system | press high
|
||
ROW BUTTONS: ["Ack","Shelve","Shelve (timed)","Unshelve"]
|
||
clicking Acknowledge at 2026-08-06T12:21:00.362Z
|
||
after click: 12:21:00.557 | gate481-alarm | EQ-6e09bcc2d907 | ACKNOWLEDGED | 750 | multi-role-test | press high
|
||
|
||
$ docker logs otopcua-dev-central-2-1 | grep AlarmCommand
|
||
[12:21:00 DBG] Mesh: AlarmCommand (correlationId=ccafe3754aca40cfa8aa0a0602c86cbb) accepted by site-b-2:4053 over gRPC
|
||
[12:21:00 DBG] … accepted by central-1:4053 … [12:21:00 DBG] … accepted by central-2:4053 …
|
||
[12:21:00 DBG] … accepted by site-a-1:4053 … [12:21:00 DBG] … accepted by site-b-1:4053 …
|
||
[12:21:00 DBG] … accepted by site-a-2:4053 …
|
||
$ for c in …; do grep "Mesh: accepted AlarmCommand" …; done → all six, same correlationId
|
||
```
|
||
|
||
One correlation id across all six recipients is the "encode once, before the fan" rule holding
|
||
for the one command whose CLR record has no correlation member of its own — the case the design
|
||
called out as the reason encoding is hoisted out of the loop.
|
||
|
||
**Check 7 — the skip, the warn, and the honest deadline.**
|
||
|
||
```
|
||
$ … UPDATE dbo.ClusterNode SET GrpcPort=NULL WHERE NodeId='site-b-2:4053' # 12:23:14
|
||
|
||
# one warn per central node, once each, within one ContactRefreshSeconds:
|
||
central-2 [12:23:20 WRN] ClusterNode site-b-2:4053 has no GrpcPort; it exposes no command endpoint
|
||
and is skipped in this dispatch set (further skips of this node are silent). Commands
|
||
will not reach it until the row carries a port
|
||
central-1 [12:23:19 WRN] …identical…
|
||
# (each node also logs the TelemetryNodeSource twin — "exposes no telemetry stream … skipped in
|
||
# this dial refresh" — expected collateral: both target sets read the same column)
|
||
|
||
# dispatch at 12:23:36 — accepted by five, site-b-2 absent:
|
||
central-2 [12:23:37 DBG] … (correlationId=88a9e0a2…) accepted by central-2 / site-a-2 / site-b-1
|
||
/ site-a-1 / central-1 :4053 over gRPC
|
||
# cumulative per-node accept counts after this dispatch — site-b-2 one behind everyone:
|
||
central-1 4 central-2 4 site-a-1 4 site-a-2 4 site-b-1 4 site-b-2 3
|
||
|
||
$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='5c8bdc9e-…' → 4 (TimedOut)
|
||
$ … SELECT NodeId, Status FROM dbo.NodeDeploymentState …
|
||
central-1|1 central-2|1 site-a-1|1 site-a-2|1 site-b-1|1 site-b-2|0
|
||
```
|
||
|
||
Row restored to `4056` and a later dispatch sealed 6/6 (`080EBB0B…`, `Status=2`, every node
|
||
`Status=1`). One operational detail worth carrying: **the skip persists for up to a full
|
||
`ContactRefreshSeconds` after the row is repaired** — a deploy issued 5 s after the restore
|
||
(`DF1C0B74…`) was still skipped and still TimedOut, because the dispatch set is only rebuilt on
|
||
the refresh tick.
|
||
|
||
**Check 8 — denials, and proof the planes were actually used.**
|
||
|
||
```
|
||
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
|
||
central-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||
central-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||
site-a-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||
site-a-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||
site-b-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||
site-b-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||
|
||
# config-serve carried real, authenticated traffic:
|
||
site-a-1 [12:20:59 INF] Fetched + verified artifact for 8d486c6b… from http://central-1:4055 (28313 bytes).
|
||
# telemetry likewise, end to end into the UI:
|
||
$ for c in …; do grep -c "Telemetry stream opened"; done → 3,4,3,3,3,5
|
||
/hosts → CLUSTER SITE-A … sa-modbus2 SITEA-modbus Modbus HEALTHY 1 host 1 tag 12:30:49 UTC 0 errors
|
||
```
|
||
|
||
The five telemetry `failed … reconnecting` warnings in the run are all accounted for: three at
|
||
12:15:20 (central dialing nodes that had not finished binding `:4056`) and two at 12:28:50 /
|
||
12:29:30 (the deliberate central-2 stop in finding 5). None is an auth failure.
|
||
|
||
### Phase 1A finding 5 — the PreStart race loses on real boots, and the memoize is what saves it
|
||
|
||
The hardening commit's premise is not hypothetical. On the run-2 boot, **central-1 lost the
|
||
race on both dependencies**:
|
||
|
||
```
|
||
$ docker logs otopcua-dev-central-1-1 | grep -A3 "Failed to resolve"
|
||
[12:15:19 ERR] Failed to resolve IMeshCommandDialer for the mesh command boundary; continuing
|
||
without it. The guards below report what that costs
|
||
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
|
||
ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
|
||
at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
|
||
[12:15:19 ERR] Failed to resolve IMeshPskProvider for the mesh command boundary; …same cause…
|
||
```
|
||
|
||
It is not rare: across the three admin boots observed in run 2 (central-1 and central-2 at
|
||
12:15, central-2 again at 12:29 after the stop/start), **two of three lost it**. Had the fix
|
||
stopped at PreStart-only resolution, those nodes would have held a permanent silent `null` —
|
||
no dialer, and a never-published known-cluster gate — and the failure would only have surfaced
|
||
the first time the coordinator singleton moved onto such a node.
|
||
|
||
**The memoize heals it, verified on the node that actually lost the race.** Run 2's dispatches
|
||
were all fanned by central-2 (the singleton host), so central-1's dialer was never exercised
|
||
by them. Central-2 was therefore stopped to move the singleton, and a dispatch driven while
|
||
central-1 alone held it:
|
||
|
||
```
|
||
$ docker compose stop central-2 # 12:28:39
|
||
$ curl … /health/ready → self akka.tcp://otopcua@central-1:4053, members 1 # 12:28:57
|
||
$ curl -X POST /api/deployments -d '{"createdBy":"1a-gate-central1-heal"}' # 12:29:01
|
||
$ docker logs otopcua-dev-central-1-1 | grep -E "accepted by|no command dialer is wired"
|
||
[12:29:02 DBG] Mesh: DispatchDeployment (correlationId=ad97d386…) accepted by site-b-1:4053 over gRPC
|
||
[12:29:02 DBG] … accepted by site-a-2 … site-b-2 … site-a-1 … central-1 … # 5 live nodes
|
||
# no "no command dialer is wired" line, and no further "Failed to resolve"
|
||
$ docker compose start central-2 → members 2, unreachable 0, status Healthy # 12:29:51
|
||
```
|
||
|
||
So the boot-time Error is **cosmetically alarming but functionally healed**, and that is now an
|
||
observed property rather than a design claim. Two follow-ups for Phase 2 rather than for 1A:
|
||
the Error text says "continuing without it", which reads as permanent and should say that the
|
||
first use retries; and a `Failed to resolve` line on a healthy node is exactly the kind of
|
||
signal a soak's PSK-drift/error grep will trip over.
|
||
|
||
**One incidental observation, not a check.** The deployment `RevisionHash` is deterministic
|
||
(three separate no-change dispatches all produced `642b9d6f79b9…`), but restoring the alarm
|
||
predicate's two text columns did **not** restore the pre-gate hash `6178a8f302aa…` — so the
|
||
artifact covers a column the `UPDATE` bumped (`Script.RowVersion` is the obvious candidate).
|
||
The rig's configuration content is byte-identical to its pre-gate state; only the hash moved.
|
||
Not investigated further — out of this program's scope.
|
||
|
||
### Run 1 record — the defect that blocked it
|
||
|
||
Branch `feat/grpc-node-command` @ `fbf7b708`. Image rebuilt from this worktree
|
||
(`/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a`) so it carries this branch;
|
||
`docker compose -f docker-dev/docker-compose.yml up -d --build` exited 0 and **all eight
|
||
containers were recreated** — `migrator` and `cluster-seed` re-ran to completion (exit 0), the
|
||
six host nodes and traefik were recreated, the `sql` container and its named volume were kept.
|
||
Nothing else touched the stack afterwards, so this run is not a concurrent-rebuild-invalidated
|
||
one. Rig-only flip: central-1/-2 `MeshTransport__Mode: "Grpc"` (hard value); **site nodes left
|
||
on the compose default `ClusterClient`** (coexistence is the point). Three Serilog
|
||
`MinimumLevel:Override` Debug scopes were added on all six nodes for this run only
|
||
(`ZB.MOM.WW.OtOpcUa.Host.Grpc`, `ZB.MOM.WW.OtOpcUa.ControlPlane.Communication`,
|
||
`Akka.Cluster.Tools.Client`) — the accept/dispatch lines the checks read are `Debug` and the
|
||
rig's default Serilog level is `Information`, so without them checks 1/3 would have had no
|
||
observable signal. Flip and overrides reverted after the gate, never committed.
|
||
|
||
**Run 1 verdict: FAIL.** Neither central node booted, so nothing that originates at central
|
||
could be driven: checks 1, 2 and 7 were **BLOCKED** outright, 3 and 4 reached only their
|
||
node-side legs, and check 8's zero-denial result was vacuous for telemetry and config-serve
|
||
(central dials both and never booted). Checks 5 and 6 **PASSED** and are the run-1 evidence
|
||
carried into the combined table above. Root cause was defect 4 — resolving `IMeshCommandDialer`
|
||
/ `IMeshPskProvider` inside the first `WithActors` starter drags in `ISecretResolver` →
|
||
`IAuditWriter` → `ActorAuditWriter`, whose constructor reads `AuditWriterActorKey` out of the
|
||
`ActorRegistry` before the singleton that registers it has run. It was **mode-independent**
|
||
(reproduced on the same image with `MeshTransport__Mode=ClusterClient`), so it was a defect of
|
||
the branch, not of the gate's flip, and it would have failed a `ClusterClient` rig equally.
|
||
|
||
### Run 1 evidence
|
||
|
||
**Provenance** — flip in effect on central only, sites untouched:
|
||
|
||
```
|
||
$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
|
||
--format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
|
||
central-1 MeshTransport__Mode=Grpc
|
||
central-2 MeshTransport__Mode=Grpc
|
||
site-a-1 MeshTransport__Mode=ClusterClient
|
||
site-b-1 MeshTransport__Mode=ClusterClient
|
||
|
||
$ docker images --format '{{.Repository}}:{{.Tag}}|{{.ID}}|{{.CreatedAt}}' | grep otopcua-host
|
||
otopcua-host:dev|69360a173f1f|2026-08-06 07:20:14 -0400 EDT # rebuilt for this run
|
||
```
|
||
|
||
**Defect 4 — the boot failure (checks 1/2/7 blocked by it).** Both central nodes, identical:
|
||
|
||
```
|
||
$ docker logs otopcua-dev-central-1-1 2>&1 | sed -n '25,45p'
|
||
[11:21:27 INF] OPC UA server started on opc.tcp://0.0.0.0:4840
|
||
[11:21:27 FTL] Unable to start AkkaHostedService - shutting down application
|
||
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
|
||
ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
|
||
at Akka.Hosting.ActorRegistry.Get[TKey]()
|
||
at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
|
||
in /src/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Audit/ActorAuditWriter.cs:line 24
|
||
...
|
||
at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_5(IServiceProvider sp)
|
||
...
|
||
at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_6(IServiceProvider sp)
|
||
...
|
||
at ZB.MOM.WW.OtOpcUa.ControlPlane.ServiceCollectionExtensions.<>c.<WithOtOpcUaControlPlaneSingletons>b__7_0(
|
||
ActorSystem system, IActorRegistry registry, IDependencyResolver resolver)
|
||
in /src/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs:line 74
|
||
at Akka.Hosting.AkkaConfigurationBuilder.StartAsync(ActorSystem sys)
|
||
at Akka.Hosting.AkkaHostedService.StartAsync(CancellationToken cancellationToken)
|
||
```
|
||
|
||
**Mode-independence, proven rather than argued** — same image, central-1's exact environment
|
||
with `MeshTransport__Mode=ClusterClient`, own hostname + self-seed so it cannot disturb the rig:
|
||
|
||
```
|
||
$ docker run -d --name scratch-cc-mode --network otopcua-dev_default --env-file c1-env.txt \
|
||
-e Cluster__PublicHostname=scratch-cc-mode \
|
||
-e Cluster__SeedNodes__0=akka.tcp://otopcua@scratch-cc-mode:4053 otopcua-host:dev
|
||
$ docker logs scratch-cc-mode 2>&1 | grep -m4 -E "MissingActorRegistry|Hosting failed|Host starting"
|
||
[11:28:45 INF] OtOpcUa.Host starting with roles=[admin,driver,cluster-main] (admin=True, driver=True)
|
||
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
|
||
ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
|
||
[11:28:47 ERR] Hosting failed to start
|
||
```
|
||
|
||
Scratch container removed after capture (`docker rm -f scratch-cc-mode`); the six real nodes
|
||
were untouched by it.
|
||
|
||
**It presents as a healthy container.** Worth carrying into Phase 2/4 readings — the failed
|
||
process does not exit, so `docker ps` shows `Up`, `RestartCount=0`, and only an HTTP probe or
|
||
the log tells the truth:
|
||
|
||
```
|
||
$ docker inspect otopcua-dev-central-1-1 -f '{{.State.Status}} restarts={{.RestartCount}}'
|
||
running restarts=0
|
||
$ docker run --rm --network otopcua-dev_default curlimages/curl -s -o /dev/null \
|
||
-w '%{http_code}\n' --max-time 5 http://central-1:9000/health/ready
|
||
000
|
||
$ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9200/health/ready
|
||
503
|
||
```
|
||
|
||
**Checks 5 + 6 — the wire-probe auth matrix** (`-import-path` onto the in-repo proto; the node
|
||
has no reflection). An empty `CommandEnvelope` has no `oneof` arm set, so `FromEnvelope` throws
|
||
and the service answers `InvalidArgument` — **after** the interceptor ran, which is exactly why
|
||
it is the positive signal here: `PermissionDenied` means the key was refused, `InvalidArgument`
|
||
means the key was accepted and only the payload was rejected:
|
||
|
||
```
|
||
$ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos
|
||
$ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \
|
||
-plaintext -import-path /protos -proto mesh_transport.proto"
|
||
|
||
# 5a no bearer -> site-a-1
|
||
Code: PermissionDenied Message: Node command authentication failed.
|
||
# 5b -H 'authorization: Bearer dev-mesh-psk-SITE-A' -> site-a-1
|
||
Code: InvalidArgument Message: Command envelope could not be decoded.
|
||
# 6a -H 'authorization: Bearer dev-mesh-psk-SITE-B' -> site-a-1
|
||
Code: PermissionDenied Message: Node command authentication failed.
|
||
# 6b -H 'authorization: Bearer dev-mesh-psk-SITE-A' -> site-b-1
|
||
Code: PermissionDenied Message: Node command authentication failed.
|
||
# 6c -H 'authorization: Bearer dev-mesh-psk-SITE-B' -> site-b-1 (positive control)
|
||
Code: InvalidArgument Message: Command envelope could not be decoded.
|
||
```
|
||
|
||
**Checks 3 + 4 — the node-side leg, driven with real envelopes** over the same authenticated
|
||
h2c path central would use. This is the first live proof that `MeshCommandAuthInterceptor` is
|
||
actually invoked on a real Kestrel pipeline (register item 4) and that
|
||
`NodeCommandGrpcService` → `NodeCommunicationActor` → EventStream → `DriverHostActor` works
|
||
end to end:
|
||
|
||
```
|
||
$ $G -H 'authorization: Bearer dev-mesh-psk-SITE-A' -H 'x-otopcua-cluster: SITE-A' \
|
||
-d '{"correlationId":"8461008e...","restartDriver":{"clusterId":"SITE-A",
|
||
"driverInstanceId":"SITEA-modbus","actorByUserName":"1a-gate",
|
||
"correlationId":"8461008e..."}}' \
|
||
site-a-1:4056 mesh_transport.v1.NodeCommandService/Execute
|
||
{}
|
||
|
||
$ docker logs otopcua-dev-site-a-1-1 2>&1 | tail
|
||
[11:31:01 DBG] Mesh: accepted RestartDriver (correlationId=8461008e37654724a292219202d78dfa) over gRPC; handed to the node communication actor.
|
||
[11:31:01 INF] DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate
|
||
[11:31:01 INF] DriverHost site-a-1:4053: spawned Modbus driver SITEA-modbus (stub=False)
|
||
[11:31:55 DBG] Mesh: accepted ReconnectDriver (correlationId=29c5dc77283f4eb7869d0c7d6686cadd) over gRPC; handed to the node communication actor.
|
||
[11:31:55 INF] DriverHost site-a-1:4053: reconnecting driver SITEA-modbus by request of 1a-gate
|
||
[11:31:55 INF] DriverInstance SITEA-modbus: ForceReconnect requested by admin; re-entering Reconnecting
|
||
[11:31:56 DBG] Mesh: accepted AlarmCommand (correlationId=39ba451173954229a71b9dd1e00ac1b9) over gRPC; handed to the node communication actor.
|
||
```
|
||
|
||
**Check 8 — interceptor denial counts across every node, whole run:**
|
||
|
||
```
|
||
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
|
||
central-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||
central-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||
site-a-1 telemetry=0 configserve=0 localdbsync=0 mesh=2 # the two deliberate negatives
|
||
site-a-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||
site-b-1 telemetry=0 configserve=0 localdbsync=0 mesh=1 # the one deliberate negative
|
||
site-b-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||
```
|
||
|
||
The `TwoNodeClusterHarness` I-1 null-provider Error (a known test-harness artifact) did **not**
|
||
appear in any rig log — `grep -c "IMeshPskProvider"` = 0 on all six nodes, as expected for
|
||
production wiring where every admin node registers the provider.
|
||
|
||
### Phase 1A defect — **CLOSED by `c0475f2d` + `e54883a8`, re-gated in run 2**
|
||
|
||
**Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter.**
|
||
`WithOtOpcUaControlPlaneSingletons`'s `WithActors` block resolves `IMeshCommandDialer` and
|
||
`IMeshPskProvider` from `IServiceProvider` to hand to `CentralCommunicationActor.Props`.
|
||
`MeshGrpcCommandDialer` takes `IMeshPskProvider`, which takes `ISecretResolver`, which
|
||
`AddZbSecrets` builds by resolving `IAuditWriter`, which on an admin node is
|
||
`ActorAuditWriter` — and its constructor does `registry.Get<AuditWriterActorKey>()`. That
|
||
singleton is registered ~60 lines later in the same builder, so at this point the key is
|
||
absent and `ActorRegistry.Get` throws `MissingActorRegistryEntryException`, failing
|
||
`AkkaHostedService.StartAsync` and the whole host.
|
||
|
||
Three things make this worth a pinned regression rather than a one-line fix note:
|
||
|
||
- **It is not mode-gated.** The two `GetService` calls run in every transport mode by design
|
||
(so a mode flip needs no restart), so the eager graph is walked on every admin boot. Proven
|
||
above with a `ClusterClient` scratch container on the same image.
|
||
- **It is invisible to the suite.** The 1A.2 container-activation test
|
||
(`It_activates_from_the_container_despite_the_optional_client_factory_parameter`) resolves
|
||
the dialer from a service collection that does not include `AddZbSecrets`, and
|
||
`TwoNodeClusterHarness` registers no `IMeshPskProvider` at all — so no existing test walks
|
||
the real graph in starter order. This is the same class as scadaproj#1 ("only visible with
|
||
the DI graph built in a real host") and as mesh Phase 6's `IClusterRoleInfo` boot crash.
|
||
- **The container lies about it.** The process does not exit, so `docker ps` reports `Up` with
|
||
`restarts=0` while the node serves no HTTP at all. Any Phase 2/4 readiness check that reads
|
||
container status rather than `/health/ready` would score this as healthy.
|
||
|
||
The shape of the fix is the one this file already uses two lines above for the coordinator
|
||
proxy — hand the actor a lazy accessor (`Func<IMeshCommandDialer?>` / `Func<IMeshPskProvider?>`,
|
||
or resolve on first use) so nothing is constructed while the `ActorSystem` is still building.
|
||
|
||
**That is what shipped.** `c0475f2d` moved both dependencies behind `Func<>` invoked in
|
||
`PreStart`, pinned by `ControlPlaneStarterOrderTests` (boots a real admin host through the
|
||
production starter order and asserts `StartAsync`; verified RED against the eager resolve in
|
||
3/3 modes). `e54883a8` then made the resolution memoize-on-first-use rather than PreStart-only,
|
||
because `PreStart` runs concurrently with the remaining starters and can still lose the race —
|
||
which **run 2 observed happening on 2 of 3 real admin boots**, so that hardening was not
|
||
theoretical. Both are re-gated above; see finding 5 for the live self-heal proof.
|
||
|
||
### Rig state after run 1 (superseded by the run-2 restoration below)
|
||
|
||
The compose flip was reverted and the six nodes recreated on the committed all-`ClusterClient`
|
||
compose. Because defect 4 was mode-independent, the branch image could not produce a healthy
|
||
central pair on *any* mode, so the rig was returned to its pre-gate image rather than left with
|
||
a dead central pair: the run-1 build was preserved as `otopcua-host:grpc-1a` (`69360a173f1f`)
|
||
and the pre-gate build (`8bd81d8aea9b`) re-tagged back to `otopcua-host:dev`.
|
||
|
||
### Rig state after run 2 — **restored, six healthy, on the branch image**
|
||
|
||
The flip was reverted (`git checkout docker-dev/docker-compose.yml`) and the six nodes
|
||
recreated on the committed all-`ClusterClient` compose. No image substitution was needed this
|
||
time: `otopcua-host:dev` is the run-2 build of the committed source (`1d8452a29018`), and it
|
||
boots a healthy central pair, so the rig runs the code under test. The run-1 workaround tag
|
||
`otopcua-host:grpc-1a` was deleted to stop anyone reusing a stale image.
|
||
|
||
```
|
||
central-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
|
||
central-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
|
||
site-a-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
|
||
site-a-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
|
||
site-b-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
|
||
site-b-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
|
||
$ curl … /health/ready → status Healthy, members 2, unreachable 0
|
||
$ grep -c "Mesh ClusterClient for cluster" → 3 on each central node (MAIN + SITE-A + SITE-B)
|
||
```
|
||
|
||
The three ClusterClients reappearing on the rollback is an incidental but useful result: the
|
||
dark switch still rolls back cleanly on the fixed branch, which is what Phase 2 will depend on
|
||
if its cutover has to be reversed.
|
||
|
||
---
|
||
|
||
## Phase 1B — node→central ack plane — **PASS**
|
||
|
||
Branch `feat/grpc-central-ack` @ `4a2a3d3a` (rebased on `master`, which carries the merged 1A).
|
||
Rig-only flip: site-a-1/-2 → `Grpc`; site-b stays `ClusterClient`; central stays `ClusterClient`
|
||
(its command direction) — acks are governed by the **node's** mode, so this isolates 1B.
|
||
|
||
| # | Check | Result |
|
||
|---|---|---|
|
||
| 1 | Deploy e2e: site-a acks arrive via `ReportApply`, `NodeDeploymentState` rows land, deployment seals | **PASS** — deployment `53BD2626…` `Status=2` (Sealed) with all six rows `Status=1` (Applied) inside ~50 ms; both site-a nodes logged `Posted the ApplyAck … to http://central-1:4055` and central-1 logged the matching `ApplyAck … from site-a-1:4053 / site-a-2:4053 … handed to the central communication actor`. **10/10 acks delivered across the whole run**, 0 drops on any node |
|
||
| 2 | Site-b acks still arrive via ClusterClient in the SAME deployment — per-pair coexistence | **PASS** — in `53BD2626…` site-b-1/-2 rows are `Applied` while **0** `ReportApply` lines name a site-b node on either central node, and site-b-1/-2 each hold a live `Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]`; conversely site-a-1/-2 hold **0** receptionist associations and created **0** ClusterClients. Each pair's only possible ack route is the one its mode selects |
|
||
| 3 | Exactly **one** ack row per site-a node per dispatch (no double-count from endpoint failover) | **PASS** — across 10 dispatches, `acks=1` for every (site-a node, deployment) pair on the happy path and on the `docker stop` failover. The **one** exception is the 4b pause drill (`458b8629…`, `acks=2` per node) and it is the documented at-least-once case, not a defect: the frozen server had in fact received the ack, and **the coordinator absorbed it without inflating the count — it reported `2/6 acks landed`, not `4/6`**. `NodeDeploymentState` holds exactly one row per node per deployment throughout |
|
||
| 4 | Kill central-1 (`docker stop`) before the apply: site-a acks fail over to central-2; restart, verify convergence | **PASS** — with central-1 stopped, both site-a nodes logged the Information failover line `… to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable)` and central-2 accepted both **within the same second**; the deployment honestly ended `Status=4` (TimedOut) with central-1 alone at `Status=0` — sealing 6/6 is structurally impossible while one of the six expected nodes is down. After `start central-1` the pair reconverged (`members 2, unreachable 0`) and the next dispatch sealed **6/6** |
|
||
| **4b** | **Hang drill** (`docker pause central-1`): the per-endpoint FRESH deadline is what lets central-2 be reached at all | **PASS** — with central-1 **paused** (hung, not refusing), both site-a nodes logged `… to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => DeadlineExceeded)` and the ack landed on central-2. **Observed failover latency ≈ 10 s** (dispatch accepted 13:03:51 → ack at 13:04:01), i.e. exactly one `AckTimeoutSeconds`. A single hoisted deadline would have entered the central-2 attempt already expired and dropped the ack; this is the live proof of the fix in `667639e6` |
|
||
| 5 | Auth probes against central `:4055` `CentralCommandService/ReportApply` | **PASS** — 7/7 as designed: no bearer ⇒ `PermissionDenied`; SITE-B's key claiming SITE-A ⇒ `PermissionDenied`; no `x-otopcua-cluster` ⇒ `PermissionDenied`; unknown cluster `SITE-Z` with a valid key ⇒ `PermissionDenied`; unauthenticated against central-**2** ⇒ `PermissionDenied`; correct key+header with a **fake** deployment id ⇒ `{}` (auth passed, coordinator ignored it — **0 rows written** for that id); malformed id ⇒ `InvalidArgument` (auth passed, payload refused). Positive control: SITE-B key + SITE-B header also reaches the decoder |
|
||
| 6 | ConfigServe (`DeploymentArtifactService`) on the same listener unaffected | **PASS** — all four site nodes **Fetched + verified** a 28 296-byte artifact from `http://central-1:4055` on a real revision change, and a direct authenticated `Fetch` returned artifact bytes; unauthenticated `Fetch` is refused by its **own** interceptor (`Unauthenticated` / "Config-serve authentication failed."), never by the mesh one — which is the method-path scoping holding. **1** config-serve denial in the whole run: the deliberate probe |
|
||
|
||
### Provenance
|
||
|
||
`docker compose -f docker-dev/docker-compose.yml up -d --build` from the worktree
|
||
`/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1b`, **exit 0**; image `otopcua-host:dev`
|
||
rebuilt to `4105cf631003` at 12:51:47 UTC. **All six host nodes plus `migrator`,
|
||
`cluster-seed` and `traefik` were recreated** (both one-shots exit 0); `sql` and its volume
|
||
were kept. Nothing else touched the stack during the run, so it is not a
|
||
concurrent-rebuild-invalidated one. Four rig-only Serilog `MinimumLevel:Override` Debug scopes
|
||
were added on all six nodes for this run only — `ZB.MOM.WW.OtOpcUa.Runtime.Communication` (the
|
||
sender's `Posted the ApplyAck` line), `ZB.MOM.WW.OtOpcUa.AdminUI.Grpc` (central's accept line),
|
||
`ZB.MOM.WW.OtOpcUa.ControlPlane.Communication` and `Akka.Cluster.Tools.Client` — because the
|
||
rig's default Serilog level is Information and checks 1–3 would otherwise have had no signal.
|
||
Flip and overrides reverted after the gate, never committed.
|
||
|
||
```
|
||
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
|
||
central-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=2
|
||
central-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=2
|
||
site-a-1 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0
|
||
site-a-2 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0
|
||
site-b-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
|
||
site-b-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
|
||
|
||
$ curl … /health/ready → Healthy, members 2, unreachable 0 (MAIN)
|
||
$ … http://<site>:8080/health/ready → 200 on all four site nodes
|
||
```
|
||
|
||
### Checks 1 + 2 evidence — the two transports, side by side in one deployment
|
||
|
||
```
|
||
$ curl -X POST http://localhost:9200/api/deployments -H 'X-Api-Key: docker-dev-deploy-key' \
|
||
-d '{"createdBy":"1b-gate-check1"}'
|
||
{"outcome":"Accepted","deploymentId":"53bd2626-ce9f-4255-a964-24e4cf3c7ccb", …} HTTP 202
|
||
|
||
$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='53bd2626-…'
|
||
2 -- Sealed
|
||
$ … SELECT NodeId, Status, AppliedAtUtc FROM dbo.NodeDeploymentState WHERE DeploymentId='53bd2626-…'
|
||
central-1:4053|1|2026-08-06 12:55:53.909 site-a-1:4053|1|2026-08-06 12:55:53.930
|
||
central-2:4053|1|2026-08-06 12:55:53.890 site-a-2:4053|1|2026-08-06 12:55:53.936
|
||
site-b-1:4053|1|2026-08-06 12:55:53.915 site-b-2:4053|1|2026-08-06 12:55:53.924
|
||
|
||
# node side — the gRPC pair:
|
||
site-a-1 [12:55:53 DBG] Posted the ApplyAck for 53bd2626… (Applied) to http://central-1:4055.
|
||
site-a-2 [12:55:53 DBG] Posted the ApplyAck for 53bd2626… (Applied) to http://central-1:4055.
|
||
# central side — same two acks, named:
|
||
central-1 [12:55:53 DBG] ApplyAck for 53bd2626… from site-a-1:4053 (Applied) handed to the central communication actor
|
||
central-1 [12:55:53 DBG] ApplyAck for 53bd2626… from site-a-2:4053 (Applied) handed to the central communication actor
|
||
central-2 (none)
|
||
|
||
# per-node ReportApply accept counts, BOTH central nodes, this deployment:
|
||
central-1 site-a-1=1 site-a-2=1 site-b-1=0 site-b-2=0 central-1=0 central-2=0
|
||
central-2 site-a-1=0 site-a-2=0 site-b-1=0 site-b-2=0 central-1=0 central-2=0
|
||
```
|
||
|
||
The site-b half is proven by exclusion plus its live association, which is the same shape 1A
|
||
used for the site pairs:
|
||
|
||
```
|
||
$ for c in site-b-1 site-b-2; do docker logs … | grep -c "Posted the ApplyAck"; done → 0, 0
|
||
site-b-1 [12:53:02 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
|
||
site-b-2 [12:53:07 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
|
||
|
||
$ for c in site-a-1 site-a-2; do docker logs … | grep -c "Connected to .*receptionist"; done → 0, 0
|
||
$ for c in site-a-1 site-a-2 site-b-1 site-b-2; do grep -c "Mesh ClusterClient for cluster"; done → 0,0,0,0
|
||
```
|
||
|
||
Site-b rows are `Applied` while nothing on either central node logged a `ReportApply` for them,
|
||
and site-a holds no receptionist association at all — so neither pair could have used the
|
||
other's transport. Coexistence is per pair, in one deployment, from one dispatch.
|
||
|
||
### Check 3 evidence — one ack per dispatch, and the duplicate that proved the absorber
|
||
|
||
Across all ten dispatches, counting `ReportApply` accepts on **both** central nodes:
|
||
|
||
```
|
||
site-a-1 dep=53bd2626… acks=1 site-a-2 dep=53bd2626… acks=1
|
||
site-a-1 dep=d0869c9e… acks=1 site-a-2 dep=d0869c9e… acks=1
|
||
site-a-1 dep=17507001… acks=1 site-a-2 dep=17507001… acks=1 # the stop-failover dispatch
|
||
site-a-1 dep=b1e837a2… acks=1 site-a-2 dep=b1e837a2… acks=1
|
||
site-a-1 dep=a394eec6… acks=1 site-a-2 dep=a394eec6… acks=1
|
||
site-a-1 dep=9bb520a4… acks=1 site-a-2 dep=9bb520a4… acks=1
|
||
site-a-1 dep=458b8629… acks=2 site-a-2 dep=458b8629… acks=2 # the 4b PAUSE dispatch
|
||
site-a-1 dep=fc2b7b64… acks=1 site-a-2 dep=fc2b7b64… acks=1
|
||
site-a-1 dep=41e4935d… acks=1 site-a-2 dep=41e4935d… acks=1
|
||
site-a-1 dep=4126636e… acks=1 site-a-2 dep=4126636e… acks=1
|
||
$ … SELECT NodeId, COUNT(*) FROM dbo.NodeDeploymentState WHERE DeploymentId='53bd2626-…' GROUP BY NodeId
|
||
central-1:4053|1 central-2:4053|1 site-a-1:4053|1 site-a-2:4053|1 site-b-1:4053|1 site-b-2:4053|1
|
||
```
|
||
|
||
A refused endpoint (`docker stop`) cannot produce a duplicate — the server never saw the call —
|
||
which is why `17507001…` is 1 per node despite failing over. The pause drill is the one shape
|
||
that can, and it did:
|
||
|
||
```
|
||
central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed … # failover target
|
||
central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed …
|
||
central-1 [13:04:33 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed … # at `docker unpause`
|
||
central-1 [13:04:33 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed …
|
||
central-1 [13:05:03 INF] Coordinator recovered in-flight deployment 458b8629… (2/6 acks landed)
|
||
```
|
||
|
||
`2/6`, not `4/6`. The frozen central had in fact received both acks and replayed them on
|
||
unpause, and `ConfigPublishCoordinator.HandleAck`'s NodeId-keyed absorption made the duplicate
|
||
free — the exact claim `GrpcApplyAckSender`'s "at-least-once walk" remark makes, observed
|
||
rather than argued.
|
||
|
||
### Check 4 evidence — refusal failover
|
||
|
||
```
|
||
$ docker compose stop central-1 # 12:57:11
|
||
$ curl -X POST … -d '{"createdBy":"1b-gate-check4-failover"}' # 12:57:30
|
||
site-a-1 [12:57:30 INF] Posted the ApplyAck for 17507001… (Applied) to http://central-2:4055
|
||
after 1 endpoint(s) failed (http://central-1:4055 => Unavailable).
|
||
site-a-2 [12:57:30 INF] …identical…
|
||
central-2 [12:57:30 DBG] ApplyAck for 17507001… from site-a-1:4053 (Applied) handed …
|
||
central-2 [12:57:30 DBG] ApplyAck for 17507001… from site-a-2:4053 (Applied) handed …
|
||
|
||
$ … SELECT NodeId, Status FROM dbo.NodeDeploymentState WHERE DeploymentId='17507001-…'
|
||
central-1:4053|0 central-2:4053|1 site-a-1:4053|1 site-a-2:4053|1
|
||
site-b-1:4053|1 site-b-2:4053|1
|
||
$ … SELECT Status FROM dbo.Deployment … → 4 (TimedOut) after the 2-minute deadline
|
||
|
||
$ docker compose start central-1 # 13:00:10
|
||
$ curl … /health/ready → Healthy, members 2, unreachable 0
|
||
$ curl -X POST … -d '{"createdBy":"1b-gate-check4-converge"}' → Status=2, all six Status=1
|
||
```
|
||
|
||
**One operational detail worth carrying.** After central-1 returned, site-a-**1**'s cached
|
||
`GrpcChannel` to `http://central-1:4055` stayed in transient failure for roughly two minutes
|
||
while site-a-**2** reached the same restarted endpoint immediately — three dispatches apart:
|
||
|
||
```
|
||
13:01:25 site-a-1 → central-2 after (central-1 => Unavailable) | site-a-2 → central-1 OK
|
||
13:01:48 site-a-1 → central-2 after (central-1 => Unavailable) | site-a-2 → central-1 OK
|
||
13:02:49 site-a-1 → central-1 OK | site-a-2 → central-1 OK
|
||
$ docker exec otopcua-dev-site-a-1-1 bash -c '</dev/tcp/central-1/4055 && echo OPEN' → OPEN
|
||
$ central-1 mesh denials in that window → 0 # the call never reached the interceptor
|
||
```
|
||
|
||
TCP was open and central-1 logged nothing, so this is the client channel's own reconnect
|
||
backoff, not a server or auth problem. **No ack was lost** — the ordered walk absorbed it every
|
||
time and said so in the log. The Phase 2 reading: a central restart shifts that node's ack load
|
||
onto the surviving endpoint for a minute or two afterwards, which a soak's failover-line count
|
||
will show as a burst rather than a fault.
|
||
|
||
### Check 4b evidence — the hang, and why the fresh deadline is the whole check
|
||
|
||
```
|
||
$ docker pause otopcua-dev-central-1-1 # 13:03:35
|
||
$ curl … /health/ready → self akka.tcp://otopcua@central-2:4053 # traefik dropped central-1
|
||
$ curl -X POST … -d '{"createdBy":"1b-gate-check4b-hang"}' # 13:03:51 (202)
|
||
site-a-1 [13:04:01 INF] Posted the ApplyAck for 458b8629… (Applied) to http://central-2:4055
|
||
after 1 endpoint(s) failed (http://central-1:4055 => DeadlineExceeded).
|
||
site-a-2 [13:04:01 INF] …identical…
|
||
central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed …
|
||
central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed …
|
||
$ … SELECT NodeId, Status, AppliedAtUtc … WHERE DeploymentId='458b8629-…'
|
||
site-a-1:4053|1|2026-08-06 13:04:01.837 site-a-2:4053|1|2026-08-06 13:04:01.774
|
||
central-1:4053|0|NULL central-2:4053|0|NULL site-b-1:4053|0|NULL site-b-2:4053|0|NULL
|
||
$ docker unpause otopcua-dev-central-1-1 # 13:04:33
|
||
```
|
||
|
||
`DeadlineExceeded` — not `Unavailable` — is the discriminator: the first endpoint burned a full
|
||
`AckTimeoutSeconds` (10 s, the default; `13:03:51 → 13:04:01`) and central-2 was **still**
|
||
dialled with a live deadline. Under the hoisted deadline this branch shipped before
|
||
`667639e6`, that second attempt would have started already expired and the ack would have been
|
||
dropped with a Warning. Observed failover latency: **≈10 s, one whole `AckTimeoutSeconds`**.
|
||
|
||
**Collateral the drill produced, recorded because it is the honest picture of a `pause`.**
|
||
Pausing a cluster member is a heavier disturbance than stopping one: central-2 marked central-1
|
||
UNREACHABLE at 13:03:46 and auto-downed it at 13:04:01, and after `unpause` the two central
|
||
nodes each held a one-member view until central-1 was restarted (13:06:36). In that window the
|
||
**two gRPC acks landed and none of the four ClusterClient acks did** — central-2 and both
|
||
site-b nodes all logged `received DispatchDeployment` at 13:03:51 yet produced no ack the
|
||
coordinator saw, and the deployment ended `TimedOut` at `2/6` naming exactly those four:
|
||
|
||
```
|
||
central-1 [13:05:51 WRN] Deployment 458b8629… timed out after 00:02:00 (2/6 acks landed).
|
||
No ack from: central-1:4053, central-2:4053, site-b-1:4053, site-b-2:4053
|
||
central-1 [13:06:36 WRN] DriverHost central-1:4053: found orphan Applying row for deployment
|
||
458b8629-…; replaying
|
||
central-1 [13:06:37 INF] DriverHost central-1:4053: applied deployment 458b8629… (children=10)
|
||
```
|
||
|
||
Stated at the strength the evidence supports: the four ClusterClient nodes received the
|
||
dispatch and none of their acks reached the coordinator, while both gRPC nodes' acks did. The
|
||
site nodes emit no apply-completion line, so whether each of the four actually finished
|
||
applying is not determinable from their logs — what is determinable is that the ack did not
|
||
arrive and the deadline said so honestly. The boot-replay backstop then repaired central-1.
|
||
Not a 1B finding either way (no gRPC ack was involved), but it is the Phase 2 auto-down
|
||
behaviour showing up early, and the split view needed a manual `restart central-1` to heal.
|
||
|
||
### Check 5 evidence — the auth matrix on the ack endpoint
|
||
|
||
`-import-path` onto the in-repo proto (the node serves no reflection). `PermissionDenied` means
|
||
the interceptor refused; anything past it (`{}` or `InvalidArgument`) means auth **passed** and
|
||
only the payload was graded:
|
||
|
||
```
|
||
$ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos
|
||
$ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \
|
||
-plaintext -import-path /protos -proto mesh_transport.proto"
|
||
$ BODY='{"deploymentId":"ffffffffffffffffffffffffffffffff","nodeId":"site-a-1:4053",
|
||
"outcome":"APPLY_OUTCOME_APPLIED","correlationId":"eeeeeeee…"}'
|
||
|
||
# 5a no bearer, cluster header present -> central-1
|
||
Code: PermissionDenied Message: Central command authentication failed.
|
||
# 5b Bearer dev-mesh-psk-SITE-B + cluster SITE-A -> central-1
|
||
Code: PermissionDenied Message: Central command authentication failed.
|
||
# 5c Bearer dev-mesh-psk-SITE-A, NO cluster header-> central-1
|
||
Code: PermissionDenied Message: Central command authentication failed.
|
||
# 5d Bearer dev-mesh-psk-SITE-A + cluster SITE-A, FAKE deployment id
|
||
{}
|
||
# 5e same, MALFORMED deployment id "not-a-guid"
|
||
Code: InvalidArgument Message: undecodable ApplyAck
|
||
# 5f Bearer dev-mesh-psk-SITE-B + cluster SITE-B (positive control)
|
||
Code: InvalidArgument Message: undecodable ApplyAck
|
||
# 5g Bearer dev-mesh-psk-SITE-A + cluster SITE-Z (unknown cluster)
|
||
Code: PermissionDenied Message: Central command authentication failed.
|
||
# 5h no bearer -> central-2
|
||
Code: PermissionDenied Message: Central command authentication failed.
|
||
```
|
||
|
||
The accepted fake ack changed nothing — the coordinator ignored a deployment it is not tracking:
|
||
|
||
```
|
||
$ … SELECT COUNT(*) FROM dbo.NodeDeploymentState WHERE DeploymentId='ffffffff-…' → 0
|
||
$ … SELECT COUNT(*) FROM dbo.Deployment WHERE DeploymentId='ffffffff-…' → 0
|
||
central-1 [13:08:08 DBG] ApplyAck for ffffffff… from site-a-1:4053 (Applied) handed to the
|
||
central communication actor
|
||
```
|
||
|
||
Central's denial log separates the three operator-facing causes even though the wire status is
|
||
deliberately uniform:
|
||
|
||
```
|
||
[13:08:07 WRN] Rejected a central command call to /mesh_transport.v1.CentralCommandService/ReportApply
|
||
claiming cluster SITE-A: no bearer token presented.
|
||
[13:08:07 WRN] … claiming cluster SITE-A: the presented bearer token did not match that cluster's
|
||
pre-shared key, no key could be established for it, or it is not a cluster this node knows…
|
||
[13:08:07 WRN] Rejected a central command call to …/ReportApply: no x-otopcua-cluster header, so there
|
||
is no cluster whose pre-shared key the caller could be graded against.
|
||
[13:08:25 WRN] … claiming cluster SITE-Z: … or it is not a cluster this node knows…
|
||
```
|
||
|
||
5g is the **known-cluster gate answering live**: a valid SITE-A key presented for `SITE-Z` is
|
||
refused in O(1), which also confirms the gate had in fact been published — see observation (b).
|
||
|
||
### Check 6 evidence — ConfigServe on the same listener
|
||
|
||
A real node-driven fetch needed a real revision change, so `ServerCluster.Notes` for `MAIN` was
|
||
edited, deployed, and **restored** (the revision hash returned to its pre-gate
|
||
`642b9d6f79b9…`, verified by a further dispatch):
|
||
|
||
```
|
||
$ … UPDATE dbo.ServerCluster SET Notes = '<original> | 1b-gate-fetch-probe' WHERE ClusterId='MAIN'
|
||
$ curl -X POST … → revisionHash 20af950fb211… (was 642b9d6f79b9…) Status=2
|
||
site-a-1 [13:09:31 INF] Fetched + verified artifact for 41e4935d… from http://central-1:4055 (28296 bytes).
|
||
site-a-2 [13:09:31 INF] …same… site-b-1 [13:09:32 INF] …same… site-b-2 [13:09:32 INF] …same…
|
||
$ … UPDATE dbo.ServerCluster SET Notes = '<original>' …
|
||
$ curl -X POST … → revisionHash 642b9d6f79b9… Status=2 # restored, byte-identical
|
||
```
|
||
|
||
Both interceptors on the one listener, each answering for its own service:
|
||
|
||
```
|
||
# authenticated Fetch, config-serve key:
|
||
$ $G -H 'authorization: Bearer configserve-docker-dev-key' -d '{"deploymentId":"FC2B7B64-…"}' \
|
||
central-1:4055 deployment_artifact.v1.DeploymentArtifactService/Fetch
|
||
{ "data": "eyJDbHVzdGVycyI6W3siQ2x1c3RlcklkIjoiTUFJTiIsIk5hbWUiOiJNYWluIGNsdXN0ZXIiL…" }
|
||
# unauthenticated Fetch — refused by the CONFIG-SERVE interceptor, not the mesh one:
|
||
Code: Unauthenticated Message: Config-serve authentication failed.
|
||
```
|
||
|
||
`Unauthenticated` here versus `PermissionDenied` on the mesh service is the method-path scoping
|
||
in `CentralCommandAuthInterceptor` holding: the two services share one `AddGrpc` pipeline and
|
||
neither interceptor grades the other's calls.
|
||
|
||
### Observations carried forward from the reviews
|
||
|
||
**(a) The I-4 shape is not exercisable on this rig, and that is a property of the rig, not a
|
||
skipped check.** `MeshPskProvider.TryGetPsk` consults `MeshTransport:ClusterPsks` **before** the
|
||
secret store and caches only store-resolved values, and docker-dev supplies
|
||
`MeshTransport__ClusterPsks__{MAIN,SITE-A,SITE-B}` on both central nodes. The
|
||
`InvalidateAll()` a contact-refresh tick performs (`ContactRefreshSeconds` = 60) therefore drops
|
||
an always-empty cache, and an ack arriving just after a tick pays no store read. No latency
|
||
delta was measurable and none should be: every ack in the run landed inside the same second it
|
||
was posted, refresh tick or not. A real deployment resolving `OT-MESH-PSK-<clusterId>` from the
|
||
store is where I-4 becomes observable — Phase 2 on a store-backed rig, not here.
|
||
|
||
**(b) The `Failed to resolve` Error fired on 4 of 4 admin boots — and healed every time.** Worse
|
||
odds than 1A's 2 of 3, and both dependencies lost the race on every boot:
|
||
|
||
```
|
||
central-1 [12:53:00 ERR] Failed to resolve IMeshCommandDialer … / IMeshPskProvider … # boot 1
|
||
central-1 [13:00:29 ERR] …both again… # boot 2 (check 4 restart)
|
||
central-1 [13:06:36 ERR] …both again… # boot 3 (4b restart)
|
||
central-2 [12:53:02 ERR] …both… # its single boot
|
||
central-1 [12:53:00 ERR] No IMeshPskProvider was supplied to this actor; the known-cluster gate
|
||
will NEVER be published and cached per-cluster keys will NEVER be dropped…
|
||
```
|
||
|
||
The second line is the one that reads worst, and 1B disproves it directly rather than by
|
||
argument: **every site-a ack central-1 accepted required that provider**, including the ones
|
||
after its third boot (13:07:36, 13:09:31, 13:09:59), and probe 5g's `SITE-Z` refusal is only
|
||
possible if `SetKnownClusters` had in fact been published. The memoize heals it; the Error text
|
||
does not say so. This is the 1A finding-5 carry-forward, now with a second, independent live
|
||
confirmation and a stronger case for the wording fix — the message contains the word "NEVER"
|
||
twice about something that self-corrects within one deploy.
|
||
|
||
**(c) Zero mesh interceptor denials outside the deliberate probes.** Whole-run counts on every
|
||
node and every interceptor:
|
||
|
||
```
|
||
central-1 telemetry=0 configserve=1 localdbsync=0 meshCentralCmd=4 nodeCmd=0
|
||
central-2 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=1 nodeCmd=0
|
||
site-a-1 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0
|
||
site-a-2 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0
|
||
site-b-1 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0
|
||
site-b-2 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0
|
||
```
|
||
|
||
`meshCentralCmd` = 4 on central-1 (probes 5a/5b/5c/5g) + 1 on central-2 (5h) = exactly the five
|
||
deliberate negatives; `configserve` = 1 = the deliberate 6b. Nothing else was refused anywhere.
|
||
|
||
**(d) Pre-existing rig noise, so a Phase 2 error grep is not surprised by it.** Site nodes log
|
||
`Unavailable … Name or service not known` from the historian client — the rig points
|
||
`ServerHistorian__Endpoint` at `http://histgw-absent.invalid:5222` on purpose (45 occurrences on
|
||
site-a-1, 1 on site-b-1, 4 on central-1 in this run). Nothing to do with the ack plane.
|
||
|
||
### Rig state after the gate — **restored, six healthy, all-ClusterClient, on the branch image**
|
||
|
||
The flip and the four Serilog overrides were reverted (`git checkout docker-dev/docker-compose.yml`,
|
||
`grep -c "GATE-1B ONLY"` → 0) and the nodes recreated on the committed compose. The simultaneous
|
||
recreate cold-started both central nodes at once and they self-formed one-member views — the
|
||
known simultaneous-cold-start split-brain the (default-OFF) `Cluster:BootstrapGuard` addresses —
|
||
healed with a single `restart central-2`:
|
||
|
||
```
|
||
central-1 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
|
||
central-2 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
|
||
site-a-1 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
|
||
site-a-2 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
|
||
site-b-1 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
|
||
site-b-2 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0
|
||
|
||
$ for h in central-1 central-2; do … /health/ready; done
|
||
central-1 Healthy members 2 unreachable 0 leader akka.tcp://otopcua@central-1:4053
|
||
central-2 Healthy members 2 unreachable 0 leader akka.tcp://otopcua@central-1:4053
|
||
$ … http://<site>:8080/health/ready → 200 ×4
|
||
|
||
$ curl -X POST … -d '{"createdBy":"1b-gate-rig-restore-verify"}'
|
||
CDDA1D80-…|2 central-1|1 central-2|1 site-a-1|1 site-a-2|1 site-b-1|1 site-b-2|1
|
||
$ for c in site-a-1 site-a-2 site-b-1 site-b-2; do grep -c "Posted the ApplyAck"; done → 0,0,0,0
|
||
$ for c in central-1 central-2; do grep -c "Mesh ClusterClient for cluster"; done → 3 per boot
|
||
```
|
||
|
||
Zero gRPC acks after the rollback and three ClusterClients per central boot: the dark switch
|
||
rolls back cleanly in the ack direction too, which is what Phase 2 depends on if its cutover has
|
||
to be reversed. Rig configuration is byte-identical to its pre-gate state (`ServerCluster.Notes`
|
||
restored; revision hash back to `642b9d6f79b9…`).
|
||
|
||
### Phase 1B defects
|
||
|
||
**None.** No product code was changed during or after this gate.
|
||
|
||
---
|
||
|
||
## Phase 2 — full-rig cutover + soak — **PASS 8/8 + baseline + PSK negatives**
|
||
|
||
Rig default flipped and committed: `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"`
|
||
(`caf2eec6`, all six node entries). Rig rebuilt from `master @ caf2eec6` — which also carries
|
||
Task 2.0 (`a94eb8a3`+`d7c9ead6`, honest PreStart-race logging, review-approved) and the DI
|
||
role-scoping pin (`2e06dfe5`+`027f0403`, review-approved) — image rebuilt, node containers
|
||
recreated, no `OTOPCUA_MESH_MODE` override in the environment. Baseline was run on the
|
||
pre-flip build first (see Baseline section, `1bb3a937`).
|
||
|
||
| # | Check | Result |
|
||
|---|---|---|
|
||
| 1 | Full rebuild all-`Grpc`: six nodes ready, deploy e2e seals, all acks via gRPC | **PASS** — deploy `949d2cc7` sealed (Status 2) 6/6 within ~1 s; every node logged `accepted DispatchDeployment … over gRPC` and `Posted the ApplyAck … to http://central-1:4055`, central logged each `ApplyAck … handed to the central communication actor`; wire-level: central holds established TCP to every node's `:4056`, node `TIME_WAIT` to central `:4055` observed. An earlier identical deploy (`8809464a`, pre-Debug-override) also sealed 6/6 in ~150 ms |
|
||
| 2 | Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes | **PASS** — via authenticated grpcurl envelopes (the 1A shape): Restart `SITEA-modbus` → site-a-1 `{}` + `restarting driver … by request of p2-gate` + `spawned` + `connected`; Reconnect + AlarmCommand accepted + handed (correlation ids `c2b1…`/`c2c1…`); site-b-1 delivery proven (`c2d1…` accepted over gRPC). AdminUI origin composed per the 1A carry-forward (no UI control exists; same `DispatchOverGrpc` path driven by check 1) |
|
||
| 3 | Kill site-a-1 mid-dispatch: per-node `Execute` failure **logged with node id + status** (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path) | **PASS** — deploy `069b57a4` 1 s after `docker kill`: central-2 WRN `DispatchDeployment … was NOT delivered to site-a-1:4053 (Unavailable: Error connecting to subchannel.). It is dropped and will NOT be retried…` — exactly **1** such line (no retry storm); 5/6 acked; `timed out after 00:02:00 (5/6 acks landed). No ack from: site-a-1:4053` at exactly +2:00; restart → FetchAndCache boot-replay restored last-known-good, timed-out deployment NOT retro-acked (ack count stayed 5) |
|
||
| 4 | Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action | **PASS** — deploy `ffc75eae` with central-1 dead: site nodes logged `Posted the ApplyAck … to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable)` (the ordered walk, per node), all live acks landed on central-2; deadline honest at +2:00 naming `central-1, site-a-1` (site-a-1 = the documented ~2-min `GrpcChannel` backoff carry-forward from its own earlier kill — loud, single-attempt). First post-restart probe caught central-1 still booting (5/6, honest TimedOut); second probe `594ecda3` **sealed 6/6** — full convergence, no manual action beyond `docker start` |
|
||
| 5 | Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; **no duplicate applies** after the pair returns (boot-replay path only) | **PASS** — deploy `e5a14c5f`: BOTH `NOT delivered to site-b-{1,2}:4053 (Unavailable…)` WRN lines in the SAME SECOND as dispatch (baseline (b): zero correlated lines, sampled-deadLetters INF); `timed out after 00:02:00 (4/6…). No ack from: site-b-1:4053, site-b-2:4053`; pair restart → boot-replay only (`FetchAndCache boot — restored served state`), rows stay 0/0, ack count stays 4 — no dupes, no resurrection |
|
||
| 6 | **Zero ClusterClient activity under `Grpc`:** `docker logs <every node> \| grep -ci "clusterclient"` = 0 new lines post-boot | **PASS** — final sweep after the full soak: `grep -ic "clusterclient\|receptionist"` = **0 on all six nodes**, covering checks 1–5, every kill/restart cycle, and the whole 135-min soak |
|
||
| 7 | Soak ≥ 2 h under periodic deploys + command traffic: 0 `PermissionDenied`/`Unauthenticated` across all six nodes (PSK drift detector), 0 unexplained `Unavailable` | **PASS** — 135 min (15:03:44Z→17:18Z), **14/14 deploy cycles sealed 6/6** (most in 0–1 s) + Restart/AlarmCommand traffic every cycle, 0 traffic failures. Grep since soak start: **PermissionDenied=0, Unauthenticated=0 on all six** (PSK drift detector clean). Review items, both explained: (a) site-a-2 `Unavailable`×3807 — ALL at DBG/INF (its WRN/ERR count was 0), the alarm-historian `SendEvent` retry loop failing DNS on the HistorianGateway host, which does not exist on this rig — pre-existing rig-shape noise, not mesh; (b) central-2 WRN×28 = exactly 14 deploys × 2 known per-deploy StartDeployment authoring warnings (`TAG-gatelong` ModbusRegion + script-compile RSS advisory) — pre-existing on the old transport. The predicted post-central-restart failover-line burst did not occur during the soak because no node was killed inside the soak window (kills all preceded it) |
|
||
| 8 | Full suite green on the cutover SHA | **PASS** — `dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo` exit 0, **zero failures across every assembly**, run in parallel with the live rig + soak traffic. Spot totals match every baseline: ControlPlane 245 (Task 2.0 set), Host 95 (role-scoping pins), Runtime 558+3skip, AdminUI 1109, Commons 346, Cluster 193, Host.IntegrationTests 212+4skip |
|
||
|
||
**Extra (plan-text requirement): PSK negatives on RUNNING containers, both services** — **PASS**:
|
||
`NodeCommandService/Execute` @ site-a-1: no bearer → `PermissionDenied: Node command
|
||
authentication failed.`; SITE-B key → same uniform denial; SITE-A key + empty envelope
|
||
(positive control) → `InvalidArgument: Command envelope could not be decoded.` (interceptor
|
||
ran, payload rejected — auth and decode are distinct). `CentralCommandService/ReportApply`
|
||
@ central-1:4055: no bearer / wrong key / SITE-A key claiming `x-otopcua-cluster: SITE-Z` →
|
||
all `PermissionDenied: Central command authentication failed.` (uniform text, unknown-cluster
|
||
claim refused).
|
||
|
||
**Gate observability note:** after check 1's first pass, the six node containers were
|
||
recreated once with an UNCOMMITTED scratchpad compose override raising Serilog default to
|
||
Debug (the mesh delivery lines are DBG; the base rig logs at Information and the 1A/1B gates
|
||
had used the same temporary raise). Same image, same SHA — no rebuild, so no void-run. All
|
||
kill drills ran after that recreation.
|
||
|
||
---
|
||
|
||
## Phase 4 — terminal gate on the post-deletion build — **PASS 8/8 — PROGRAM CLOSED**
|
||
|
||
Run 2026-08-06 (log times UTC, 23:53–00:27 spanning midnight into 08-07). `master @ 0b194003`
|
||
(= merge `9420f8ef` + one docs-only commit; the Phase 3 deletion merge is `2fa880b6`).
|
||
**Provenance guard PASS:** image `otopcua-host:dev` = `d2a290fc5711`, built 2026-08-06
|
||
17:55:10 EDT — later than both the deletion merge (16:57:05 EDT) and the last code merge
|
||
`9420f8ef` (17:28:48 EDT), so the boundary checks ran on the post-deletion binary. No rebuild
|
||
during the gate. The six node containers were recreated twice before any drill (once for the
|
||
gate's Debug scopes, once to widen them to the ack plane — the "Posted the ApplyAck" sender
|
||
logs in `Runtime.Communication` and the central receiver in `AdminUI.Grpc`, neither covered
|
||
by the 1A/2 scope set); same image both times, so no void-run. Check 1 ran before the second
|
||
recreation (interceptor probes — log-reset does not affect them); checks 2–7 after it, so
|
||
the final greps cover every drill. Override reverted after the gate, never committed.
|
||
|
||
| # | Check | Result |
|
||
|---|---|---|
|
||
| 1 | PSK negatives on both services (no key / wrong key / wrong cluster's key); **unset server key ⇒ all calls rejected** (fail-closed proven live) | **PASS** — 7/7: `Execute` @ site-a-1 no-bearer → `PermissionDenied: Node command authentication failed.`, SITE-B key → same uniform text, SITE-A key + empty envelope → `InvalidArgument` (interceptor ran, payload rejected); `ReportApply` @ central-1 no-bearer / garbage key / SITE-A key claiming `x-otopcua-cluster: SITE-Z` → all `PermissionDenied: Central command authentication failed.` (uniform, unknown-cluster refused = no key configured for that cluster ⇒ every call rejected), SITE-A key + SITE-A claim → `InvalidArgument: undecodable ApplyAck` (positive control). Unset-key fail-closed at the earliest gate: scratch container on the rig image, `Grpc` mode, `GrpcPsk` removed → `Hosting failed to start`, `OptionsValidationException` naming `MeshTransport:GrpcPsk` + the env form (scratch removed after capture) |
|
||
| 2 | Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand) | **PASS** — deploy `22dfec4b` fanned as ONE envelope: all six nodes logged `accepted DispatchDeployment (correlationId=c532b8f7…)` in the same second. **Restart + Reconnect driven from the real `/hosts` AdminUI panel for the first time in any gate** (the #546 wire-in closed the 1A "structurally undrivable" carry-forward): each fanned to all six under a single correlation id (`41157cfb…` / `cb81bc5e…`), and BOTH SITE-A nodes acted (`restarting driver SITEA-modbus by request of multi-role-test` + `spawned`, `ForceReconnect requested by admin`) — actor = the logged-in operator, i.e. the full `AdminOperationsActor.Dispatch → DispatchOverGrpc` origin path live. AlarmCommand: authenticated envelopes accepted + handed on site-b-1 and central-1 |
|
||
| 3 | Ack matrix: every node acks exactly once, deployment seals | **PASS** — for `22dfec4b`: every node logged exactly ONE `Posted the ApplyAck … (Applied) to http://central-1:4055`, central-1 served exactly 6 `ReportApply` 200s and handed 6 acks to the central communication actor, `Deployment.Status=2` with 6/6 `NodeDeploymentState` Applied inside a ~30 ms spread |
|
||
| 4 | Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry) | **PASS** — kill central-1 → deploy `3edf043d`: all five live nodes posted `to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable)` (the ordered walk, one loud INF per node), deadline honest at exactly +2:00 naming only central-1 (5/6). Restart → first re-deploy hit the documented ~2-min `GrpcChannel` backoff (ONE `was NOT delivered … will NOT be retried` WRN, honest TimedOut), next dispatch **sealed 6/6** — no manual action beyond `docker start`. Kill site-a-1 + dispatch in the same second → exactly **1** `NOT delivered to site-a-1 (Unavailable…)` line (no retry storm), 5/6 acks, deadline at exactly +2:00 naming only site-a-1; restart → `FetchAndCache boot — restored served state` and the timed-out deployment NOT retro-acked (count stayed 5) |
|
||
| 5 | Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart) | **PASS** — deploy `f311cf5b` dispatched and the coordinator host (central-2) killed **in the same second** with the deployment open; **central-1's recovered coordinator fired the deadline at exactly +2:00** (`timed out … (5/6 acks landed). No ack from: site-b-2:4053` — the deliberately-stopped node), with central-2's own ack — posted before its death — preserved in the 5. After restarting both: per-node `Posted the ApplyAck` count for `f311cf5b` exactly 1 on the five that acked / 0 on site-b-2, DB count stayed 5 (no retro-ack, no dupes), site-b-2 boot-replayed last-known-good, and reconverge deploy `12a46546` **sealed 6/6** |
|
||
| 6 | **Boundary check: no Akka association between any site container and central** | **PASS** — `netstat` in every node's netns: exactly THREE established `:4053` associations rig-wide, each strictly pair-internal (central-1↔central-2, site-a-1↔site-a-2, site-b-1↔site-b-2 by container IP); **zero site↔central `:4053` connections**. Baseline (a)'s unauthenticated cross-mesh association is gone at the TCP level |
|
||
| 7 | Full-rig restart (pairs together): clean convergence; `grep -ri "receptionist\|clusterclient"` over all node logs = 0 lines | **PASS, with one non-mesh observation** — all six restarted in one `compose restart`; both SITE pairs re-formed cleanly within 7 s; the CENTRAL pair hit the **documented simultaneous-cold-start split-brain** (same-second mutual `InitJoinNack` → both self-formed — the exact scenario the Phase 7 bootstrap guard `d1dac87f` ships for, opt-in and default-OFF, not enabled on this rig; pre-existing pair-formation behavior, zero mesh-transport involvement) and was healed by the standard operator action, a single-node restart of central-2, which rejoined as Secondary (`memberCount=2 unreachableCount=0`, leader central-1 — the Phase 7 rejoin drill behavior holding). Transport verdict clean both ways: `grep -icE "receptionist|clusterclient"` = **0 on all six nodes** over logs covering every drill, and post-restart deploy `11007721` **sealed 6/6**. Whole-gate PSK-drift sweep: **zero** denials on all four interceptors across all six nodes for the entire post-recreation run |
|
||
| 8 | `MeshGrpcBoundaryTests` + full suite green on the merge SHA | **PASS** — targeted `MeshGrpcBoundaryTests` 3/3; terminal full suite on `master @ 0b194003`: `dotnet test` direct exit-code capture (no pipe) **RC=0, 50/50 suites, 9,333 passed / 0 failed** — identical totals to the Phase-3 DoD baseline, run in parallel with the live rig per the Phase 2 precedent |
|
||
|
||
### Phase 4 carry-forward closures
|
||
|
||
| Item | Disposition |
|
||
|---|---|
|
||
| 1A check-3 AdminUI origin (closed by composition in 1A/2) | **Now closed by direct evidence** — #546 wired `DriverStatusPanel` into `/hosts`, and this gate drove Restart + Reconnect from the real page (fan + act, operator identity carried) |
|
||
| `Failed to resolve` Error-text cosmetics (1A finding 5 → Phase 2) | **Closed in Phase 2** Task 2.0 (`a94eb8a3`+`d7c9ead6`, honest PreStart-race logging) |
|
||
| ~2-min `GrpcChannel` backoff after a peer dies (1B → Phase 2 soak note) | **Behavior confirmed three more times this gate** (post-central-1-restart re-deploy; central-2→site-a-1 dispatch skip at 00:10:21 after the drill-B kill). Always loud (one WRN, single attempt), never lossy beyond the honest deadline, always healed by the next dispatch after the window. Stands as an operator note in `docs/Redundancy.md` territory — expect one TimedOut deploy within ~2 min of any node kill/restart |
|
||
| `GrpcPort` NULL→repair skip persists ≤ `ContactRefreshSeconds` (1A check 7) | Operational note stands (dispatch set rebuilds on the refresh tick); no Phase 4 re-check needed — the mechanism is unchanged since 1A proved it live |
|
||
| I-4 cold-PSK-cache latency after a refresh tick (1B → Phase 2 → Phase 4) | **Re-carried out of the program with rationale, unchanged:** still structurally unmeasurable on docker-dev — the rig configures `MeshTransport:ClusterPsks`, which answers ahead of the store, so the cache never holds a store-resolved value and `InvalidateAll()` clears nothing. Every ack in every phase landed within the second it was posted. To be measured once on the first store-backed deployment (a real `${secret:OT-MESH-PSK-…}` fleet); until then it is an unquantified-latency note, not a correctness gap — the fail-closed behavior on a cold cache is pinned by unit tests |
|
||
|
||
No new defects. The central-pair cold-start split-brain in check 7 is pre-existing, documented
|
||
(Phase 7 bootstrap guard, deliberately opt-in), reproduced here only because a same-second
|
||
pair restart is exactly its trigger — recorded as the reason an operator restarting a central
|
||
pair together should either stagger the two nodes or enable `Cluster:BootstrapGuard`.
|
||
|
||
**Program CLOSED.** Both baseline gaps are shut on the wire: (a) zero cross-mesh Akka
|
||
associations and zero receptionist/ClusterClient log lines on the post-deletion binary;
|
||
(b) an unreachable node now costs one loud per-node WRN in the dispatch second plus an honest
|
||
deadline, instead of a sampled dead-letter counter and silence.
|
||
|
||
### Carry-forwards
|
||
|
||
Anything a phase gate could not drive gets named here with why + what proves it instead, and
|
||
is re-checked in the next gate (the ScadaBridge proportionate-gate rule).
|
||
|
||
| Item | Deferred from | Why it could not be driven | Proven instead by | Status |
|
||
|---|---|---|---|---|
|
||
| Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-`GrpcPort` skip | 1A run 1 | Both central nodes failed to boot (defect 4) | — | **CLOSED in 1A run 2** — all three PASS live |
|
||
| Check 4 acknowledge half — the condition actually acknowledging | 1A run 1 | The rig's one `ScriptedAlarm` sits Normal and its owning central node was down | — | **CLOSED in 1A run 2** — acked from the real `/alerts` page, `ACKNOWLEDGED` row observed |
|
||
| Check 8 for telemetry + config-serve | 1A run 1 | Central dials both planes and never booted, so neither carried a call — 0 denials was vacuous | — | **CLOSED in 1A run 2** — both planes carried authenticated traffic with 0 denials |
|
||
| Check 3 central half — `RestartDriver`/`ReconnectDriver` **originating at the AdminUI** | 1A runs 1 + 2 | **Structurally undrivable, not a rig problem:** `<DriverStatusPanel` has 0 usages repo-wide (the 5 name hits are comments), no rendered page carries a Restart/Reconnect control, and the only non-script-analysis HTTP POST is `/api/deployments`. Pre-existing on `master`, not caused by this program | Composition the controller accepted: run 1's live node-side leg for both commands + run 2's check 4, which drives the identical `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path from the AdminUI for another command type; per-type difference is `ToEnvelope`, covered by the T0.2 goldens | **CLOSED by composition.** The orphaned panel itself is out of this program's scope — file separately if the control is meant to exist |
|
||
| `Failed to resolve` Error text says "continuing without it", which reads permanent although first use retries; and it will trip a soak's error grep on a healthy node | 1A run 2 (finding 5) | Cosmetic/observability, not behaviour | The self-heal proven live (finding 5), and **re-confirmed independently in 1B**: 4 of 4 admin boots lost the race, yet every site-a ack central-1 accepted and probe 5g's `SITE-Z` refusal both require the provider it claimed would "NEVER" be published | Phase 2 |
|
||
| I-4 cold-PSK-cache latency after a contact-refresh tick | 1B | **Not exercisable on docker-dev, structurally:** `TryGetPsk` reads `MeshTransport:ClusterPsks` ahead of the store and caches only store-resolved values, and the rig configures that map — so `InvalidateAll()` clears an always-empty cache and no ack pays a store read | Every ack in the run landed inside the second it was posted, tick or no tick (1B observation (a)) | Phase 2, on a store-backed rig |
|
||
| A restarted central's cached `GrpcChannel` can stay in transient failure for ~2 min on the node that saw it die, while a sibling node reaches the same endpoint at once | 1B check 4 | Not a defect — the ordered walk absorbed it and logged the failover each time; no ack lost. TCP was open and central logged nothing, so it is the client channel's backoff | 3 dispatches across the window, 2 failed over and the 3rd went direct (1B check 4 evidence) | Phase 2 soak — expect a burst of failover lines after any central restart, not a fault |
|
||
|
||
### Defects the gates caught that the suite did not
|
||
|
||
Recorded per phase as they occur, with root cause + the pinned regression test.
|
||
|
||
| # | Phase | Defect | Root cause | Status |
|
||
|---|---|---|---|---|
|
||
| 4 | 1A gate run 1 | **An admin node cannot boot at all** — `MissingActorRegistryEntryException: AuditWriterActorKey`, both central nodes, every transport mode | The first `WithActors` starter eagerly resolves `IMeshCommandDialer`/`IMeshPskProvider`, whose graph reaches `ActorAuditWriter`, whose ctor reads an `ActorRegistry` key registered ~60 lines later in the same builder | **CLOSED** — `c0475f2d` (`Func<>` + `PreStart`, pinned red-first by `ControlPlaneStarterOrderTests`) + `e54883a8` (memoize-on-first-use). Re-gated in run 2: 6/6 nodes boot, 0 `Hosting failed to start` |
|
||
| 5 | 1A gate run 2 | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider` with the same `AuditWriterActorKey` cause | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede `AuditWriterActorKey`'s registration | **Not a defect — it is the case `e54883a8` was written for, and the memoize demonstrably heals it.** Proven live by moving the coordinator singleton onto the node that lost the race (stop central-2) and watching it fan a dispatch to all five live nodes with no dialer error. Residual is cosmetic: the Error text and its effect on soak greps → Phase 2 |
|