docs(mesh): Phase 1A live gate re-run — PASS
Re-ran the 1A gate on the docker-dev rig at e54883a8 after the defect-4 boot
fix. Image rebuilt (otopcua-host:dev -> 1d8452a29018); six host nodes plus
both one-shots recreated; run 1's preserved image was stale and deliberately
not reused. VERDICT: PASS.
Boot: 6/6 nodes healthy INCLUDING the central pair, 0 Hosting failed to start,
/health/ready 200 on every node, MAIN mesh 2/2.
Check 1 PASS: one DispatchDeployment, one correlation id, accepted by all six
nodes; central logs 6 accepted / 0 failed; zero ClusterClient lines on either
central node with Akka.Cluster.Tools.Client at Debug.
Check 2 PASS: deployment Sealed with all six NodeDeploymentState rows Applied
inside ~90 ms, while the four site nodes hold receptionist associations — the
only ack path they have in 1A. Mixed transport proven.
Check 4 PASS: Ack clicked on the real /alerts page against a live ACTIVATED
row; central fanned one AlarmCommand to all six over gRPC and the condition
acknowledged 0.54 s later carrying the operator identity.
Check 7 PASS: with site-b-2's GrpcPort NULLed, both central nodes warned once
and the next dispatch reached exactly five nodes; the deployment then honestly
ended TimedOut with site-b-2 alone unapplied. Row restored, 6/6 sealed after.
Check 8 PASS and no longer vacuous: 0 denials on all four interceptors, and
this time config-serve (28 KB verified artifact fetches from :4055) and
telemetry (streams open on all six, live driver health on /hosts) both carried
real traffic.
Check 3 closed by composition, with the reason now established rather than
assumed: <DriverStatusPanel has 0 usages repo-wide, no rendered page carries a
Restart/Reconnect control, and the only non-script-analysis POST is
/api/deployments. Run 1 drove the node-side leg for both commands; check 4
drives the identical AdminOperationsActor.Dispatch -> DispatchOverGrpc origin
path from the AdminUI.
Checks 5 + 6 stand from run 1 — nothing in the fix touches the interceptor.
New finding 5, recorded because it is load-bearing for the fix's design: the
PreStart resolve LOSES its race on real boots — 2 of 3 observed admin boots
logged Failed to resolve for both dependencies with the same
AuditWriterActorKey cause. The memoize-on-first-use hardening is therefore not
theoretical, and its heal was proven live by moving the coordinator singleton
onto the node that lost the race (stop central-2) and watching central-1 fan a
dispatch to all five live nodes with no dialer error. Residual is cosmetic and
carried to Phase 2: the Error text reads as permanent, and it will trip a soak
error-grep on a healthy node.
Defect 4 marked CLOSED; run 1's record kept in full underneath the combined
table. Rig restored: flip reverted, six nodes healthy on all-ClusterClient on
the committed image, three mesh ClusterClients back on each central node.
This commit is contained in:
Joseph Doherty
@@ -154,7 +154,270 @@ Rig left running on `ClusterClient` (its normal state) after the gate.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Phase 1A — central→node command plane — **FAIL (blocked at the central boot boundary)**
|
## Phase 1A — central→node command plane — **PASS (on the re-run at `e54883a8`)**
|
||||||
|
|
||||||
|
Two runs. **Run 1 @ `fbf7b708` FAILED** at the central boot boundary (defect 4) and could not
|
||||||
|
drive checks 1/2/7 or the central halves of 3/4; its checks 5 + 6 and the node-side halves of
|
||||||
|
3 + 4 stand as recorded and were not repeated. **Run 2 @ `e54883a8`** — after the boot fix
|
||||||
|
(`c0475f2d`) and its review-hardening follow-up (`e54883a8`) — drove every carried check. The
|
||||||
|
result table below is the **combined, current** verdict; run 1's failure record is preserved
|
||||||
|
underneath it because the defect it caught is the most valuable thing this gate produced.
|
||||||
|
|
||||||
|
| # | Check | Verdict | Run |
|
||||||
|
|---|---|---|---|
|
||||||
|
| — | **Boot:** all six nodes healthy INCLUDING the central pair, on the branch image | **PASS** — 6/6 `running restarts=0`, 0 `Hosting failed to start`, `/health/ready` 200 ×6, MAIN mesh `memberCount=2 unreachableCount=0`. **But the `Resolve` Error line the fix logs DID fire — see finding 5** | 2 |
|
||||||
|
| 1 | Deploy dispatch reaches **all six** driver nodes over gRPC; zero `SendToAll`/ClusterClient on central | **PASS** — 6/6 nodes accepted one `DispatchDeployment` under a single shared `correlationId=cf499b42…`; central's own per-node outcome log records 6 accepted / 0 failed; `grep -ci clusterclient` = **0** on both central nodes | 2 |
|
||||||
|
| 2 | Acks return over ClusterClient (site mode unchanged) and the deployment **seals** | **PASS** — deployment `8DF9AAE0…` `Status=2` (Sealed) with all six `NodeDeploymentState` rows `Status=1` (Applied) inside ~90 ms; all four site nodes hold live `Connected to [akka.tcp://…/system/receptionist]` associations, which is their only ack path in 1A | 2 |
|
||||||
|
| 3 | `RestartDriver` + `ReconnectDriver` land on the target site node | **PASS (composed)** — node-side leg driven live in run 1 (both commands acted on site-a-1). The AdminUI half **cannot be driven and this is now proven, not assumed**: `<DriverStatusPanel` has **0** usages repo-wide, seven rendered pages contain no Restart/Reconnect text or button, and the only non-script-analysis HTTP POST in the whole app is `/api/deployments`. Closed by composition — check 4 proves the *same* `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path live from central for an admin-originated command | 1 + 2 |
|
||||||
|
| 4 | `AlarmCommand` from the AdminUI lands and the condition acknowledges | **PASS** — Ack clicked on the real `/alerts` page against a live `gate481-alarm` ACTIVATED row; central fanned **one** `AlarmCommand` (`correlationId=ccafe375…`, minted once) to all six over gRPC, and the condition acknowledged — a new `ACKNOWLEDGED` row appeared 0.54 s later carrying the operator's identity (`multi-role-test`) | 2 |
|
||||||
|
| 5 | Unauthenticated ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | **PASS** — 3/3 probes as designed (run 1; not repeated — nothing in the fix touches the interceptor) | 1 |
|
||||||
|
| 6 | **Wrong-key isolation** | **PASS** — both directions rejected, with a positive control proving the rejection is the key and not reachability (run 1) | 1 |
|
||||||
|
| 7 | NULL `GrpcPort` node skipped with the warn-once; fan continues | **PASS** — with `site-b-2:4053` NULLed, both central nodes logged the warn **once** each and the next dispatch (`correlationId=88a9e0a2…`) was accepted by exactly **5** nodes, site-b-2 absent; the deployment then honestly ended `Status=4` (TimedOut) with site-b-2 alone at `Status=0`. Row restored; a later dispatch sealed 6/6 | 2 |
|
||||||
|
| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures | **PASS, and no longer vacuous** — 0 denials on all four interceptors across all six nodes, **and all three planes carried real traffic this time**: site nodes fetched + SHA-verified 28 KB artifacts from `http://central-1:4055`, `Telemetry stream opened` on every node with live driver health rendering on `/hosts`, LocalDb sync active on the site pairs | 2 |
|
||||||
|
|
||||||
|
### Run 2 provenance
|
||||||
|
|
||||||
|
`docker compose -f docker-dev/docker-compose.yml up -d --build` from the worktree, **exit 0**;
|
||||||
|
image `otopcua-host:dev` rebuilt to `1d8452a29018` at 12:15 UTC (run 1's preserved
|
||||||
|
`otopcua-host:grpc-1a` was stale and deliberately **not** reused). The six host nodes plus
|
||||||
|
`migrator` and `cluster-seed` were recreated (both one-shots exit 0); `sql` and its volume were
|
||||||
|
kept, and `traefik` was reused unchanged (its service definition did not change). Nothing else
|
||||||
|
touched the stack during the run, so it is not concurrent-rebuild-invalidated. Same rig-only
|
||||||
|
flip and same three Serilog Debug overrides as run 1; both reverted afterwards, never committed.
|
||||||
|
|
||||||
|
```
|
||||||
|
$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
|
||||||
|
--format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
|
||||||
|
central-1 MeshTransport__Mode=Grpc
|
||||||
|
central-2 MeshTransport__Mode=Grpc
|
||||||
|
site-a-1 MeshTransport__Mode=ClusterClient
|
||||||
|
site-b-1 MeshTransport__Mode=ClusterClient
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run 2 evidence
|
||||||
|
|
||||||
|
**Boot — the fix works, and the race it was hardened against is real.**
|
||||||
|
|
||||||
|
```
|
||||||
|
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
|
||||||
|
central-1 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=2
|
||||||
|
central-2 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0
|
||||||
|
site-a-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
|
||||||
|
site-a-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
|
||||||
|
site-b-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
|
||||||
|
site-b-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
|
||||||
|
```
|
||||||
|
|
||||||
|
See **finding 5** below for the two `resolveErr` lines on central-1 — they are the PreStart
|
||||||
|
race losing on a real boot, and the memoize retry healing it was verified live rather than
|
||||||
|
argued.
|
||||||
|
|
||||||
|
**Check 1 — the fan, both ends.** One envelope, one correlation id, six recipients:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ for c in …; do docker logs otopcua-dev-$c-1 | grep "Mesh: accepted DispatchDeployment" | tail -1; done
|
||||||
|
central-1 [12:16:49 DBG] Mesh: accepted DispatchDeployment (correlationId=cf499b42c6914a578035b2b5bc1020f7) over gRPC; handed to the node communication actor.
|
||||||
|
central-2 [12:16:49 DBG] …same correlationId…
|
||||||
|
site-a-1 [12:16:49 DBG] …same correlationId…
|
||||||
|
site-a-2 [12:16:49 DBG] …same correlationId…
|
||||||
|
site-b-1 [12:16:49 DBG] …same correlationId…
|
||||||
|
site-b-2 [12:16:49 DBG] …same correlationId…
|
||||||
|
|
||||||
|
$ docker logs otopcua-dev-central-2-1 | grep "accepted by" # the node hosting the coordinator
|
||||||
|
[12:16:49 DBG] Mesh: DispatchDeployment (correlationId=cf499b42…) accepted by central-2:4053 over gRPC
|
||||||
|
[12:16:49 DBG] … accepted by site-b-1:4053 … [12:16:49 DBG] … accepted by site-a-2:4053 …
|
||||||
|
[12:16:49 DBG] … accepted by site-a-1:4053 … [12:16:49 DBG] … accepted by central-1:4053 …
|
||||||
|
[12:16:49 DBG] … accepted by site-b-2:4053 …
|
||||||
|
$ docker logs … | grep -c "was NOT delivered to" → 0 on both central nodes
|
||||||
|
|
||||||
|
# Zero ClusterClient on the command side, both central nodes:
|
||||||
|
central-1 MeshClusterClientCreated=0 clusterclientAnyLine=0
|
||||||
|
central-2 MeshClusterClientCreated=0 clusterclientAnyLine=0
|
||||||
|
```
|
||||||
|
|
||||||
|
The `clusterclientAnyLine=0` count is over the whole log with
|
||||||
|
`Serilog:MinimumLevel:Override:Akka.Cluster.Tools.Client=Debug` in force, so it covers Akka's
|
||||||
|
own ClusterClient internals and not merely this repo's creation message.
|
||||||
|
|
||||||
|
**Check 2 — mixed transport, end to end.** Commands out over gRPC (above), acks back over
|
||||||
|
ClusterClient:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='8df9aae0-…'
|
||||||
|
2 -- Sealed
|
||||||
|
$ … SELECT NodeId, Status, AppliedAtUtc FROM dbo.NodeDeploymentState WHERE DeploymentId='8df9aae0-…'
|
||||||
|
central-1:4053|1|2026-08-06 12:16:49.939 site-a-1:4053|1|2026-08-06 12:16:49.977
|
||||||
|
central-2:4053|1|2026-08-06 12:16:49.915 site-a-2:4053|1|2026-08-06 12:16:49.950
|
||||||
|
site-b-1:4053|1|2026-08-06 12:16:49.957 site-b-2:4053|1|2026-08-06 12:16:49.967
|
||||||
|
|
||||||
|
$ for c in site-a-1 site-a-2 site-b-1 site-b-2; do docker logs … | grep "receptionist" | tail -1; done
|
||||||
|
site-a-1 [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
|
||||||
|
site-a-2 [12:15:20 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
|
||||||
|
site-b-1 [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
|
||||||
|
site-b-2 [12:15:25 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
|
||||||
|
$ grep -c "No central ClusterClient" → 0 on all six # no ack was dropped for want of a client
|
||||||
|
```
|
||||||
|
|
||||||
|
A site node has **no** gRPC ack path in 1A (that is 1B), so the receptionist association above
|
||||||
|
is not merely present, it is the only route those four Applied rows could have taken.
|
||||||
|
|
||||||
|
**Check 3 — why the AdminUI half is undrivable, established rather than asserted:**
|
||||||
|
|
||||||
|
```
|
||||||
|
$ grep -rn "<DriverStatusPanel" --include="*.razor" --include="*.cs" src/ | wc -l
|
||||||
|
0 # the component holding the buttons is referenced nowhere
|
||||||
|
$ grep -rn "DriverStatusPanel" --include="*.razor" src/ | grep -v "Shared/Drivers/DriverStatusPanel.razor:"
|
||||||
|
…5 hits, all of them code COMMENTS in Alerts.razor / Hosts.razor ("mirrors DriverStatusPanel…")
|
||||||
|
|
||||||
|
$ node scan-ui.js # every page a Designer/Administrator/Operator can reach
|
||||||
|
/ | reconnect/restart text: false | buttons: ["Sign out"]
|
||||||
|
/hosts | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
|
||||||
|
/fleet | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
|
||||||
|
/clusters | reconnect/restart text: false | buttons: ["Sign out"]
|
||||||
|
/clusters/SITE-A | reconnect/restart text: false | buttons: ["Sign out"]
|
||||||
|
/alerts | reconnect/restart text: false | buttons: ["Sign out","Clear"]
|
||||||
|
/deployments | reconnect/restart text: false | buttons: ["Sign out","Deploy current configuration"]
|
||||||
|
|
||||||
|
$ grep -rn "MapPost|MapPut|MapDelete" src/…/AdminUI src/…/Host
|
||||||
|
…/Api/DeployApiEndpoints.cs:43: app.MapPost("/api/deployments", …)
|
||||||
|
…/ScriptAnalysis/ScriptAnalysisEndpoints.cs:19-24: /diagnostics /completions /hover /signature-help /format /inlay-hints
|
||||||
|
```
|
||||||
|
|
||||||
|
There is no button and no endpoint. What closes the check is that `RestartDriver`,
|
||||||
|
`ReconnectDriver` and `AlarmCommand` all leave central through the **same three lines** —
|
||||||
|
`AdminOperationsActor.Dispatch` → `MeshCommand` → `CentralCommunicationActor.DispatchOverGrpc`
|
||||||
|
— and check 4 drove that path live from the AdminUI for `AlarmCommand`, while run 1 drove the
|
||||||
|
node-side decode-and-act for the two driver commands. The only per-type difference left is
|
||||||
|
`MeshTransportDtoMapper.ToEnvelope`, which the T0.2 goldens cover member-for-member.
|
||||||
|
|
||||||
|
**Check 4 — AdminUI ack, fanned by central, acknowledged by the owner.** The rig's one
|
||||||
|
scripted alarm sits Normal indefinitely (`hr200 > 70000`), so a transition was manufactured by
|
||||||
|
editing its predicate to an always-true form and deploying; the `/alerts` page was already open
|
||||||
|
when it fired, the Ack button was clicked, and the predicate + hash were restored afterwards
|
||||||
|
(verified byte-identical by `SELECT`) with a further deploy:
|
||||||
|
|
||||||
|
```
|
||||||
|
page open at 2026-08-06T12:20:59.036Z initial: empty window
|
||||||
|
ROWS: 12:21:00.017 gate481-alarm | PressHigh | EQ-6e09bcc2d907 | ACTIVATED | 750 | system | press high
|
||||||
|
ROW BUTTONS: ["Ack","Shelve","Shelve (timed)","Unshelve"]
|
||||||
|
clicking Acknowledge at 2026-08-06T12:21:00.362Z
|
||||||
|
after click: 12:21:00.557 | gate481-alarm | EQ-6e09bcc2d907 | ACKNOWLEDGED | 750 | multi-role-test | press high
|
||||||
|
|
||||||
|
$ docker logs otopcua-dev-central-2-1 | grep AlarmCommand
|
||||||
|
[12:21:00 DBG] Mesh: AlarmCommand (correlationId=ccafe3754aca40cfa8aa0a0602c86cbb) accepted by site-b-2:4053 over gRPC
|
||||||
|
[12:21:00 DBG] … accepted by central-1:4053 … [12:21:00 DBG] … accepted by central-2:4053 …
|
||||||
|
[12:21:00 DBG] … accepted by site-a-1:4053 … [12:21:00 DBG] … accepted by site-b-1:4053 …
|
||||||
|
[12:21:00 DBG] … accepted by site-a-2:4053 …
|
||||||
|
$ for c in …; do grep "Mesh: accepted AlarmCommand" …; done → all six, same correlationId
|
||||||
|
```
|
||||||
|
|
||||||
|
One correlation id across all six recipients is the "encode once, before the fan" rule holding
|
||||||
|
for the one command whose CLR record has no correlation member of its own — the case the design
|
||||||
|
called out as the reason encoding is hoisted out of the loop.
|
||||||
|
|
||||||
|
**Check 7 — the skip, the warn, and the honest deadline.**
|
||||||
|
|
||||||
|
```
|
||||||
|
$ … UPDATE dbo.ClusterNode SET GrpcPort=NULL WHERE NodeId='site-b-2:4053' # 12:23:14
|
||||||
|
|
||||||
|
# one warn per central node, once each, within one ContactRefreshSeconds:
|
||||||
|
central-2 [12:23:20 WRN] ClusterNode site-b-2:4053 has no GrpcPort; it exposes no command endpoint
|
||||||
|
and is skipped in this dispatch set (further skips of this node are silent). Commands
|
||||||
|
will not reach it until the row carries a port
|
||||||
|
central-1 [12:23:19 WRN] …identical…
|
||||||
|
# (each node also logs the TelemetryNodeSource twin — "exposes no telemetry stream … skipped in
|
||||||
|
# this dial refresh" — expected collateral: both target sets read the same column)
|
||||||
|
|
||||||
|
# dispatch at 12:23:36 — accepted by five, site-b-2 absent:
|
||||||
|
central-2 [12:23:37 DBG] … (correlationId=88a9e0a2…) accepted by central-2 / site-a-2 / site-b-1
|
||||||
|
/ site-a-1 / central-1 :4053 over gRPC
|
||||||
|
# cumulative per-node accept counts after this dispatch — site-b-2 one behind everyone:
|
||||||
|
central-1 4 central-2 4 site-a-1 4 site-a-2 4 site-b-1 4 site-b-2 3
|
||||||
|
|
||||||
|
$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='5c8bdc9e-…' → 4 (TimedOut)
|
||||||
|
$ … SELECT NodeId, Status FROM dbo.NodeDeploymentState …
|
||||||
|
central-1|1 central-2|1 site-a-1|1 site-a-2|1 site-b-1|1 site-b-2|0
|
||||||
|
```
|
||||||
|
|
||||||
|
Row restored to `4056` and a later dispatch sealed 6/6 (`080EBB0B…`, `Status=2`, every node
|
||||||
|
`Status=1`). One operational detail worth carrying: **the skip persists for up to a full
|
||||||
|
`ContactRefreshSeconds` after the row is repaired** — a deploy issued 5 s after the restore
|
||||||
|
(`DF1C0B74…`) was still skipped and still TimedOut, because the dispatch set is only rebuilt on
|
||||||
|
the refresh tick.
|
||||||
|
|
||||||
|
**Check 8 — denials, and proof the planes were actually used.**
|
||||||
|
|
||||||
|
```
|
||||||
|
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
|
||||||
|
central-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||||||
|
central-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||||||
|
site-a-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||||||
|
site-a-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||||||
|
site-b-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||||||
|
site-b-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
|
||||||
|
|
||||||
|
# config-serve carried real, authenticated traffic:
|
||||||
|
site-a-1 [12:20:59 INF] Fetched + verified artifact for 8d486c6b… from http://central-1:4055 (28313 bytes).
|
||||||
|
# telemetry likewise, end to end into the UI:
|
||||||
|
$ for c in …; do grep -c "Telemetry stream opened"; done → 3,4,3,3,3,5
|
||||||
|
/hosts → CLUSTER SITE-A … sa-modbus2 SITEA-modbus Modbus HEALTHY 1 host 1 tag 12:30:49 UTC 0 errors
|
||||||
|
```
|
||||||
|
|
||||||
|
The five telemetry `failed … reconnecting` warnings in the run are all accounted for: three at
|
||||||
|
12:15:20 (central dialing nodes that had not finished binding `:4056`) and two at 12:28:50 /
|
||||||
|
12:29:30 (the deliberate central-2 stop in finding 5). None is an auth failure.
|
||||||
|
|
||||||
|
### Phase 1A finding 5 — the PreStart race loses on real boots, and the memoize is what saves it
|
||||||
|
|
||||||
|
The hardening commit's premise is not hypothetical. On the run-2 boot, **central-1 lost the
|
||||||
|
race on both dependencies**:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ docker logs otopcua-dev-central-1-1 | grep -A3 "Failed to resolve"
|
||||||
|
[12:15:19 ERR] Failed to resolve IMeshCommandDialer for the mesh command boundary; continuing
|
||||||
|
without it. The guards below report what that costs
|
||||||
|
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
|
||||||
|
ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
|
||||||
|
at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
|
||||||
|
[12:15:19 ERR] Failed to resolve IMeshPskProvider for the mesh command boundary; …same cause…
|
||||||
|
```
|
||||||
|
|
||||||
|
It is not rare: across the three admin boots observed in run 2 (central-1 and central-2 at
|
||||||
|
12:15, central-2 again at 12:29 after the stop/start), **two of three lost it**. Had the fix
|
||||||
|
stopped at PreStart-only resolution, those nodes would have held a permanent silent `null` —
|
||||||
|
no dialer, and a never-published known-cluster gate — and the failure would only have surfaced
|
||||||
|
the first time the coordinator singleton moved onto such a node.
|
||||||
|
|
||||||
|
**The memoize heals it, verified on the node that actually lost the race.** Run 2's dispatches
|
||||||
|
were all fanned by central-2 (the singleton host), so central-1's dialer was never exercised
|
||||||
|
by them. Central-2 was therefore stopped to move the singleton, and a dispatch driven while
|
||||||
|
central-1 alone held it:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ docker compose stop central-2 # 12:28:39
|
||||||
|
$ curl … /health/ready → self akka.tcp://otopcua@central-1:4053, members 1 # 12:28:57
|
||||||
|
$ curl -X POST /api/deployments -d '{"createdBy":"1a-gate-central1-heal"}' # 12:29:01
|
||||||
|
$ docker logs otopcua-dev-central-1-1 | grep -E "accepted by|no command dialer is wired"
|
||||||
|
[12:29:02 DBG] Mesh: DispatchDeployment (correlationId=ad97d386…) accepted by site-b-1:4053 over gRPC
|
||||||
|
[12:29:02 DBG] … accepted by site-a-2 … site-b-2 … site-a-1 … central-1 … # 5 live nodes
|
||||||
|
# no "no command dialer is wired" line, and no further "Failed to resolve"
|
||||||
|
$ docker compose start central-2 → members 2, unreachable 0, status Healthy # 12:29:51
|
||||||
|
```
|
||||||
|
|
||||||
|
So the boot-time Error is **cosmetically alarming but functionally healed**, and that is now an
|
||||||
|
observed property rather than a design claim. Two follow-ups for Phase 2 rather than for 1A:
|
||||||
|
the Error text says "continuing without it", which reads as permanent and should say that the
|
||||||
|
first use retries; and a `Failed to resolve` line on a healthy node is exactly the kind of
|
||||||
|
signal a soak's PSK-drift/error grep will trip over.
|
||||||
|
|
||||||
|
**One incidental observation, not a check.** The deployment `RevisionHash` is deterministic
|
||||||
|
(three separate no-change dispatches all produced `642b9d6f79b9…`), but restoring the alarm
|
||||||
|
predicate's two text columns did **not** restore the pre-gate hash `6178a8f302aa…` — so the
|
||||||
|
artifact covers a column the `UPDATE` bumped (`Script.RowVersion` is the obvious candidate).
|
||||||
|
The rig's configuration content is byte-identical to its pre-gate state; only the hash moved.
|
||||||
|
Not investigated further — out of this program's scope.
|
||||||
|
|
||||||
|
### Run 1 record — the defect that blocked it
|
||||||
|
|
||||||
Branch `feat/grpc-node-command` @ `fbf7b708`. Image rebuilt from this worktree
|
Branch `feat/grpc-node-command` @ `fbf7b708`. Image rebuilt from this worktree
|
||||||
(`/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a`) so it carries this branch;
|
(`/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a`) so it carries this branch;
|
||||||
@@ -170,26 +433,18 @@ on the compose default `ClusterClient`** (coexistence is the point). Three Seril
|
|||||||
rig's default Serilog level is `Information`, so without them checks 1/3 would have had no
|
rig's default Serilog level is `Information`, so without them checks 1/3 would have had no
|
||||||
observable signal. Flip and overrides reverted after the gate, never committed.
|
observable signal. Flip and overrides reverted after the gate, never committed.
|
||||||
|
|
||||||
**Verdict: FAIL.** Neither central node boots on this branch, so nothing that originates at
|
**Run 1 verdict: FAIL.** Neither central node booted, so nothing that originates at central
|
||||||
central could be driven. Root cause is defect 4 below — resolving `IMeshCommandDialer` /
|
could be driven: checks 1, 2 and 7 were **BLOCKED** outright, 3 and 4 reached only their
|
||||||
`IMeshPskProvider` inside the first `WithActors` starter drags in `ISecretResolver` →
|
node-side legs, and check 8's zero-denial result was vacuous for telemetry and config-serve
|
||||||
|
(central dials both and never booted). Checks 5 and 6 **PASSED** and are the run-1 evidence
|
||||||
|
carried into the combined table above. Root cause was defect 4 — resolving `IMeshCommandDialer`
|
||||||
|
/ `IMeshPskProvider` inside the first `WithActors` starter drags in `ISecretResolver` →
|
||||||
`IAuditWriter` → `ActorAuditWriter`, whose constructor reads `AuditWriterActorKey` out of the
|
`IAuditWriter` → `ActorAuditWriter`, whose constructor reads `AuditWriterActorKey` out of the
|
||||||
`ActorRegistry` before the singleton that registers it has run. It is **mode-independent**
|
`ActorRegistry` before the singleton that registers it has run. It was **mode-independent**
|
||||||
(reproduced on the same image with `MeshTransport__Mode=ClusterClient`), so it is a defect of
|
(reproduced on the same image with `MeshTransport__Mode=ClusterClient`), so it was a defect of
|
||||||
the branch, not of the gate's flip, and it would have failed a `ClusterClient` rig equally.
|
the branch, not of the gate's flip, and it would have failed a `ClusterClient` rig equally.
|
||||||
|
|
||||||
| # | Check | Result |
|
### Run 1 evidence
|
||||||
|---|---|---|
|
|
||||||
| 1 | Deploy dispatch from AdminUI reaches **all six** driver nodes over gRPC (per-node `Execute` log lines on every node; zero `SendToAll` lines on central) | **BLOCKED** — `POST /api/deployments` is unreachable: traefik answers `503` because neither central node has a live HTTP surface (`curl http://central-1:9000/health/ready` and `central-2` both return `000`, connection refused). Nothing dispatched. |
|
|
||||||
| 2 | Acks still return over ClusterClient (site mode unchanged) and the deployment **seals** — mixed-transport deploy proven end to end | **BLOCKED** — no deployment could be dispatched (check 1). The most recent `dbo.Deployment` row is still the pre-gate `03540DA5-…` (`Status=2`, Sealed, `2026-08-06 06:11:22`), i.e. nothing was attempted during this run. |
|
|
||||||
| 3 | `RestartDriver` + `ReconnectDriver` from AdminUI land on the target site node (EventStream delivery observed via driver restart in logs) | **PARTIAL PASS (node-side leg only)** — both commands were delivered to site-a-1 over real Kestrel h2c with the correct PSK and both acted: `DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate` followed by a fresh `spawned Modbus driver SITEA-modbus`, and `reconnecting driver … ForceReconnect requested by admin; re-entering Reconnecting`. The **AdminUI half was not driven** — see the two carry-forwards below (central down; and the UI control is unreachable on this branch anyway). |
|
|
||||||
| 4 | `AlarmCommand` (ack a scripted alarm from AdminUI) lands and the condition acknowledges | **PARTIAL PASS (envelope leg only)** — the alarm arm decoded and was delivered node-side (`Mesh: accepted AlarmCommand (correlationId=39ba4511…) over gRPC`), which is the arm with no CLR correlation member and therefore the one worth wiring live. **The acknowledge itself was NOT observed**: the rig's single `dbo.ScriptedAlarm` row (`gate481-alarm`, equipment `EQ-6e09bcc2d907`) is not owned by SITE-A, and its owning node is a central one, which is down. |
|
|
||||||
| 5 | Unauthenticated `grpcurl` to a site node `:4056` `mesh_transport.v1.NodeCommandService/Execute` ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | **PASS** — 3/3 probes graded as designed: no bearer ⇒ `PermissionDenied`, correct `dev-mesh-psk-SITE-A` + empty envelope ⇒ `InvalidArgument` (**auth passed**, the decode is what failed), and a full `RestartDriver` envelope with the same key ⇒ `{}` + observed driver restart. |
|
|
||||||
| 6 | **Wrong-key isolation:** site-b's PSK presented to site-a ⇒ `PermissionDenied` (per-cluster scoping is real, not decorative) | **PASS** — both directions rejected: `dev-mesh-psk-SITE-B` → site-a-1 ⇒ `PermissionDenied`, `dev-mesh-psk-SITE-A` → site-b-1 ⇒ `PermissionDenied`, while `dev-mesh-psk-SITE-B` → site-b-1 reaches the decoder (`InvalidArgument`) — so the rejection is the key, not the reachability. |
|
|
||||||
| 7 | A node with NULL `GrpcPort` is skipped with the warn-once, fan continues to the others | **BLOCKED** — the skip is computed by `CentralCommunicationActor.AdoptGrpcTargets` on a contact refresh; no central node is running to refresh. The rig's `ClusterNode` rows were left untouched (all six still `GrpcPort=4056`) rather than mutated for a check that could not be observed. |
|
|
||||||
| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures on their interceptors across the run | **PARTIAL PASS** — 0 `Rejected a telemetry stream call` / `Rejected a config-serve call` / `Rejected a LocalDb sync call` lines across all six nodes, and the only `Rejected a mesh command call` lines are the 3 deliberate negative probes (2 on site-a-1, 1 on site-b-1). **But telemetry and config-serve carried no traffic at all** in this run — central dials both and never booted (site-a-1 logs 0 telemetry lines) — so "unaffected" is established for LocalDb sync only. |
|
|
||||||
|
|
||||||
### Phase 1A evidence
|
|
||||||
|
|
||||||
**Provenance** — flip in effect on central only, sites untouched:
|
**Provenance** — flip in effect on central only, sites untouched:
|
||||||
|
|
||||||
@@ -322,7 +577,7 @@ The `TwoNodeClusterHarness` I-1 null-provider Error (a known test-harness artifa
|
|||||||
appear in any rig log — `grep -c "IMeshPskProvider"` = 0 on all six nodes, as expected for
|
appear in any rig log — `grep -c "IMeshPskProvider"` = 0 on all six nodes, as expected for
|
||||||
production wiring where every admin node registers the provider.
|
production wiring where every admin node registers the provider.
|
||||||
|
|
||||||
### Phase 1A defect
|
### Phase 1A defect — **CLOSED by `c0475f2d` + `e54883a8`, re-gated in run 2**
|
||||||
|
|
||||||
**Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter.**
|
**Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter.**
|
||||||
`WithOtOpcUaControlPlaneSingletons`'s `WithActors` block resolves `IMeshCommandDialer` and
|
`WithOtOpcUaControlPlaneSingletons`'s `WithActors` block resolves `IMeshCommandDialer` and
|
||||||
@@ -351,20 +606,46 @@ Three things make this worth a pinned regression rather than a one-line fix note
|
|||||||
|
|
||||||
The shape of the fix is the one this file already uses two lines above for the coordinator
|
The shape of the fix is the one this file already uses two lines above for the coordinator
|
||||||
proxy — hand the actor a lazy accessor (`Func<IMeshCommandDialer?>` / `Func<IMeshPskProvider?>`,
|
proxy — hand the actor a lazy accessor (`Func<IMeshCommandDialer?>` / `Func<IMeshPskProvider?>`,
|
||||||
or resolve on first use) so nothing is constructed while the `ActorSystem` is still building —
|
or resolve on first use) so nothing is constructed while the `ActorSystem` is still building.
|
||||||
but the fix and its red-first pin belong to 1A, not to this record.
|
|
||||||
|
|
||||||
### Rig state after the gate
|
**That is what shipped.** `c0475f2d` moved both dependencies behind `Func<>` invoked in
|
||||||
|
`PreStart`, pinned by `ControlPlaneStarterOrderTests` (boots a real admin host through the
|
||||||
|
production starter order and asserts `StartAsync`; verified RED against the eager resolve in
|
||||||
|
3/3 modes). `e54883a8` then made the resolution memoize-on-first-use rather than PreStart-only,
|
||||||
|
because `PreStart` runs concurrently with the remaining starters and can still lose the race —
|
||||||
|
which **run 2 observed happening on 2 of 3 real admin boots**, so that hardening was not
|
||||||
|
theoretical. Both are re-gated above; see finding 5 for the live self-heal proof.
|
||||||
|
|
||||||
The compose flip was reverted (`git checkout docker-dev/docker-compose.yml`) and the six nodes
|
### Rig state after run 1 (superseded by the run-2 restoration below)
|
||||||
recreated on the committed all-`ClusterClient` compose. Because defect 4 is mode-independent,
|
|
||||||
the branch image cannot produce a healthy central pair on *any* mode, so the rig was returned
|
The compose flip was reverted and the six nodes recreated on the committed all-`ClusterClient`
|
||||||
to its pre-gate image rather than left with a dead central pair: the 1A build was preserved as
|
compose. Because defect 4 was mode-independent, the branch image could not produce a healthy
|
||||||
`otopcua-host:grpc-1a` (`69360a173f1f`) and the pre-gate build (`8bd81d8aea9b`, 04:52) re-tagged
|
central pair on *any* mode, so the rig was returned to its pre-gate image rather than left with
|
||||||
back to `otopcua-host:dev`. **Anyone re-running this gate must rebuild — the running rig is
|
a dead central pair: the run-1 build was preserved as `otopcua-host:grpc-1a` (`69360a173f1f`)
|
||||||
NOT on the branch under test.** Verified after restore: six nodes `running restarts=0`, zero
|
and the pre-gate build (`8bd81d8aea9b`) re-tagged back to `otopcua-host:dev`.
|
||||||
`Hosting failed to start` on all six, `/health/ready` 200 on every node, MAIN mesh
|
|
||||||
`memberCount=2 unreachableCount=0`, AdminUI 200 on `http://localhost:9200`.
|
### Rig state after run 2 — **restored, six healthy, on the branch image**
|
||||||
|
|
||||||
|
The flip was reverted (`git checkout docker-dev/docker-compose.yml`) and the six nodes
|
||||||
|
recreated on the committed all-`ClusterClient` compose. No image substitution was needed this
|
||||||
|
time: `otopcua-host:dev` is the run-2 build of the committed source (`1d8452a29018`), and it
|
||||||
|
boots a healthy central pair, so the rig runs the code under test. The run-1 workaround tag
|
||||||
|
`otopcua-host:grpc-1a` was deleted to stop anyone reusing a stale image.
|
||||||
|
|
||||||
|
```
|
||||||
|
central-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
|
||||||
|
central-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
|
||||||
|
site-a-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
|
||||||
|
site-a-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
|
||||||
|
site-b-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
|
||||||
|
site-b-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
|
||||||
|
$ curl … /health/ready → status Healthy, members 2, unreachable 0
|
||||||
|
$ grep -c "Mesh ClusterClient for cluster" → 3 on each central node (MAIN + SITE-A + SITE-B)
|
||||||
|
```
|
||||||
|
|
||||||
|
The three ClusterClients reappearing on the rollback is an incidental but useful result: the
|
||||||
|
dark switch still rolls back cleanly on the fixed branch, which is what Phase 2 will depend on
|
||||||
|
if its cutover has to be reversed.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -425,12 +706,13 @@ checks 6–7.
|
|||||||
Anything a phase gate could not drive gets named here with why + what proves it instead, and
|
Anything a phase gate could not drive gets named here with why + what proves it instead, and
|
||||||
is re-checked in the next gate (the ScadaBridge proportionate-gate rule).
|
is re-checked in the next gate (the ScadaBridge proportionate-gate rule).
|
||||||
|
|
||||||
| Item | Deferred from | Why it could not be driven | Proven instead by | Re-check in |
|
| Item | Deferred from | Why it could not be driven | Proven instead by | Status |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-`GrpcPort` skip | 1A | Both central nodes fail to boot (defect 4); no AdminUI, no deploy API, no contact refresh | Nothing live. Unit coverage only: the fan Theory, the gate-⊋-fan fact, `A_slow_dial_does_not_park_the_actors_mailbox` | **1A gate re-run after defect 4 is fixed** — these are the checks 1A exists to prove and must not be carried past it |
|
| Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-`GrpcPort` skip | 1A run 1 | Both central nodes failed to boot (defect 4) | — | **CLOSED in 1A run 2** — all three PASS live |
|
||||||
| Check 3 central half — `RestartDriver`/`ReconnectDriver` **originating at the AdminUI** | 1A | (a) central down; (b) independently, `DriverStatusPanel.razor` — the only component holding the Reconnect/Restart buttons — is referenced by **no page or component in the AdminUI**, so the control is unreachable in the running UI on this branch and on `master` | The node-side leg driven live with real envelopes (check 3 evidence); central's encode covered by the T0.2 mapper goldens | 1A re-run; the orphaned panel is pre-existing and out of this program's scope — file separately if it is to be fixed |
|
| Check 4 acknowledge half — the condition actually acknowledging | 1A run 1 | The rig's one `ScriptedAlarm` sits Normal and its owning central node was down | — | **CLOSED in 1A run 2** — acked from the real `/alerts` page, `ACKNOWLEDGED` row observed |
|
||||||
| Check 4 acknowledge half — the condition actually acknowledging | 1A | The rig's one `ScriptedAlarm` (`gate481-alarm`) is not owned by a SITE cluster, and its owning central node is down | The alarm arm decoding + delivering node-side (check 4 evidence) | 1A re-run, seeding a SITE-A-owned scripted alarm first |
|
| Check 8 for telemetry + config-serve | 1A run 1 | Central dials both planes and never booted, so neither carried a call — 0 denials was vacuous | — | **CLOSED in 1A run 2** — both planes carried authenticated traffic with 0 denials |
|
||||||
| Check 8 for telemetry + config-serve | 1A | Central dials both planes and never booted, so neither carried a single call — 0 denials is vacuous for them | LocalDb sync (pair-local) did carry traffic with 0 denials | 1A re-run |
|
| Check 3 central half — `RestartDriver`/`ReconnectDriver` **originating at the AdminUI** | 1A runs 1 + 2 | **Structurally undrivable, not a rig problem:** `<DriverStatusPanel` has 0 usages repo-wide (the 5 name hits are comments), no rendered page carries a Restart/Reconnect control, and the only non-script-analysis HTTP POST is `/api/deployments`. Pre-existing on `master`, not caused by this program | Composition the controller accepted: run 1's live node-side leg for both commands + run 2's check 4, which drives the identical `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path from the AdminUI for another command type; per-type difference is `ToEnvelope`, covered by the T0.2 goldens | **CLOSED by composition.** The orphaned panel itself is out of this program's scope — file separately if the control is meant to exist |
|
||||||
|
| `Failed to resolve` Error text says "continuing without it", which reads permanent although first use retries; and it will trip a soak's error grep on a healthy node | 1A run 2 (finding 5) | Cosmetic/observability, not behaviour | The self-heal proven live (finding 5) | Phase 2 |
|
||||||
|
|
||||||
### Defects the gates caught that the suite did not
|
### Defects the gates caught that the suite did not
|
||||||
|
|
||||||
@@ -438,4 +720,5 @@ Recorded per phase as they occur, with root cause + the pinned regression test.
|
|||||||
|
|
||||||
| # | Phase | Defect | Root cause | Status |
|
| # | Phase | Defect | Root cause | Status |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| 4 | 1A gate | **An admin node cannot boot at all** — `MissingActorRegistryEntryException: AuditWriterActorKey`, both central nodes, every transport mode | The first `WithActors` starter eagerly resolves `IMeshCommandDialer`/`IMeshPskProvider`, whose graph reaches `ActorAuditWriter`, whose ctor reads an `ActorRegistry` key registered ~60 lines later in the same builder | **OPEN — blocks 1A.** Full analysis + evidence in the Phase 1A section above. No product change was made by this gate |
|
| 4 | 1A gate run 1 | **An admin node cannot boot at all** — `MissingActorRegistryEntryException: AuditWriterActorKey`, both central nodes, every transport mode | The first `WithActors` starter eagerly resolves `IMeshCommandDialer`/`IMeshPskProvider`, whose graph reaches `ActorAuditWriter`, whose ctor reads an `ActorRegistry` key registered ~60 lines later in the same builder | **CLOSED** — `c0475f2d` (`Func<>` + `PreStart`, pinned red-first by `ControlPlaneStarterOrderTests`) + `e54883a8` (memoize-on-first-use). Re-gated in run 2: 6/6 nodes boot, 0 `Hosting failed to start` |
|
||||||
|
| 5 | 1A gate run 2 | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider` with the same `AuditWriterActorKey` cause | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede `AuditWriterActorKey`'s registration | **Not a defect — it is the case `e54883a8` was written for, and the memoize demonstrably heals it.** Proven live by moving the coordinator singleton onto the node that lost the race (stop central-2) and watching it fan a dispatch to all five live nodes with no dialer error. Residual is cosmetic: the Error text and its effect on soak greps → Phase 2 |
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ posture).
|
|||||||
| Phase | Scope | Branch | Status | PR / merge SHA | Live gate |
|
| Phase | Scope | Branch | Status | PR / merge SHA | Live gate |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **MERGED to `master`** @ `10b6d209` (2026-08-06) — all 6 tasks complete @ `23555ed0`; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed | `10b6d209` | **PASS 4/4** |
|
| 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **MERGED to `master`** @ `10b6d209` (2026-08-06) — all 6 tasks complete @ `23555ed0`; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed | `10b6d209` | **PASS 4/4** |
|
||||||
| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **1A.3 gate RAN 2026-08-06 @ `fbf7b708` and FAILED — defect 4 blocks the phase.** Both central nodes fail to boot (mode-independent), so checks 1/2/7 could not be driven at all and 3/4 only node-side; 5/6 PASS. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅. **Next: fix defect 4 with a red-first pin, then RE-RUN the 1A gate** | — | **FAIL** (5 of 8 not provable; 2 PASS, 2 partial, 3 blocked) |
|
| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **COMPLETE @ `e54883a8` — gate re-run PASSED 2026-08-06.** Run 1 @ `fbf7b708` FAILED on defect 4 (no admin node could boot, mode-independent); fixed by `c0475f2d` + `e54883a8` and re-gated. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅ | — | **PASS 8/8** (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards) |
|
||||||
| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebases on 1A | `feat/grpc-central-ack` | **1B.1 ✅ `14dd07d2` + 1B.2 ✅ `a5a789d4` (+fixes `e54711a2`/`4d607575`)** — all reviews ✅ (1B.2 on Opus: 1 Critical found+fixed, see defect log); awaiting 1A merge → rebase → 1B.3 gate | — | PENDING |
|
| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebases on 1A | `feat/grpc-central-ack` | **1B.1 ✅ `14dd07d2` + 1B.2 ✅ `a5a789d4` (+fixes `e54711a2`/`4d607575`)** — all reviews ✅ (1B.2 on Opus: 1 Critical found+fixed, see defect log); awaiting 1A merge → rebase → 1B.3 gate | — | PENDING |
|
||||||
| 2 | Baseline reproduction on pre-flip build, then full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills | (rig commit) | NOT STARTED | — | PENDING |
|
| 2 | Baseline reproduction on pre-flip build, then full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills | (rig commit) | NOT STARTED | — | PENDING |
|
||||||
| 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — |
|
| 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — |
|
||||||
@@ -129,7 +129,8 @@ Defects found during execution (live-gate or review), with root cause + pinned t
|
|||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| 1 | 1B.2 | **Failover silently defeated on a hanging central** (CRITICAL, caught by Opus code review): the ack deadline was one absolute `DateTime` shared across the endpoint walk — a hung (not refused) central-1 burned the whole budget and central-2 was dialed with an expired deadline | deadline hoisted outside the `foreach`; a kill-style drill can't see it (refused = fast-fail), only a hang discriminates | `e54711a2`; pinned by `Each_endpoint_gets_a_FRESH_deadline` (mutation-verified); 1B.3 gains a `docker pause` hang drill |
|
| 1 | 1B.2 | **Failover silently defeated on a hanging central** (CRITICAL, caught by Opus code review): the ack deadline was one absolute `DateTime` shared across the endpoint walk — a hung (not refused) central-1 burned the whole budget and central-2 was dialed with an expired deadline | deadline hoisted outside the `foreach`; a kill-style drill can't see it (refused = fast-fail), only a hang discriminates | `e54711a2`; pinned by `Each_endpoint_gets_a_FRESH_deadline` (mutation-verified); 1B.3 gains a `docker pause` hang drill |
|
||||||
| 2 | 1A.2 | **Dark-switch rollback deactivated the known-cluster gate** (Opus review, I-2): gate published only under `Mode=Grpc` while the interceptor + service are port-mapped in every mode — a rollback to ClusterClient/Dps would silently restore the timing oracle + pre-auth flood | gate activation coupled to transport mode instead of the endpoint it protects | `48dc2120` — gate follows the endpoint; pinned by the every-mode Theory + Dps guard pin `86633f14` |
|
| 2 | 1A.2 | **Dark-switch rollback deactivated the known-cluster gate** (Opus review, I-2): gate published only under `Mode=Grpc` while the interceptor + service are port-mapped in every mode — a rollback to ClusterClient/Dps would silently restore the timing oracle + pre-auth flood | gate activation coupled to transport mode instead of the endpoint it protects | `48dc2120` — gate follows the endpoint; pinned by the every-mode Theory + Dps guard pin `86633f14` |
|
||||||
| 4 | 1A.3 (live gate) | **An admin node cannot boot** (BLOCKS 1A): both central nodes die at startup with `MissingActorRegistryEntryException: ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey`, taking `AkkaHostedService` and the whole host with them. Mode-INDEPENDENT — reproduced on the same image with `MeshTransport__Mode=ClusterClient`, so it is not the gate's flip. Driver-only nodes are unaffected (they register neither `IMeshPskProvider` nor `ActorAuditWriter`) | `WithOtOpcUaControlPlaneSingletons`'s first `WithActors` starter eagerly resolves `IMeshCommandDialer` + `IMeshPskProvider` to pass into `CentralCommunicationActor.Props`. That graph is dialer → `IMeshPskProvider` → `ISecretResolver` → (`AddZbSecrets`) → `IAuditWriter` → `ActorAuditWriter`, whose ctor calls `registry.Get<AuditWriterActorKey>()` — and `WithSingleton<AuditWriterActorKey>` is registered ~60 lines later in the same builder. Same class as scadaproj#1 and mesh Phase 6's `IClusterRoleInfo` crash: only visible with the DI graph built in a real host, which is why the container-activation test (no `AddZbSecrets` in its collection) and `TwoNodeClusterHarness` (no provider at all) both miss it. Aggravating: the failed process does not exit, so `docker ps` reports `Up restarts=0` while the node serves no HTTP | **OPEN.** Fix + red-first pin belong to 1A (the gate made no product change). Suggested shape: hand the actor lazy accessors, mirroring the `Func<>` the same block already uses for the coordinator proxy. Pin must build the graph in starter order (a Host.IntegrationTests boot of an admin-role host), not merely activate the dialer |
|
| 5 | 1A.3 (live gate, run 2) | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider`, same `AuditWriterActorKey` cause. NOT a defect: it is exactly the case the review-prescribed hardening was written for | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede starter #4 | **Closed by `e54883a8`, and the heal is PROVEN LIVE** — the coordinator singleton was moved onto the node that lost the race (stop central-2) and it fanned a dispatch to all five live nodes with no dialer error. Residual is cosmetic (Error text reads permanent; will trip a soak error-grep on a healthy node) → Phase 2 |
|
||||||
|
| 4 | 1A.3 (live gate, run 1) | **An admin node cannot boot** (BLOCKED 1A): both central nodes die at startup with `MissingActorRegistryEntryException: ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey`, taking `AkkaHostedService` and the whole host with them. Mode-INDEPENDENT — reproduced on the same image with `MeshTransport__Mode=ClusterClient`, so it is not the gate's flip. Driver-only nodes are unaffected (they register neither `IMeshPskProvider` nor `ActorAuditWriter`) | `WithOtOpcUaControlPlaneSingletons`'s first `WithActors` starter eagerly resolves `IMeshCommandDialer` + `IMeshPskProvider` to pass into `CentralCommunicationActor.Props`. That graph is dialer → `IMeshPskProvider` → `ISecretResolver` → (`AddZbSecrets`) → `IAuditWriter` → `ActorAuditWriter`, whose ctor calls `registry.Get<AuditWriterActorKey>()` — and `WithSingleton<AuditWriterActorKey>` is registered ~60 lines later in the same builder. Same class as scadaproj#1 and mesh Phase 6's `IClusterRoleInfo` crash: only visible with the DI graph built in a real host, which is why the container-activation test (no `AddZbSecrets` in its collection) and `TwoNodeClusterHarness` (no provider at all) both miss it. Aggravating: the failed process does not exit, so `docker ps` reports `Up restarts=0` while the node serves no HTTP | **OPEN.** Fix + red-first pin belong to 1A (the gate made no product change). Suggested shape: hand the actor lazy accessors, mirroring the `Func<>` the same block already uses for the coordinator proxy. Pin must build the graph in starter order (a Host.IntegrationTests boot of an admin-role host), not merely activate the dialer |
|
||||||
| 3 | 1A.2 | **Fan parked the actor mailbox** (self-caught by the implementer): dialing from the receive ran `TryGetPsk`'s cold-cache 3 s block pre-await (cold after EVERY refresh due to `InvalidateAll`), stalling deployment acks behind the fan | async methods run synchronously to the first await | `Task.Run` hop (token None — a canceled scheduling would read as a DB failure via PipeTo); pinned red-first by `A_slow_dial_does_not_park_the_actors_mailbox` |
|
| 3 | 1A.2 | **Fan parked the actor mailbox** (self-caught by the implementer): dialing from the receive ran `TryGetPsk`'s cold-cache 3 s block pre-await (cold after EVERY refresh due to `InvalidateAll`), stalling deployment acks behind the fan | async methods run synchronously to the first await | `Task.Run` hop (token None — a canceled scheduling would read as a DB failure via PipeTo); pinned red-first by `A_slow_dial_does_not_park_the_actors_mailbox` |
|
||||||
|
|
||||||
## Carry-forward register
|
## Carry-forward register
|
||||||
@@ -142,9 +143,10 @@ Checks a gate could not drive, named + carried to the next gate (proportionate-g
|
|||||||
| I-4 measurement: ack burst within ~3 s after a contact-refresh tick (cold PSK cache on the inbound auth path) | 1A.2 review | bounded 3 s store read + `Lazy` stampede suppression | 1B.3 gate (observe) |
|
| I-4 measurement: ack burst within ~3 s after a contact-refresh tick (cold PSK cache on the inbound auth path) | 1A.2 review | bounded 3 s store read + `Lazy` stampede suppression | 1B.3 gate (observe) |
|
||||||
| `TwoNodeClusterHarness` logs the I-1 null-provider Error at every spawn (harness registers no `IMeshPskProvider`) — EXPECTED, not a rig fault | 1A.2 re-review | production pairing is `hasAdmin`-unconditional in one block | note for 1B.3/Phase 2 readers |
|
| `TwoNodeClusterHarness` logs the I-1 null-provider Error at every spawn (harness registers no `IMeshPskProvider`) — EXPECTED, not a rig fault | 1A.2 re-review | production pairing is `hasAdmin`-unconditional in one block | note for 1B.3/Phase 2 readers |
|
||||||
| Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter` | Phase 2 (or 1B.3) — note defect 4 shows the missing pin is not merely role-scoping but **starter-order**: the container-activation test resolves from a collection without `AddZbSecrets` |
|
| Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter` | Phase 2 (or 1B.3) — note defect 4 shows the missing pin is not merely role-scoping but **starter-order**: the container-activation test resolves from a collection without `AddZbSecrets` |
|
||||||
| 1A checks 1/2/7 (deploy fan over gRPC, mixed-transport seal, NULL-`GrpcPort` skip) and the central halves of 3/4 | 1A.3 gate | Nothing live — unit coverage only | **1A gate re-run after defect 4**; must not be carried past 1A |
|
| 1A checks 1/2/7 (deploy fan over gRPC, mixed-transport seal, NULL-`GrpcPort` skip) and the central half of 4 | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — all driven live and PASS |
|
||||||
| Check 8 is vacuous for telemetry + config-serve — central never booted, so neither plane carried a call | 1A.3 gate | LocalDb sync did carry traffic, 0 denials | 1A gate re-run |
|
| Check 8 is vacuous for telemetry + config-serve — central never booted, so neither plane carried a call | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — both planes carried authenticated traffic (28 KB artifact fetches from `:4055`, telemetry streams open on all six) with 0 denials |
|
||||||
| **`DriverStatusPanel.razor` is referenced by no page or component** — the Reconnect/Restart buttons are unreachable in the running AdminUI (pre-existing on `master`, not caused by this program) | 1A.3 gate | Node-side leg driven with real envelopes | Out of program scope — file separately if the control is meant to exist |
|
| **`DriverStatusPanel.razor` is referenced by no page or component** — the Reconnect/Restart buttons are unreachable in the running AdminUI (pre-existing on `master`, not caused by this program; re-verified in run 2: 0 `<DriverStatusPanel` usages, no Restart/Reconnect control on any rendered page, no such HTTP endpoint) | 1A.3 gate | Check 3 **closed by composition**: run 1's live node-side leg for both commands + run 2's check 4 driving the identical `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path from the AdminUI | Out of program scope — file separately if the control is meant to exist |
|
||||||
|
| `Failed to resolve` Error text reads as permanent though first use retries, and will trip a soak error-grep on a healthy node | 1A.3 gate run 2 (defect 5) | Self-heal proven live | Phase 2 |
|
||||||
|
|
||||||
## Issues filed by this program
|
## Issues filed by this program
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user