diff --git a/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md b/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md index d1811c24..e18b860b 100644 --- a/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md +++ b/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md @@ -154,7 +154,270 @@ Rig left running on `ClusterClient` (its normal state) after the gate. --- -## Phase 1A — central→node command plane — **FAIL (blocked at the central boot boundary)** +## Phase 1A — central→node command plane — **PASS (on the re-run at `e54883a8`)** + +Two runs. **Run 1 @ `fbf7b708` FAILED** at the central boot boundary (defect 4) and could not +drive checks 1/2/7 or the central halves of 3/4; its checks 5 + 6 and the node-side halves of +3 + 4 stand as recorded and were not repeated. **Run 2 @ `e54883a8`** — after the boot fix +(`c0475f2d`) and its review-hardening follow-up (`e54883a8`) — drove every carried check. The +result table below is the **combined, current** verdict; run 1's failure record is preserved +underneath it because the defect it caught is the most valuable thing this gate produced. + +| # | Check | Verdict | Run | +|---|---|---|---| +| — | **Boot:** all six nodes healthy INCLUDING the central pair, on the branch image | **PASS** — 6/6 `running restarts=0`, 0 `Hosting failed to start`, `/health/ready` 200 ×6, MAIN mesh `memberCount=2 unreachableCount=0`. **But the `Resolve` Error line the fix logs DID fire — see finding 5** | 2 | +| 1 | Deploy dispatch reaches **all six** driver nodes over gRPC; zero `SendToAll`/ClusterClient on central | **PASS** — 6/6 nodes accepted one `DispatchDeployment` under a single shared `correlationId=cf499b42…`; central's own per-node outcome log records 6 accepted / 0 failed; `grep -ci clusterclient` = **0** on both central nodes | 2 | +| 2 | Acks return over ClusterClient (site mode unchanged) and the deployment **seals** | **PASS** — deployment `8DF9AAE0…` `Status=2` (Sealed) with all six `NodeDeploymentState` rows `Status=1` (Applied) inside ~90 ms; all four site nodes hold live `Connected to [akka.tcp://…/system/receptionist]` associations, which is their only ack path in 1A | 2 | +| 3 | `RestartDriver` + `ReconnectDriver` land on the target site node | **PASS (composed)** — node-side leg driven live in run 1 (both commands acted on site-a-1). The AdminUI half **cannot be driven and this is now proven, not assumed**: ` 70000`), so a transition was manufactured by +editing its predicate to an always-true form and deploying; the `/alerts` page was already open +when it fired, the Ack button was clicked, and the predicate + hash were restored afterwards +(verified byte-identical by `SELECT`) with a further deploy: + +``` +page open at 2026-08-06T12:20:59.036Z initial: empty window +ROWS: 12:21:00.017 gate481-alarm | PressHigh | EQ-6e09bcc2d907 | ACTIVATED | 750 | system | press high +ROW BUTTONS: ["Ack","Shelve","Shelve (timed)","Unshelve"] +clicking Acknowledge at 2026-08-06T12:21:00.362Z +after click: 12:21:00.557 | gate481-alarm | EQ-6e09bcc2d907 | ACKNOWLEDGED | 750 | multi-role-test | press high + +$ docker logs otopcua-dev-central-2-1 | grep AlarmCommand +[12:21:00 DBG] Mesh: AlarmCommand (correlationId=ccafe3754aca40cfa8aa0a0602c86cbb) accepted by site-b-2:4053 over gRPC +[12:21:00 DBG] … accepted by central-1:4053 … [12:21:00 DBG] … accepted by central-2:4053 … +[12:21:00 DBG] … accepted by site-a-1:4053 … [12:21:00 DBG] … accepted by site-b-1:4053 … +[12:21:00 DBG] … accepted by site-a-2:4053 … +$ for c in …; do grep "Mesh: accepted AlarmCommand" …; done → all six, same correlationId +``` + +One correlation id across all six recipients is the "encode once, before the fan" rule holding +for the one command whose CLR record has no correlation member of its own — the case the design +called out as the reason encoding is hoisted out of the loop. + +**Check 7 — the skip, the warn, and the honest deadline.** + +``` +$ … UPDATE dbo.ClusterNode SET GrpcPort=NULL WHERE NodeId='site-b-2:4053' # 12:23:14 + +# one warn per central node, once each, within one ContactRefreshSeconds: +central-2 [12:23:20 WRN] ClusterNode site-b-2:4053 has no GrpcPort; it exposes no command endpoint + and is skipped in this dispatch set (further skips of this node are silent). Commands + will not reach it until the row carries a port +central-1 [12:23:19 WRN] …identical… +# (each node also logs the TelemetryNodeSource twin — "exposes no telemetry stream … skipped in +# this dial refresh" — expected collateral: both target sets read the same column) + +# dispatch at 12:23:36 — accepted by five, site-b-2 absent: +central-2 [12:23:37 DBG] … (correlationId=88a9e0a2…) accepted by central-2 / site-a-2 / site-b-1 + / site-a-1 / central-1 :4053 over gRPC +# cumulative per-node accept counts after this dispatch — site-b-2 one behind everyone: +central-1 4 central-2 4 site-a-1 4 site-a-2 4 site-b-1 4 site-b-2 3 + +$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='5c8bdc9e-…' → 4 (TimedOut) +$ … SELECT NodeId, Status FROM dbo.NodeDeploymentState … +central-1|1 central-2|1 site-a-1|1 site-a-2|1 site-b-1|1 site-b-2|0 +``` + +Row restored to `4056` and a later dispatch sealed 6/6 (`080EBB0B…`, `Status=2`, every node +`Status=1`). One operational detail worth carrying: **the skip persists for up to a full +`ContactRefreshSeconds` after the row is repaired** — a deploy issued 5 s after the restore +(`DF1C0B74…`) was still skipped and still TimedOut, because the dispatch set is only rebuilt on +the refresh tick. + +**Check 8 — denials, and proof the planes were actually used.** + +``` +$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done +central-1 telemetry=0 configserve=0 localdbsync=0 mesh=0 +central-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 +site-a-1 telemetry=0 configserve=0 localdbsync=0 mesh=0 +site-a-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 +site-b-1 telemetry=0 configserve=0 localdbsync=0 mesh=0 +site-b-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 + +# config-serve carried real, authenticated traffic: +site-a-1 [12:20:59 INF] Fetched + verified artifact for 8d486c6b… from http://central-1:4055 (28313 bytes). +# telemetry likewise, end to end into the UI: +$ for c in …; do grep -c "Telemetry stream opened"; done → 3,4,3,3,3,5 +/hosts → CLUSTER SITE-A … sa-modbus2 SITEA-modbus Modbus HEALTHY 1 host 1 tag 12:30:49 UTC 0 errors +``` + +The five telemetry `failed … reconnecting` warnings in the run are all accounted for: three at +12:15:20 (central dialing nodes that had not finished binding `:4056`) and two at 12:28:50 / +12:29:30 (the deliberate central-2 stop in finding 5). None is an auth failure. + +### Phase 1A finding 5 — the PreStart race loses on real boots, and the memoize is what saves it + +The hardening commit's premise is not hypothetical. On the run-2 boot, **central-1 lost the +race on both dependencies**: + +``` +$ docker logs otopcua-dev-central-1-1 | grep -A3 "Failed to resolve" +[12:15:19 ERR] Failed to resolve IMeshCommandDialer for the mesh command boundary; continuing + without it. The guards below report what that costs +Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key + ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey + at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry) +[12:15:19 ERR] Failed to resolve IMeshPskProvider for the mesh command boundary; …same cause… +``` + +It is not rare: across the three admin boots observed in run 2 (central-1 and central-2 at +12:15, central-2 again at 12:29 after the stop/start), **two of three lost it**. Had the fix +stopped at PreStart-only resolution, those nodes would have held a permanent silent `null` — +no dialer, and a never-published known-cluster gate — and the failure would only have surfaced +the first time the coordinator singleton moved onto such a node. + +**The memoize heals it, verified on the node that actually lost the race.** Run 2's dispatches +were all fanned by central-2 (the singleton host), so central-1's dialer was never exercised +by them. Central-2 was therefore stopped to move the singleton, and a dispatch driven while +central-1 alone held it: + +``` +$ docker compose stop central-2 # 12:28:39 +$ curl … /health/ready → self akka.tcp://otopcua@central-1:4053, members 1 # 12:28:57 +$ curl -X POST /api/deployments -d '{"createdBy":"1a-gate-central1-heal"}' # 12:29:01 +$ docker logs otopcua-dev-central-1-1 | grep -E "accepted by|no command dialer is wired" +[12:29:02 DBG] Mesh: DispatchDeployment (correlationId=ad97d386…) accepted by site-b-1:4053 over gRPC +[12:29:02 DBG] … accepted by site-a-2 … site-b-2 … site-a-1 … central-1 … # 5 live nodes +# no "no command dialer is wired" line, and no further "Failed to resolve" +$ docker compose start central-2 → members 2, unreachable 0, status Healthy # 12:29:51 +``` + +So the boot-time Error is **cosmetically alarming but functionally healed**, and that is now an +observed property rather than a design claim. Two follow-ups for Phase 2 rather than for 1A: +the Error text says "continuing without it", which reads as permanent and should say that the +first use retries; and a `Failed to resolve` line on a healthy node is exactly the kind of +signal a soak's PSK-drift/error grep will trip over. + +**One incidental observation, not a check.** The deployment `RevisionHash` is deterministic +(three separate no-change dispatches all produced `642b9d6f79b9…`), but restoring the alarm +predicate's two text columns did **not** restore the pre-gate hash `6178a8f302aa…` — so the +artifact covers a column the `UPDATE` bumped (`Script.RowVersion` is the obvious candidate). +The rig's configuration content is byte-identical to its pre-gate state; only the hash moved. +Not investigated further — out of this program's scope. + +### Run 1 record — the defect that blocked it Branch `feat/grpc-node-command` @ `fbf7b708`. Image rebuilt from this worktree (`/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a`) so it carries this branch; @@ -170,26 +433,18 @@ on the compose default `ClusterClient`** (coexistence is the point). Three Seril rig's default Serilog level is `Information`, so without them checks 1/3 would have had no observable signal. Flip and overrides reverted after the gate, never committed. -**Verdict: FAIL.** Neither central node boots on this branch, so nothing that originates at -central could be driven. Root cause is defect 4 below — resolving `IMeshCommandDialer` / -`IMeshPskProvider` inside the first `WithActors` starter drags in `ISecretResolver` → +**Run 1 verdict: FAIL.** Neither central node booted, so nothing that originates at central +could be driven: checks 1, 2 and 7 were **BLOCKED** outright, 3 and 4 reached only their +node-side legs, and check 8's zero-denial result was vacuous for telemetry and config-serve +(central dials both and never booted). Checks 5 and 6 **PASSED** and are the run-1 evidence +carried into the combined table above. Root cause was defect 4 — resolving `IMeshCommandDialer` +/ `IMeshPskProvider` inside the first `WithActors` starter drags in `ISecretResolver` → `IAuditWriter` → `ActorAuditWriter`, whose constructor reads `AuditWriterActorKey` out of the -`ActorRegistry` before the singleton that registers it has run. It is **mode-independent** -(reproduced on the same image with `MeshTransport__Mode=ClusterClient`), so it is a defect of +`ActorRegistry` before the singleton that registers it has run. It was **mode-independent** +(reproduced on the same image with `MeshTransport__Mode=ClusterClient`), so it was a defect of the branch, not of the gate's flip, and it would have failed a `ClusterClient` rig equally. -| # | Check | Result | -|---|---|---| -| 1 | Deploy dispatch from AdminUI reaches **all six** driver nodes over gRPC (per-node `Execute` log lines on every node; zero `SendToAll` lines on central) | **BLOCKED** — `POST /api/deployments` is unreachable: traefik answers `503` because neither central node has a live HTTP surface (`curl http://central-1:9000/health/ready` and `central-2` both return `000`, connection refused). Nothing dispatched. | -| 2 | Acks still return over ClusterClient (site mode unchanged) and the deployment **seals** — mixed-transport deploy proven end to end | **BLOCKED** — no deployment could be dispatched (check 1). The most recent `dbo.Deployment` row is still the pre-gate `03540DA5-…` (`Status=2`, Sealed, `2026-08-06 06:11:22`), i.e. nothing was attempted during this run. | -| 3 | `RestartDriver` + `ReconnectDriver` from AdminUI land on the target site node (EventStream delivery observed via driver restart in logs) | **PARTIAL PASS (node-side leg only)** — both commands were delivered to site-a-1 over real Kestrel h2c with the correct PSK and both acted: `DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate` followed by a fresh `spawned Modbus driver SITEA-modbus`, and `reconnecting driver … ForceReconnect requested by admin; re-entering Reconnecting`. The **AdminUI half was not driven** — see the two carry-forwards below (central down; and the UI control is unreachable on this branch anyway). | -| 4 | `AlarmCommand` (ack a scripted alarm from AdminUI) lands and the condition acknowledges | **PARTIAL PASS (envelope leg only)** — the alarm arm decoded and was delivered node-side (`Mesh: accepted AlarmCommand (correlationId=39ba4511…) over gRPC`), which is the arm with no CLR correlation member and therefore the one worth wiring live. **The acknowledge itself was NOT observed**: the rig's single `dbo.ScriptedAlarm` row (`gate481-alarm`, equipment `EQ-6e09bcc2d907`) is not owned by SITE-A, and its owning node is a central one, which is down. | -| 5 | Unauthenticated `grpcurl` to a site node `:4056` `mesh_transport.v1.NodeCommandService/Execute` ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | **PASS** — 3/3 probes graded as designed: no bearer ⇒ `PermissionDenied`, correct `dev-mesh-psk-SITE-A` + empty envelope ⇒ `InvalidArgument` (**auth passed**, the decode is what failed), and a full `RestartDriver` envelope with the same key ⇒ `{}` + observed driver restart. | -| 6 | **Wrong-key isolation:** site-b's PSK presented to site-a ⇒ `PermissionDenied` (per-cluster scoping is real, not decorative) | **PASS** — both directions rejected: `dev-mesh-psk-SITE-B` → site-a-1 ⇒ `PermissionDenied`, `dev-mesh-psk-SITE-A` → site-b-1 ⇒ `PermissionDenied`, while `dev-mesh-psk-SITE-B` → site-b-1 reaches the decoder (`InvalidArgument`) — so the rejection is the key, not the reachability. | -| 7 | A node with NULL `GrpcPort` is skipped with the warn-once, fan continues to the others | **BLOCKED** — the skip is computed by `CentralCommunicationActor.AdoptGrpcTargets` on a contact refresh; no central node is running to refresh. The rig's `ClusterNode` rows were left untouched (all six still `GrpcPort=4056`) rather than mutated for a check that could not be observed. | -| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures on their interceptors across the run | **PARTIAL PASS** — 0 `Rejected a telemetry stream call` / `Rejected a config-serve call` / `Rejected a LocalDb sync call` lines across all six nodes, and the only `Rejected a mesh command call` lines are the 3 deliberate negative probes (2 on site-a-1, 1 on site-b-1). **But telemetry and config-serve carried no traffic at all** in this run — central dials both and never booted (site-a-1 logs 0 telemetry lines) — so "unaffected" is established for LocalDb sync only. | - -### Phase 1A evidence +### Run 1 evidence **Provenance** — flip in effect on central only, sites untouched: @@ -322,7 +577,7 @@ The `TwoNodeClusterHarness` I-1 null-provider Error (a known test-harness artifa appear in any rig log — `grep -c "IMeshPskProvider"` = 0 on all six nodes, as expected for production wiring where every admin node registers the provider. -### Phase 1A defect +### Phase 1A defect — **CLOSED by `c0475f2d` + `e54883a8`, re-gated in run 2** **Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter.** `WithOtOpcUaControlPlaneSingletons`'s `WithActors` block resolves `IMeshCommandDialer` and @@ -351,20 +606,46 @@ Three things make this worth a pinned regression rather than a one-line fix note The shape of the fix is the one this file already uses two lines above for the coordinator proxy — hand the actor a lazy accessor (`Func` / `Func`, -or resolve on first use) so nothing is constructed while the `ActorSystem` is still building — -but the fix and its red-first pin belong to 1A, not to this record. +or resolve on first use) so nothing is constructed while the `ActorSystem` is still building. -### Rig state after the gate +**That is what shipped.** `c0475f2d` moved both dependencies behind `Func<>` invoked in +`PreStart`, pinned by `ControlPlaneStarterOrderTests` (boots a real admin host through the +production starter order and asserts `StartAsync`; verified RED against the eager resolve in +3/3 modes). `e54883a8` then made the resolution memoize-on-first-use rather than PreStart-only, +because `PreStart` runs concurrently with the remaining starters and can still lose the race — +which **run 2 observed happening on 2 of 3 real admin boots**, so that hardening was not +theoretical. Both are re-gated above; see finding 5 for the live self-heal proof. -The compose flip was reverted (`git checkout docker-dev/docker-compose.yml`) and the six nodes -recreated on the committed all-`ClusterClient` compose. Because defect 4 is mode-independent, -the branch image cannot produce a healthy central pair on *any* mode, so the rig was returned -to its pre-gate image rather than left with a dead central pair: the 1A build was preserved as -`otopcua-host:grpc-1a` (`69360a173f1f`) and the pre-gate build (`8bd81d8aea9b`, 04:52) re-tagged -back to `otopcua-host:dev`. **Anyone re-running this gate must rebuild — the running rig is -NOT on the branch under test.** Verified after restore: six nodes `running restarts=0`, zero -`Hosting failed to start` on all six, `/health/ready` 200 on every node, MAIN mesh -`memberCount=2 unreachableCount=0`, AdminUI 200 on `http://localhost:9200`. +### Rig state after run 1 (superseded by the run-2 restoration below) + +The compose flip was reverted and the six nodes recreated on the committed all-`ClusterClient` +compose. Because defect 4 was mode-independent, the branch image could not produce a healthy +central pair on *any* mode, so the rig was returned to its pre-gate image rather than left with +a dead central pair: the run-1 build was preserved as `otopcua-host:grpc-1a` (`69360a173f1f`) +and the pre-gate build (`8bd81d8aea9b`) re-tagged back to `otopcua-host:dev`. + +### Rig state after run 2 — **restored, six healthy, on the branch image** + +The flip was reverted (`git checkout docker-dev/docker-compose.yml`) and the six nodes +recreated on the committed all-`ClusterClient` compose. No image substitution was needed this +time: `otopcua-host:dev` is the run-2 build of the committed source (`1d8452a29018`), and it +boots a healthy central pair, so the rig runs the code under test. The run-1 workaround tag +`otopcua-host:grpc-1a` was deleted to stop anyone reusing a stale image. + +``` +central-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 +central-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 +site-a-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 +site-a-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 +site-b-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 +site-b-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 +$ curl … /health/ready → status Healthy, members 2, unreachable 0 +$ grep -c "Mesh ClusterClient for cluster" → 3 on each central node (MAIN + SITE-A + SITE-B) +``` + +The three ClusterClients reappearing on the rollback is an incidental but useful result: the +dark switch still rolls back cleanly on the fixed branch, which is what Phase 2 will depend on +if its cutover has to be reversed. --- @@ -425,12 +706,13 @@ checks 6–7. Anything a phase gate could not drive gets named here with why + what proves it instead, and is re-checked in the next gate (the ScadaBridge proportionate-gate rule). -| Item | Deferred from | Why it could not be driven | Proven instead by | Re-check in | +| Item | Deferred from | Why it could not be driven | Proven instead by | Status | |---|---|---|---|---| -| Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-`GrpcPort` skip | 1A | Both central nodes fail to boot (defect 4); no AdminUI, no deploy API, no contact refresh | Nothing live. Unit coverage only: the fan Theory, the gate-⊋-fan fact, `A_slow_dial_does_not_park_the_actors_mailbox` | **1A gate re-run after defect 4 is fixed** — these are the checks 1A exists to prove and must not be carried past it | -| Check 3 central half — `RestartDriver`/`ReconnectDriver` **originating at the AdminUI** | 1A | (a) central down; (b) independently, `DriverStatusPanel.razor` — the only component holding the Reconnect/Restart buttons — is referenced by **no page or component in the AdminUI**, so the control is unreachable in the running UI on this branch and on `master` | The node-side leg driven live with real envelopes (check 3 evidence); central's encode covered by the T0.2 mapper goldens | 1A re-run; the orphaned panel is pre-existing and out of this program's scope — file separately if it is to be fixed | -| Check 4 acknowledge half — the condition actually acknowledging | 1A | The rig's one `ScriptedAlarm` (`gate481-alarm`) is not owned by a SITE cluster, and its owning central node is down | The alarm arm decoding + delivering node-side (check 4 evidence) | 1A re-run, seeding a SITE-A-owned scripted alarm first | -| Check 8 for telemetry + config-serve | 1A | Central dials both planes and never booted, so neither carried a single call — 0 denials is vacuous for them | LocalDb sync (pair-local) did carry traffic with 0 denials | 1A re-run | +| Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-`GrpcPort` skip | 1A run 1 | Both central nodes failed to boot (defect 4) | — | **CLOSED in 1A run 2** — all three PASS live | +| Check 4 acknowledge half — the condition actually acknowledging | 1A run 1 | The rig's one `ScriptedAlarm` sits Normal and its owning central node was down | — | **CLOSED in 1A run 2** — acked from the real `/alerts` page, `ACKNOWLEDGED` row observed | +| Check 8 for telemetry + config-serve | 1A run 1 | Central dials both planes and never booted, so neither carried a call — 0 denials was vacuous | — | **CLOSED in 1A run 2** — both planes carried authenticated traffic with 0 denials | +| Check 3 central half — `RestartDriver`/`ReconnectDriver` **originating at the AdminUI** | 1A runs 1 + 2 | **Structurally undrivable, not a rig problem:** `` + `PreStart`, pinned red-first by `ControlPlaneStarterOrderTests`) + `e54883a8` (memoize-on-first-use). Re-gated in run 2: 6/6 nodes boot, 0 `Hosting failed to start` | +| 5 | 1A gate run 2 | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider` with the same `AuditWriterActorKey` cause | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede `AuditWriterActorKey`'s registration | **Not a defect — it is the case `e54883a8` was written for, and the memoize demonstrably heals it.** Proven live by moving the coordinator singleton onto the node that lost the race (stop central-2) and watching it fan a dispatch to all five live nodes with no dialer error. Residual is cosmetic: the Error text and its effect on soak greps → Phase 2 | diff --git a/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md b/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md index 89709602..de08ff80 100644 --- a/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md +++ b/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md @@ -23,7 +23,7 @@ posture). | Phase | Scope | Branch | Status | PR / merge SHA | Live gate | |---|---|---|---|---|---| | 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **MERGED to `master`** @ `10b6d209` (2026-08-06) — all 6 tasks complete @ `23555ed0`; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed | `10b6d209` | **PASS 4/4** | -| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **1A.3 gate RAN 2026-08-06 @ `fbf7b708` and FAILED — defect 4 blocks the phase.** Both central nodes fail to boot (mode-independent), so checks 1/2/7 could not be driven at all and 3/4 only node-side; 5/6 PASS. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅. **Next: fix defect 4 with a red-first pin, then RE-RUN the 1A gate** | — | **FAIL** (5 of 8 not provable; 2 PASS, 2 partial, 3 blocked) | +| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **COMPLETE @ `e54883a8` — gate re-run PASSED 2026-08-06.** Run 1 @ `fbf7b708` FAILED on defect 4 (no admin node could boot, mode-independent); fixed by `c0475f2d` + `e54883a8` and re-gated. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅ | — | **PASS 8/8** (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards) | | 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebases on 1A | `feat/grpc-central-ack` | **1B.1 ✅ `14dd07d2` + 1B.2 ✅ `a5a789d4` (+fixes `e54711a2`/`4d607575`)** — all reviews ✅ (1B.2 on Opus: 1 Critical found+fixed, see defect log); awaiting 1A merge → rebase → 1B.3 gate | — | PENDING | | 2 | Baseline reproduction on pre-flip build, then full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills | (rig commit) | NOT STARTED | — | PENDING | | 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — | @@ -129,7 +129,8 @@ Defects found during execution (live-gate or review), with root cause + pinned t |---|---|---|---|---| | 1 | 1B.2 | **Failover silently defeated on a hanging central** (CRITICAL, caught by Opus code review): the ack deadline was one absolute `DateTime` shared across the endpoint walk — a hung (not refused) central-1 burned the whole budget and central-2 was dialed with an expired deadline | deadline hoisted outside the `foreach`; a kill-style drill can't see it (refused = fast-fail), only a hang discriminates | `e54711a2`; pinned by `Each_endpoint_gets_a_FRESH_deadline` (mutation-verified); 1B.3 gains a `docker pause` hang drill | | 2 | 1A.2 | **Dark-switch rollback deactivated the known-cluster gate** (Opus review, I-2): gate published only under `Mode=Grpc` while the interceptor + service are port-mapped in every mode — a rollback to ClusterClient/Dps would silently restore the timing oracle + pre-auth flood | gate activation coupled to transport mode instead of the endpoint it protects | `48dc2120` — gate follows the endpoint; pinned by the every-mode Theory + Dps guard pin `86633f14` | -| 4 | 1A.3 (live gate) | **An admin node cannot boot** (BLOCKS 1A): both central nodes die at startup with `MissingActorRegistryEntryException: ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey`, taking `AkkaHostedService` and the whole host with them. Mode-INDEPENDENT — reproduced on the same image with `MeshTransport__Mode=ClusterClient`, so it is not the gate's flip. Driver-only nodes are unaffected (they register neither `IMeshPskProvider` nor `ActorAuditWriter`) | `WithOtOpcUaControlPlaneSingletons`'s first `WithActors` starter eagerly resolves `IMeshCommandDialer` + `IMeshPskProvider` to pass into `CentralCommunicationActor.Props`. That graph is dialer → `IMeshPskProvider` → `ISecretResolver` → (`AddZbSecrets`) → `IAuditWriter` → `ActorAuditWriter`, whose ctor calls `registry.Get()` — and `WithSingleton` is registered ~60 lines later in the same builder. Same class as scadaproj#1 and mesh Phase 6's `IClusterRoleInfo` crash: only visible with the DI graph built in a real host, which is why the container-activation test (no `AddZbSecrets` in its collection) and `TwoNodeClusterHarness` (no provider at all) both miss it. Aggravating: the failed process does not exit, so `docker ps` reports `Up restarts=0` while the node serves no HTTP | **OPEN.** Fix + red-first pin belong to 1A (the gate made no product change). Suggested shape: hand the actor lazy accessors, mirroring the `Func<>` the same block already uses for the coordinator proxy. Pin must build the graph in starter order (a Host.IntegrationTests boot of an admin-role host), not merely activate the dialer | +| 5 | 1A.3 (live gate, run 2) | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider`, same `AuditWriterActorKey` cause. NOT a defect: it is exactly the case the review-prescribed hardening was written for | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede starter #4 | **Closed by `e54883a8`, and the heal is PROVEN LIVE** — the coordinator singleton was moved onto the node that lost the race (stop central-2) and it fanned a dispatch to all five live nodes with no dialer error. Residual is cosmetic (Error text reads permanent; will trip a soak error-grep on a healthy node) → Phase 2 | +| 4 | 1A.3 (live gate, run 1) | **An admin node cannot boot** (BLOCKED 1A): both central nodes die at startup with `MissingActorRegistryEntryException: ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey`, taking `AkkaHostedService` and the whole host with them. Mode-INDEPENDENT — reproduced on the same image with `MeshTransport__Mode=ClusterClient`, so it is not the gate's flip. Driver-only nodes are unaffected (they register neither `IMeshPskProvider` nor `ActorAuditWriter`) | `WithOtOpcUaControlPlaneSingletons`'s first `WithActors` starter eagerly resolves `IMeshCommandDialer` + `IMeshPskProvider` to pass into `CentralCommunicationActor.Props`. That graph is dialer → `IMeshPskProvider` → `ISecretResolver` → (`AddZbSecrets`) → `IAuditWriter` → `ActorAuditWriter`, whose ctor calls `registry.Get()` — and `WithSingleton` is registered ~60 lines later in the same builder. Same class as scadaproj#1 and mesh Phase 6's `IClusterRoleInfo` crash: only visible with the DI graph built in a real host, which is why the container-activation test (no `AddZbSecrets` in its collection) and `TwoNodeClusterHarness` (no provider at all) both miss it. Aggravating: the failed process does not exit, so `docker ps` reports `Up restarts=0` while the node serves no HTTP | **OPEN.** Fix + red-first pin belong to 1A (the gate made no product change). Suggested shape: hand the actor lazy accessors, mirroring the `Func<>` the same block already uses for the coordinator proxy. Pin must build the graph in starter order (a Host.IntegrationTests boot of an admin-role host), not merely activate the dialer | | 3 | 1A.2 | **Fan parked the actor mailbox** (self-caught by the implementer): dialing from the receive ran `TryGetPsk`'s cold-cache 3 s block pre-await (cold after EVERY refresh due to `InvalidateAll`), stalling deployment acks behind the fan | async methods run synchronously to the first await | `Task.Run` hop (token None — a canceled scheduling would read as a DB failure via PipeTo); pinned red-first by `A_slow_dial_does_not_park_the_actors_mailbox` | ## Carry-forward register @@ -142,9 +143,10 @@ Checks a gate could not drive, named + carried to the next gate (proportionate-g | I-4 measurement: ack burst within ~3 s after a contact-refresh tick (cold PSK cache on the inbound auth path) | 1A.2 review | bounded 3 s store read + `Lazy` stampede suppression | 1B.3 gate (observe) | | `TwoNodeClusterHarness` logs the I-1 null-provider Error at every spawn (harness registers no `IMeshPskProvider`) — EXPECTED, not a rig fault | 1A.2 re-review | production pairing is `hasAdmin`-unconditional in one block | note for 1B.3/Phase 2 readers | | Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter` | Phase 2 (or 1B.3) — note defect 4 shows the missing pin is not merely role-scoping but **starter-order**: the container-activation test resolves from a collection without `AddZbSecrets` | -| 1A checks 1/2/7 (deploy fan over gRPC, mixed-transport seal, NULL-`GrpcPort` skip) and the central halves of 3/4 | 1A.3 gate | Nothing live — unit coverage only | **1A gate re-run after defect 4**; must not be carried past 1A | -| Check 8 is vacuous for telemetry + config-serve — central never booted, so neither plane carried a call | 1A.3 gate | LocalDb sync did carry traffic, 0 denials | 1A gate re-run | -| **`DriverStatusPanel.razor` is referenced by no page or component** — the Reconnect/Restart buttons are unreachable in the running AdminUI (pre-existing on `master`, not caused by this program) | 1A.3 gate | Node-side leg driven with real envelopes | Out of program scope — file separately if the control is meant to exist | +| 1A checks 1/2/7 (deploy fan over gRPC, mixed-transport seal, NULL-`GrpcPort` skip) and the central half of 4 | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — all driven live and PASS | +| Check 8 is vacuous for telemetry + config-serve — central never booted, so neither plane carried a call | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — both planes carried authenticated traffic (28 KB artifact fetches from `:4055`, telemetry streams open on all six) with 0 denials | +| **`DriverStatusPanel.razor` is referenced by no page or component** — the Reconnect/Restart buttons are unreachable in the running AdminUI (pre-existing on `master`, not caused by this program; re-verified in run 2: 0 `