diff --git a/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md b/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md index 9dc56776..d1811c24 100644 --- a/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md +++ b/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md @@ -154,22 +154,217 @@ Rig left running on `ClusterClient` (its normal state) after the gate. --- -## Phase 1A — central→node command plane — **PENDING** +## Phase 1A — central→node command plane — **FAIL (blocked at the central boot boundary)** -Branch `feat/grpc-node-command` @ `________`. Rig-only flip: central-1/-2 → -`MeshTransport__Mode=Grpc`; **site nodes remain `ClusterClient`** (coexistence is the point). -Flip reverted after the gate, never committed. +Branch `feat/grpc-node-command` @ `fbf7b708`. Image rebuilt from this worktree +(`/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a`) so it carries this branch; +`docker compose -f docker-dev/docker-compose.yml up -d --build` exited 0 and **all eight +containers were recreated** — `migrator` and `cluster-seed` re-ran to completion (exit 0), the +six host nodes and traefik were recreated, the `sql` container and its named volume were kept. +Nothing else touched the stack afterwards, so this run is not a concurrent-rebuild-invalidated +one. Rig-only flip: central-1/-2 `MeshTransport__Mode: "Grpc"` (hard value); **site nodes left +on the compose default `ClusterClient`** (coexistence is the point). Three Serilog +`MinimumLevel:Override` Debug scopes were added on all six nodes for this run only +(`ZB.MOM.WW.OtOpcUa.Host.Grpc`, `ZB.MOM.WW.OtOpcUa.ControlPlane.Communication`, +`Akka.Cluster.Tools.Client`) — the accept/dispatch lines the checks read are `Debug` and the +rig's default Serilog level is `Information`, so without them checks 1/3 would have had no +observable signal. Flip and overrides reverted after the gate, never committed. + +**Verdict: FAIL.** Neither central node boots on this branch, so nothing that originates at +central could be driven. Root cause is defect 4 below — resolving `IMeshCommandDialer` / +`IMeshPskProvider` inside the first `WithActors` starter drags in `ISecretResolver` → +`IAuditWriter` → `ActorAuditWriter`, whose constructor reads `AuditWriterActorKey` out of the +`ActorRegistry` before the singleton that registers it has run. It is **mode-independent** +(reproduced on the same image with `MeshTransport__Mode=ClusterClient`), so it is a defect of +the branch, not of the gate's flip, and it would have failed a `ClusterClient` rig equally. | # | Check | Result | |---|---|---| -| 1 | Deploy dispatch from AdminUI reaches **all six** driver nodes over gRPC (per-node `Execute` log lines on every node; zero `SendToAll` lines on central) | PENDING | -| 2 | Acks still return over ClusterClient (site mode unchanged) and the deployment **seals** — mixed-transport deploy proven end to end | PENDING | -| 3 | `RestartDriver` + `ReconnectDriver` from AdminUI land on the target site node (EventStream delivery observed via driver restart in logs) | PENDING | -| 4 | `AlarmCommand` (ack a scripted alarm from AdminUI) lands and the condition acknowledges | PENDING | -| 5 | Unauthenticated `grpcurl` to a site node `:4056` `mesh_transport.v1.NodeCommandService/Execute` ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | PENDING | -| 6 | **Wrong-key isolation:** site-b's PSK presented to site-a ⇒ `PermissionDenied` (per-cluster scoping is real, not decorative) | PENDING | -| 7 | A node with NULL `GrpcPort` is skipped with the warn-once, fan continues to the others | PENDING | -| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures on their interceptors across the run | PENDING | +| 1 | Deploy dispatch from AdminUI reaches **all six** driver nodes over gRPC (per-node `Execute` log lines on every node; zero `SendToAll` lines on central) | **BLOCKED** — `POST /api/deployments` is unreachable: traefik answers `503` because neither central node has a live HTTP surface (`curl http://central-1:9000/health/ready` and `central-2` both return `000`, connection refused). Nothing dispatched. | +| 2 | Acks still return over ClusterClient (site mode unchanged) and the deployment **seals** — mixed-transport deploy proven end to end | **BLOCKED** — no deployment could be dispatched (check 1). The most recent `dbo.Deployment` row is still the pre-gate `03540DA5-…` (`Status=2`, Sealed, `2026-08-06 06:11:22`), i.e. nothing was attempted during this run. | +| 3 | `RestartDriver` + `ReconnectDriver` from AdminUI land on the target site node (EventStream delivery observed via driver restart in logs) | **PARTIAL PASS (node-side leg only)** — both commands were delivered to site-a-1 over real Kestrel h2c with the correct PSK and both acted: `DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate` followed by a fresh `spawned Modbus driver SITEA-modbus`, and `reconnecting driver … ForceReconnect requested by admin; re-entering Reconnecting`. The **AdminUI half was not driven** — see the two carry-forwards below (central down; and the UI control is unreachable on this branch anyway). | +| 4 | `AlarmCommand` (ack a scripted alarm from AdminUI) lands and the condition acknowledges | **PARTIAL PASS (envelope leg only)** — the alarm arm decoded and was delivered node-side (`Mesh: accepted AlarmCommand (correlationId=39ba4511…) over gRPC`), which is the arm with no CLR correlation member and therefore the one worth wiring live. **The acknowledge itself was NOT observed**: the rig's single `dbo.ScriptedAlarm` row (`gate481-alarm`, equipment `EQ-6e09bcc2d907`) is not owned by SITE-A, and its owning node is a central one, which is down. | +| 5 | Unauthenticated `grpcurl` to a site node `:4056` `mesh_transport.v1.NodeCommandService/Execute` ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | **PASS** — 3/3 probes graded as designed: no bearer ⇒ `PermissionDenied`, correct `dev-mesh-psk-SITE-A` + empty envelope ⇒ `InvalidArgument` (**auth passed**, the decode is what failed), and a full `RestartDriver` envelope with the same key ⇒ `{}` + observed driver restart. | +| 6 | **Wrong-key isolation:** site-b's PSK presented to site-a ⇒ `PermissionDenied` (per-cluster scoping is real, not decorative) | **PASS** — both directions rejected: `dev-mesh-psk-SITE-B` → site-a-1 ⇒ `PermissionDenied`, `dev-mesh-psk-SITE-A` → site-b-1 ⇒ `PermissionDenied`, while `dev-mesh-psk-SITE-B` → site-b-1 reaches the decoder (`InvalidArgument`) — so the rejection is the key, not the reachability. | +| 7 | A node with NULL `GrpcPort` is skipped with the warn-once, fan continues to the others | **BLOCKED** — the skip is computed by `CentralCommunicationActor.AdoptGrpcTargets` on a contact refresh; no central node is running to refresh. The rig's `ClusterNode` rows were left untouched (all six still `GrpcPort=4056`) rather than mutated for a check that could not be observed. | +| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures on their interceptors across the run | **PARTIAL PASS** — 0 `Rejected a telemetry stream call` / `Rejected a config-serve call` / `Rejected a LocalDb sync call` lines across all six nodes, and the only `Rejected a mesh command call` lines are the 3 deliberate negative probes (2 on site-a-1, 1 on site-b-1). **But telemetry and config-serve carried no traffic at all** in this run — central dials both and never booted (site-a-1 logs 0 telemetry lines) — so "unaffected" is established for LocalDb sync only. | + +### Phase 1A evidence + +**Provenance** — flip in effect on central only, sites untouched: + +``` +$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \ + --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done +central-1 MeshTransport__Mode=Grpc +central-2 MeshTransport__Mode=Grpc +site-a-1 MeshTransport__Mode=ClusterClient +site-b-1 MeshTransport__Mode=ClusterClient + +$ docker images --format '{{.Repository}}:{{.Tag}}|{{.ID}}|{{.CreatedAt}}' | grep otopcua-host +otopcua-host:dev|69360a173f1f|2026-08-06 07:20:14 -0400 EDT # rebuilt for this run +``` + +**Defect 4 — the boot failure (checks 1/2/7 blocked by it).** Both central nodes, identical: + +``` +$ docker logs otopcua-dev-central-1-1 2>&1 | sed -n '25,45p' +[11:21:27 INF] OPC UA server started on opc.tcp://0.0.0.0:4840 +[11:21:27 FTL] Unable to start AkkaHostedService - shutting down application +Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key + ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey + at Akka.Hosting.ActorRegistry.Get[TKey]() + at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry) + in /src/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Audit/ActorAuditWriter.cs:line 24 + ... + at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.b__0_5(IServiceProvider sp) + ... + at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.b__0_6(IServiceProvider sp) + ... + at ZB.MOM.WW.OtOpcUa.ControlPlane.ServiceCollectionExtensions.<>c.b__7_0( + ActorSystem system, IActorRegistry registry, IDependencyResolver resolver) + in /src/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs:line 74 + at Akka.Hosting.AkkaConfigurationBuilder.StartAsync(ActorSystem sys) + at Akka.Hosting.AkkaHostedService.StartAsync(CancellationToken cancellationToken) +``` + +**Mode-independence, proven rather than argued** — same image, central-1's exact environment +with `MeshTransport__Mode=ClusterClient`, own hostname + self-seed so it cannot disturb the rig: + +``` +$ docker run -d --name scratch-cc-mode --network otopcua-dev_default --env-file c1-env.txt \ + -e Cluster__PublicHostname=scratch-cc-mode \ + -e Cluster__SeedNodes__0=akka.tcp://otopcua@scratch-cc-mode:4053 otopcua-host:dev +$ docker logs scratch-cc-mode 2>&1 | grep -m4 -E "MissingActorRegistry|Hosting failed|Host starting" +[11:28:45 INF] OtOpcUa.Host starting with roles=[admin,driver,cluster-main] (admin=True, driver=True) +Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key + ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey +[11:28:47 ERR] Hosting failed to start +``` + +Scratch container removed after capture (`docker rm -f scratch-cc-mode`); the six real nodes +were untouched by it. + +**It presents as a healthy container.** Worth carrying into Phase 2/4 readings — the failed +process does not exit, so `docker ps` shows `Up`, `RestartCount=0`, and only an HTTP probe or +the log tells the truth: + +``` +$ docker inspect otopcua-dev-central-1-1 -f '{{.State.Status}} restarts={{.RestartCount}}' +running restarts=0 +$ docker run --rm --network otopcua-dev_default curlimages/curl -s -o /dev/null \ + -w '%{http_code}\n' --max-time 5 http://central-1:9000/health/ready +000 +$ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9200/health/ready +503 +``` + +**Checks 5 + 6 — the wire-probe auth matrix** (`-import-path` onto the in-repo proto; the node +has no reflection). An empty `CommandEnvelope` has no `oneof` arm set, so `FromEnvelope` throws +and the service answers `InvalidArgument` — **after** the interceptor ran, which is exactly why +it is the positive signal here: `PermissionDenied` means the key was refused, `InvalidArgument` +means the key was accepted and only the payload was rejected: + +``` +$ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos +$ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \ + -plaintext -import-path /protos -proto mesh_transport.proto" + +# 5a no bearer -> site-a-1 +Code: PermissionDenied Message: Node command authentication failed. +# 5b -H 'authorization: Bearer dev-mesh-psk-SITE-A' -> site-a-1 +Code: InvalidArgument Message: Command envelope could not be decoded. +# 6a -H 'authorization: Bearer dev-mesh-psk-SITE-B' -> site-a-1 +Code: PermissionDenied Message: Node command authentication failed. +# 6b -H 'authorization: Bearer dev-mesh-psk-SITE-A' -> site-b-1 +Code: PermissionDenied Message: Node command authentication failed. +# 6c -H 'authorization: Bearer dev-mesh-psk-SITE-B' -> site-b-1 (positive control) +Code: InvalidArgument Message: Command envelope could not be decoded. +``` + +**Checks 3 + 4 — the node-side leg, driven with real envelopes** over the same authenticated +h2c path central would use. This is the first live proof that `MeshCommandAuthInterceptor` is +actually invoked on a real Kestrel pipeline (register item 4) and that +`NodeCommandGrpcService` → `NodeCommunicationActor` → EventStream → `DriverHostActor` works +end to end: + +``` +$ $G -H 'authorization: Bearer dev-mesh-psk-SITE-A' -H 'x-otopcua-cluster: SITE-A' \ + -d '{"correlationId":"8461008e...","restartDriver":{"clusterId":"SITE-A", + "driverInstanceId":"SITEA-modbus","actorByUserName":"1a-gate", + "correlationId":"8461008e..."}}' \ + site-a-1:4056 mesh_transport.v1.NodeCommandService/Execute +{} + +$ docker logs otopcua-dev-site-a-1-1 2>&1 | tail +[11:31:01 DBG] Mesh: accepted RestartDriver (correlationId=8461008e37654724a292219202d78dfa) over gRPC; handed to the node communication actor. +[11:31:01 INF] DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate +[11:31:01 INF] DriverHost site-a-1:4053: spawned Modbus driver SITEA-modbus (stub=False) +[11:31:55 DBG] Mesh: accepted ReconnectDriver (correlationId=29c5dc77283f4eb7869d0c7d6686cadd) over gRPC; handed to the node communication actor. +[11:31:55 INF] DriverHost site-a-1:4053: reconnecting driver SITEA-modbus by request of 1a-gate +[11:31:55 INF] DriverInstance SITEA-modbus: ForceReconnect requested by admin; re-entering Reconnecting +[11:31:56 DBG] Mesh: accepted AlarmCommand (correlationId=39ba451173954229a71b9dd1e00ac1b9) over gRPC; handed to the node communication actor. +``` + +**Check 8 — interceptor denial counts across every node, whole run:** + +``` +$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done +central-1 telemetry=0 configserve=0 localdbsync=0 mesh=0 +central-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 +site-a-1 telemetry=0 configserve=0 localdbsync=0 mesh=2 # the two deliberate negatives +site-a-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 +site-b-1 telemetry=0 configserve=0 localdbsync=0 mesh=1 # the one deliberate negative +site-b-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 +``` + +The `TwoNodeClusterHarness` I-1 null-provider Error (a known test-harness artifact) did **not** +appear in any rig log — `grep -c "IMeshPskProvider"` = 0 on all six nodes, as expected for +production wiring where every admin node registers the provider. + +### Phase 1A defect + +**Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter.** +`WithOtOpcUaControlPlaneSingletons`'s `WithActors` block resolves `IMeshCommandDialer` and +`IMeshPskProvider` from `IServiceProvider` to hand to `CentralCommunicationActor.Props`. +`MeshGrpcCommandDialer` takes `IMeshPskProvider`, which takes `ISecretResolver`, which +`AddZbSecrets` builds by resolving `IAuditWriter`, which on an admin node is +`ActorAuditWriter` — and its constructor does `registry.Get()`. That +singleton is registered ~60 lines later in the same builder, so at this point the key is +absent and `ActorRegistry.Get` throws `MissingActorRegistryEntryException`, failing +`AkkaHostedService.StartAsync` and the whole host. + +Three things make this worth a pinned regression rather than a one-line fix note: + +- **It is not mode-gated.** The two `GetService` calls run in every transport mode by design + (so a mode flip needs no restart), so the eager graph is walked on every admin boot. Proven + above with a `ClusterClient` scratch container on the same image. +- **It is invisible to the suite.** The 1A.2 container-activation test + (`It_activates_from_the_container_despite_the_optional_client_factory_parameter`) resolves + the dialer from a service collection that does not include `AddZbSecrets`, and + `TwoNodeClusterHarness` registers no `IMeshPskProvider` at all — so no existing test walks + the real graph in starter order. This is the same class as scadaproj#1 ("only visible with + the DI graph built in a real host") and as mesh Phase 6's `IClusterRoleInfo` boot crash. +- **The container lies about it.** The process does not exit, so `docker ps` reports `Up` with + `restarts=0` while the node serves no HTTP at all. Any Phase 2/4 readiness check that reads + container status rather than `/health/ready` would score this as healthy. + +The shape of the fix is the one this file already uses two lines above for the coordinator +proxy — hand the actor a lazy accessor (`Func` / `Func`, +or resolve on first use) so nothing is constructed while the `ActorSystem` is still building — +but the fix and its red-first pin belong to 1A, not to this record. + +### Rig state after the gate + +The compose flip was reverted (`git checkout docker-dev/docker-compose.yml`) and the six nodes +recreated on the committed all-`ClusterClient` compose. Because defect 4 is mode-independent, +the branch image cannot produce a healthy central pair on *any* mode, so the rig was returned +to its pre-gate image rather than left with a dead central pair: the 1A build was preserved as +`otopcua-host:grpc-1a` (`69360a173f1f`) and the pre-gate build (`8bd81d8aea9b`, 04:52) re-tagged +back to `otopcua-host:dev`. **Anyone re-running this gate must rebuild — the running rig is +NOT on the branch under test.** Verified after restore: six nodes `running restarts=0`, zero +`Hosting failed to start` on all six, `/health/ready` 200 on every node, MAIN mesh +`memberCount=2 unreachableCount=0`, AdminUI 200 on `http://localhost:9200`. --- @@ -228,8 +423,19 @@ checks 6–7. ### Carry-forwards Anything a phase gate could not drive gets named here with why + what proves it instead, and -is re-checked in the next gate (the ScadaBridge proportionate-gate rule). Currently: none. +is re-checked in the next gate (the ScadaBridge proportionate-gate rule). + +| Item | Deferred from | Why it could not be driven | Proven instead by | Re-check in | +|---|---|---|---|---| +| Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-`GrpcPort` skip | 1A | Both central nodes fail to boot (defect 4); no AdminUI, no deploy API, no contact refresh | Nothing live. Unit coverage only: the fan Theory, the gate-⊋-fan fact, `A_slow_dial_does_not_park_the_actors_mailbox` | **1A gate re-run after defect 4 is fixed** — these are the checks 1A exists to prove and must not be carried past it | +| Check 3 central half — `RestartDriver`/`ReconnectDriver` **originating at the AdminUI** | 1A | (a) central down; (b) independently, `DriverStatusPanel.razor` — the only component holding the Reconnect/Restart buttons — is referenced by **no page or component in the AdminUI**, so the control is unreachable in the running UI on this branch and on `master` | The node-side leg driven live with real envelopes (check 3 evidence); central's encode covered by the T0.2 mapper goldens | 1A re-run; the orphaned panel is pre-existing and out of this program's scope — file separately if it is to be fixed | +| Check 4 acknowledge half — the condition actually acknowledging | 1A | The rig's one `ScriptedAlarm` (`gate481-alarm`) is not owned by a SITE cluster, and its owning central node is down | The alarm arm decoding + delivering node-side (check 4 evidence) | 1A re-run, seeding a SITE-A-owned scripted alarm first | +| Check 8 for telemetry + config-serve | 1A | Central dials both planes and never booted, so neither carried a single call — 0 denials is vacuous for them | LocalDb sync (pair-local) did carry traffic with 0 denials | 1A re-run | ### Defects the gates caught that the suite did not Recorded per phase as they occur, with root cause + the pinned regression test. + +| # | Phase | Defect | Root cause | Status | +|---|---|---|---|---| +| 4 | 1A gate | **An admin node cannot boot at all** — `MissingActorRegistryEntryException: AuditWriterActorKey`, both central nodes, every transport mode | The first `WithActors` starter eagerly resolves `IMeshCommandDialer`/`IMeshPskProvider`, whose graph reaches `ActorAuditWriter`, whose ctor reads an `ActorRegistry` key registered ~60 lines later in the same builder | **OPEN — blocks 1A.** Full analysis + evidence in the Phase 1A section above. No product change was made by this gate | diff --git a/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md b/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md index 591b0142..89709602 100644 --- a/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md +++ b/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md @@ -23,7 +23,7 @@ posture). | Phase | Scope | Branch | Status | PR / merge SHA | Live gate | |---|---|---|---|---|---| | 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **MERGED to `master`** @ `10b6d209` (2026-08-06) — all 6 tasks complete @ `23555ed0`; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed | `10b6d209` | **PASS 4/4** | -| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`)** — all reviews spec ✅ + code ✅ (1A.2 on Opus, fix loop closed); 1A.3 gate next | — | PENDING | +| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **1A.3 gate RAN 2026-08-06 @ `fbf7b708` and FAILED — defect 4 blocks the phase.** Both central nodes fail to boot (mode-independent), so checks 1/2/7 could not be driven at all and 3/4 only node-side; 5/6 PASS. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅. **Next: fix defect 4 with a red-first pin, then RE-RUN the 1A gate** | — | **FAIL** (5 of 8 not provable; 2 PASS, 2 partial, 3 blocked) | | 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebases on 1A | `feat/grpc-central-ack` | **1B.1 ✅ `14dd07d2` + 1B.2 ✅ `a5a789d4` (+fixes `e54711a2`/`4d607575`)** — all reviews ✅ (1B.2 on Opus: 1 Critical found+fixed, see defect log); awaiting 1A merge → rebase → 1B.3 gate | — | PENDING | | 2 | Baseline reproduction on pre-flip build, then full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills | (rig commit) | NOT STARTED | — | PENDING | | 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — | @@ -129,6 +129,7 @@ Defects found during execution (live-gate or review), with root cause + pinned t |---|---|---|---|---| | 1 | 1B.2 | **Failover silently defeated on a hanging central** (CRITICAL, caught by Opus code review): the ack deadline was one absolute `DateTime` shared across the endpoint walk — a hung (not refused) central-1 burned the whole budget and central-2 was dialed with an expired deadline | deadline hoisted outside the `foreach`; a kill-style drill can't see it (refused = fast-fail), only a hang discriminates | `e54711a2`; pinned by `Each_endpoint_gets_a_FRESH_deadline` (mutation-verified); 1B.3 gains a `docker pause` hang drill | | 2 | 1A.2 | **Dark-switch rollback deactivated the known-cluster gate** (Opus review, I-2): gate published only under `Mode=Grpc` while the interceptor + service are port-mapped in every mode — a rollback to ClusterClient/Dps would silently restore the timing oracle + pre-auth flood | gate activation coupled to transport mode instead of the endpoint it protects | `48dc2120` — gate follows the endpoint; pinned by the every-mode Theory + Dps guard pin `86633f14` | +| 4 | 1A.3 (live gate) | **An admin node cannot boot** (BLOCKS 1A): both central nodes die at startup with `MissingActorRegistryEntryException: ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey`, taking `AkkaHostedService` and the whole host with them. Mode-INDEPENDENT — reproduced on the same image with `MeshTransport__Mode=ClusterClient`, so it is not the gate's flip. Driver-only nodes are unaffected (they register neither `IMeshPskProvider` nor `ActorAuditWriter`) | `WithOtOpcUaControlPlaneSingletons`'s first `WithActors` starter eagerly resolves `IMeshCommandDialer` + `IMeshPskProvider` to pass into `CentralCommunicationActor.Props`. That graph is dialer → `IMeshPskProvider` → `ISecretResolver` → (`AddZbSecrets`) → `IAuditWriter` → `ActorAuditWriter`, whose ctor calls `registry.Get()` — and `WithSingleton` is registered ~60 lines later in the same builder. Same class as scadaproj#1 and mesh Phase 6's `IClusterRoleInfo` crash: only visible with the DI graph built in a real host, which is why the container-activation test (no `AddZbSecrets` in its collection) and `TwoNodeClusterHarness` (no provider at all) both miss it. Aggravating: the failed process does not exit, so `docker ps` reports `Up restarts=0` while the node serves no HTTP | **OPEN.** Fix + red-first pin belong to 1A (the gate made no product change). Suggested shape: hand the actor lazy accessors, mirroring the `Func<>` the same block already uses for the coordinator proxy. Pin must build the graph in starter order (a Host.IntegrationTests boot of an admin-role host), not merely activate the dialer | | 3 | 1A.2 | **Fan parked the actor mailbox** (self-caught by the implementer): dialing from the receive ran `TryGetPsk`'s cold-cache 3 s block pre-await (cold after EVERY refresh due to `InvalidateAll`), stalling deployment acks behind the fan | async methods run synchronously to the first await | `Task.Run` hop (token None — a canceled scheduling would read as a DB failure via PipeTo); pinned red-first by `A_slow_dial_does_not_park_the_actors_mailbox` | ## Carry-forward register @@ -140,15 +141,18 @@ Checks a gate could not drive, named + carried to the next gate (proportionate-g | Hang-style central failover drill (`docker pause`, not `docker stop` — only a hang discriminates the per-endpoint-deadline fix) | 1B.2 review | unit fact `Each_endpoint_gets_a_FRESH_deadline` | 1B.3 gate (MUST include) | | I-4 measurement: ack burst within ~3 s after a contact-refresh tick (cold PSK cache on the inbound auth path) | 1A.2 review | bounded 3 s store read + `Lazy` stampede suppression | 1B.3 gate (observe) | | `TwoNodeClusterHarness` logs the I-1 null-provider Error at every spawn (harness registers no `IMeshPskProvider`) — EXPECTED, not a rig fault | 1A.2 re-review | production pairing is `hasAdmin`-unconditional in one block | note for 1B.3/Phase 2 readers | -| Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter` | Phase 2 (or 1B.3) | +| Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter` | Phase 2 (or 1B.3) — note defect 4 shows the missing pin is not merely role-scoping but **starter-order**: the container-activation test resolves from a collection without `AddZbSecrets` | +| 1A checks 1/2/7 (deploy fan over gRPC, mixed-transport seal, NULL-`GrpcPort` skip) and the central halves of 3/4 | 1A.3 gate | Nothing live — unit coverage only | **1A gate re-run after defect 4**; must not be carried past 1A | +| Check 8 is vacuous for telemetry + config-serve — central never booted, so neither plane carried a call | 1A.3 gate | LocalDb sync did carry traffic, 0 denials | 1A gate re-run | +| **`DriverStatusPanel.razor` is referenced by no page or component** — the Reconnect/Restart buttons are unreachable in the running AdminUI (pre-existing on `master`, not caused by this program) | 1A.3 gate | Node-side leg driven with real envelopes | Out of program scope — file separately if the control is meant to exist | ## Issues filed by this program | Issue | What | Status | |---|---|---| | [lmxopcua#542](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/542) | `FleetStatusBroadcaster` derives "fleet" from central's own gossip ring — silently central-pair-blind since mesh Phase 6; pre-existing, transport-independent, out of this program's scope | FILED 2026-08-06 | -| *(to file in 1A.3)* | `ConfigPublishCoordinatorTimeoutTests.DeadlineElapsed_for_current_deployment_marks_TimedOut` is racy on its own terms (150 ms wall-clock deadline; row inserted after spawn) — flaked 2/~11 full runs once 1A.2 grew the suite; green at base. Fix the test (inject clock / raise deadline) | to file | -| *(to file in 1A.3)* | Mesh command/ack planes are h2c-only — the per-cluster PSK crosses the network in cleartext on a write surface (precedent: every sibling gRPC plane; trusted-site-link assumption). Decide: accepted risk, or a `UseTls`/scheme option | to file | +| [lmxopcua#544](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/544) | `ConfigPublishCoordinatorTimeoutTests.DeadlineElapsed_for_current_deployment_marks_TimedOut` is racy on its own terms (150 ms wall-clock deadline; row inserted after spawn) — flaked 2/~11 full runs once 1A.2 grew the suite; green at base. Fix the test (inject clock / raise deadline) | FILED 2026-08-06 (1A.3) | +| [lmxopcua#545](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/545) | Mesh command/ack planes are h2c-only — the per-cluster PSK crosses the network in cleartext on a write surface (precedent: every sibling gRPC plane; trusted-site-link assumption). Decide: accepted risk, or a `UseTls`/scheme option | FILED 2026-08-06 (1A.3) | | [lmxopcua#543](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/543) | The three pre-existing auth interceptors (`TelemetryStream`/`ConfigServe`/`LocalDbSync`) treat a whitespace-only key as SET (`IsNullOrEmpty`), so a whitespace `Telemetry:ApiKey` etc. opens the endpoint to a whitespace bearer; the two new mesh interceptors use `IsNullOrWhiteSpace`. Backport is a behaviour change to shipped endpoints — own issue, out of this program's scope | FILED 2026-08-06 | ## Known constraints carried from the research (do not rediscover)