# ClusterClient → gRPC migration — live gate record Rig: `docker-dev/` (central-1/-2 = `admin,driver,cluster-MAIN`; site-a-1/-2, site-b-1/-2; sql + traefik + migrator + cluster-seed), rebuilt from the branch under test via `docker compose -f docker-dev/docker-compose.yml up -d --build`. Recorded check-by-check in the family's live-gate format. Later phases' sections are appended as those phases land; every check row gets a **PASS**/**FAIL** verdict plus one quantitative justification clause, with fenced evidence (real commands, real output). A run invalidated by a concurrent rig rebuild is **void** — note it and re-run (the ScadaBridge void-run rule). Provenance discipline: every phase section opens with branch + commit SHA + "image rebuilt, all N containers recreated", and Phase 4 additionally verifies the image build timestamp post-dates the deletion merge (a stale image invalidates the boundary checks). Plan: `2026-08-06-clusterclient-to-grpc.md`. Tracking: `2026-08-06-clusterclient-to-grpc-tracking.md`. --- ## Baseline (run on the PRE-change build, before the Phase 2 flip) The two gaps this program closes, **reproduced rather than assumed**: **(a) Unauthenticated Akka remoting crosses the WAN boundary.** From the host: ```bash # A live TCP association from a site container to central's Akka port: docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN' # And the receptionist actually accepting cross-mesh traffic (central log after any AdminUI # driver restart targeting site-a): docker logs otopcua-dev-central-1-1 2>&1 | grep -i "clusterclient\|receptionist" | tail -5 ``` Expected: `OPEN`; receptionist/client lines present. Nothing on this path presents any credential. **(b) Silent drop on an unreachable cluster.** Stop the site-b pair, dispatch a deployment (`POST http://localhost:9200/api/deployments`, `X-Api-Key: docker-dev-deploy-key`), observe: no per-node dispatch error surfaces anywhere; site-b's `NodeDeploymentState` rows sit `Applying` until the 2-minute deadline marks the deployment `TimedOut`/partially failed. The deadline is the only witness. Restart the pair. - [x] Baseline (a) recorded with evidence — **REPRODUCED** 2026-08-06, rig on `master @ 46be4c8d` (post-1B image, all-ClusterClient default, no mode override; `MeshTransport__Mode=ClusterClient` confirmed via `printenv` in the site container) - [x] Baseline (b) recorded with evidence — **REPRODUCED** same run ### Baseline (a) evidence — unauthenticated Akka remoting crosses the boundary ``` $ docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN' OPEN # no credential presented on this path $ docker logs otopcua-dev-central-1-1 | grep -i "clusterclient\|receptionist" | tail -5 [13:12:51 INF] Connected to [akka://otopcua/system/receptionist] [13:12:51 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s) [13:12:53 INF] Connected to [akka://otopcua/system/receptionist] [13:12:54 INF] Connected to [akka.tcp://otopcua@site-a-1:4053/system/receptionist] [13:12:57 INF] Connected to [akka.tcp://otopcua@site-b-1:4053/system/receptionist] ``` ### Baseline (b) evidence — silent drop on an unreachable cluster Site-b pair stopped 13:53:5x UTC; dispatch accepted 13:53:54 (`POST /api/deployments` → `{"outcome":"Accepted","deploymentId":"c6cd0798-…"}`, HTTP 202). ``` # 20 s after dispatch — 4/6 applied within ~15 ms, site-b rows parked, deployment open: NodeDeploymentState: central-1/-2, site-a-1/-2 → Status 1 @ 13:53:54.9xx site-b-1/-2 → Status 0, AppliedAtUtc NULL Deployment c6cd0798 → Status 1 (in progress) # The site-b fan leg is not merely unlogged — it is actively suppressed. The SITE-B # ClusterClient's SendToAll lands in deadLetters at INF, under Akka's sampled generic # dead-letter counter, naming no node and no failure: [13:53:54 INF] Message [DispatchDeployment] wrapped in [$SendToAll] … to [akka://otopcua/deadLetters] was not delivered. [467] dead letters encountered, of which 456 were not logged. … Message content: DispatchDeployment { DeploymentId = c6cd0798…, … } # Dispatch-to-deadline window (13:53:50–13:55:53): ambient Akka association-gating noise for # the downed pair exists (transport chatter, fires with or without a dispatch), but ZERO lines # correlate site-b with the deployment/dispatch/command: $ docker logs --since …09:53:50 --until …09:55:53 otopcua-dev-central-1-1 \ | grep -i "site-b" | grep -icE "deploy|dispatch|command|c6cd0798" 0 # The deadline is the only witness, exactly 2 minutes later: [13:55:54 WRN] Deployment c6cd0798… timed out after 00:02:00 (4/6 acks landed). No ack from: site-b-1:4053, site-b-2:4053. Each is an enabled, non-maintenance ClusterNode row — start the node, or set ClusterNode.MaintenanceMode = 1 … Deployment c6cd0798 → Status 4; site-b rows stay Status 0/NULL (a timed-out deployment is not retro-acked; boot replay serves the latest artifact without resurrecting the dead record). ``` Pair restarted after the drill; site-b-1 back healthy. Note for later log reads on this host: `docker logs --since/--until` parses bare timestamps as HOST-LOCAL time (EDT here, UTC−4) while the app writes UTC — a UTC-shaped `--since` silently returns nothing (it points 4 h into the future). Convert to local first; earlier in this run that mistake made a window look empty. --- ## Phase 0 — contract + auth + config groundwork — **PASS** Branch `feat/grpc-phase0-contract-auth` @ `c2bfad4a`. Image rebuilt, all six containers recreated (`docker compose -f docker-dev/docker-compose.yml up -d --build`, run from the worktree at `/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-phase0` so the image carries this branch). The rig was already running (from the main checkout's compose file) before this run; the rebuild recreated all eight containers (sql kept, migrator + cluster-seed one-shots re-ran, central-1/-2 + site-a-1/-2 + site-b-1/-2 + traefik recreated) under the fixed `otopcua-dev` project name, so this run is not a concurrent-rebuild-invalidated one — nothing else touched the stack afterwards. | # | Check | Result | |---|---|---| | 1 | All six nodes boot on `ClusterClient` with the dark `Grpc` config present (PSKs, endpoints, ClusterPsks map); no validator fires | **PASS** — 6/6 nodes `restarts=0`, `docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}'` reports `running restarts=0` on every node; `docker logs | grep -iE "OptionsValidationException\|MeshTransport.*(is empty\|refuses to start)"` = 0 lines on all six | | 2 | Deliberate misconfig: a scratch container with `MeshTransport__Mode=Grpc` and **no** `GrpcPsk` refuses to boot, error names `MeshTransport:GrpcPsk` + the env form | **PASS** — `Hosting failed to start` + `OptionsValidationException` naming `MeshTransport:GrpcPsk` and `MeshTransport__GrpcPsk`, below | | 3 | Same with empty `CentralGrpcEndpoints` — refuses, names the key | **PASS** — `OptionsValidationException` naming `MeshTransport:CentralGrpcEndpoints` and `MeshTransport__CentralGrpcEndpoints__0`, below | | 4 | Full suite green, 0 warnings, on the branch | **PASS** — `dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo`: 0 failures on a clean run (see evidence below); no `.cs` files changed on this branch since the last full rebuild, so the pre-existing ~868-warning baseline is unchanged (nothing to newly warn about) | Note: the new gRPC services are not yet mapped in Phase 0, so no wire probes here — the interceptor negative/positive matrix is unit-tested and wire-probed in 1A/1B. **Check 1 evidence** — no validator fired on any of the six nodes, all dark `Grpc` keys present: ``` $ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' otopcua-dev-$c-1 done running restarts=0 # central-1 running restarts=0 # central-2 running restarts=0 # site-a-1 running restarts=0 # site-a-2 running restarts=0 # site-b-1 running restarts=0 # site-b-2 $ docker logs otopcua-dev-central-1-1 2>&1 | grep -iE "MeshTransport|ClusterClient" [08:53:26 INF] Mesh ClusterClient for cluster MAIN created with 2 contact point(s) [08:53:26 INF] Mesh ClusterClient for cluster SITE-A created with 2 contact point(s) [08:53:26 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s) $ docker logs otopcua-dev-site-a-1-1 2>&1 | grep -iE "cluster.client|receptionist" [08:53:26 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist] $ docker inspect otopcua-dev-site-a-1-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep MeshTransport MeshTransport__CentralGrpcEndpoints__1=http://central-2:4055 MeshTransport__CentralGrpcEndpoints__0=http://central-1:4055 MeshTransport__Mode=ClusterClient MeshTransport__GrpcPsk=dev-mesh-psk-SITE-A MeshTransport__CentralContactPoints__1=akka.tcp://otopcua@central-2:4053 MeshTransport__CentralContactPoints__0=akka.tcp://otopcua@central-1:4053 $ curl -sf http://localhost:9200/health/ready {"status":"Healthy", ..., "akka":{"status":"Healthy","description":"Akka cluster member status: Up", "data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","selfRoles":["admin","cluster-MAIN","driver"], "memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}}} ``` **Check 2 evidence** — scratch container from `otopcua-host:dev`, site-a-1's real environment with `MeshTransport__GrpcPsk` removed and `Mode` forced to `Grpc`: ``` $ docker run --rm --name scratch-check2-noPsk --network otopcua-dev_default \ --env-file check2-env.txt otopcua-host:dev [08:54:23 ERR] Hosting failed to start Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:GrpcPsk is empty. Under Grpc this per-cluster pre-shared key is the whole authentication boundary of the command plane in both directions — without it this node's inbound command service fails closed and every ack it posts to central is rejected. Supply it via the environment (MeshTransport__GrpcPsk), in production as ${secret:OT-MESH-PSK-}. at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name) ... ``` **Check 3 evidence** — same base env, `GrpcPsk` present, both `CentralGrpcEndpoints` indices unset: ``` $ docker run -d --name scratch-check3-noEndpoints --network otopcua-dev_default \ --env-file check3-env.txt otopcua-host:dev $ docker logs scratch-check3-noEndpoints 2>&1 | grep -A2 "OptionsValidationException" [08:55:55 ERR] Hosting failed to start Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:CentralGrpcEndpoints is empty. Under Grpc a driver node has nowhere to post its deployment ack, and every deployment would time out naming a node that is running perfectly. Supply them via the environment (MeshTransport__CentralGrpcEndpoints__0). ``` Both scratch containers removed after capture (`docker rm -f scratch-check2-noPsk scratch-check3-noEndpoints`); the rig's six real nodes were never affected (still `restarts=0` after both runs). **Check 4 evidence** — full suite, clean re-run (a first run hit one race-condition flake in `RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed`, which the test's own name identifies as exercising a race; re-ran in isolation and it passed, then a full second run passed clean end to end — pre-existing timing sensitivity, not a regression from this branch, which touched only `docker-dev/docker-compose.yml` and `docs/Configuration.md`, no `.cs` files): ``` $ dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo ... Passed! - Failed: 0, Passed: 212, Skipped: 4, Total: 216, Duration: 3m56s - ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.dll (net10.0) $ echo $? 0 ``` Rig left running on `ClusterClient` (its normal state) after the gate. --- ## Phase 1A — central→node command plane — **PASS (on the re-run at `e54883a8`)** Two runs. **Run 1 @ `fbf7b708` FAILED** at the central boot boundary (defect 4) and could not drive checks 1/2/7 or the central halves of 3/4; its checks 5 + 6 and the node-side halves of 3 + 4 stand as recorded and were not repeated. **Run 2 @ `e54883a8`** — after the boot fix (`c0475f2d`) and its review-hardening follow-up (`e54883a8`) — drove every carried check. The result table below is the **combined, current** verdict; run 1's failure record is preserved underneath it because the defect it caught is the most valuable thing this gate produced. | # | Check | Verdict | Run | |---|---|---|---| | — | **Boot:** all six nodes healthy INCLUDING the central pair, on the branch image | **PASS** — 6/6 `running restarts=0`, 0 `Hosting failed to start`, `/health/ready` 200 ×6, MAIN mesh `memberCount=2 unreachableCount=0`. **But the `Resolve` Error line the fix logs DID fire — see finding 5** | 2 | | 1 | Deploy dispatch reaches **all six** driver nodes over gRPC; zero `SendToAll`/ClusterClient on central | **PASS** — 6/6 nodes accepted one `DispatchDeployment` under a single shared `correlationId=cf499b42…`; central's own per-node outcome log records 6 accepted / 0 failed; `grep -ci clusterclient` = **0** on both central nodes | 2 | | 2 | Acks return over ClusterClient (site mode unchanged) and the deployment **seals** | **PASS** — deployment `8DF9AAE0…` `Status=2` (Sealed) with all six `NodeDeploymentState` rows `Status=1` (Applied) inside ~90 ms; all four site nodes hold live `Connected to [akka.tcp://…/system/receptionist]` associations, which is their only ack path in 1A | 2 | | 3 | `RestartDriver` + `ReconnectDriver` land on the target site node | **PASS (composed)** — node-side leg driven live in run 1 (both commands acted on site-a-1). The AdminUI half **cannot be driven and this is now proven, not assumed**: ` 70000`), so a transition was manufactured by editing its predicate to an always-true form and deploying; the `/alerts` page was already open when it fired, the Ack button was clicked, and the predicate + hash were restored afterwards (verified byte-identical by `SELECT`) with a further deploy: ``` page open at 2026-08-06T12:20:59.036Z initial: empty window ROWS: 12:21:00.017 gate481-alarm | PressHigh | EQ-6e09bcc2d907 | ACTIVATED | 750 | system | press high ROW BUTTONS: ["Ack","Shelve","Shelve (timed)","Unshelve"] clicking Acknowledge at 2026-08-06T12:21:00.362Z after click: 12:21:00.557 | gate481-alarm | EQ-6e09bcc2d907 | ACKNOWLEDGED | 750 | multi-role-test | press high $ docker logs otopcua-dev-central-2-1 | grep AlarmCommand [12:21:00 DBG] Mesh: AlarmCommand (correlationId=ccafe3754aca40cfa8aa0a0602c86cbb) accepted by site-b-2:4053 over gRPC [12:21:00 DBG] … accepted by central-1:4053 … [12:21:00 DBG] … accepted by central-2:4053 … [12:21:00 DBG] … accepted by site-a-1:4053 … [12:21:00 DBG] … accepted by site-b-1:4053 … [12:21:00 DBG] … accepted by site-a-2:4053 … $ for c in …; do grep "Mesh: accepted AlarmCommand" …; done → all six, same correlationId ``` One correlation id across all six recipients is the "encode once, before the fan" rule holding for the one command whose CLR record has no correlation member of its own — the case the design called out as the reason encoding is hoisted out of the loop. **Check 7 — the skip, the warn, and the honest deadline.** ``` $ … UPDATE dbo.ClusterNode SET GrpcPort=NULL WHERE NodeId='site-b-2:4053' # 12:23:14 # one warn per central node, once each, within one ContactRefreshSeconds: central-2 [12:23:20 WRN] ClusterNode site-b-2:4053 has no GrpcPort; it exposes no command endpoint and is skipped in this dispatch set (further skips of this node are silent). Commands will not reach it until the row carries a port central-1 [12:23:19 WRN] …identical… # (each node also logs the TelemetryNodeSource twin — "exposes no telemetry stream … skipped in # this dial refresh" — expected collateral: both target sets read the same column) # dispatch at 12:23:36 — accepted by five, site-b-2 absent: central-2 [12:23:37 DBG] … (correlationId=88a9e0a2…) accepted by central-2 / site-a-2 / site-b-1 / site-a-1 / central-1 :4053 over gRPC # cumulative per-node accept counts after this dispatch — site-b-2 one behind everyone: central-1 4 central-2 4 site-a-1 4 site-a-2 4 site-b-1 4 site-b-2 3 $ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='5c8bdc9e-…' → 4 (TimedOut) $ … SELECT NodeId, Status FROM dbo.NodeDeploymentState … central-1|1 central-2|1 site-a-1|1 site-a-2|1 site-b-1|1 site-b-2|0 ``` Row restored to `4056` and a later dispatch sealed 6/6 (`080EBB0B…`, `Status=2`, every node `Status=1`). One operational detail worth carrying: **the skip persists for up to a full `ContactRefreshSeconds` after the row is repaired** — a deploy issued 5 s after the restore (`DF1C0B74…`) was still skipped and still TimedOut, because the dispatch set is only rebuilt on the refresh tick. **Check 8 — denials, and proof the planes were actually used.** ``` $ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done central-1 telemetry=0 configserve=0 localdbsync=0 mesh=0 central-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 site-a-1 telemetry=0 configserve=0 localdbsync=0 mesh=0 site-a-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 site-b-1 telemetry=0 configserve=0 localdbsync=0 mesh=0 site-b-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 # config-serve carried real, authenticated traffic: site-a-1 [12:20:59 INF] Fetched + verified artifact for 8d486c6b… from http://central-1:4055 (28313 bytes). # telemetry likewise, end to end into the UI: $ for c in …; do grep -c "Telemetry stream opened"; done → 3,4,3,3,3,5 /hosts → CLUSTER SITE-A … sa-modbus2 SITEA-modbus Modbus HEALTHY 1 host 1 tag 12:30:49 UTC 0 errors ``` The five telemetry `failed … reconnecting` warnings in the run are all accounted for: three at 12:15:20 (central dialing nodes that had not finished binding `:4056`) and two at 12:28:50 / 12:29:30 (the deliberate central-2 stop in finding 5). None is an auth failure. ### Phase 1A finding 5 — the PreStart race loses on real boots, and the memoize is what saves it The hardening commit's premise is not hypothetical. On the run-2 boot, **central-1 lost the race on both dependencies**: ``` $ docker logs otopcua-dev-central-1-1 | grep -A3 "Failed to resolve" [12:15:19 ERR] Failed to resolve IMeshCommandDialer for the mesh command boundary; continuing without it. The guards below report what that costs Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry) [12:15:19 ERR] Failed to resolve IMeshPskProvider for the mesh command boundary; …same cause… ``` It is not rare: across the three admin boots observed in run 2 (central-1 and central-2 at 12:15, central-2 again at 12:29 after the stop/start), **two of three lost it**. Had the fix stopped at PreStart-only resolution, those nodes would have held a permanent silent `null` — no dialer, and a never-published known-cluster gate — and the failure would only have surfaced the first time the coordinator singleton moved onto such a node. **The memoize heals it, verified on the node that actually lost the race.** Run 2's dispatches were all fanned by central-2 (the singleton host), so central-1's dialer was never exercised by them. Central-2 was therefore stopped to move the singleton, and a dispatch driven while central-1 alone held it: ``` $ docker compose stop central-2 # 12:28:39 $ curl … /health/ready → self akka.tcp://otopcua@central-1:4053, members 1 # 12:28:57 $ curl -X POST /api/deployments -d '{"createdBy":"1a-gate-central1-heal"}' # 12:29:01 $ docker logs otopcua-dev-central-1-1 | grep -E "accepted by|no command dialer is wired" [12:29:02 DBG] Mesh: DispatchDeployment (correlationId=ad97d386…) accepted by site-b-1:4053 over gRPC [12:29:02 DBG] … accepted by site-a-2 … site-b-2 … site-a-1 … central-1 … # 5 live nodes # no "no command dialer is wired" line, and no further "Failed to resolve" $ docker compose start central-2 → members 2, unreachable 0, status Healthy # 12:29:51 ``` So the boot-time Error is **cosmetically alarming but functionally healed**, and that is now an observed property rather than a design claim. Two follow-ups for Phase 2 rather than for 1A: the Error text says "continuing without it", which reads as permanent and should say that the first use retries; and a `Failed to resolve` line on a healthy node is exactly the kind of signal a soak's PSK-drift/error grep will trip over. **One incidental observation, not a check.** The deployment `RevisionHash` is deterministic (three separate no-change dispatches all produced `642b9d6f79b9…`), but restoring the alarm predicate's two text columns did **not** restore the pre-gate hash `6178a8f302aa…` — so the artifact covers a column the `UPDATE` bumped (`Script.RowVersion` is the obvious candidate). The rig's configuration content is byte-identical to its pre-gate state; only the hash moved. Not investigated further — out of this program's scope. ### Run 1 record — the defect that blocked it Branch `feat/grpc-node-command` @ `fbf7b708`. Image rebuilt from this worktree (`/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a`) so it carries this branch; `docker compose -f docker-dev/docker-compose.yml up -d --build` exited 0 and **all eight containers were recreated** — `migrator` and `cluster-seed` re-ran to completion (exit 0), the six host nodes and traefik were recreated, the `sql` container and its named volume were kept. Nothing else touched the stack afterwards, so this run is not a concurrent-rebuild-invalidated one. Rig-only flip: central-1/-2 `MeshTransport__Mode: "Grpc"` (hard value); **site nodes left on the compose default `ClusterClient`** (coexistence is the point). Three Serilog `MinimumLevel:Override` Debug scopes were added on all six nodes for this run only (`ZB.MOM.WW.OtOpcUa.Host.Grpc`, `ZB.MOM.WW.OtOpcUa.ControlPlane.Communication`, `Akka.Cluster.Tools.Client`) — the accept/dispatch lines the checks read are `Debug` and the rig's default Serilog level is `Information`, so without them checks 1/3 would have had no observable signal. Flip and overrides reverted after the gate, never committed. **Run 1 verdict: FAIL.** Neither central node booted, so nothing that originates at central could be driven: checks 1, 2 and 7 were **BLOCKED** outright, 3 and 4 reached only their node-side legs, and check 8's zero-denial result was vacuous for telemetry and config-serve (central dials both and never booted). Checks 5 and 6 **PASSED** and are the run-1 evidence carried into the combined table above. Root cause was defect 4 — resolving `IMeshCommandDialer` / `IMeshPskProvider` inside the first `WithActors` starter drags in `ISecretResolver` → `IAuditWriter` → `ActorAuditWriter`, whose constructor reads `AuditWriterActorKey` out of the `ActorRegistry` before the singleton that registers it has run. It was **mode-independent** (reproduced on the same image with `MeshTransport__Mode=ClusterClient`), so it was a defect of the branch, not of the gate's flip, and it would have failed a `ClusterClient` rig equally. ### Run 1 evidence **Provenance** — flip in effect on central only, sites untouched: ``` $ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \ --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done central-1 MeshTransport__Mode=Grpc central-2 MeshTransport__Mode=Grpc site-a-1 MeshTransport__Mode=ClusterClient site-b-1 MeshTransport__Mode=ClusterClient $ docker images --format '{{.Repository}}:{{.Tag}}|{{.ID}}|{{.CreatedAt}}' | grep otopcua-host otopcua-host:dev|69360a173f1f|2026-08-06 07:20:14 -0400 EDT # rebuilt for this run ``` **Defect 4 — the boot failure (checks 1/2/7 blocked by it).** Both central nodes, identical: ``` $ docker logs otopcua-dev-central-1-1 2>&1 | sed -n '25,45p' [11:21:27 INF] OPC UA server started on opc.tcp://0.0.0.0:4840 [11:21:27 FTL] Unable to start AkkaHostedService - shutting down application Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey at Akka.Hosting.ActorRegistry.Get[TKey]() at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry) in /src/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Audit/ActorAuditWriter.cs:line 24 ... at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.b__0_5(IServiceProvider sp) ... at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.b__0_6(IServiceProvider sp) ... at ZB.MOM.WW.OtOpcUa.ControlPlane.ServiceCollectionExtensions.<>c.b__7_0( ActorSystem system, IActorRegistry registry, IDependencyResolver resolver) in /src/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs:line 74 at Akka.Hosting.AkkaConfigurationBuilder.StartAsync(ActorSystem sys) at Akka.Hosting.AkkaHostedService.StartAsync(CancellationToken cancellationToken) ``` **Mode-independence, proven rather than argued** — same image, central-1's exact environment with `MeshTransport__Mode=ClusterClient`, own hostname + self-seed so it cannot disturb the rig: ``` $ docker run -d --name scratch-cc-mode --network otopcua-dev_default --env-file c1-env.txt \ -e Cluster__PublicHostname=scratch-cc-mode \ -e Cluster__SeedNodes__0=akka.tcp://otopcua@scratch-cc-mode:4053 otopcua-host:dev $ docker logs scratch-cc-mode 2>&1 | grep -m4 -E "MissingActorRegistry|Hosting failed|Host starting" [11:28:45 INF] OtOpcUa.Host starting with roles=[admin,driver,cluster-main] (admin=True, driver=True) Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey [11:28:47 ERR] Hosting failed to start ``` Scratch container removed after capture (`docker rm -f scratch-cc-mode`); the six real nodes were untouched by it. **It presents as a healthy container.** Worth carrying into Phase 2/4 readings — the failed process does not exit, so `docker ps` shows `Up`, `RestartCount=0`, and only an HTTP probe or the log tells the truth: ``` $ docker inspect otopcua-dev-central-1-1 -f '{{.State.Status}} restarts={{.RestartCount}}' running restarts=0 $ docker run --rm --network otopcua-dev_default curlimages/curl -s -o /dev/null \ -w '%{http_code}\n' --max-time 5 http://central-1:9000/health/ready 000 $ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9200/health/ready 503 ``` **Checks 5 + 6 — the wire-probe auth matrix** (`-import-path` onto the in-repo proto; the node has no reflection). An empty `CommandEnvelope` has no `oneof` arm set, so `FromEnvelope` throws and the service answers `InvalidArgument` — **after** the interceptor ran, which is exactly why it is the positive signal here: `PermissionDenied` means the key was refused, `InvalidArgument` means the key was accepted and only the payload was rejected: ``` $ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos $ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \ -plaintext -import-path /protos -proto mesh_transport.proto" # 5a no bearer -> site-a-1 Code: PermissionDenied Message: Node command authentication failed. # 5b -H 'authorization: Bearer dev-mesh-psk-SITE-A' -> site-a-1 Code: InvalidArgument Message: Command envelope could not be decoded. # 6a -H 'authorization: Bearer dev-mesh-psk-SITE-B' -> site-a-1 Code: PermissionDenied Message: Node command authentication failed. # 6b -H 'authorization: Bearer dev-mesh-psk-SITE-A' -> site-b-1 Code: PermissionDenied Message: Node command authentication failed. # 6c -H 'authorization: Bearer dev-mesh-psk-SITE-B' -> site-b-1 (positive control) Code: InvalidArgument Message: Command envelope could not be decoded. ``` **Checks 3 + 4 — the node-side leg, driven with real envelopes** over the same authenticated h2c path central would use. This is the first live proof that `MeshCommandAuthInterceptor` is actually invoked on a real Kestrel pipeline (register item 4) and that `NodeCommandGrpcService` → `NodeCommunicationActor` → EventStream → `DriverHostActor` works end to end: ``` $ $G -H 'authorization: Bearer dev-mesh-psk-SITE-A' -H 'x-otopcua-cluster: SITE-A' \ -d '{"correlationId":"8461008e...","restartDriver":{"clusterId":"SITE-A", "driverInstanceId":"SITEA-modbus","actorByUserName":"1a-gate", "correlationId":"8461008e..."}}' \ site-a-1:4056 mesh_transport.v1.NodeCommandService/Execute {} $ docker logs otopcua-dev-site-a-1-1 2>&1 | tail [11:31:01 DBG] Mesh: accepted RestartDriver (correlationId=8461008e37654724a292219202d78dfa) over gRPC; handed to the node communication actor. [11:31:01 INF] DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate [11:31:01 INF] DriverHost site-a-1:4053: spawned Modbus driver SITEA-modbus (stub=False) [11:31:55 DBG] Mesh: accepted ReconnectDriver (correlationId=29c5dc77283f4eb7869d0c7d6686cadd) over gRPC; handed to the node communication actor. [11:31:55 INF] DriverHost site-a-1:4053: reconnecting driver SITEA-modbus by request of 1a-gate [11:31:55 INF] DriverInstance SITEA-modbus: ForceReconnect requested by admin; re-entering Reconnecting [11:31:56 DBG] Mesh: accepted AlarmCommand (correlationId=39ba451173954229a71b9dd1e00ac1b9) over gRPC; handed to the node communication actor. ``` **Check 8 — interceptor denial counts across every node, whole run:** ``` $ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done central-1 telemetry=0 configserve=0 localdbsync=0 mesh=0 central-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 site-a-1 telemetry=0 configserve=0 localdbsync=0 mesh=2 # the two deliberate negatives site-a-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 site-b-1 telemetry=0 configserve=0 localdbsync=0 mesh=1 # the one deliberate negative site-b-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 ``` The `TwoNodeClusterHarness` I-1 null-provider Error (a known test-harness artifact) did **not** appear in any rig log — `grep -c "IMeshPskProvider"` = 0 on all six nodes, as expected for production wiring where every admin node registers the provider. ### Phase 1A defect — **CLOSED by `c0475f2d` + `e54883a8`, re-gated in run 2** **Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter.** `WithOtOpcUaControlPlaneSingletons`'s `WithActors` block resolves `IMeshCommandDialer` and `IMeshPskProvider` from `IServiceProvider` to hand to `CentralCommunicationActor.Props`. `MeshGrpcCommandDialer` takes `IMeshPskProvider`, which takes `ISecretResolver`, which `AddZbSecrets` builds by resolving `IAuditWriter`, which on an admin node is `ActorAuditWriter` — and its constructor does `registry.Get()`. That singleton is registered ~60 lines later in the same builder, so at this point the key is absent and `ActorRegistry.Get` throws `MissingActorRegistryEntryException`, failing `AkkaHostedService.StartAsync` and the whole host. Three things make this worth a pinned regression rather than a one-line fix note: - **It is not mode-gated.** The two `GetService` calls run in every transport mode by design (so a mode flip needs no restart), so the eager graph is walked on every admin boot. Proven above with a `ClusterClient` scratch container on the same image. - **It is invisible to the suite.** The 1A.2 container-activation test (`It_activates_from_the_container_despite_the_optional_client_factory_parameter`) resolves the dialer from a service collection that does not include `AddZbSecrets`, and `TwoNodeClusterHarness` registers no `IMeshPskProvider` at all — so no existing test walks the real graph in starter order. This is the same class as scadaproj#1 ("only visible with the DI graph built in a real host") and as mesh Phase 6's `IClusterRoleInfo` boot crash. - **The container lies about it.** The process does not exit, so `docker ps` reports `Up` with `restarts=0` while the node serves no HTTP at all. Any Phase 2/4 readiness check that reads container status rather than `/health/ready` would score this as healthy. The shape of the fix is the one this file already uses two lines above for the coordinator proxy — hand the actor a lazy accessor (`Func` / `Func`, or resolve on first use) so nothing is constructed while the `ActorSystem` is still building. **That is what shipped.** `c0475f2d` moved both dependencies behind `Func<>` invoked in `PreStart`, pinned by `ControlPlaneStarterOrderTests` (boots a real admin host through the production starter order and asserts `StartAsync`; verified RED against the eager resolve in 3/3 modes). `e54883a8` then made the resolution memoize-on-first-use rather than PreStart-only, because `PreStart` runs concurrently with the remaining starters and can still lose the race — which **run 2 observed happening on 2 of 3 real admin boots**, so that hardening was not theoretical. Both are re-gated above; see finding 5 for the live self-heal proof. ### Rig state after run 1 (superseded by the run-2 restoration below) The compose flip was reverted and the six nodes recreated on the committed all-`ClusterClient` compose. Because defect 4 was mode-independent, the branch image could not produce a healthy central pair on *any* mode, so the rig was returned to its pre-gate image rather than left with a dead central pair: the run-1 build was preserved as `otopcua-host:grpc-1a` (`69360a173f1f`) and the pre-gate build (`8bd81d8aea9b`) re-tagged back to `otopcua-host:dev`. ### Rig state after run 2 — **restored, six healthy, on the branch image** The flip was reverted (`git checkout docker-dev/docker-compose.yml`) and the six nodes recreated on the committed all-`ClusterClient` compose. No image substitution was needed this time: `otopcua-host:dev` is the run-2 build of the committed source (`1d8452a29018`), and it boots a healthy central pair, so the rig runs the code under test. The run-1 workaround tag `otopcua-host:grpc-1a` was deleted to stop anyone reusing a stale image. ``` central-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 central-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 site-a-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 site-a-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 site-b-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 site-b-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 $ curl … /health/ready → status Healthy, members 2, unreachable 0 $ grep -c "Mesh ClusterClient for cluster" → 3 on each central node (MAIN + SITE-A + SITE-B) ``` The three ClusterClients reappearing on the rollback is an incidental but useful result: the dark switch still rolls back cleanly on the fixed branch, which is what Phase 2 will depend on if its cutover has to be reversed. --- ## Phase 1B — node→central ack plane — **PASS** Branch `feat/grpc-central-ack` @ `4a2a3d3a` (rebased on `master`, which carries the merged 1A). Rig-only flip: site-a-1/-2 → `Grpc`; site-b stays `ClusterClient`; central stays `ClusterClient` (its command direction) — acks are governed by the **node's** mode, so this isolates 1B. | # | Check | Result | |---|---|---| | 1 | Deploy e2e: site-a acks arrive via `ReportApply`, `NodeDeploymentState` rows land, deployment seals | **PASS** — deployment `53BD2626…` `Status=2` (Sealed) with all six rows `Status=1` (Applied) inside ~50 ms; both site-a nodes logged `Posted the ApplyAck … to http://central-1:4055` and central-1 logged the matching `ApplyAck … from site-a-1:4053 / site-a-2:4053 … handed to the central communication actor`. **10/10 acks delivered across the whole run**, 0 drops on any node | | 2 | Site-b acks still arrive via ClusterClient in the SAME deployment — per-pair coexistence | **PASS** — in `53BD2626…` site-b-1/-2 rows are `Applied` while **0** `ReportApply` lines name a site-b node on either central node, and site-b-1/-2 each hold a live `Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]`; conversely site-a-1/-2 hold **0** receptionist associations and created **0** ClusterClients. Each pair's only possible ack route is the one its mode selects | | 3 | Exactly **one** ack row per site-a node per dispatch (no double-count from endpoint failover) | **PASS** — across 10 dispatches, `acks=1` for every (site-a node, deployment) pair on the happy path and on the `docker stop` failover. The **one** exception is the 4b pause drill (`458b8629…`, `acks=2` per node) and it is the documented at-least-once case, not a defect: the frozen server had in fact received the ack, and **the coordinator absorbed it without inflating the count — it reported `2/6 acks landed`, not `4/6`**. `NodeDeploymentState` holds exactly one row per node per deployment throughout | | 4 | Kill central-1 (`docker stop`) before the apply: site-a acks fail over to central-2; restart, verify convergence | **PASS** — with central-1 stopped, both site-a nodes logged the Information failover line `… to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable)` and central-2 accepted both **within the same second**; the deployment honestly ended `Status=4` (TimedOut) with central-1 alone at `Status=0` — sealing 6/6 is structurally impossible while one of the six expected nodes is down. After `start central-1` the pair reconverged (`members 2, unreachable 0`) and the next dispatch sealed **6/6** | | **4b** | **Hang drill** (`docker pause central-1`): the per-endpoint FRESH deadline is what lets central-2 be reached at all | **PASS** — with central-1 **paused** (hung, not refusing), both site-a nodes logged `… to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => DeadlineExceeded)` and the ack landed on central-2. **Observed failover latency ≈ 10 s** (dispatch accepted 13:03:51 → ack at 13:04:01), i.e. exactly one `AckTimeoutSeconds`. A single hoisted deadline would have entered the central-2 attempt already expired and dropped the ack; this is the live proof of the fix in `667639e6` | | 5 | Auth probes against central `:4055` `CentralCommandService/ReportApply` | **PASS** — 7/7 as designed: no bearer ⇒ `PermissionDenied`; SITE-B's key claiming SITE-A ⇒ `PermissionDenied`; no `x-otopcua-cluster` ⇒ `PermissionDenied`; unknown cluster `SITE-Z` with a valid key ⇒ `PermissionDenied`; unauthenticated against central-**2** ⇒ `PermissionDenied`; correct key+header with a **fake** deployment id ⇒ `{}` (auth passed, coordinator ignored it — **0 rows written** for that id); malformed id ⇒ `InvalidArgument` (auth passed, payload refused). Positive control: SITE-B key + SITE-B header also reaches the decoder | | 6 | ConfigServe (`DeploymentArtifactService`) on the same listener unaffected | **PASS** — all four site nodes **Fetched + verified** a 28 296-byte artifact from `http://central-1:4055` on a real revision change, and a direct authenticated `Fetch` returned artifact bytes; unauthenticated `Fetch` is refused by its **own** interceptor (`Unauthenticated` / "Config-serve authentication failed."), never by the mesh one — which is the method-path scoping holding. **1** config-serve denial in the whole run: the deliberate probe | ### Provenance `docker compose -f docker-dev/docker-compose.yml up -d --build` from the worktree `/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1b`, **exit 0**; image `otopcua-host:dev` rebuilt to `4105cf631003` at 12:51:47 UTC. **All six host nodes plus `migrator`, `cluster-seed` and `traefik` were recreated** (both one-shots exit 0); `sql` and its volume were kept. Nothing else touched the stack during the run, so it is not a concurrent-rebuild-invalidated one. Four rig-only Serilog `MinimumLevel:Override` Debug scopes were added on all six nodes for this run only — `ZB.MOM.WW.OtOpcUa.Runtime.Communication` (the sender's `Posted the ApplyAck` line), `ZB.MOM.WW.OtOpcUa.AdminUI.Grpc` (central's accept line), `ZB.MOM.WW.OtOpcUa.ControlPlane.Communication` and `Akka.Cluster.Tools.Client` — because the rig's default Serilog level is Information and checks 1–3 would otherwise have had no signal. Flip and overrides reverted after the gate, never committed. ``` $ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done central-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=2 central-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=2 site-a-1 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0 site-a-2 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0 site-b-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0 site-b-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0 $ curl … /health/ready → Healthy, members 2, unreachable 0 (MAIN) $ … http://:8080/health/ready → 200 on all four site nodes ``` ### Checks 1 + 2 evidence — the two transports, side by side in one deployment ``` $ curl -X POST http://localhost:9200/api/deployments -H 'X-Api-Key: docker-dev-deploy-key' \ -d '{"createdBy":"1b-gate-check1"}' {"outcome":"Accepted","deploymentId":"53bd2626-ce9f-4255-a964-24e4cf3c7ccb", …} HTTP 202 $ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='53bd2626-…' 2 -- Sealed $ … SELECT NodeId, Status, AppliedAtUtc FROM dbo.NodeDeploymentState WHERE DeploymentId='53bd2626-…' central-1:4053|1|2026-08-06 12:55:53.909 site-a-1:4053|1|2026-08-06 12:55:53.930 central-2:4053|1|2026-08-06 12:55:53.890 site-a-2:4053|1|2026-08-06 12:55:53.936 site-b-1:4053|1|2026-08-06 12:55:53.915 site-b-2:4053|1|2026-08-06 12:55:53.924 # node side — the gRPC pair: site-a-1 [12:55:53 DBG] Posted the ApplyAck for 53bd2626… (Applied) to http://central-1:4055. site-a-2 [12:55:53 DBG] Posted the ApplyAck for 53bd2626… (Applied) to http://central-1:4055. # central side — same two acks, named: central-1 [12:55:53 DBG] ApplyAck for 53bd2626… from site-a-1:4053 (Applied) handed to the central communication actor central-1 [12:55:53 DBG] ApplyAck for 53bd2626… from site-a-2:4053 (Applied) handed to the central communication actor central-2 (none) # per-node ReportApply accept counts, BOTH central nodes, this deployment: central-1 site-a-1=1 site-a-2=1 site-b-1=0 site-b-2=0 central-1=0 central-2=0 central-2 site-a-1=0 site-a-2=0 site-b-1=0 site-b-2=0 central-1=0 central-2=0 ``` The site-b half is proven by exclusion plus its live association, which is the same shape 1A used for the site pairs: ``` $ for c in site-b-1 site-b-2; do docker logs … | grep -c "Posted the ApplyAck"; done → 0, 0 site-b-1 [12:53:02 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist] site-b-2 [12:53:07 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist] $ for c in site-a-1 site-a-2; do docker logs … | grep -c "Connected to .*receptionist"; done → 0, 0 $ for c in site-a-1 site-a-2 site-b-1 site-b-2; do grep -c "Mesh ClusterClient for cluster"; done → 0,0,0,0 ``` Site-b rows are `Applied` while nothing on either central node logged a `ReportApply` for them, and site-a holds no receptionist association at all — so neither pair could have used the other's transport. Coexistence is per pair, in one deployment, from one dispatch. ### Check 3 evidence — one ack per dispatch, and the duplicate that proved the absorber Across all ten dispatches, counting `ReportApply` accepts on **both** central nodes: ``` site-a-1 dep=53bd2626… acks=1 site-a-2 dep=53bd2626… acks=1 site-a-1 dep=d0869c9e… acks=1 site-a-2 dep=d0869c9e… acks=1 site-a-1 dep=17507001… acks=1 site-a-2 dep=17507001… acks=1 # the stop-failover dispatch site-a-1 dep=b1e837a2… acks=1 site-a-2 dep=b1e837a2… acks=1 site-a-1 dep=a394eec6… acks=1 site-a-2 dep=a394eec6… acks=1 site-a-1 dep=9bb520a4… acks=1 site-a-2 dep=9bb520a4… acks=1 site-a-1 dep=458b8629… acks=2 site-a-2 dep=458b8629… acks=2 # the 4b PAUSE dispatch site-a-1 dep=fc2b7b64… acks=1 site-a-2 dep=fc2b7b64… acks=1 site-a-1 dep=41e4935d… acks=1 site-a-2 dep=41e4935d… acks=1 site-a-1 dep=4126636e… acks=1 site-a-2 dep=4126636e… acks=1 $ … SELECT NodeId, COUNT(*) FROM dbo.NodeDeploymentState WHERE DeploymentId='53bd2626-…' GROUP BY NodeId central-1:4053|1 central-2:4053|1 site-a-1:4053|1 site-a-2:4053|1 site-b-1:4053|1 site-b-2:4053|1 ``` A refused endpoint (`docker stop`) cannot produce a duplicate — the server never saw the call — which is why `17507001…` is 1 per node despite failing over. The pause drill is the one shape that can, and it did: ``` central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed … # failover target central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed … central-1 [13:04:33 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed … # at `docker unpause` central-1 [13:04:33 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed … central-1 [13:05:03 INF] Coordinator recovered in-flight deployment 458b8629… (2/6 acks landed) ``` `2/6`, not `4/6`. The frozen central had in fact received both acks and replayed them on unpause, and `ConfigPublishCoordinator.HandleAck`'s NodeId-keyed absorption made the duplicate free — the exact claim `GrpcApplyAckSender`'s "at-least-once walk" remark makes, observed rather than argued. ### Check 4 evidence — refusal failover ``` $ docker compose stop central-1 # 12:57:11 $ curl -X POST … -d '{"createdBy":"1b-gate-check4-failover"}' # 12:57:30 site-a-1 [12:57:30 INF] Posted the ApplyAck for 17507001… (Applied) to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable). site-a-2 [12:57:30 INF] …identical… central-2 [12:57:30 DBG] ApplyAck for 17507001… from site-a-1:4053 (Applied) handed … central-2 [12:57:30 DBG] ApplyAck for 17507001… from site-a-2:4053 (Applied) handed … $ … SELECT NodeId, Status FROM dbo.NodeDeploymentState WHERE DeploymentId='17507001-…' central-1:4053|0 central-2:4053|1 site-a-1:4053|1 site-a-2:4053|1 site-b-1:4053|1 site-b-2:4053|1 $ … SELECT Status FROM dbo.Deployment … → 4 (TimedOut) after the 2-minute deadline $ docker compose start central-1 # 13:00:10 $ curl … /health/ready → Healthy, members 2, unreachable 0 $ curl -X POST … -d '{"createdBy":"1b-gate-check4-converge"}' → Status=2, all six Status=1 ``` **One operational detail worth carrying.** After central-1 returned, site-a-**1**'s cached `GrpcChannel` to `http://central-1:4055` stayed in transient failure for roughly two minutes while site-a-**2** reached the same restarted endpoint immediately — three dispatches apart: ``` 13:01:25 site-a-1 → central-2 after (central-1 => Unavailable) | site-a-2 → central-1 OK 13:01:48 site-a-1 → central-2 after (central-1 => Unavailable) | site-a-2 → central-1 OK 13:02:49 site-a-1 → central-1 OK | site-a-2 → central-1 OK $ docker exec otopcua-dev-site-a-1-1 bash -c ' DeadlineExceeded). site-a-2 [13:04:01 INF] …identical… central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed … central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed … $ … SELECT NodeId, Status, AppliedAtUtc … WHERE DeploymentId='458b8629-…' site-a-1:4053|1|2026-08-06 13:04:01.837 site-a-2:4053|1|2026-08-06 13:04:01.774 central-1:4053|0|NULL central-2:4053|0|NULL site-b-1:4053|0|NULL site-b-2:4053|0|NULL $ docker unpause otopcua-dev-central-1-1 # 13:04:33 ``` `DeadlineExceeded` — not `Unavailable` — is the discriminator: the first endpoint burned a full `AckTimeoutSeconds` (10 s, the default; `13:03:51 → 13:04:01`) and central-2 was **still** dialled with a live deadline. Under the hoisted deadline this branch shipped before `667639e6`, that second attempt would have started already expired and the ack would have been dropped with a Warning. Observed failover latency: **≈10 s, one whole `AckTimeoutSeconds`**. **Collateral the drill produced, recorded because it is the honest picture of a `pause`.** Pausing a cluster member is a heavier disturbance than stopping one: central-2 marked central-1 UNREACHABLE at 13:03:46 and auto-downed it at 13:04:01, and after `unpause` the two central nodes each held a one-member view until central-1 was restarted (13:06:36). In that window the **two gRPC acks landed and none of the four ClusterClient acks did** — central-2 and both site-b nodes all logged `received DispatchDeployment` at 13:03:51 yet produced no ack the coordinator saw, and the deployment ended `TimedOut` at `2/6` naming exactly those four: ``` central-1 [13:05:51 WRN] Deployment 458b8629… timed out after 00:02:00 (2/6 acks landed). No ack from: central-1:4053, central-2:4053, site-b-1:4053, site-b-2:4053 central-1 [13:06:36 WRN] DriverHost central-1:4053: found orphan Applying row for deployment 458b8629-…; replaying central-1 [13:06:37 INF] DriverHost central-1:4053: applied deployment 458b8629… (children=10) ``` Stated at the strength the evidence supports: the four ClusterClient nodes received the dispatch and none of their acks reached the coordinator, while both gRPC nodes' acks did. The site nodes emit no apply-completion line, so whether each of the four actually finished applying is not determinable from their logs — what is determinable is that the ack did not arrive and the deadline said so honestly. The boot-replay backstop then repaired central-1. Not a 1B finding either way (no gRPC ack was involved), but it is the Phase 2 auto-down behaviour showing up early, and the split view needed a manual `restart central-1` to heal. ### Check 5 evidence — the auth matrix on the ack endpoint `-import-path` onto the in-repo proto (the node serves no reflection). `PermissionDenied` means the interceptor refused; anything past it (`{}` or `InvalidArgument`) means auth **passed** and only the payload was graded: ``` $ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos $ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \ -plaintext -import-path /protos -proto mesh_transport.proto" $ BODY='{"deploymentId":"ffffffffffffffffffffffffffffffff","nodeId":"site-a-1:4053", "outcome":"APPLY_OUTCOME_APPLIED","correlationId":"eeeeeeee…"}' # 5a no bearer, cluster header present -> central-1 Code: PermissionDenied Message: Central command authentication failed. # 5b Bearer dev-mesh-psk-SITE-B + cluster SITE-A -> central-1 Code: PermissionDenied Message: Central command authentication failed. # 5c Bearer dev-mesh-psk-SITE-A, NO cluster header-> central-1 Code: PermissionDenied Message: Central command authentication failed. # 5d Bearer dev-mesh-psk-SITE-A + cluster SITE-A, FAKE deployment id {} # 5e same, MALFORMED deployment id "not-a-guid" Code: InvalidArgument Message: undecodable ApplyAck # 5f Bearer dev-mesh-psk-SITE-B + cluster SITE-B (positive control) Code: InvalidArgument Message: undecodable ApplyAck # 5g Bearer dev-mesh-psk-SITE-A + cluster SITE-Z (unknown cluster) Code: PermissionDenied Message: Central command authentication failed. # 5h no bearer -> central-2 Code: PermissionDenied Message: Central command authentication failed. ``` The accepted fake ack changed nothing — the coordinator ignored a deployment it is not tracking: ``` $ … SELECT COUNT(*) FROM dbo.NodeDeploymentState WHERE DeploymentId='ffffffff-…' → 0 $ … SELECT COUNT(*) FROM dbo.Deployment WHERE DeploymentId='ffffffff-…' → 0 central-1 [13:08:08 DBG] ApplyAck for ffffffff… from site-a-1:4053 (Applied) handed to the central communication actor ``` Central's denial log separates the three operator-facing causes even though the wire status is deliberately uniform: ``` [13:08:07 WRN] Rejected a central command call to /mesh_transport.v1.CentralCommandService/ReportApply claiming cluster SITE-A: no bearer token presented. [13:08:07 WRN] … claiming cluster SITE-A: the presented bearer token did not match that cluster's pre-shared key, no key could be established for it, or it is not a cluster this node knows… [13:08:07 WRN] Rejected a central command call to …/ReportApply: no x-otopcua-cluster header, so there is no cluster whose pre-shared key the caller could be graded against. [13:08:25 WRN] … claiming cluster SITE-Z: … or it is not a cluster this node knows… ``` 5g is the **known-cluster gate answering live**: a valid SITE-A key presented for `SITE-Z` is refused in O(1), which also confirms the gate had in fact been published — see observation (b). ### Check 6 evidence — ConfigServe on the same listener A real node-driven fetch needed a real revision change, so `ServerCluster.Notes` for `MAIN` was edited, deployed, and **restored** (the revision hash returned to its pre-gate `642b9d6f79b9…`, verified by a further dispatch): ``` $ … UPDATE dbo.ServerCluster SET Notes = ' | 1b-gate-fetch-probe' WHERE ClusterId='MAIN' $ curl -X POST … → revisionHash 20af950fb211… (was 642b9d6f79b9…) Status=2 site-a-1 [13:09:31 INF] Fetched + verified artifact for 41e4935d… from http://central-1:4055 (28296 bytes). site-a-2 [13:09:31 INF] …same… site-b-1 [13:09:32 INF] …same… site-b-2 [13:09:32 INF] …same… $ … UPDATE dbo.ServerCluster SET Notes = '' … $ curl -X POST … → revisionHash 642b9d6f79b9… Status=2 # restored, byte-identical ``` Both interceptors on the one listener, each answering for its own service: ``` # authenticated Fetch, config-serve key: $ $G -H 'authorization: Bearer configserve-docker-dev-key' -d '{"deploymentId":"FC2B7B64-…"}' \ central-1:4055 deployment_artifact.v1.DeploymentArtifactService/Fetch { "data": "eyJDbHVzdGVycyI6W3siQ2x1c3RlcklkIjoiTUFJTiIsIk5hbWUiOiJNYWluIGNsdXN0ZXIiL…" } # unauthenticated Fetch — refused by the CONFIG-SERVE interceptor, not the mesh one: Code: Unauthenticated Message: Config-serve authentication failed. ``` `Unauthenticated` here versus `PermissionDenied` on the mesh service is the method-path scoping in `CentralCommandAuthInterceptor` holding: the two services share one `AddGrpc` pipeline and neither interceptor grades the other's calls. ### Observations carried forward from the reviews **(a) The I-4 shape is not exercisable on this rig, and that is a property of the rig, not a skipped check.** `MeshPskProvider.TryGetPsk` consults `MeshTransport:ClusterPsks` **before** the secret store and caches only store-resolved values, and docker-dev supplies `MeshTransport__ClusterPsks__{MAIN,SITE-A,SITE-B}` on both central nodes. The `InvalidateAll()` a contact-refresh tick performs (`ContactRefreshSeconds` = 60) therefore drops an always-empty cache, and an ack arriving just after a tick pays no store read. No latency delta was measurable and none should be: every ack in the run landed inside the same second it was posted, refresh tick or not. A real deployment resolving `OT-MESH-PSK-` from the store is where I-4 becomes observable — Phase 2 on a store-backed rig, not here. **(b) The `Failed to resolve` Error fired on 4 of 4 admin boots — and healed every time.** Worse odds than 1A's 2 of 3, and both dependencies lost the race on every boot: ``` central-1 [12:53:00 ERR] Failed to resolve IMeshCommandDialer … / IMeshPskProvider … # boot 1 central-1 [13:00:29 ERR] …both again… # boot 2 (check 4 restart) central-1 [13:06:36 ERR] …both again… # boot 3 (4b restart) central-2 [12:53:02 ERR] …both… # its single boot central-1 [12:53:00 ERR] No IMeshPskProvider was supplied to this actor; the known-cluster gate will NEVER be published and cached per-cluster keys will NEVER be dropped… ``` The second line is the one that reads worst, and 1B disproves it directly rather than by argument: **every site-a ack central-1 accepted required that provider**, including the ones after its third boot (13:07:36, 13:09:31, 13:09:59), and probe 5g's `SITE-Z` refusal is only possible if `SetKnownClusters` had in fact been published. The memoize heals it; the Error text does not say so. This is the 1A finding-5 carry-forward, now with a second, independent live confirmation and a stronger case for the wording fix — the message contains the word "NEVER" twice about something that self-corrects within one deploy. **(c) Zero mesh interceptor denials outside the deliberate probes.** Whole-run counts on every node and every interceptor: ``` central-1 telemetry=0 configserve=1 localdbsync=0 meshCentralCmd=4 nodeCmd=0 central-2 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=1 nodeCmd=0 site-a-1 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0 site-a-2 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0 site-b-1 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0 site-b-2 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0 ``` `meshCentralCmd` = 4 on central-1 (probes 5a/5b/5c/5g) + 1 on central-2 (5h) = exactly the five deliberate negatives; `configserve` = 1 = the deliberate 6b. Nothing else was refused anywhere. **(d) Pre-existing rig noise, so a Phase 2 error grep is not surprised by it.** Site nodes log `Unavailable … Name or service not known` from the historian client — the rig points `ServerHistorian__Endpoint` at `http://histgw-absent.invalid:5222` on purpose (45 occurrences on site-a-1, 1 on site-b-1, 4 on central-1 in this run). Nothing to do with the ack plane. ### Rig state after the gate — **restored, six healthy, all-ClusterClient, on the branch image** The flip and the four Serilog overrides were reverted (`git checkout docker-dev/docker-compose.yml`, `grep -c "GATE-1B ONLY"` → 0) and the nodes recreated on the committed compose. The simultaneous recreate cold-started both central nodes at once and they self-formed one-member views — the known simultaneous-cold-start split-brain the (default-OFF) `Cluster:BootstrapGuard` addresses — healed with a single `restart central-2`: ``` central-1 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0 central-2 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0 site-a-1 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0 site-a-2 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0 site-b-1 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0 site-b-2 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0 $ for h in central-1 central-2; do … /health/ready; done central-1 Healthy members 2 unreachable 0 leader akka.tcp://otopcua@central-1:4053 central-2 Healthy members 2 unreachable 0 leader akka.tcp://otopcua@central-1:4053 $ … http://:8080/health/ready → 200 ×4 $ curl -X POST … -d '{"createdBy":"1b-gate-rig-restore-verify"}' CDDA1D80-…|2 central-1|1 central-2|1 site-a-1|1 site-a-2|1 site-b-1|1 site-b-2|1 $ for c in site-a-1 site-a-2 site-b-1 site-b-2; do grep -c "Posted the ApplyAck"; done → 0,0,0,0 $ for c in central-1 central-2; do grep -c "Mesh ClusterClient for cluster"; done → 3 per boot ``` Zero gRPC acks after the rollback and three ClusterClients per central boot: the dark switch rolls back cleanly in the ack direction too, which is what Phase 2 depends on if its cutover has to be reversed. Rig configuration is byte-identical to its pre-gate state (`ServerCluster.Notes` restored; revision hash back to `642b9d6f79b9…`). ### Phase 1B defects **None.** No product code was changed during or after this gate. --- ## Phase 2 — full-rig cutover + soak — **PENDING** Rig default flipped and committed: `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"`. Branch/SHA `________`. **Run the Baseline section first on the pre-flip build.** | # | Check | Result | |---|---|---| | 1 | Full rebuild all-`Grpc`: six nodes ready, deploy e2e seals, all acks via gRPC | PENDING | | 2 | Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes | PENDING | | 3 | Kill site-a-1 mid-dispatch: per-node `Execute` failure **logged with node id + status** (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path) | PENDING | | 4 | Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action | PENDING | | 5 | Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; **no duplicate applies** after the pair returns (boot-replay path only) | PENDING | | 6 | **Zero ClusterClient activity under `Grpc`:** `docker logs \| grep -ci "clusterclient"` = 0 new lines post-boot | PENDING | | 7 | Soak ≥ 2 h under periodic deploys + command traffic: 0 `PermissionDenied`/`Unauthenticated` across all six nodes (PSK drift detector), 0 unexplained `Unavailable` | PENDING | | 8 | Full suite green on the cutover SHA | PENDING | --- ## Phase 4 — terminal gate on the post-deletion build — **PENDING** `master` @ `________` (after the Phase 3 deletion merge). **Provenance guard:** image build timestamp verified LATER than the merge commit time — a pre-deletion image would invalidate checks 6–7. | # | Check | Result | |---|---|---| | 1 | PSK negatives on both services (no key / wrong key / wrong cluster's key); **unset server key ⇒ all calls rejected** (fail-closed proven live) | PENDING | | 2 | Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand) | PENDING | | 3 | Ack matrix: every node acks exactly once, deployment seals | PENDING | | 4 | Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry) | PENDING | | 5 | Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart) | PENDING | | 6 | **Boundary check: no Akka association between any site container and central** — the Baseline (a) probe now shows no cross-mesh `:4053` traffic (pair-internal remoting only) | PENDING | | 7 | Full-rig restart (pairs together): clean convergence; `grep -ri "receptionist\|clusterclient"` over all node logs = 0 lines | PENDING | | 8 | `MeshGrpcBoundaryTests` + full suite green on the merge SHA | PENDING | ### Carry-forwards Anything a phase gate could not drive gets named here with why + what proves it instead, and is re-checked in the next gate (the ScadaBridge proportionate-gate rule). | Item | Deferred from | Why it could not be driven | Proven instead by | Status | |---|---|---|---|---| | Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-`GrpcPort` skip | 1A run 1 | Both central nodes failed to boot (defect 4) | — | **CLOSED in 1A run 2** — all three PASS live | | Check 4 acknowledge half — the condition actually acknowledging | 1A run 1 | The rig's one `ScriptedAlarm` sits Normal and its owning central node was down | — | **CLOSED in 1A run 2** — acked from the real `/alerts` page, `ACKNOWLEDGED` row observed | | Check 8 for telemetry + config-serve | 1A run 1 | Central dials both planes and never booted, so neither carried a call — 0 denials was vacuous | — | **CLOSED in 1A run 2** — both planes carried authenticated traffic with 0 denials | | Check 3 central half — `RestartDriver`/`ReconnectDriver` **originating at the AdminUI** | 1A runs 1 + 2 | **Structurally undrivable, not a rig problem:** `` + `PreStart`, pinned red-first by `ControlPlaneStarterOrderTests`) + `e54883a8` (memoize-on-first-use). Re-gated in run 2: 6/6 nodes boot, 0 `Hosting failed to start` | | 5 | 1A gate run 2 | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider` with the same `AuditWriterActorKey` cause | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede `AuditWriterActorKey`'s registration | **Not a defect — it is the case `e54883a8` was written for, and the memoize demonstrably heals it.** Proven live by moving the coordinator singleton onto the node that lost the race (stop central-2) and watching it fan a dispatch to all five live nodes with no dialer error. Residual is cosmetic: the Error text and its effect on soak greps → Phase 2 |