# ClusterClient → gRPC migration — live gate record Rig: `docker-dev/` (central-1/-2 = `admin,driver,cluster-MAIN`; site-a-1/-2, site-b-1/-2; sql + traefik + migrator + cluster-seed), rebuilt from the branch under test via `docker compose -f docker-dev/docker-compose.yml up -d --build`. Recorded check-by-check in the family's live-gate format. Later phases' sections are appended as those phases land; every check row gets a **PASS**/**FAIL** verdict plus one quantitative justification clause, with fenced evidence (real commands, real output). A run invalidated by a concurrent rig rebuild is **void** — note it and re-run (the ScadaBridge void-run rule). Provenance discipline: every phase section opens with branch + commit SHA + "image rebuilt, all N containers recreated", and Phase 4 additionally verifies the image build timestamp post-dates the deletion merge (a stale image invalidates the boundary checks). Plan: `2026-08-06-clusterclient-to-grpc.md`. Tracking: `2026-08-06-clusterclient-to-grpc-tracking.md`. --- ## Baseline (run on the PRE-change build, before the Phase 2 flip) The two gaps this program closes, **reproduced rather than assumed**: **(a) Unauthenticated Akka remoting crosses the WAN boundary.** From the host: ```bash # A live TCP association from a site container to central's Akka port: docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN' # And the receptionist actually accepting cross-mesh traffic (central log after any AdminUI # driver restart targeting site-a): docker logs otopcua-dev-central-1-1 2>&1 | grep -i "clusterclient\|receptionist" | tail -5 ``` Expected: `OPEN`; receptionist/client lines present. Nothing on this path presents any credential. **(b) Silent drop on an unreachable cluster.** Stop the site-b pair, dispatch a deployment (`POST http://localhost:9200/api/deployments`, `X-Api-Key: docker-dev-deploy-key`), observe: no per-node dispatch error surfaces anywhere; site-b's `NodeDeploymentState` rows sit `Applying` until the 2-minute deadline marks the deployment `TimedOut`/partially failed. The deadline is the only witness. Restart the pair. - [x] Baseline (a) recorded with evidence — **REPRODUCED** 2026-08-06, rig on `master @ 46be4c8d` (post-1B image, all-ClusterClient default, no mode override; `MeshTransport__Mode=ClusterClient` confirmed via `printenv` in the site container) - [x] Baseline (b) recorded with evidence — **REPRODUCED** same run ### Baseline (a) evidence — unauthenticated Akka remoting crosses the boundary ``` $ docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN' OPEN # no credential presented on this path $ docker logs otopcua-dev-central-1-1 | grep -i "clusterclient\|receptionist" | tail -5 [13:12:51 INF] Connected to [akka://otopcua/system/receptionist] [13:12:51 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s) [13:12:53 INF] Connected to [akka://otopcua/system/receptionist] [13:12:54 INF] Connected to [akka.tcp://otopcua@site-a-1:4053/system/receptionist] [13:12:57 INF] Connected to [akka.tcp://otopcua@site-b-1:4053/system/receptionist] ``` ### Baseline (b) evidence — silent drop on an unreachable cluster Site-b pair stopped 13:53:5x UTC; dispatch accepted 13:53:54 (`POST /api/deployments` → `{"outcome":"Accepted","deploymentId":"c6cd0798-…"}`, HTTP 202). ``` # 20 s after dispatch — 4/6 applied within ~15 ms, site-b rows parked, deployment open: NodeDeploymentState: central-1/-2, site-a-1/-2 → Status 1 @ 13:53:54.9xx site-b-1/-2 → Status 0, AppliedAtUtc NULL Deployment c6cd0798 → Status 1 (in progress) # The site-b fan leg is not merely unlogged — it is actively suppressed. The SITE-B # ClusterClient's SendToAll lands in deadLetters at INF, under Akka's sampled generic # dead-letter counter, naming no node and no failure: [13:53:54 INF] Message [DispatchDeployment] wrapped in [$SendToAll] … to [akka://otopcua/deadLetters] was not delivered. [467] dead letters encountered, of which 456 were not logged. … Message content: DispatchDeployment { DeploymentId = c6cd0798…, … } # Dispatch-to-deadline window (13:53:50–13:55:53): ambient Akka association-gating noise for # the downed pair exists (transport chatter, fires with or without a dispatch), but ZERO lines # correlate site-b with the deployment/dispatch/command: $ docker logs --since …09:53:50 --until …09:55:53 otopcua-dev-central-1-1 \ | grep -i "site-b" | grep -icE "deploy|dispatch|command|c6cd0798" 0 # The deadline is the only witness, exactly 2 minutes later: [13:55:54 WRN] Deployment c6cd0798… timed out after 00:02:00 (4/6 acks landed). No ack from: site-b-1:4053, site-b-2:4053. Each is an enabled, non-maintenance ClusterNode row — start the node, or set ClusterNode.MaintenanceMode = 1 … Deployment c6cd0798 → Status 4; site-b rows stay Status 0/NULL (a timed-out deployment is not retro-acked; boot replay serves the latest artifact without resurrecting the dead record). ``` Pair restarted after the drill; site-b-1 back healthy. Note for later log reads on this host: `docker logs --since/--until` parses bare timestamps as HOST-LOCAL time (EDT here, UTC−4) while the app writes UTC — a UTC-shaped `--since` silently returns nothing (it points 4 h into the future). Convert to local first; earlier in this run that mistake made a window look empty. --- ## Phase 0 — contract + auth + config groundwork — **PASS** Branch `feat/grpc-phase0-contract-auth` @ `c2bfad4a`. Image rebuilt, all six containers recreated (`docker compose -f docker-dev/docker-compose.yml up -d --build`, run from the worktree at `/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-phase0` so the image carries this branch). The rig was already running (from the main checkout's compose file) before this run; the rebuild recreated all eight containers (sql kept, migrator + cluster-seed one-shots re-ran, central-1/-2 + site-a-1/-2 + site-b-1/-2 + traefik recreated) under the fixed `otopcua-dev` project name, so this run is not a concurrent-rebuild-invalidated one — nothing else touched the stack afterwards. | # | Check | Result | |---|---|---| | 1 | All six nodes boot on `ClusterClient` with the dark `Grpc` config present (PSKs, endpoints, ClusterPsks map); no validator fires | **PASS** — 6/6 nodes `restarts=0`, `docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}'` reports `running restarts=0` on every node; `docker logs | grep -iE "OptionsValidationException\|MeshTransport.*(is empty\|refuses to start)"` = 0 lines on all six | | 2 | Deliberate misconfig: a scratch container with `MeshTransport__Mode=Grpc` and **no** `GrpcPsk` refuses to boot, error names `MeshTransport:GrpcPsk` + the env form | **PASS** — `Hosting failed to start` + `OptionsValidationException` naming `MeshTransport:GrpcPsk` and `MeshTransport__GrpcPsk`, below | | 3 | Same with empty `CentralGrpcEndpoints` — refuses, names the key | **PASS** — `OptionsValidationException` naming `MeshTransport:CentralGrpcEndpoints` and `MeshTransport__CentralGrpcEndpoints__0`, below | | 4 | Full suite green, 0 warnings, on the branch | **PASS** — `dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo`: 0 failures on a clean run (see evidence below); no `.cs` files changed on this branch since the last full rebuild, so the pre-existing ~868-warning baseline is unchanged (nothing to newly warn about) | Note: the new gRPC services are not yet mapped in Phase 0, so no wire probes here — the interceptor negative/positive matrix is unit-tested and wire-probed in 1A/1B. **Check 1 evidence** — no validator fired on any of the six nodes, all dark `Grpc` keys present: ``` $ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do docker inspect -f '{{.State.Status}} restarts={{.RestartCount}}' otopcua-dev-$c-1 done running restarts=0 # central-1 running restarts=0 # central-2 running restarts=0 # site-a-1 running restarts=0 # site-a-2 running restarts=0 # site-b-1 running restarts=0 # site-b-2 $ docker logs otopcua-dev-central-1-1 2>&1 | grep -iE "MeshTransport|ClusterClient" [08:53:26 INF] Mesh ClusterClient for cluster MAIN created with 2 contact point(s) [08:53:26 INF] Mesh ClusterClient for cluster SITE-A created with 2 contact point(s) [08:53:26 INF] Mesh ClusterClient for cluster SITE-B created with 2 contact point(s) $ docker logs otopcua-dev-site-a-1-1 2>&1 | grep -iE "cluster.client|receptionist" [08:53:26 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist] $ docker inspect otopcua-dev-site-a-1-1 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep MeshTransport MeshTransport__CentralGrpcEndpoints__1=http://central-2:4055 MeshTransport__CentralGrpcEndpoints__0=http://central-1:4055 MeshTransport__Mode=ClusterClient MeshTransport__GrpcPsk=dev-mesh-psk-SITE-A MeshTransport__CentralContactPoints__1=akka.tcp://otopcua@central-2:4053 MeshTransport__CentralContactPoints__0=akka.tcp://otopcua@central-1:4053 $ curl -sf http://localhost:9200/health/ready {"status":"Healthy", ..., "akka":{"status":"Healthy","description":"Akka cluster member status: Up", "data":{"selfAddress":"akka.tcp://otopcua@central-2:4053","selfRoles":["admin","cluster-MAIN","driver"], "memberCount":2,"unreachableCount":0,"leader":"akka.tcp://otopcua@central-1:4053"}}} ``` **Check 2 evidence** — scratch container from `otopcua-host:dev`, site-a-1's real environment with `MeshTransport__GrpcPsk` removed and `Mode` forced to `Grpc`: ``` $ docker run --rm --name scratch-check2-noPsk --network otopcua-dev_default \ --env-file check2-env.txt otopcua-host:dev [08:54:23 ERR] Hosting failed to start Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:GrpcPsk is empty. Under Grpc this per-cluster pre-shared key is the whole authentication boundary of the command plane in both directions — without it this node's inbound command service fails closed and every ack it posts to central is rejected. Supply it via the environment (MeshTransport__GrpcPsk), in production as ${secret:OT-MESH-PSK-}. at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name) ... ``` **Check 3 evidence** — same base env, `GrpcPsk` present, both `CentralGrpcEndpoints` indices unset: ``` $ docker run -d --name scratch-check3-noEndpoints --network otopcua-dev_default \ --env-file check3-env.txt otopcua-host:dev $ docker logs scratch-check3-noEndpoints 2>&1 | grep -A2 "OptionsValidationException" [08:55:55 ERR] Hosting failed to start Microsoft.Extensions.Options.OptionsValidationException: MeshTransport:CentralGrpcEndpoints is empty. Under Grpc a driver node has nowhere to post its deployment ack, and every deployment would time out naming a node that is running perfectly. Supply them via the environment (MeshTransport__CentralGrpcEndpoints__0). ``` Both scratch containers removed after capture (`docker rm -f scratch-check2-noPsk scratch-check3-noEndpoints`); the rig's six real nodes were never affected (still `restarts=0` after both runs). **Check 4 evidence** — full suite, clean re-run (a first run hit one race-condition flake in `RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed`, which the test's own name identifies as exercising a race; re-ran in isolation and it passed, then a full second run passed clean end to end — pre-existing timing sensitivity, not a regression from this branch, which touched only `docker-dev/docker-compose.yml` and `docs/Configuration.md`, no `.cs` files): ``` $ dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo ... Passed! - Failed: 0, Passed: 212, Skipped: 4, Total: 216, Duration: 3m56s - ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.dll (net10.0) $ echo $? 0 ``` Rig left running on `ClusterClient` (its normal state) after the gate. --- ## Phase 1A — central→node command plane — **PASS (on the re-run at `e54883a8`)** Two runs. **Run 1 @ `fbf7b708` FAILED** at the central boot boundary (defect 4) and could not drive checks 1/2/7 or the central halves of 3/4; its checks 5 + 6 and the node-side halves of 3 + 4 stand as recorded and were not repeated. **Run 2 @ `e54883a8`** — after the boot fix (`c0475f2d`) and its review-hardening follow-up (`e54883a8`) — drove every carried check. The result table below is the **combined, current** verdict; run 1's failure record is preserved underneath it because the defect it caught is the most valuable thing this gate produced. | # | Check | Verdict | Run | |---|---|---|---| | — | **Boot:** all six nodes healthy INCLUDING the central pair, on the branch image | **PASS** — 6/6 `running restarts=0`, 0 `Hosting failed to start`, `/health/ready` 200 ×6, MAIN mesh `memberCount=2 unreachableCount=0`. **But the `Resolve` Error line the fix logs DID fire — see finding 5** | 2 | | 1 | Deploy dispatch reaches **all six** driver nodes over gRPC; zero `SendToAll`/ClusterClient on central | **PASS** — 6/6 nodes accepted one `DispatchDeployment` under a single shared `correlationId=cf499b42…`; central's own per-node outcome log records 6 accepted / 0 failed; `grep -ci clusterclient` = **0** on both central nodes | 2 | | 2 | Acks return over ClusterClient (site mode unchanged) and the deployment **seals** | **PASS** — deployment `8DF9AAE0…` `Status=2` (Sealed) with all six `NodeDeploymentState` rows `Status=1` (Applied) inside ~90 ms; all four site nodes hold live `Connected to [akka.tcp://…/system/receptionist]` associations, which is their only ack path in 1A | 2 | | 3 | `RestartDriver` + `ReconnectDriver` land on the target site node | **PASS (composed)** — node-side leg driven live in run 1 (both commands acted on site-a-1). The AdminUI half **cannot be driven and this is now proven, not assumed**: ` 70000`), so a transition was manufactured by editing its predicate to an always-true form and deploying; the `/alerts` page was already open when it fired, the Ack button was clicked, and the predicate + hash were restored afterwards (verified byte-identical by `SELECT`) with a further deploy: ``` page open at 2026-08-06T12:20:59.036Z initial: empty window ROWS: 12:21:00.017 gate481-alarm | PressHigh | EQ-6e09bcc2d907 | ACTIVATED | 750 | system | press high ROW BUTTONS: ["Ack","Shelve","Shelve (timed)","Unshelve"] clicking Acknowledge at 2026-08-06T12:21:00.362Z after click: 12:21:00.557 | gate481-alarm | EQ-6e09bcc2d907 | ACKNOWLEDGED | 750 | multi-role-test | press high $ docker logs otopcua-dev-central-2-1 | grep AlarmCommand [12:21:00 DBG] Mesh: AlarmCommand (correlationId=ccafe3754aca40cfa8aa0a0602c86cbb) accepted by site-b-2:4053 over gRPC [12:21:00 DBG] … accepted by central-1:4053 … [12:21:00 DBG] … accepted by central-2:4053 … [12:21:00 DBG] … accepted by site-a-1:4053 … [12:21:00 DBG] … accepted by site-b-1:4053 … [12:21:00 DBG] … accepted by site-a-2:4053 … $ for c in …; do grep "Mesh: accepted AlarmCommand" …; done → all six, same correlationId ``` One correlation id across all six recipients is the "encode once, before the fan" rule holding for the one command whose CLR record has no correlation member of its own — the case the design called out as the reason encoding is hoisted out of the loop. **Check 7 — the skip, the warn, and the honest deadline.** ``` $ … UPDATE dbo.ClusterNode SET GrpcPort=NULL WHERE NodeId='site-b-2:4053' # 12:23:14 # one warn per central node, once each, within one ContactRefreshSeconds: central-2 [12:23:20 WRN] ClusterNode site-b-2:4053 has no GrpcPort; it exposes no command endpoint and is skipped in this dispatch set (further skips of this node are silent). Commands will not reach it until the row carries a port central-1 [12:23:19 WRN] …identical… # (each node also logs the TelemetryNodeSource twin — "exposes no telemetry stream … skipped in # this dial refresh" — expected collateral: both target sets read the same column) # dispatch at 12:23:36 — accepted by five, site-b-2 absent: central-2 [12:23:37 DBG] … (correlationId=88a9e0a2…) accepted by central-2 / site-a-2 / site-b-1 / site-a-1 / central-1 :4053 over gRPC # cumulative per-node accept counts after this dispatch — site-b-2 one behind everyone: central-1 4 central-2 4 site-a-1 4 site-a-2 4 site-b-1 4 site-b-2 3 $ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='5c8bdc9e-…' → 4 (TimedOut) $ … SELECT NodeId, Status FROM dbo.NodeDeploymentState … central-1|1 central-2|1 site-a-1|1 site-a-2|1 site-b-1|1 site-b-2|0 ``` Row restored to `4056` and a later dispatch sealed 6/6 (`080EBB0B…`, `Status=2`, every node `Status=1`). One operational detail worth carrying: **the skip persists for up to a full `ContactRefreshSeconds` after the row is repaired** — a deploy issued 5 s after the restore (`DF1C0B74…`) was still skipped and still TimedOut, because the dispatch set is only rebuilt on the refresh tick. **Check 8 — denials, and proof the planes were actually used.** ``` $ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done central-1 telemetry=0 configserve=0 localdbsync=0 mesh=0 central-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 site-a-1 telemetry=0 configserve=0 localdbsync=0 mesh=0 site-a-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 site-b-1 telemetry=0 configserve=0 localdbsync=0 mesh=0 site-b-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 # config-serve carried real, authenticated traffic: site-a-1 [12:20:59 INF] Fetched + verified artifact for 8d486c6b… from http://central-1:4055 (28313 bytes). # telemetry likewise, end to end into the UI: $ for c in …; do grep -c "Telemetry stream opened"; done → 3,4,3,3,3,5 /hosts → CLUSTER SITE-A … sa-modbus2 SITEA-modbus Modbus HEALTHY 1 host 1 tag 12:30:49 UTC 0 errors ``` The five telemetry `failed … reconnecting` warnings in the run are all accounted for: three at 12:15:20 (central dialing nodes that had not finished binding `:4056`) and two at 12:28:50 / 12:29:30 (the deliberate central-2 stop in finding 5). None is an auth failure. ### Phase 1A finding 5 — the PreStart race loses on real boots, and the memoize is what saves it The hardening commit's premise is not hypothetical. On the run-2 boot, **central-1 lost the race on both dependencies**: ``` $ docker logs otopcua-dev-central-1-1 | grep -A3 "Failed to resolve" [12:15:19 ERR] Failed to resolve IMeshCommandDialer for the mesh command boundary; continuing without it. The guards below report what that costs Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry) [12:15:19 ERR] Failed to resolve IMeshPskProvider for the mesh command boundary; …same cause… ``` It is not rare: across the three admin boots observed in run 2 (central-1 and central-2 at 12:15, central-2 again at 12:29 after the stop/start), **two of three lost it**. Had the fix stopped at PreStart-only resolution, those nodes would have held a permanent silent `null` — no dialer, and a never-published known-cluster gate — and the failure would only have surfaced the first time the coordinator singleton moved onto such a node. **The memoize heals it, verified on the node that actually lost the race.** Run 2's dispatches were all fanned by central-2 (the singleton host), so central-1's dialer was never exercised by them. Central-2 was therefore stopped to move the singleton, and a dispatch driven while central-1 alone held it: ``` $ docker compose stop central-2 # 12:28:39 $ curl … /health/ready → self akka.tcp://otopcua@central-1:4053, members 1 # 12:28:57 $ curl -X POST /api/deployments -d '{"createdBy":"1a-gate-central1-heal"}' # 12:29:01 $ docker logs otopcua-dev-central-1-1 | grep -E "accepted by|no command dialer is wired" [12:29:02 DBG] Mesh: DispatchDeployment (correlationId=ad97d386…) accepted by site-b-1:4053 over gRPC [12:29:02 DBG] … accepted by site-a-2 … site-b-2 … site-a-1 … central-1 … # 5 live nodes # no "no command dialer is wired" line, and no further "Failed to resolve" $ docker compose start central-2 → members 2, unreachable 0, status Healthy # 12:29:51 ``` So the boot-time Error is **cosmetically alarming but functionally healed**, and that is now an observed property rather than a design claim. Two follow-ups for Phase 2 rather than for 1A: the Error text says "continuing without it", which reads as permanent and should say that the first use retries; and a `Failed to resolve` line on a healthy node is exactly the kind of signal a soak's PSK-drift/error grep will trip over. **One incidental observation, not a check.** The deployment `RevisionHash` is deterministic (three separate no-change dispatches all produced `642b9d6f79b9…`), but restoring the alarm predicate's two text columns did **not** restore the pre-gate hash `6178a8f302aa…` — so the artifact covers a column the `UPDATE` bumped (`Script.RowVersion` is the obvious candidate). The rig's configuration content is byte-identical to its pre-gate state; only the hash moved. Not investigated further — out of this program's scope. ### Run 1 record — the defect that blocked it Branch `feat/grpc-node-command` @ `fbf7b708`. Image rebuilt from this worktree (`/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a`) so it carries this branch; `docker compose -f docker-dev/docker-compose.yml up -d --build` exited 0 and **all eight containers were recreated** — `migrator` and `cluster-seed` re-ran to completion (exit 0), the six host nodes and traefik were recreated, the `sql` container and its named volume were kept. Nothing else touched the stack afterwards, so this run is not a concurrent-rebuild-invalidated one. Rig-only flip: central-1/-2 `MeshTransport__Mode: "Grpc"` (hard value); **site nodes left on the compose default `ClusterClient`** (coexistence is the point). Three Serilog `MinimumLevel:Override` Debug scopes were added on all six nodes for this run only (`ZB.MOM.WW.OtOpcUa.Host.Grpc`, `ZB.MOM.WW.OtOpcUa.ControlPlane.Communication`, `Akka.Cluster.Tools.Client`) — the accept/dispatch lines the checks read are `Debug` and the rig's default Serilog level is `Information`, so without them checks 1/3 would have had no observable signal. Flip and overrides reverted after the gate, never committed. **Run 1 verdict: FAIL.** Neither central node booted, so nothing that originates at central could be driven: checks 1, 2 and 7 were **BLOCKED** outright, 3 and 4 reached only their node-side legs, and check 8's zero-denial result was vacuous for telemetry and config-serve (central dials both and never booted). Checks 5 and 6 **PASSED** and are the run-1 evidence carried into the combined table above. Root cause was defect 4 — resolving `IMeshCommandDialer` / `IMeshPskProvider` inside the first `WithActors` starter drags in `ISecretResolver` → `IAuditWriter` → `ActorAuditWriter`, whose constructor reads `AuditWriterActorKey` out of the `ActorRegistry` before the singleton that registers it has run. It was **mode-independent** (reproduced on the same image with `MeshTransport__Mode=ClusterClient`), so it was a defect of the branch, not of the gate's flip, and it would have failed a `ClusterClient` rig equally. ### Run 1 evidence **Provenance** — flip in effect on central only, sites untouched: ``` $ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \ --format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done central-1 MeshTransport__Mode=Grpc central-2 MeshTransport__Mode=Grpc site-a-1 MeshTransport__Mode=ClusterClient site-b-1 MeshTransport__Mode=ClusterClient $ docker images --format '{{.Repository}}:{{.Tag}}|{{.ID}}|{{.CreatedAt}}' | grep otopcua-host otopcua-host:dev|69360a173f1f|2026-08-06 07:20:14 -0400 EDT # rebuilt for this run ``` **Defect 4 — the boot failure (checks 1/2/7 blocked by it).** Both central nodes, identical: ``` $ docker logs otopcua-dev-central-1-1 2>&1 | sed -n '25,45p' [11:21:27 INF] OPC UA server started on opc.tcp://0.0.0.0:4840 [11:21:27 FTL] Unable to start AkkaHostedService - shutting down application Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey at Akka.Hosting.ActorRegistry.Get[TKey]() at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry) in /src/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Audit/ActorAuditWriter.cs:line 24 ... at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.b__0_5(IServiceProvider sp) ... at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.b__0_6(IServiceProvider sp) ... at ZB.MOM.WW.OtOpcUa.ControlPlane.ServiceCollectionExtensions.<>c.b__7_0( ActorSystem system, IActorRegistry registry, IDependencyResolver resolver) in /src/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs:line 74 at Akka.Hosting.AkkaConfigurationBuilder.StartAsync(ActorSystem sys) at Akka.Hosting.AkkaHostedService.StartAsync(CancellationToken cancellationToken) ``` **Mode-independence, proven rather than argued** — same image, central-1's exact environment with `MeshTransport__Mode=ClusterClient`, own hostname + self-seed so it cannot disturb the rig: ``` $ docker run -d --name scratch-cc-mode --network otopcua-dev_default --env-file c1-env.txt \ -e Cluster__PublicHostname=scratch-cc-mode \ -e Cluster__SeedNodes__0=akka.tcp://otopcua@scratch-cc-mode:4053 otopcua-host:dev $ docker logs scratch-cc-mode 2>&1 | grep -m4 -E "MissingActorRegistry|Hosting failed|Host starting" [11:28:45 INF] OtOpcUa.Host starting with roles=[admin,driver,cluster-main] (admin=True, driver=True) Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey [11:28:47 ERR] Hosting failed to start ``` Scratch container removed after capture (`docker rm -f scratch-cc-mode`); the six real nodes were untouched by it. **It presents as a healthy container.** Worth carrying into Phase 2/4 readings — the failed process does not exit, so `docker ps` shows `Up`, `RestartCount=0`, and only an HTTP probe or the log tells the truth: ``` $ docker inspect otopcua-dev-central-1-1 -f '{{.State.Status}} restarts={{.RestartCount}}' running restarts=0 $ docker run --rm --network otopcua-dev_default curlimages/curl -s -o /dev/null \ -w '%{http_code}\n' --max-time 5 http://central-1:9000/health/ready 000 $ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9200/health/ready 503 ``` **Checks 5 + 6 — the wire-probe auth matrix** (`-import-path` onto the in-repo proto; the node has no reflection). An empty `CommandEnvelope` has no `oneof` arm set, so `FromEnvelope` throws and the service answers `InvalidArgument` — **after** the interceptor ran, which is exactly why it is the positive signal here: `PermissionDenied` means the key was refused, `InvalidArgument` means the key was accepted and only the payload was rejected: ``` $ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos $ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \ -plaintext -import-path /protos -proto mesh_transport.proto" # 5a no bearer -> site-a-1 Code: PermissionDenied Message: Node command authentication failed. # 5b -H 'authorization: Bearer dev-mesh-psk-SITE-A' -> site-a-1 Code: InvalidArgument Message: Command envelope could not be decoded. # 6a -H 'authorization: Bearer dev-mesh-psk-SITE-B' -> site-a-1 Code: PermissionDenied Message: Node command authentication failed. # 6b -H 'authorization: Bearer dev-mesh-psk-SITE-A' -> site-b-1 Code: PermissionDenied Message: Node command authentication failed. # 6c -H 'authorization: Bearer dev-mesh-psk-SITE-B' -> site-b-1 (positive control) Code: InvalidArgument Message: Command envelope could not be decoded. ``` **Checks 3 + 4 — the node-side leg, driven with real envelopes** over the same authenticated h2c path central would use. This is the first live proof that `MeshCommandAuthInterceptor` is actually invoked on a real Kestrel pipeline (register item 4) and that `NodeCommandGrpcService` → `NodeCommunicationActor` → EventStream → `DriverHostActor` works end to end: ``` $ $G -H 'authorization: Bearer dev-mesh-psk-SITE-A' -H 'x-otopcua-cluster: SITE-A' \ -d '{"correlationId":"8461008e...","restartDriver":{"clusterId":"SITE-A", "driverInstanceId":"SITEA-modbus","actorByUserName":"1a-gate", "correlationId":"8461008e..."}}' \ site-a-1:4056 mesh_transport.v1.NodeCommandService/Execute {} $ docker logs otopcua-dev-site-a-1-1 2>&1 | tail [11:31:01 DBG] Mesh: accepted RestartDriver (correlationId=8461008e37654724a292219202d78dfa) over gRPC; handed to the node communication actor. [11:31:01 INF] DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate [11:31:01 INF] DriverHost site-a-1:4053: spawned Modbus driver SITEA-modbus (stub=False) [11:31:55 DBG] Mesh: accepted ReconnectDriver (correlationId=29c5dc77283f4eb7869d0c7d6686cadd) over gRPC; handed to the node communication actor. [11:31:55 INF] DriverHost site-a-1:4053: reconnecting driver SITEA-modbus by request of 1a-gate [11:31:55 INF] DriverInstance SITEA-modbus: ForceReconnect requested by admin; re-entering Reconnecting [11:31:56 DBG] Mesh: accepted AlarmCommand (correlationId=39ba451173954229a71b9dd1e00ac1b9) over gRPC; handed to the node communication actor. ``` **Check 8 — interceptor denial counts across every node, whole run:** ``` $ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done central-1 telemetry=0 configserve=0 localdbsync=0 mesh=0 central-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 site-a-1 telemetry=0 configserve=0 localdbsync=0 mesh=2 # the two deliberate negatives site-a-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 site-b-1 telemetry=0 configserve=0 localdbsync=0 mesh=1 # the one deliberate negative site-b-2 telemetry=0 configserve=0 localdbsync=0 mesh=0 ``` The `TwoNodeClusterHarness` I-1 null-provider Error (a known test-harness artifact) did **not** appear in any rig log — `grep -c "IMeshPskProvider"` = 0 on all six nodes, as expected for production wiring where every admin node registers the provider. ### Phase 1A defect — **CLOSED by `c0475f2d` + `e54883a8`, re-gated in run 2** **Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter.** `WithOtOpcUaControlPlaneSingletons`'s `WithActors` block resolves `IMeshCommandDialer` and `IMeshPskProvider` from `IServiceProvider` to hand to `CentralCommunicationActor.Props`. `MeshGrpcCommandDialer` takes `IMeshPskProvider`, which takes `ISecretResolver`, which `AddZbSecrets` builds by resolving `IAuditWriter`, which on an admin node is `ActorAuditWriter` — and its constructor does `registry.Get()`. That singleton is registered ~60 lines later in the same builder, so at this point the key is absent and `ActorRegistry.Get` throws `MissingActorRegistryEntryException`, failing `AkkaHostedService.StartAsync` and the whole host. Three things make this worth a pinned regression rather than a one-line fix note: - **It is not mode-gated.** The two `GetService` calls run in every transport mode by design (so a mode flip needs no restart), so the eager graph is walked on every admin boot. Proven above with a `ClusterClient` scratch container on the same image. - **It is invisible to the suite.** The 1A.2 container-activation test (`It_activates_from_the_container_despite_the_optional_client_factory_parameter`) resolves the dialer from a service collection that does not include `AddZbSecrets`, and `TwoNodeClusterHarness` registers no `IMeshPskProvider` at all — so no existing test walks the real graph in starter order. This is the same class as scadaproj#1 ("only visible with the DI graph built in a real host") and as mesh Phase 6's `IClusterRoleInfo` boot crash. - **The container lies about it.** The process does not exit, so `docker ps` reports `Up` with `restarts=0` while the node serves no HTTP at all. Any Phase 2/4 readiness check that reads container status rather than `/health/ready` would score this as healthy. The shape of the fix is the one this file already uses two lines above for the coordinator proxy — hand the actor a lazy accessor (`Func` / `Func`, or resolve on first use) so nothing is constructed while the `ActorSystem` is still building. **That is what shipped.** `c0475f2d` moved both dependencies behind `Func<>` invoked in `PreStart`, pinned by `ControlPlaneStarterOrderTests` (boots a real admin host through the production starter order and asserts `StartAsync`; verified RED against the eager resolve in 3/3 modes). `e54883a8` then made the resolution memoize-on-first-use rather than PreStart-only, because `PreStart` runs concurrently with the remaining starters and can still lose the race — which **run 2 observed happening on 2 of 3 real admin boots**, so that hardening was not theoretical. Both are re-gated above; see finding 5 for the live self-heal proof. ### Rig state after run 1 (superseded by the run-2 restoration below) The compose flip was reverted and the six nodes recreated on the committed all-`ClusterClient` compose. Because defect 4 was mode-independent, the branch image could not produce a healthy central pair on *any* mode, so the rig was returned to its pre-gate image rather than left with a dead central pair: the run-1 build was preserved as `otopcua-host:grpc-1a` (`69360a173f1f`) and the pre-gate build (`8bd81d8aea9b`) re-tagged back to `otopcua-host:dev`. ### Rig state after run 2 — **restored, six healthy, on the branch image** The flip was reverted (`git checkout docker-dev/docker-compose.yml`) and the six nodes recreated on the committed all-`ClusterClient` compose. No image substitution was needed this time: `otopcua-host:dev` is the run-2 build of the committed source (`1d8452a29018`), and it boots a healthy central pair, so the rig runs the code under test. The run-1 workaround tag `otopcua-host:grpc-1a` was deleted to stop anyone reusing a stale image. ``` central-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 central-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 site-a-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 site-a-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 site-b-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 site-b-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0 $ curl … /health/ready → status Healthy, members 2, unreachable 0 $ grep -c "Mesh ClusterClient for cluster" → 3 on each central node (MAIN + SITE-A + SITE-B) ``` The three ClusterClients reappearing on the rollback is an incidental but useful result: the dark switch still rolls back cleanly on the fixed branch, which is what Phase 2 will depend on if its cutover has to be reversed. --- ## Phase 1B — node→central ack plane — **PASS** Branch `feat/grpc-central-ack` @ `4a2a3d3a` (rebased on `master`, which carries the merged 1A). Rig-only flip: site-a-1/-2 → `Grpc`; site-b stays `ClusterClient`; central stays `ClusterClient` (its command direction) — acks are governed by the **node's** mode, so this isolates 1B. | # | Check | Result | |---|---|---| | 1 | Deploy e2e: site-a acks arrive via `ReportApply`, `NodeDeploymentState` rows land, deployment seals | **PASS** — deployment `53BD2626…` `Status=2` (Sealed) with all six rows `Status=1` (Applied) inside ~50 ms; both site-a nodes logged `Posted the ApplyAck … to http://central-1:4055` and central-1 logged the matching `ApplyAck … from site-a-1:4053 / site-a-2:4053 … handed to the central communication actor`. **10/10 acks delivered across the whole run**, 0 drops on any node | | 2 | Site-b acks still arrive via ClusterClient in the SAME deployment — per-pair coexistence | **PASS** — in `53BD2626…` site-b-1/-2 rows are `Applied` while **0** `ReportApply` lines name a site-b node on either central node, and site-b-1/-2 each hold a live `Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]`; conversely site-a-1/-2 hold **0** receptionist associations and created **0** ClusterClients. Each pair's only possible ack route is the one its mode selects | | 3 | Exactly **one** ack row per site-a node per dispatch (no double-count from endpoint failover) | **PASS** — across 10 dispatches, `acks=1` for every (site-a node, deployment) pair on the happy path and on the `docker stop` failover. The **one** exception is the 4b pause drill (`458b8629…`, `acks=2` per node) and it is the documented at-least-once case, not a defect: the frozen server had in fact received the ack, and **the coordinator absorbed it without inflating the count — it reported `2/6 acks landed`, not `4/6`**. `NodeDeploymentState` holds exactly one row per node per deployment throughout | | 4 | Kill central-1 (`docker stop`) before the apply: site-a acks fail over to central-2; restart, verify convergence | **PASS** — with central-1 stopped, both site-a nodes logged the Information failover line `… to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable)` and central-2 accepted both **within the same second**; the deployment honestly ended `Status=4` (TimedOut) with central-1 alone at `Status=0` — sealing 6/6 is structurally impossible while one of the six expected nodes is down. After `start central-1` the pair reconverged (`members 2, unreachable 0`) and the next dispatch sealed **6/6** | | **4b** | **Hang drill** (`docker pause central-1`): the per-endpoint FRESH deadline is what lets central-2 be reached at all | **PASS** — with central-1 **paused** (hung, not refusing), both site-a nodes logged `… to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => DeadlineExceeded)` and the ack landed on central-2. **Observed failover latency ≈ 10 s** (dispatch accepted 13:03:51 → ack at 13:04:01), i.e. exactly one `AckTimeoutSeconds`. A single hoisted deadline would have entered the central-2 attempt already expired and dropped the ack; this is the live proof of the fix in `667639e6` | | 5 | Auth probes against central `:4055` `CentralCommandService/ReportApply` | **PASS** — 7/7 as designed: no bearer ⇒ `PermissionDenied`; SITE-B's key claiming SITE-A ⇒ `PermissionDenied`; no `x-otopcua-cluster` ⇒ `PermissionDenied`; unknown cluster `SITE-Z` with a valid key ⇒ `PermissionDenied`; unauthenticated against central-**2** ⇒ `PermissionDenied`; correct key+header with a **fake** deployment id ⇒ `{}` (auth passed, coordinator ignored it — **0 rows written** for that id); malformed id ⇒ `InvalidArgument` (auth passed, payload refused). Positive control: SITE-B key + SITE-B header also reaches the decoder | | 6 | ConfigServe (`DeploymentArtifactService`) on the same listener unaffected | **PASS** — all four site nodes **Fetched + verified** a 28 296-byte artifact from `http://central-1:4055` on a real revision change, and a direct authenticated `Fetch` returned artifact bytes; unauthenticated `Fetch` is refused by its **own** interceptor (`Unauthenticated` / "Config-serve authentication failed."), never by the mesh one — which is the method-path scoping holding. **1** config-serve denial in the whole run: the deliberate probe | ### Provenance `docker compose -f docker-dev/docker-compose.yml up -d --build` from the worktree `/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1b`, **exit 0**; image `otopcua-host:dev` rebuilt to `4105cf631003` at 12:51:47 UTC. **All six host nodes plus `migrator`, `cluster-seed` and `traefik` were recreated** (both one-shots exit 0); `sql` and its volume were kept. Nothing else touched the stack during the run, so it is not a concurrent-rebuild-invalidated one. Four rig-only Serilog `MinimumLevel:Override` Debug scopes were added on all six nodes for this run only — `ZB.MOM.WW.OtOpcUa.Runtime.Communication` (the sender's `Posted the ApplyAck` line), `ZB.MOM.WW.OtOpcUa.AdminUI.Grpc` (central's accept line), `ZB.MOM.WW.OtOpcUa.ControlPlane.Communication` and `Akka.Cluster.Tools.Client` — because the rig's default Serilog level is Information and checks 1–3 would otherwise have had no signal. Flip and overrides reverted after the gate, never committed. ``` $ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done central-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=2 central-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=2 site-a-1 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0 site-a-2 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0 site-b-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0 site-b-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0 $ curl … /health/ready → Healthy, members 2, unreachable 0 (MAIN) $ … http://:8080/health/ready → 200 on all four site nodes ``` ### Checks 1 + 2 evidence — the two transports, side by side in one deployment ``` $ curl -X POST http://localhost:9200/api/deployments -H 'X-Api-Key: docker-dev-deploy-key' \ -d '{"createdBy":"1b-gate-check1"}' {"outcome":"Accepted","deploymentId":"53bd2626-ce9f-4255-a964-24e4cf3c7ccb", …} HTTP 202 $ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='53bd2626-…' 2 -- Sealed $ … SELECT NodeId, Status, AppliedAtUtc FROM dbo.NodeDeploymentState WHERE DeploymentId='53bd2626-…' central-1:4053|1|2026-08-06 12:55:53.909 site-a-1:4053|1|2026-08-06 12:55:53.930 central-2:4053|1|2026-08-06 12:55:53.890 site-a-2:4053|1|2026-08-06 12:55:53.936 site-b-1:4053|1|2026-08-06 12:55:53.915 site-b-2:4053|1|2026-08-06 12:55:53.924 # node side — the gRPC pair: site-a-1 [12:55:53 DBG] Posted the ApplyAck for 53bd2626… (Applied) to http://central-1:4055. site-a-2 [12:55:53 DBG] Posted the ApplyAck for 53bd2626… (Applied) to http://central-1:4055. # central side — same two acks, named: central-1 [12:55:53 DBG] ApplyAck for 53bd2626… from site-a-1:4053 (Applied) handed to the central communication actor central-1 [12:55:53 DBG] ApplyAck for 53bd2626… from site-a-2:4053 (Applied) handed to the central communication actor central-2 (none) # per-node ReportApply accept counts, BOTH central nodes, this deployment: central-1 site-a-1=1 site-a-2=1 site-b-1=0 site-b-2=0 central-1=0 central-2=0 central-2 site-a-1=0 site-a-2=0 site-b-1=0 site-b-2=0 central-1=0 central-2=0 ``` The site-b half is proven by exclusion plus its live association, which is the same shape 1A used for the site pairs: ``` $ for c in site-b-1 site-b-2; do docker logs … | grep -c "Posted the ApplyAck"; done → 0, 0 site-b-1 [12:53:02 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist] site-b-2 [12:53:07 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist] $ for c in site-a-1 site-a-2; do docker logs … | grep -c "Connected to .*receptionist"; done → 0, 0 $ for c in site-a-1 site-a-2 site-b-1 site-b-2; do grep -c "Mesh ClusterClient for cluster"; done → 0,0,0,0 ``` Site-b rows are `Applied` while nothing on either central node logged a `ReportApply` for them, and site-a holds no receptionist association at all — so neither pair could have used the other's transport. Coexistence is per pair, in one deployment, from one dispatch. ### Check 3 evidence — one ack per dispatch, and the duplicate that proved the absorber Across all ten dispatches, counting `ReportApply` accepts on **both** central nodes: ``` site-a-1 dep=53bd2626… acks=1 site-a-2 dep=53bd2626… acks=1 site-a-1 dep=d0869c9e… acks=1 site-a-2 dep=d0869c9e… acks=1 site-a-1 dep=17507001… acks=1 site-a-2 dep=17507001… acks=1 # the stop-failover dispatch site-a-1 dep=b1e837a2… acks=1 site-a-2 dep=b1e837a2… acks=1 site-a-1 dep=a394eec6… acks=1 site-a-2 dep=a394eec6… acks=1 site-a-1 dep=9bb520a4… acks=1 site-a-2 dep=9bb520a4… acks=1 site-a-1 dep=458b8629… acks=2 site-a-2 dep=458b8629… acks=2 # the 4b PAUSE dispatch site-a-1 dep=fc2b7b64… acks=1 site-a-2 dep=fc2b7b64… acks=1 site-a-1 dep=41e4935d… acks=1 site-a-2 dep=41e4935d… acks=1 site-a-1 dep=4126636e… acks=1 site-a-2 dep=4126636e… acks=1 $ … SELECT NodeId, COUNT(*) FROM dbo.NodeDeploymentState WHERE DeploymentId='53bd2626-…' GROUP BY NodeId central-1:4053|1 central-2:4053|1 site-a-1:4053|1 site-a-2:4053|1 site-b-1:4053|1 site-b-2:4053|1 ``` A refused endpoint (`docker stop`) cannot produce a duplicate — the server never saw the call — which is why `17507001…` is 1 per node despite failing over. The pause drill is the one shape that can, and it did: ``` central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed … # failover target central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed … central-1 [13:04:33 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed … # at `docker unpause` central-1 [13:04:33 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed … central-1 [13:05:03 INF] Coordinator recovered in-flight deployment 458b8629… (2/6 acks landed) ``` `2/6`, not `4/6`. The frozen central had in fact received both acks and replayed them on unpause, and `ConfigPublishCoordinator.HandleAck`'s NodeId-keyed absorption made the duplicate free — the exact claim `GrpcApplyAckSender`'s "at-least-once walk" remark makes, observed rather than argued. ### Check 4 evidence — refusal failover ``` $ docker compose stop central-1 # 12:57:11 $ curl -X POST … -d '{"createdBy":"1b-gate-check4-failover"}' # 12:57:30 site-a-1 [12:57:30 INF] Posted the ApplyAck for 17507001… (Applied) to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable). site-a-2 [12:57:30 INF] …identical… central-2 [12:57:30 DBG] ApplyAck for 17507001… from site-a-1:4053 (Applied) handed … central-2 [12:57:30 DBG] ApplyAck for 17507001… from site-a-2:4053 (Applied) handed … $ … SELECT NodeId, Status FROM dbo.NodeDeploymentState WHERE DeploymentId='17507001-…' central-1:4053|0 central-2:4053|1 site-a-1:4053|1 site-a-2:4053|1 site-b-1:4053|1 site-b-2:4053|1 $ … SELECT Status FROM dbo.Deployment … → 4 (TimedOut) after the 2-minute deadline $ docker compose start central-1 # 13:00:10 $ curl … /health/ready → Healthy, members 2, unreachable 0 $ curl -X POST … -d '{"createdBy":"1b-gate-check4-converge"}' → Status=2, all six Status=1 ``` **One operational detail worth carrying.** After central-1 returned, site-a-**1**'s cached `GrpcChannel` to `http://central-1:4055` stayed in transient failure for roughly two minutes while site-a-**2** reached the same restarted endpoint immediately — three dispatches apart: ``` 13:01:25 site-a-1 → central-2 after (central-1 => Unavailable) | site-a-2 → central-1 OK 13:01:48 site-a-1 → central-2 after (central-1 => Unavailable) | site-a-2 → central-1 OK 13:02:49 site-a-1 → central-1 OK | site-a-2 → central-1 OK $ docker exec otopcua-dev-site-a-1-1 bash -c ' DeadlineExceeded). site-a-2 [13:04:01 INF] …identical… central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-1:4053 (Applied) handed … central-2 [13:04:01 DBG] ApplyAck for 458b8629… from site-a-2:4053 (Applied) handed … $ … SELECT NodeId, Status, AppliedAtUtc … WHERE DeploymentId='458b8629-…' site-a-1:4053|1|2026-08-06 13:04:01.837 site-a-2:4053|1|2026-08-06 13:04:01.774 central-1:4053|0|NULL central-2:4053|0|NULL site-b-1:4053|0|NULL site-b-2:4053|0|NULL $ docker unpause otopcua-dev-central-1-1 # 13:04:33 ``` `DeadlineExceeded` — not `Unavailable` — is the discriminator: the first endpoint burned a full `AckTimeoutSeconds` (10 s, the default; `13:03:51 → 13:04:01`) and central-2 was **still** dialled with a live deadline. Under the hoisted deadline this branch shipped before `667639e6`, that second attempt would have started already expired and the ack would have been dropped with a Warning. Observed failover latency: **≈10 s, one whole `AckTimeoutSeconds`**. **Collateral the drill produced, recorded because it is the honest picture of a `pause`.** Pausing a cluster member is a heavier disturbance than stopping one: central-2 marked central-1 UNREACHABLE at 13:03:46 and auto-downed it at 13:04:01, and after `unpause` the two central nodes each held a one-member view until central-1 was restarted (13:06:36). In that window the **two gRPC acks landed and none of the four ClusterClient acks did** — central-2 and both site-b nodes all logged `received DispatchDeployment` at 13:03:51 yet produced no ack the coordinator saw, and the deployment ended `TimedOut` at `2/6` naming exactly those four: ``` central-1 [13:05:51 WRN] Deployment 458b8629… timed out after 00:02:00 (2/6 acks landed). No ack from: central-1:4053, central-2:4053, site-b-1:4053, site-b-2:4053 central-1 [13:06:36 WRN] DriverHost central-1:4053: found orphan Applying row for deployment 458b8629-…; replaying central-1 [13:06:37 INF] DriverHost central-1:4053: applied deployment 458b8629… (children=10) ``` Stated at the strength the evidence supports: the four ClusterClient nodes received the dispatch and none of their acks reached the coordinator, while both gRPC nodes' acks did. The site nodes emit no apply-completion line, so whether each of the four actually finished applying is not determinable from their logs — what is determinable is that the ack did not arrive and the deadline said so honestly. The boot-replay backstop then repaired central-1. Not a 1B finding either way (no gRPC ack was involved), but it is the Phase 2 auto-down behaviour showing up early, and the split view needed a manual `restart central-1` to heal. ### Check 5 evidence — the auth matrix on the ack endpoint `-import-path` onto the in-repo proto (the node serves no reflection). `PermissionDenied` means the interceptor refused; anything past it (`{}` or `InvalidArgument`) means auth **passed** and only the payload was graded: ``` $ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos $ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \ -plaintext -import-path /protos -proto mesh_transport.proto" $ BODY='{"deploymentId":"ffffffffffffffffffffffffffffffff","nodeId":"site-a-1:4053", "outcome":"APPLY_OUTCOME_APPLIED","correlationId":"eeeeeeee…"}' # 5a no bearer, cluster header present -> central-1 Code: PermissionDenied Message: Central command authentication failed. # 5b Bearer dev-mesh-psk-SITE-B + cluster SITE-A -> central-1 Code: PermissionDenied Message: Central command authentication failed. # 5c Bearer dev-mesh-psk-SITE-A, NO cluster header-> central-1 Code: PermissionDenied Message: Central command authentication failed. # 5d Bearer dev-mesh-psk-SITE-A + cluster SITE-A, FAKE deployment id {} # 5e same, MALFORMED deployment id "not-a-guid" Code: InvalidArgument Message: undecodable ApplyAck # 5f Bearer dev-mesh-psk-SITE-B + cluster SITE-B (positive control) Code: InvalidArgument Message: undecodable ApplyAck # 5g Bearer dev-mesh-psk-SITE-A + cluster SITE-Z (unknown cluster) Code: PermissionDenied Message: Central command authentication failed. # 5h no bearer -> central-2 Code: PermissionDenied Message: Central command authentication failed. ``` The accepted fake ack changed nothing — the coordinator ignored a deployment it is not tracking: ``` $ … SELECT COUNT(*) FROM dbo.NodeDeploymentState WHERE DeploymentId='ffffffff-…' → 0 $ … SELECT COUNT(*) FROM dbo.Deployment WHERE DeploymentId='ffffffff-…' → 0 central-1 [13:08:08 DBG] ApplyAck for ffffffff… from site-a-1:4053 (Applied) handed to the central communication actor ``` Central's denial log separates the three operator-facing causes even though the wire status is deliberately uniform: ``` [13:08:07 WRN] Rejected a central command call to /mesh_transport.v1.CentralCommandService/ReportApply claiming cluster SITE-A: no bearer token presented. [13:08:07 WRN] … claiming cluster SITE-A: the presented bearer token did not match that cluster's pre-shared key, no key could be established for it, or it is not a cluster this node knows… [13:08:07 WRN] Rejected a central command call to …/ReportApply: no x-otopcua-cluster header, so there is no cluster whose pre-shared key the caller could be graded against. [13:08:25 WRN] … claiming cluster SITE-Z: … or it is not a cluster this node knows… ``` 5g is the **known-cluster gate answering live**: a valid SITE-A key presented for `SITE-Z` is refused in O(1), which also confirms the gate had in fact been published — see observation (b). ### Check 6 evidence — ConfigServe on the same listener A real node-driven fetch needed a real revision change, so `ServerCluster.Notes` for `MAIN` was edited, deployed, and **restored** (the revision hash returned to its pre-gate `642b9d6f79b9…`, verified by a further dispatch): ``` $ … UPDATE dbo.ServerCluster SET Notes = ' | 1b-gate-fetch-probe' WHERE ClusterId='MAIN' $ curl -X POST … → revisionHash 20af950fb211… (was 642b9d6f79b9…) Status=2 site-a-1 [13:09:31 INF] Fetched + verified artifact for 41e4935d… from http://central-1:4055 (28296 bytes). site-a-2 [13:09:31 INF] …same… site-b-1 [13:09:32 INF] …same… site-b-2 [13:09:32 INF] …same… $ … UPDATE dbo.ServerCluster SET Notes = '' … $ curl -X POST … → revisionHash 642b9d6f79b9… Status=2 # restored, byte-identical ``` Both interceptors on the one listener, each answering for its own service: ``` # authenticated Fetch, config-serve key: $ $G -H 'authorization: Bearer configserve-docker-dev-key' -d '{"deploymentId":"FC2B7B64-…"}' \ central-1:4055 deployment_artifact.v1.DeploymentArtifactService/Fetch { "data": "eyJDbHVzdGVycyI6W3siQ2x1c3RlcklkIjoiTUFJTiIsIk5hbWUiOiJNYWluIGNsdXN0ZXIiL…" } # unauthenticated Fetch — refused by the CONFIG-SERVE interceptor, not the mesh one: Code: Unauthenticated Message: Config-serve authentication failed. ``` `Unauthenticated` here versus `PermissionDenied` on the mesh service is the method-path scoping in `CentralCommandAuthInterceptor` holding: the two services share one `AddGrpc` pipeline and neither interceptor grades the other's calls. ### Observations carried forward from the reviews **(a) The I-4 shape is not exercisable on this rig, and that is a property of the rig, not a skipped check.** `MeshPskProvider.TryGetPsk` consults `MeshTransport:ClusterPsks` **before** the secret store and caches only store-resolved values, and docker-dev supplies `MeshTransport__ClusterPsks__{MAIN,SITE-A,SITE-B}` on both central nodes. The `InvalidateAll()` a contact-refresh tick performs (`ContactRefreshSeconds` = 60) therefore drops an always-empty cache, and an ack arriving just after a tick pays no store read. No latency delta was measurable and none should be: every ack in the run landed inside the same second it was posted, refresh tick or not. A real deployment resolving `OT-MESH-PSK-` from the store is where I-4 becomes observable — Phase 2 on a store-backed rig, not here. **(b) The `Failed to resolve` Error fired on 4 of 4 admin boots — and healed every time.** Worse odds than 1A's 2 of 3, and both dependencies lost the race on every boot: ``` central-1 [12:53:00 ERR] Failed to resolve IMeshCommandDialer … / IMeshPskProvider … # boot 1 central-1 [13:00:29 ERR] …both again… # boot 2 (check 4 restart) central-1 [13:06:36 ERR] …both again… # boot 3 (4b restart) central-2 [12:53:02 ERR] …both… # its single boot central-1 [12:53:00 ERR] No IMeshPskProvider was supplied to this actor; the known-cluster gate will NEVER be published and cached per-cluster keys will NEVER be dropped… ``` The second line is the one that reads worst, and 1B disproves it directly rather than by argument: **every site-a ack central-1 accepted required that provider**, including the ones after its third boot (13:07:36, 13:09:31, 13:09:59), and probe 5g's `SITE-Z` refusal is only possible if `SetKnownClusters` had in fact been published. The memoize heals it; the Error text does not say so. This is the 1A finding-5 carry-forward, now with a second, independent live confirmation and a stronger case for the wording fix — the message contains the word "NEVER" twice about something that self-corrects within one deploy. **(c) Zero mesh interceptor denials outside the deliberate probes.** Whole-run counts on every node and every interceptor: ``` central-1 telemetry=0 configserve=1 localdbsync=0 meshCentralCmd=4 nodeCmd=0 central-2 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=1 nodeCmd=0 site-a-1 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0 site-a-2 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0 site-b-1 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0 site-b-2 telemetry=0 configserve=0 localdbsync=0 meshCentralCmd=0 nodeCmd=0 ``` `meshCentralCmd` = 4 on central-1 (probes 5a/5b/5c/5g) + 1 on central-2 (5h) = exactly the five deliberate negatives; `configserve` = 1 = the deliberate 6b. Nothing else was refused anywhere. **(d) Pre-existing rig noise, so a Phase 2 error grep is not surprised by it.** Site nodes log `Unavailable … Name or service not known` from the historian client — the rig points `ServerHistorian__Endpoint` at `http://histgw-absent.invalid:5222` on purpose (45 occurrences on site-a-1, 1 on site-b-1, 4 on central-1 in this run). Nothing to do with the ack plane. ### Rig state after the gate — **restored, six healthy, all-ClusterClient, on the branch image** The flip and the four Serilog overrides were reverted (`git checkout docker-dev/docker-compose.yml`, `grep -c "GATE-1B ONLY"` → 0) and the nodes recreated on the committed compose. The simultaneous recreate cold-started both central nodes at once and they self-formed one-member views — the known simultaneous-cold-start split-brain the (default-OFF) `Cluster:BootstrapGuard` addresses — healed with a single `restart central-2`: ``` central-1 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0 central-2 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0 site-a-1 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0 site-a-2 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0 site-b-1 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0 site-b-2 running restarts=0 mode=ClusterClient image=4105cf631003 hostingFailed=0 $ for h in central-1 central-2; do … /health/ready; done central-1 Healthy members 2 unreachable 0 leader akka.tcp://otopcua@central-1:4053 central-2 Healthy members 2 unreachable 0 leader akka.tcp://otopcua@central-1:4053 $ … http://:8080/health/ready → 200 ×4 $ curl -X POST … -d '{"createdBy":"1b-gate-rig-restore-verify"}' CDDA1D80-…|2 central-1|1 central-2|1 site-a-1|1 site-a-2|1 site-b-1|1 site-b-2|1 $ for c in site-a-1 site-a-2 site-b-1 site-b-2; do grep -c "Posted the ApplyAck"; done → 0,0,0,0 $ for c in central-1 central-2; do grep -c "Mesh ClusterClient for cluster"; done → 3 per boot ``` Zero gRPC acks after the rollback and three ClusterClients per central boot: the dark switch rolls back cleanly in the ack direction too, which is what Phase 2 depends on if its cutover has to be reversed. Rig configuration is byte-identical to its pre-gate state (`ServerCluster.Notes` restored; revision hash back to `642b9d6f79b9…`). ### Phase 1B defects **None.** No product code was changed during or after this gate. --- ## Phase 2 — full-rig cutover + soak — **PASS 8/8 + baseline + PSK negatives** Rig default flipped and committed: `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"` (`caf2eec6`, all six node entries). Rig rebuilt from `master @ caf2eec6` — which also carries Task 2.0 (`a94eb8a3`+`d7c9ead6`, honest PreStart-race logging, review-approved) and the DI role-scoping pin (`2e06dfe5`+`027f0403`, review-approved) — image rebuilt, node containers recreated, no `OTOPCUA_MESH_MODE` override in the environment. Baseline was run on the pre-flip build first (see Baseline section, `1bb3a937`). | # | Check | Result | |---|---|---| | 1 | Full rebuild all-`Grpc`: six nodes ready, deploy e2e seals, all acks via gRPC | **PASS** — deploy `949d2cc7` sealed (Status 2) 6/6 within ~1 s; every node logged `accepted DispatchDeployment … over gRPC` and `Posted the ApplyAck … to http://central-1:4055`, central logged each `ApplyAck … handed to the central communication actor`; wire-level: central holds established TCP to every node's `:4056`, node `TIME_WAIT` to central `:4055` observed. An earlier identical deploy (`8809464a`, pre-Debug-override) also sealed 6/6 in ~150 ms | | 2 | Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes | **PASS** — via authenticated grpcurl envelopes (the 1A shape): Restart `SITEA-modbus` → site-a-1 `{}` + `restarting driver … by request of p2-gate` + `spawned` + `connected`; Reconnect + AlarmCommand accepted + handed (correlation ids `c2b1…`/`c2c1…`); site-b-1 delivery proven (`c2d1…` accepted over gRPC). AdminUI origin composed per the 1A carry-forward (no UI control exists; same `DispatchOverGrpc` path driven by check 1) | | 3 | Kill site-a-1 mid-dispatch: per-node `Execute` failure **logged with node id + status** (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path) | **PASS** — deploy `069b57a4` 1 s after `docker kill`: central-2 WRN `DispatchDeployment … was NOT delivered to site-a-1:4053 (Unavailable: Error connecting to subchannel.). It is dropped and will NOT be retried…` — exactly **1** such line (no retry storm); 5/6 acked; `timed out after 00:02:00 (5/6 acks landed). No ack from: site-a-1:4053` at exactly +2:00; restart → FetchAndCache boot-replay restored last-known-good, timed-out deployment NOT retro-acked (ack count stayed 5) | | 4 | Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action | **PASS** — deploy `ffc75eae` with central-1 dead: site nodes logged `Posted the ApplyAck … to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable)` (the ordered walk, per node), all live acks landed on central-2; deadline honest at +2:00 naming `central-1, site-a-1` (site-a-1 = the documented ~2-min `GrpcChannel` backoff carry-forward from its own earlier kill — loud, single-attempt). First post-restart probe caught central-1 still booting (5/6, honest TimedOut); second probe `594ecda3` **sealed 6/6** — full convergence, no manual action beyond `docker start` | | 5 | Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; **no duplicate applies** after the pair returns (boot-replay path only) | **PASS** — deploy `e5a14c5f`: BOTH `NOT delivered to site-b-{1,2}:4053 (Unavailable…)` WRN lines in the SAME SECOND as dispatch (baseline (b): zero correlated lines, sampled-deadLetters INF); `timed out after 00:02:00 (4/6…). No ack from: site-b-1:4053, site-b-2:4053`; pair restart → boot-replay only (`FetchAndCache boot — restored served state`), rows stay 0/0, ack count stays 4 — no dupes, no resurrection | | 6 | **Zero ClusterClient activity under `Grpc`:** `docker logs \| grep -ci "clusterclient"` = 0 new lines post-boot | **PASS** — final sweep after the full soak: `grep -ic "clusterclient\|receptionist"` = **0 on all six nodes**, covering checks 1–5, every kill/restart cycle, and the whole 135-min soak | | 7 | Soak ≥ 2 h under periodic deploys + command traffic: 0 `PermissionDenied`/`Unauthenticated` across all six nodes (PSK drift detector), 0 unexplained `Unavailable` | **PASS** — 135 min (15:03:44Z→17:18Z), **14/14 deploy cycles sealed 6/6** (most in 0–1 s) + Restart/AlarmCommand traffic every cycle, 0 traffic failures. Grep since soak start: **PermissionDenied=0, Unauthenticated=0 on all six** (PSK drift detector clean). Review items, both explained: (a) site-a-2 `Unavailable`×3807 — ALL at DBG/INF (its WRN/ERR count was 0), the alarm-historian `SendEvent` retry loop failing DNS on the HistorianGateway host, which does not exist on this rig — pre-existing rig-shape noise, not mesh; (b) central-2 WRN×28 = exactly 14 deploys × 2 known per-deploy StartDeployment authoring warnings (`TAG-gatelong` ModbusRegion + script-compile RSS advisory) — pre-existing on the old transport. The predicted post-central-restart failover-line burst did not occur during the soak because no node was killed inside the soak window (kills all preceded it) | | 8 | Full suite green on the cutover SHA | **PASS** — `dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo` exit 0, **zero failures across every assembly**, run in parallel with the live rig + soak traffic. Spot totals match every baseline: ControlPlane 245 (Task 2.0 set), Host 95 (role-scoping pins), Runtime 558+3skip, AdminUI 1109, Commons 346, Cluster 193, Host.IntegrationTests 212+4skip | **Extra (plan-text requirement): PSK negatives on RUNNING containers, both services** — **PASS**: `NodeCommandService/Execute` @ site-a-1: no bearer → `PermissionDenied: Node command authentication failed.`; SITE-B key → same uniform denial; SITE-A key + empty envelope (positive control) → `InvalidArgument: Command envelope could not be decoded.` (interceptor ran, payload rejected — auth and decode are distinct). `CentralCommandService/ReportApply` @ central-1:4055: no bearer / wrong key / SITE-A key claiming `x-otopcua-cluster: SITE-Z` → all `PermissionDenied: Central command authentication failed.` (uniform text, unknown-cluster claim refused). **Gate observability note:** after check 1's first pass, the six node containers were recreated once with an UNCOMMITTED scratchpad compose override raising Serilog default to Debug (the mesh delivery lines are DBG; the base rig logs at Information and the 1A/1B gates had used the same temporary raise). Same image, same SHA — no rebuild, so no void-run. All kill drills ran after that recreation. --- ## Phase 4 — terminal gate on the post-deletion build — **PASS 8/8 — PROGRAM CLOSED** Run 2026-08-06 (log times UTC, 23:53–00:27 spanning midnight into 08-07). `master @ 0b194003` (= merge `9420f8ef` + one docs-only commit; the Phase 3 deletion merge is `2fa880b6`). **Provenance guard PASS:** image `otopcua-host:dev` = `d2a290fc5711`, built 2026-08-06 17:55:10 EDT — later than both the deletion merge (16:57:05 EDT) and the last code merge `9420f8ef` (17:28:48 EDT), so the boundary checks ran on the post-deletion binary. No rebuild during the gate. The six node containers were recreated twice before any drill (once for the gate's Debug scopes, once to widen them to the ack plane — the "Posted the ApplyAck" sender logs in `Runtime.Communication` and the central receiver in `AdminUI.Grpc`, neither covered by the 1A/2 scope set); same image both times, so no void-run. Check 1 ran before the second recreation (interceptor probes — log-reset does not affect them); checks 2–7 after it, so the final greps cover every drill. Override reverted after the gate, never committed. | # | Check | Result | |---|---|---| | 1 | PSK negatives on both services (no key / wrong key / wrong cluster's key); **unset server key ⇒ all calls rejected** (fail-closed proven live) | **PASS** — 7/7: `Execute` @ site-a-1 no-bearer → `PermissionDenied: Node command authentication failed.`, SITE-B key → same uniform text, SITE-A key + empty envelope → `InvalidArgument` (interceptor ran, payload rejected); `ReportApply` @ central-1 no-bearer / garbage key / SITE-A key claiming `x-otopcua-cluster: SITE-Z` → all `PermissionDenied: Central command authentication failed.` (uniform, unknown-cluster refused = no key configured for that cluster ⇒ every call rejected), SITE-A key + SITE-A claim → `InvalidArgument: undecodable ApplyAck` (positive control). Unset-key fail-closed at the earliest gate: scratch container on the rig image, `Grpc` mode, `GrpcPsk` removed → `Hosting failed to start`, `OptionsValidationException` naming `MeshTransport:GrpcPsk` + the env form (scratch removed after capture) | | 2 | Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand) | **PASS** — deploy `22dfec4b` fanned as ONE envelope: all six nodes logged `accepted DispatchDeployment (correlationId=c532b8f7…)` in the same second. **Restart + Reconnect driven from the real `/hosts` AdminUI panel for the first time in any gate** (the #546 wire-in closed the 1A "structurally undrivable" carry-forward): each fanned to all six under a single correlation id (`41157cfb…` / `cb81bc5e…`), and BOTH SITE-A nodes acted (`restarting driver SITEA-modbus by request of multi-role-test` + `spawned`, `ForceReconnect requested by admin`) — actor = the logged-in operator, i.e. the full `AdminOperationsActor.Dispatch → DispatchOverGrpc` origin path live. AlarmCommand: authenticated envelopes accepted + handed on site-b-1 and central-1 | | 3 | Ack matrix: every node acks exactly once, deployment seals | **PASS** — for `22dfec4b`: every node logged exactly ONE `Posted the ApplyAck … (Applied) to http://central-1:4055`, central-1 served exactly 6 `ReportApply` 200s and handed 6 acks to the central communication actor, `Deployment.Status=2` with 6/6 `NodeDeploymentState` Applied inside a ~30 ms spread | | 4 | Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry) | **PASS** — kill central-1 → deploy `3edf043d`: all five live nodes posted `to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable)` (the ordered walk, one loud INF per node), deadline honest at exactly +2:00 naming only central-1 (5/6). Restart → first re-deploy hit the documented ~2-min `GrpcChannel` backoff (ONE `was NOT delivered … will NOT be retried` WRN, honest TimedOut), next dispatch **sealed 6/6** — no manual action beyond `docker start`. Kill site-a-1 + dispatch in the same second → exactly **1** `NOT delivered to site-a-1 (Unavailable…)` line (no retry storm), 5/6 acks, deadline at exactly +2:00 naming only site-a-1; restart → `FetchAndCache boot — restored served state` and the timed-out deployment NOT retro-acked (count stayed 5) | | 5 | Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart) | **PASS** — deploy `f311cf5b` dispatched and the coordinator host (central-2) killed **in the same second** with the deployment open; **central-1's recovered coordinator fired the deadline at exactly +2:00** (`timed out … (5/6 acks landed). No ack from: site-b-2:4053` — the deliberately-stopped node), with central-2's own ack — posted before its death — preserved in the 5. After restarting both: per-node `Posted the ApplyAck` count for `f311cf5b` exactly 1 on the five that acked / 0 on site-b-2, DB count stayed 5 (no retro-ack, no dupes), site-b-2 boot-replayed last-known-good, and reconverge deploy `12a46546` **sealed 6/6** | | 6 | **Boundary check: no Akka association between any site container and central** | **PASS** — `netstat` in every node's netns: exactly THREE established `:4053` associations rig-wide, each strictly pair-internal (central-1↔central-2, site-a-1↔site-a-2, site-b-1↔site-b-2 by container IP); **zero site↔central `:4053` connections**. Baseline (a)'s unauthenticated cross-mesh association is gone at the TCP level | | 7 | Full-rig restart (pairs together): clean convergence; `grep -ri "receptionist\|clusterclient"` over all node logs = 0 lines | **PASS, with one non-mesh observation** — all six restarted in one `compose restart`; both SITE pairs re-formed cleanly within 7 s; the CENTRAL pair hit the **documented simultaneous-cold-start split-brain** (same-second mutual `InitJoinNack` → both self-formed — the exact scenario the Phase 7 bootstrap guard `d1dac87f` ships for, opt-in and default-OFF, not enabled on this rig; pre-existing pair-formation behavior, zero mesh-transport involvement) and was healed by the standard operator action, a single-node restart of central-2, which rejoined as Secondary (`memberCount=2 unreachableCount=0`, leader central-1 — the Phase 7 rejoin drill behavior holding). Transport verdict clean both ways: `grep -icE "receptionist|clusterclient"` = **0 on all six nodes** over logs covering every drill, and post-restart deploy `11007721` **sealed 6/6**. Whole-gate PSK-drift sweep: **zero** denials on all four interceptors across all six nodes for the entire post-recreation run | | 8 | `MeshGrpcBoundaryTests` + full suite green on the merge SHA | **PASS** — targeted `MeshGrpcBoundaryTests` 3/3; terminal full suite on `master @ 0b194003`: `dotnet test` direct exit-code capture (no pipe) **RC=0, 50/50 suites, 9,333 passed / 0 failed** — identical totals to the Phase-3 DoD baseline, run in parallel with the live rig per the Phase 2 precedent | ### Phase 4 carry-forward closures | Item | Disposition | |---|---| | 1A check-3 AdminUI origin (closed by composition in 1A/2) | **Now closed by direct evidence** — #546 wired `DriverStatusPanel` into `/hosts`, and this gate drove Restart + Reconnect from the real page (fan + act, operator identity carried) | | `Failed to resolve` Error-text cosmetics (1A finding 5 → Phase 2) | **Closed in Phase 2** Task 2.0 (`a94eb8a3`+`d7c9ead6`, honest PreStart-race logging) | | ~2-min `GrpcChannel` backoff after a peer dies (1B → Phase 2 soak note) | **Behavior confirmed three more times this gate** (post-central-1-restart re-deploy; central-2→site-a-1 dispatch skip at 00:10:21 after the drill-B kill). Always loud (one WRN, single attempt), never lossy beyond the honest deadline, always healed by the next dispatch after the window. Stands as an operator note in `docs/Redundancy.md` territory — expect one TimedOut deploy within ~2 min of any node kill/restart | | `GrpcPort` NULL→repair skip persists ≤ `ContactRefreshSeconds` (1A check 7) | Operational note stands (dispatch set rebuilds on the refresh tick); no Phase 4 re-check needed — the mechanism is unchanged since 1A proved it live | | I-4 cold-PSK-cache latency after a refresh tick (1B → Phase 2 → Phase 4) | **Re-carried out of the program with rationale, unchanged:** still structurally unmeasurable on docker-dev — the rig configures `MeshTransport:ClusterPsks`, which answers ahead of the store, so the cache never holds a store-resolved value and `InvalidateAll()` clears nothing. Every ack in every phase landed within the second it was posted. To be measured once on the first store-backed deployment (a real `${secret:OT-MESH-PSK-…}` fleet); until then it is an unquantified-latency note, not a correctness gap — the fail-closed behavior on a cold cache is pinned by unit tests | No new defects. The central-pair cold-start split-brain in check 7 is pre-existing, documented (Phase 7 bootstrap guard, deliberately opt-in), reproduced here only because a same-second pair restart is exactly its trigger — recorded as the reason an operator restarting a central pair together should either stagger the two nodes or enable `Cluster:BootstrapGuard`. **Program CLOSED.** Both baseline gaps are shut on the wire: (a) zero cross-mesh Akka associations and zero receptionist/ClusterClient log lines on the post-deletion binary; (b) an unreachable node now costs one loud per-node WRN in the dispatch second plus an honest deadline, instead of a sampled dead-letter counter and silence. ### Carry-forwards Anything a phase gate could not drive gets named here with why + what proves it instead, and is re-checked in the next gate (the ScadaBridge proportionate-gate rule). | Item | Deferred from | Why it could not be driven | Proven instead by | Status | |---|---|---|---|---| | Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-`GrpcPort` skip | 1A run 1 | Both central nodes failed to boot (defect 4) | — | **CLOSED in 1A run 2** — all three PASS live | | Check 4 acknowledge half — the condition actually acknowledging | 1A run 1 | The rig's one `ScriptedAlarm` sits Normal and its owning central node was down | — | **CLOSED in 1A run 2** — acked from the real `/alerts` page, `ACKNOWLEDGED` row observed | | Check 8 for telemetry + config-serve | 1A run 1 | Central dials both planes and never booted, so neither carried a call — 0 denials was vacuous | — | **CLOSED in 1A run 2** — both planes carried authenticated traffic with 0 denials | | Check 3 central half — `RestartDriver`/`ReconnectDriver` **originating at the AdminUI** | 1A runs 1 + 2 | **Structurally undrivable, not a rig problem:** `` + `PreStart`, pinned red-first by `ControlPlaneStarterOrderTests`) + `e54883a8` (memoize-on-first-use). Re-gated in run 2: 6/6 nodes boot, 0 `Hosting failed to start` | | 5 | 1A gate run 2 | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider` with the same `AuditWriterActorKey` cause | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede `AuditWriterActorKey`'s registration | **Not a defect — it is the case `e54883a8` was written for, and the memoize demonstrably heals it.** Proven live by moving the coordinator singleton onto the node that lost the race (stop central-2) and watching it fan a dispatch to all five live nodes with no dialer error. Residual is cosmetic: the Error text and its effect on soak greps → Phase 2 |