docs(plans): ClusterClient→gRPC Phase 4 terminal live gate PASS 8/8 — PROGRAM CLOSED
Ran on master @ 0b194003, post-deletion image d2a290fc5711 (provenance-guarded).
PSK matrix 7/7 incl. unset-key boot refusal; command matrix with Restart/Reconnect
driven from the real /hosts panel (#546 closed the 1A composition carry-forward by
direct evidence); exactly-once ack matrix sealed 6/6; central-kill + site-kill drills
honest (one loud WRN, +2:00 deadlines, boot-replay, no retro-ack); mid-deploy
coordinator kill: recovered coordinator fired the deadline at exactly +2:00, all
landed acks preserved, zero dupes; boundary: 3 established :4053 associations, all
pair-internal, zero site↔central; full-rig restart transport grep = 0 on all six
(one non-mesh observation: central pair reproduced the documented guard-off
cold-start split-brain, healed by single-node rejoin); MeshGrpcBoundaryTests 3/3;
terminal full suite RC=0 direct capture, 50/50 suites, 9,333/0.
I-4 cold-PSK-cache re-carried out of the program (unmeasurable on docker-dev).
No new defects.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
@@ -1123,22 +1123,50 @@ kill drills ran after that recreation.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — terminal gate on the post-deletion build — **PENDING**
|
||||
## Phase 4 — terminal gate on the post-deletion build — **PASS 8/8 — PROGRAM CLOSED**
|
||||
|
||||
`master` @ `________` (after the Phase 3 deletion merge). **Provenance guard:** image build
|
||||
timestamp verified LATER than the merge commit time — a pre-deletion image would invalidate
|
||||
checks 6–7.
|
||||
Run 2026-08-06 (log times UTC, 23:53–00:27 spanning midnight into 08-07). `master @ 0b194003`
|
||||
(= merge `9420f8ef` + one docs-only commit; the Phase 3 deletion merge is `2fa880b6`).
|
||||
**Provenance guard PASS:** image `otopcua-host:dev` = `d2a290fc5711`, built 2026-08-06
|
||||
17:55:10 EDT — later than both the deletion merge (16:57:05 EDT) and the last code merge
|
||||
`9420f8ef` (17:28:48 EDT), so the boundary checks ran on the post-deletion binary. No rebuild
|
||||
during the gate. The six node containers were recreated twice before any drill (once for the
|
||||
gate's Debug scopes, once to widen them to the ack plane — the "Posted the ApplyAck" sender
|
||||
logs in `Runtime.Communication` and the central receiver in `AdminUI.Grpc`, neither covered
|
||||
by the 1A/2 scope set); same image both times, so no void-run. Check 1 ran before the second
|
||||
recreation (interceptor probes — log-reset does not affect them); checks 2–7 after it, so
|
||||
the final greps cover every drill. Override reverted after the gate, never committed.
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | PSK negatives on both services (no key / wrong key / wrong cluster's key); **unset server key ⇒ all calls rejected** (fail-closed proven live) | PENDING |
|
||||
| 2 | Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand) | PENDING |
|
||||
| 3 | Ack matrix: every node acks exactly once, deployment seals | PENDING |
|
||||
| 4 | Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry) | PENDING |
|
||||
| 5 | Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart) | PENDING |
|
||||
| 6 | **Boundary check: no Akka association between any site container and central** — the Baseline (a) probe now shows no cross-mesh `:4053` traffic (pair-internal remoting only) | PENDING |
|
||||
| 7 | Full-rig restart (pairs together): clean convergence; `grep -ri "receptionist\|clusterclient"` over all node logs = 0 lines | PENDING |
|
||||
| 8 | `MeshGrpcBoundaryTests` + full suite green on the merge SHA | PENDING |
|
||||
| 1 | PSK negatives on both services (no key / wrong key / wrong cluster's key); **unset server key ⇒ all calls rejected** (fail-closed proven live) | **PASS** — 7/7: `Execute` @ site-a-1 no-bearer → `PermissionDenied: Node command authentication failed.`, SITE-B key → same uniform text, SITE-A key + empty envelope → `InvalidArgument` (interceptor ran, payload rejected); `ReportApply` @ central-1 no-bearer / garbage key / SITE-A key claiming `x-otopcua-cluster: SITE-Z` → all `PermissionDenied: Central command authentication failed.` (uniform, unknown-cluster refused = no key configured for that cluster ⇒ every call rejected), SITE-A key + SITE-A claim → `InvalidArgument: undecodable ApplyAck` (positive control). Unset-key fail-closed at the earliest gate: scratch container on the rig image, `Grpc` mode, `GrpcPsk` removed → `Hosting failed to start`, `OptionsValidationException` naming `MeshTransport:GrpcPsk` + the env form (scratch removed after capture) |
|
||||
| 2 | Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand) | **PASS** — deploy `22dfec4b` fanned as ONE envelope: all six nodes logged `accepted DispatchDeployment (correlationId=c532b8f7…)` in the same second. **Restart + Reconnect driven from the real `/hosts` AdminUI panel for the first time in any gate** (the #546 wire-in closed the 1A "structurally undrivable" carry-forward): each fanned to all six under a single correlation id (`41157cfb…` / `cb81bc5e…`), and BOTH SITE-A nodes acted (`restarting driver SITEA-modbus by request of multi-role-test` + `spawned`, `ForceReconnect requested by admin`) — actor = the logged-in operator, i.e. the full `AdminOperationsActor.Dispatch → DispatchOverGrpc` origin path live. AlarmCommand: authenticated envelopes accepted + handed on site-b-1 and central-1 |
|
||||
| 3 | Ack matrix: every node acks exactly once, deployment seals | **PASS** — for `22dfec4b`: every node logged exactly ONE `Posted the ApplyAck … (Applied) to http://central-1:4055`, central-1 served exactly 6 `ReportApply` 200s and handed 6 acks to the central communication actor, `Deployment.Status=2` with 6/6 `NodeDeploymentState` Applied inside a ~30 ms spread |
|
||||
| 4 | Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry) | **PASS** — kill central-1 → deploy `3edf043d`: all five live nodes posted `to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable)` (the ordered walk, one loud INF per node), deadline honest at exactly +2:00 naming only central-1 (5/6). Restart → first re-deploy hit the documented ~2-min `GrpcChannel` backoff (ONE `was NOT delivered … will NOT be retried` WRN, honest TimedOut), next dispatch **sealed 6/6** — no manual action beyond `docker start`. Kill site-a-1 + dispatch in the same second → exactly **1** `NOT delivered to site-a-1 (Unavailable…)` line (no retry storm), 5/6 acks, deadline at exactly +2:00 naming only site-a-1; restart → `FetchAndCache boot — restored served state` and the timed-out deployment NOT retro-acked (count stayed 5) |
|
||||
| 5 | Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart) | **PASS** — deploy `f311cf5b` dispatched and the coordinator host (central-2) killed **in the same second** with the deployment open; **central-1's recovered coordinator fired the deadline at exactly +2:00** (`timed out … (5/6 acks landed). No ack from: site-b-2:4053` — the deliberately-stopped node), with central-2's own ack — posted before its death — preserved in the 5. After restarting both: per-node `Posted the ApplyAck` count for `f311cf5b` exactly 1 on the five that acked / 0 on site-b-2, DB count stayed 5 (no retro-ack, no dupes), site-b-2 boot-replayed last-known-good, and reconverge deploy `12a46546` **sealed 6/6** |
|
||||
| 6 | **Boundary check: no Akka association between any site container and central** | **PASS** — `netstat` in every node's netns: exactly THREE established `:4053` associations rig-wide, each strictly pair-internal (central-1↔central-2, site-a-1↔site-a-2, site-b-1↔site-b-2 by container IP); **zero site↔central `:4053` connections**. Baseline (a)'s unauthenticated cross-mesh association is gone at the TCP level |
|
||||
| 7 | Full-rig restart (pairs together): clean convergence; `grep -ri "receptionist\|clusterclient"` over all node logs = 0 lines | **PASS, with one non-mesh observation** — all six restarted in one `compose restart`; both SITE pairs re-formed cleanly within 7 s; the CENTRAL pair hit the **documented simultaneous-cold-start split-brain** (same-second mutual `InitJoinNack` → both self-formed — the exact scenario the Phase 7 bootstrap guard `d1dac87f` ships for, opt-in and default-OFF, not enabled on this rig; pre-existing pair-formation behavior, zero mesh-transport involvement) and was healed by the standard operator action, a single-node restart of central-2, which rejoined as Secondary (`memberCount=2 unreachableCount=0`, leader central-1 — the Phase 7 rejoin drill behavior holding). Transport verdict clean both ways: `grep -icE "receptionist|clusterclient"` = **0 on all six nodes** over logs covering every drill, and post-restart deploy `11007721` **sealed 6/6**. Whole-gate PSK-drift sweep: **zero** denials on all four interceptors across all six nodes for the entire post-recreation run |
|
||||
| 8 | `MeshGrpcBoundaryTests` + full suite green on the merge SHA | **PASS** — targeted `MeshGrpcBoundaryTests` 3/3; terminal full suite on `master @ 0b194003`: `dotnet test` direct exit-code capture (no pipe) **RC=0, 50/50 suites, 9,333 passed / 0 failed** — identical totals to the Phase-3 DoD baseline, run in parallel with the live rig per the Phase 2 precedent |
|
||||
|
||||
### Phase 4 carry-forward closures
|
||||
|
||||
| Item | Disposition |
|
||||
|---|---|
|
||||
| 1A check-3 AdminUI origin (closed by composition in 1A/2) | **Now closed by direct evidence** — #546 wired `DriverStatusPanel` into `/hosts`, and this gate drove Restart + Reconnect from the real page (fan + act, operator identity carried) |
|
||||
| `Failed to resolve` Error-text cosmetics (1A finding 5 → Phase 2) | **Closed in Phase 2** Task 2.0 (`a94eb8a3`+`d7c9ead6`, honest PreStart-race logging) |
|
||||
| ~2-min `GrpcChannel` backoff after a peer dies (1B → Phase 2 soak note) | **Behavior confirmed three more times this gate** (post-central-1-restart re-deploy; central-2→site-a-1 dispatch skip at 00:10:21 after the drill-B kill). Always loud (one WRN, single attempt), never lossy beyond the honest deadline, always healed by the next dispatch after the window. Stands as an operator note in `docs/Redundancy.md` territory — expect one TimedOut deploy within ~2 min of any node kill/restart |
|
||||
| `GrpcPort` NULL→repair skip persists ≤ `ContactRefreshSeconds` (1A check 7) | Operational note stands (dispatch set rebuilds on the refresh tick); no Phase 4 re-check needed — the mechanism is unchanged since 1A proved it live |
|
||||
| I-4 cold-PSK-cache latency after a refresh tick (1B → Phase 2 → Phase 4) | **Re-carried out of the program with rationale, unchanged:** still structurally unmeasurable on docker-dev — the rig configures `MeshTransport:ClusterPsks`, which answers ahead of the store, so the cache never holds a store-resolved value and `InvalidateAll()` clears nothing. Every ack in every phase landed within the second it was posted. To be measured once on the first store-backed deployment (a real `${secret:OT-MESH-PSK-…}` fleet); until then it is an unquantified-latency note, not a correctness gap — the fail-closed behavior on a cold cache is pinned by unit tests |
|
||||
|
||||
No new defects. The central-pair cold-start split-brain in check 7 is pre-existing, documented
|
||||
(Phase 7 bootstrap guard, deliberately opt-in), reproduced here only because a same-second
|
||||
pair restart is exactly its trigger — recorded as the reason an operator restarting a central
|
||||
pair together should either stagger the two nodes or enable `Cluster:BootstrapGuard`.
|
||||
|
||||
**Program CLOSED.** Both baseline gaps are shut on the wire: (a) zero cross-mesh Akka
|
||||
associations and zero receptionist/ClusterClient log lines on the post-deletion binary;
|
||||
(b) an unreachable node now costs one loud per-node WRN in the dispatch second plus an honest
|
||||
deadline, instead of a sampled dead-letter counter and silence.
|
||||
|
||||
### Carry-forwards
|
||||
|
||||
|
||||
@@ -3,6 +3,11 @@
|
||||
**This is the single place to see program state.** Update it at every phase boundary, merge,
|
||||
defect, and decision. Everything else (design, plan, live gate) is referenced from here.
|
||||
|
||||
> **PROGRAM CLOSED 2026-08-06** — all phases 0/1A/1B/2/3/4 complete and merged; the Phase 4
|
||||
> terminal live gate PASSED 8/8 on `master @ 0b194003` (see the phase board and the live-gate
|
||||
> doc). Sole item leaving the program open-ended: the I-4 cold-PSK-cache latency measurement,
|
||||
> re-carried to the first store-backed deployment (structurally unmeasurable on docker-dev).
|
||||
|
||||
**Program goal:** all central↔site mesh traffic on gRPC with per-cluster PSK auth;
|
||||
`ClusterClient`/`ClusterClientReceptionist` deleted; Akka remoting never crosses the mesh
|
||||
boundary. Mirrors ScadaBridge's completed 2026-07-22 migration (same cluster boxes, same end
|
||||
@@ -26,8 +31,8 @@ posture).
|
||||
| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **COMPLETE @ `e54883a8` — gate re-run PASSED 2026-08-06.** Run 1 @ `fbf7b708` FAILED on defect 4 (no admin node could boot, mode-independent); fixed by `c0475f2d` + `e54883a8` and re-gated. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅. **MERGED to `master`** @ `b360239e` (2026-08-06) | `b360239e` | **PASS 8/8** (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards) |
|
||||
| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebased on merged 1A | `feat/grpc-central-ack` | **MERGED to `master`** @ `7203d135` (2026-08-06, branch tip `cf229a95` incl. the phases 2-4 plan update); 1B.1 ✅ + 1B.2 ✅ (Opus: deadline Critical found+fixed) + gate **PASS 7/7** incl. the 4b hang drill (~10 s failover = exactly one AckTimeoutSeconds — live proof of the fix); at-least-once duplicate absorbed idempotently exactly as traced | `7203d135` | **PASS 7/7** |
|
||||
| 2 | **COMPLETE — gate PASS 8/8 + baseline + PSK negatives** (2026-08-06, all on `master`, all review chains closed). Task 2.0 ✅ (`a94eb8a3`+`d7c9ead6`, small chain: one Important fixed — non-vacuous once-per-incarnation pin — then APPROVED; healthy racing boot = ZERO Warn/Error, proven live on both centrals 4/4 boots). Register-item-3 role-scoping pin ✅ (`2e06dfe5`+`027f0403`, review found+fixed a Critical phantom-block false pass in the scan — braceless `if (hasAdmin)` at Program.cs:146). Baseline REPRODUCED pre-flip (`1bb3a937`): cross-mesh `:4053` open unauthenticated; the site-b drop is actively suppressed into SAMPLED deadLetters INF (456/467 unlogged) with the deploy deadline as sole witness. Cutover `caf2eec6` (rig default → `Grpc`, committed). Gate: deploy e2e all-gRPC sealed 6/6; command matrix live incl. real driver restart; kill-site-a-1 → per-node WRN naming node+status, exactly 1 attempt, honest deadline, boot-replay-only recovery; kill-central-1 → ack ordered-walk failover logged per node, reconvergence 6/6; site-b-pair drill → BOTH failures logged in the dispatch second (the baseline contrast); clusterclient/receptionist lines = 0 on all six through the whole gate; 135-min soak 14/14 cycles, denied=0/unauth=0, review items explained (historian-sidecar-absent DBG noise; 14×2 known per-deploy authoring WRNs); **full suite exit 0 on the cutover SHA, run against the live rig** | `caf2eec6` (cutover) + gate records `1bb3a937`/`bcc91268`/`eaf14e01`+final | **PASS 8/8** |
|
||||
| 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | **MERGED to `master` @ `2fa880b6` and pushed** (2026-08-06, user-approved; branch tip `f8a43dfe`, 11 commits off `master @ 55104526`). Phase 4 NOT started — user has not yet said proceed. Task 3.1 ✅ high-risk chain: 7 ordered steps `0db40a5e`/`ab5b958d`/`5d7ecdf7`/`d67a7e5d`/`85ca86e9`/`af3b0c31`/`4d2dfb5b`; spec review ✅ (all KEEPs verified: `LoadContactsFromDb`, DPS+frame-size canary, `deployment-acks` DPS ingress, Kestrel block, one-ctor interceptors, no retries, proto untouched; `MeshGrpcBoundaryTests`' no-association assert verified falsifiable); code review ✅ Approved, one Important fixed (`cd713557` — `MeshPaths` full-path constants were dead-in-production + doc claimed a wire-contract role they no longer have; collapsed to `*Name` constants). Retired-mode fail-closed: `Mode=ClusterClient` now FAILS validation naming `MeshTransport:Mode` + allowed values (pinned); `SplitTopologyTransportValidator` requires exactly `Grpc`. Task 3.2 ✅ docs sweep `406b1ae5` + review-fix `8e25ed55` (Configuration/Redundancy/Telemetry/config-db-schema/CLAUDE.md; deferment.md had no mention); docs-accuracy review ✅ Approved — every config key/port/behavior claim verified against source, two pre-existing Minor stalenesses fixed (Telemetry.md Phase 5 gate status; config-db-schema Phase 6 gossip premise). scadaproj index updated on its `main` and PUSHED (`0e416ca`, merge-state-accurate: Phase 3 recorded as pending merge). Grep-gates: `CentralContactPoints` in src/tests/docker-dev = 0; `clusterclient\|receptionist` = historical/fail-closed/retirement mentions only (classified by both reviewers). **DoD full-suite: run 3 = `dotnet test` exit 0, 50/50 suites, 9,326 passed / 0 failed / 57 skipped, on `8e25ed55` (docs-only commits followed).** Runs 1–2 each had ONE unrelated single-test failure (three distinct tests across the two runs: `RouteNativeAlarmAck_on_non_primary_is_dropped`, `Rediscovery_flag_persists_on_subsequent_health_publishes`, `LoggerLifecycleTests.ConfigureLogging_DisposesPreviousLogger…`) — all in files the 55-file branch diff never touches, all 3/3 green in isolation (9/9 total): pre-existing under-load timing flakes, not Phase 3 regressions. ⚠️ Process find: run 1's "exit 0" captured `tail`'s exit code, not `dotnet test`'s (`cmd \| tail; $?` trap) — fixed for runs 2–3; **Phase 2's check-8 "full suite exit 0" used the same pipeline shape and its log did not survive, so that exit-code claim is unverifiable in hindsight** (its per-suite pass lines were observed at the time; Phase 4's terminal run on the merge SHA, with correct capture, supersedes it) | `2fa880b6` | n/a (Phase 4 is the terminal live gate) |
|
||||
| 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | NOT STARTED | — | PENDING |
|
||||
| 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | **MERGED to `master` @ `2fa880b6` and pushed** (2026-08-06, user-approved; branch tip `f8a43dfe`, 11 commits off `master @ 55104526`). Phase 4 ran later the same day on the user's go-ahead — see its row. Task 3.1 ✅ high-risk chain: 7 ordered steps `0db40a5e`/`ab5b958d`/`5d7ecdf7`/`d67a7e5d`/`85ca86e9`/`af3b0c31`/`4d2dfb5b`; spec review ✅ (all KEEPs verified: `LoadContactsFromDb`, DPS+frame-size canary, `deployment-acks` DPS ingress, Kestrel block, one-ctor interceptors, no retries, proto untouched; `MeshGrpcBoundaryTests`' no-association assert verified falsifiable); code review ✅ Approved, one Important fixed (`cd713557` — `MeshPaths` full-path constants were dead-in-production + doc claimed a wire-contract role they no longer have; collapsed to `*Name` constants). Retired-mode fail-closed: `Mode=ClusterClient` now FAILS validation naming `MeshTransport:Mode` + allowed values (pinned); `SplitTopologyTransportValidator` requires exactly `Grpc`. Task 3.2 ✅ docs sweep `406b1ae5` + review-fix `8e25ed55` (Configuration/Redundancy/Telemetry/config-db-schema/CLAUDE.md; deferment.md had no mention); docs-accuracy review ✅ Approved — every config key/port/behavior claim verified against source, two pre-existing Minor stalenesses fixed (Telemetry.md Phase 5 gate status; config-db-schema Phase 6 gossip premise). scadaproj index updated on its `main` and PUSHED (`0e416ca`, merge-state-accurate: Phase 3 recorded as pending merge). Grep-gates: `CentralContactPoints` in src/tests/docker-dev = 0; `clusterclient\|receptionist` = historical/fail-closed/retirement mentions only (classified by both reviewers). **DoD full-suite: run 3 = `dotnet test` exit 0, 50/50 suites, 9,326 passed / 0 failed / 57 skipped, on `8e25ed55` (docs-only commits followed).** Runs 1–2 each had ONE unrelated single-test failure (three distinct tests across the two runs: `RouteNativeAlarmAck_on_non_primary_is_dropped`, `Rediscovery_flag_persists_on_subsequent_health_publishes`, `LoggerLifecycleTests.ConfigureLogging_DisposesPreviousLogger…`) — all in files the 55-file branch diff never touches, all 3/3 green in isolation (9/9 total): pre-existing under-load timing flakes, not Phase 3 regressions. ⚠️ Process find: run 1's "exit 0" captured `tail`'s exit code, not `dotnet test`'s (`cmd \| tail; $?` trap) — fixed for runs 2–3; **Phase 2's check-8 "full suite exit 0" used the same pipeline shape and its log did not survive, so that exit-code claim is unverifiable in hindsight** (its per-suite pass lines were observed at the time; Phase 4's terminal run on the merge SHA, with correct capture, supersedes it) | `2fa880b6` | n/a (Phase 4 is the terminal live gate) |
|
||||
| 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | **COMPLETE — gate PASS 8/8, 2026-08-06, PROGRAM CLOSED.** Ran on `master @ 0b194003` (Phase 3 merge `2fa880b6` + the #542–#546 fix merges + docs); provenance guard PASS (image `d2a290fc5711` built 17:55:10 EDT > deletion merge 16:57:05 > last code merge `9420f8ef` 17:28:48). Full record in the live-gate doc; headline results: PSK matrix 7/7 incl. unset-key boot refusal naming `MeshTransport:GrpcPsk`; command matrix with **Restart+Reconnect driven from the real `/hosts` panel for the first time** (#546 closed the 1A composition carry-forward by direct evidence — fan to six under one correlation id, both SITE-A nodes acted, operator identity carried); ack matrix exactly-once ×6 sealed in ~30 ms; central-kill ack failover (ordered walk to central-2, one INF per node) + site-kill dispatch honesty (exactly 1 WRN, +2:00 deadline, boot-replay, no retro-ack); **mid-deploy coordinator kill: central-2 killed in the dispatch second, central-1's recovered coordinator fired the deadline at exactly +2:00 with all 5 landed acks preserved (incl. the dead node's own), zero dupes across both restarts, reconverge sealed 6/6**; boundary: exactly 3 established `:4053` associations rig-wide, all pair-internal, zero site↔central; full-rig restart → transport grep `receptionist\|clusterclient` = **0 on all six**, post-restart deploy sealed 6/6 (one non-mesh observation: the CENTRAL pair reproduced the documented guard-off simultaneous-cold-start split-brain — Phase 7 territory, opt-in `Cluster:BootstrapGuard` exists, healed by single-node rejoin; both SITE pairs re-formed clean); `MeshGrpcBoundaryTests` 3/3; **terminal full suite RC=0 direct capture, 50/50 suites, 9,333 passed / 0 failed** (supersedes Phase 2's unverifiable check-8 exit code). Whole-gate PSK-drift sweep 0 denials ×6 nodes ×4 interceptors. I-4 cold-PSK-cache re-carried out of the program (docker-dev structurally cannot measure it; first store-backed deployment measures it). No new defects. Debug override reverted; rig left on the committed compose | n/a (gate only) | **PASS 8/8** |
|
||||
|
||||
**Execution rules recap:** phase order 0 → (1A ∥ 1B) → 2 → 3 → 4; 1A∥1B only in separate git
|
||||
worktrees, one Opus subagent per worktree, sequential tasks within a worktree; 1A merges
|
||||
@@ -145,13 +150,13 @@ Checks a gate could not drive, named + carried to the next gate (proportionate-g
|
||||
| Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter`; starter-order half now covered by `ControlPlaneStarterOrderTests` | **CLOSED in Phase 2** (`2e06dfe5`+`027f0403`): source-scan pin in Host.Tests, review-hardened — a Critical phantom-block false pass (braceless `if (hasAdmin)` at Program.cs:146 made the scanner adopt the `if (hasDriver)` block's brace) was found and fixed, both bite-checks proven |
|
||||
| `Resolve`-failure Error wording says "continuing without it" (reads permanent — the memoize heals it at first use) and the line fires on real boots (2/3 in the gate) — reword or whitelist it before the Phase 2 soak error-grep, else a healthy central scores as failing the soak | 1A.3 re-run | heal proven live: singleton moved onto the race-losing node, fan clean, no dialer-missing error | **CLOSED by Task 2.0** (`a94eb8a3`+`d7c9ead6`): PreStart resolve logs Debug-only; Error moved to first-use sites (per-dispatch for the dialer, once-per-incarnation for the PSK provider); healthy racing boot = 0 Warn/Error, pinned + proven live (soak grep ran clean with NO whitelist entry for this path) |
|
||||
| A repaired `GrpcPort` takes effect only at the next contact refresh (≤60 s): a deploy 5 s post-restore was still skipped and honestly TimedOut — by design, operator-visible; document if it surprises in Phase 2 | 1A.3 re-run | check 7 evidence | Phase 2: not exercised (no GrpcPort repair drill was in the check table) and nothing surprised; behaviour unchanged by design — note stands for operators |
|
||||
| I-4 (cold PSK cache on the inbound auth path after each refresh tick) is structurally unmeasurable on docker-dev — `ClusterPsks` config map answers ahead of the store, so the cache is always empty and `InvalidateAll` clears nothing | 1B.3 gate | every ack landed within the posting second | **Re-carried to Phase 4** with the same rationale — docker-dev still configures `ClusterPsks`, no store-backed rig existed during Phase 2 |
|
||||
| After a central restart, a node's cached `GrpcChannel` can sit in transient failure ~2 min (client backoff; TCP open, sibling reaches the same endpoint) — no ack lost, the walk absorbs it loudly. Expect a failover-line burst after any central restart in the Phase 2 soak | 1B.3 gate | 3 dispatches: 2 failed over, 3rd direct | **Phase 2 OBSERVED, on the command dial**: after the check-3 site-a-1 kill, central-2's cached channel dropped the check-4 dispatch to the already-healthy site-a-1 (loud, single-attempt, honest deadline) ~3 min post-kill; healed by the next deploy which sealed 6/6. Soak itself had no kills, so no burst inside the soak window. Behaviour understood; carry the operator note to Phase 4's drills |
|
||||
| I-4 (cold PSK cache on the inbound auth path after each refresh tick) is structurally unmeasurable on docker-dev — `ClusterPsks` config map answers ahead of the store, so the cache is always empty and `InvalidateAll` clears nothing | 1B.3 gate | every ack landed within the posting second | **Re-carried OUT of the program at Phase 4** (2026-08-06): docker-dev still configures `ClusterPsks`, so the cache never holds a store-resolved value and the latency cannot exist on this rig. Every ack in every gate landed within its posting second. To be measured once on the first store-backed deployment; cold-cache fail-closed behavior is pinned by unit tests |
|
||||
| After a central restart, a node's cached `GrpcChannel` can sit in transient failure ~2 min (client backoff; TCP open, sibling reaches the same endpoint) — no ack lost, the walk absorbs it loudly. Expect a failover-line burst after any central restart in the Phase 2 soak | 1B.3 gate | 3 dispatches: 2 failed over, 3rd direct | **Phase 2 OBSERVED, on the command dial**: after the check-3 site-a-1 kill, central-2's cached channel dropped the check-4 dispatch to the already-healthy site-a-1 (loud, single-attempt, honest deadline) ~3 min post-kill; healed by the next deploy which sealed 6/6. **Phase 4 confirmed three more occurrences** (post-central-1-restart re-deploy; central-2→site-a-1 skip after the drill-B kill) — always loud, never lossy beyond the honest deadline, healed by the next dispatch after the ~2-min window. Standing operator note: expect one TimedOut deploy within ~2 min of any node kill/restart |
|
||||
| **The PreStart race fires on essentially EVERY admin boot** (4/4 in the 1B gate, 2/3 in 1A) and logs two FALSE Errors ("continuing without it", "gate will NEVER be published") that the memoize immediately heals — the wording fix is now a MUST before the Phase 2 soak error-grep, not a nicety | 1B.3 gate obs (b) | gate published + acks accepted on every one of those boots | **CLOSED — Task 2.0 shipped as the Phase 2 opening task** (`a94eb8a3`+`d7c9ead6`) and verified live: both centrals booted with zero resolve-path Warn/Error on the cutover rig |
|
||||
| `RevisionHash` moved on byte-identical config after a direct SQL `UPDATE` bumped `Script.RowVersion` (hash covers a column content-identity does not); determinism itself holds (3 no-change dispatches → identical hash) | 1A.3 re-run | incidental, out of program scope | note only |
|
||||
| 1A checks 1/2/7 (deploy fan over gRPC, mixed-transport seal, NULL-`GrpcPort` skip) and the central half of 4 | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — all driven live and PASS |
|
||||
| Check 8 is vacuous for telemetry + config-serve — central never booted, so neither plane carried a call | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — both planes carried authenticated traffic (28 KB artifact fetches from `:4055`, telemetry streams open on all six) with 0 denials |
|
||||
| **`DriverStatusPanel.razor` is referenced by no page or component** — the Reconnect/Restart buttons are unreachable in the running AdminUI (pre-existing on `master`, not caused by this program; re-verified in run 2: 0 `<DriverStatusPanel` usages, no Restart/Reconnect control on any rendered page, no such HTTP endpoint) | 1A.3 gate | Check 3 **closed by composition**: run 1's live node-side leg for both commands + run 2's check 4 driving the identical `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path from the AdminUI | Out of program scope — file separately if the control is meant to exist |
|
||||
| **`DriverStatusPanel.razor` is referenced by no page or component** — the Reconnect/Restart buttons are unreachable in the running AdminUI (pre-existing on `master`, not caused by this program; re-verified in run 2: 0 `<DriverStatusPanel` usages, no Restart/Reconnect control on any rendered page, no such HTTP endpoint) | 1A.3 gate | Check 3 **closed by composition**: run 1's live node-side leg for both commands + run 2's check 4 driving the identical `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path from the AdminUI | Filed as **#546**, FIXED + merged (`9420f8ef`) — panel wired into `/hosts`. **Phase 4 then drove Restart + Reconnect from the real page live** (fan to six, both SITE-A nodes acted, operator identity carried), converting the composition closure into direct evidence |
|
||||
| `Failed to resolve` Error text reads as permanent though first use retries, and will trip a soak error-grep on a healthy node | 1A.3 gate run 2 (defect 5) | Self-heal proven live | Phase 2 |
|
||||
|
||||
## Issues filed by this program
|
||||
|
||||
Reference in New Issue
Block a user