From 5510452662448f828f5fc4a851c8eeb19dbfeeaf Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 6 Aug 2026 13:26:56 -0400 Subject: [PATCH] =?UTF-8?q?docs(plans):=20Phase=202=20COMPLETE=20=E2=80=94?= =?UTF-8?q?=20gate=20PASS=208/8,=20soak=2014/14=20clean,=20full=20suite=20?= =?UTF-8?q?exit=200=20on=20cutover=20SHA;=20carry-forwards=20closed/re-car?= =?UTF-8?q?ried?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1 --- .../2026-08-06-clusterclient-to-grpc-live-gate.md | 6 +++--- .../2026-08-06-clusterclient-to-grpc-tracking.md | 14 +++++++------- .../2026-08-06-clusterclient-to-grpc.md.tasks.json | 8 ++++++-- 3 files changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md b/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md index 19095e2d..ef5acf59 100644 --- a/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md +++ b/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md @@ -1086,7 +1086,7 @@ restored; revision hash back to `642b9d6f79b9…`). --- -## Phase 2 — full-rig cutover + soak — **PENDING** +## Phase 2 — full-rig cutover + soak — **PASS 8/8 + baseline + PSK negatives** Rig default flipped and committed: `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"` (`caf2eec6`, all six node entries). Rig rebuilt from `master @ caf2eec6` — which also carries @@ -1102,8 +1102,8 @@ pre-flip build first (see Baseline section, `1bb3a937`). | 3 | Kill site-a-1 mid-dispatch: per-node `Execute` failure **logged with node id + status** (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path) | **PASS** — deploy `069b57a4` 1 s after `docker kill`: central-2 WRN `DispatchDeployment … was NOT delivered to site-a-1:4053 (Unavailable: Error connecting to subchannel.). It is dropped and will NOT be retried…` — exactly **1** such line (no retry storm); 5/6 acked; `timed out after 00:02:00 (5/6 acks landed). No ack from: site-a-1:4053` at exactly +2:00; restart → FetchAndCache boot-replay restored last-known-good, timed-out deployment NOT retro-acked (ack count stayed 5) | | 4 | Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action | **PASS** — deploy `ffc75eae` with central-1 dead: site nodes logged `Posted the ApplyAck … to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable)` (the ordered walk, per node), all live acks landed on central-2; deadline honest at +2:00 naming `central-1, site-a-1` (site-a-1 = the documented ~2-min `GrpcChannel` backoff carry-forward from its own earlier kill — loud, single-attempt). First post-restart probe caught central-1 still booting (5/6, honest TimedOut); second probe `594ecda3` **sealed 6/6** — full convergence, no manual action beyond `docker start` | | 5 | Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; **no duplicate applies** after the pair returns (boot-replay path only) | **PASS** — deploy `e5a14c5f`: BOTH `NOT delivered to site-b-{1,2}:4053 (Unavailable…)` WRN lines in the SAME SECOND as dispatch (baseline (b): zero correlated lines, sampled-deadLetters INF); `timed out after 00:02:00 (4/6…). No ack from: site-b-1:4053, site-b-2:4053`; pair restart → boot-replay only (`FetchAndCache boot — restored served state`), rows stay 0/0, ack count stays 4 — no dupes, no resurrection | -| 6 | **Zero ClusterClient activity under `Grpc`:** `docker logs \| grep -ci "clusterclient"` = 0 new lines post-boot | **PASS (interim + final at soak end)** — `grep -ic "clusterclient\|receptionist"` = **0 on all six nodes** after checks 1–5 (incl. every kill/restart cycle) | -| 7 | Soak ≥ 2 h under periodic deploys + command traffic: 0 `PermissionDenied`/`Unauthenticated` across all six nodes (PSK drift detector), 0 unexplained `Unavailable` | **RUNNING** — driver started 15:03:44Z, 135-min target: deploy+seal check per 10-min cycle + Restart(site-a)/AlarmCommand(site-b) traffic; whitelist-aware final grep (excludes the standby `/health/active` 503 + `Health check cluster-primary` ERR noise — Health 0.3.0 by-design standby behaviour, pre-existing) | +| 6 | **Zero ClusterClient activity under `Grpc`:** `docker logs \| grep -ci "clusterclient"` = 0 new lines post-boot | **PASS** — final sweep after the full soak: `grep -ic "clusterclient\|receptionist"` = **0 on all six nodes**, covering checks 1–5, every kill/restart cycle, and the whole 135-min soak | +| 7 | Soak ≥ 2 h under periodic deploys + command traffic: 0 `PermissionDenied`/`Unauthenticated` across all six nodes (PSK drift detector), 0 unexplained `Unavailable` | **PASS** — 135 min (15:03:44Z→17:18Z), **14/14 deploy cycles sealed 6/6** (most in 0–1 s) + Restart/AlarmCommand traffic every cycle, 0 traffic failures. Grep since soak start: **PermissionDenied=0, Unauthenticated=0 on all six** (PSK drift detector clean). Review items, both explained: (a) site-a-2 `Unavailable`×3807 — ALL at DBG/INF (its WRN/ERR count was 0), the alarm-historian `SendEvent` retry loop failing DNS on the HistorianGateway host, which does not exist on this rig — pre-existing rig-shape noise, not mesh; (b) central-2 WRN×28 = exactly 14 deploys × 2 known per-deploy StartDeployment authoring warnings (`TAG-gatelong` ModbusRegion + script-compile RSS advisory) — pre-existing on the old transport. The predicted post-central-restart failover-line burst did not occur during the soak because no node was killed inside the soak window (kills all preceded it) | | 8 | Full suite green on the cutover SHA | **PASS** — `dotnet test ZB.MOM.WW.OtOpcUa.slnx --nologo` exit 0, **zero failures across every assembly**, run in parallel with the live rig + soak traffic. Spot totals match every baseline: ControlPlane 245 (Task 2.0 set), Host 95 (role-scoping pins), Runtime 558+3skip, AdminUI 1109, Commons 346, Cluster 193, Host.IntegrationTests 212+4skip | **Extra (plan-text requirement): PSK negatives on RUNNING containers, both services** — **PASS**: diff --git a/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md b/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md index c31f7ced..30d37b03 100644 --- a/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md +++ b/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md @@ -25,7 +25,7 @@ posture). | 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **MERGED to `master`** @ `10b6d209` (2026-08-06) — all 6 tasks complete @ `23555ed0`; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed | `10b6d209` | **PASS 4/4** | | 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **COMPLETE @ `e54883a8` — gate re-run PASSED 2026-08-06.** Run 1 @ `fbf7b708` FAILED on defect 4 (no admin node could boot, mode-independent); fixed by `c0475f2d` + `e54883a8` and re-gated. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅. **MERGED to `master`** @ `b360239e` (2026-08-06) | `b360239e` | **PASS 8/8** (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards) | | 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebased on merged 1A | `feat/grpc-central-ack` | **MERGED to `master`** @ `7203d135` (2026-08-06, branch tip `cf229a95` incl. the phases 2-4 plan update); 1B.1 ✅ + 1B.2 ✅ (Opus: deadline Critical found+fixed) + gate **PASS 7/7** incl. the 4b hang drill (~10 s failover = exactly one AckTimeoutSeconds — live proof of the fix); at-least-once duplicate absorbed idempotently exactly as traced | `7203d135` | **PASS 7/7** | -| 2 | **Opens with Task 2.0 (MUST FIX): honest PreStart-race logging** (both gates: fires ~every admin boot, two false Errors — soak error-grep impossible without it). Then baseline reproduction on pre-flip build, full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills. **Targeted-test policy in force (plan Phase 2 header): per-task unit runs targeted; ONE full-suite run per phase at the DoD** | (rig commit; 2.0 on master) | NOT STARTED | — | PENDING | +| 2 | **COMPLETE — gate PASS 8/8 + baseline + PSK negatives** (2026-08-06, all on `master`, all review chains closed). Task 2.0 ✅ (`a94eb8a3`+`d7c9ead6`, small chain: one Important fixed — non-vacuous once-per-incarnation pin — then APPROVED; healthy racing boot = ZERO Warn/Error, proven live on both centrals 4/4 boots). Register-item-3 role-scoping pin ✅ (`2e06dfe5`+`027f0403`, review found+fixed a Critical phantom-block false pass in the scan — braceless `if (hasAdmin)` at Program.cs:146). Baseline REPRODUCED pre-flip (`1bb3a937`): cross-mesh `:4053` open unauthenticated; the site-b drop is actively suppressed into SAMPLED deadLetters INF (456/467 unlogged) with the deploy deadline as sole witness. Cutover `caf2eec6` (rig default → `Grpc`, committed). Gate: deploy e2e all-gRPC sealed 6/6; command matrix live incl. real driver restart; kill-site-a-1 → per-node WRN naming node+status, exactly 1 attempt, honest deadline, boot-replay-only recovery; kill-central-1 → ack ordered-walk failover logged per node, reconvergence 6/6; site-b-pair drill → BOTH failures logged in the dispatch second (the baseline contrast); clusterclient/receptionist lines = 0 on all six through the whole gate; 135-min soak 14/14 cycles, denied=0/unauth=0, review items explained (historian-sidecar-absent DBG noise; 14×2 known per-deploy authoring WRNs); **full suite exit 0 on the cutover SHA, run against the live rig** | `caf2eec6` (cutover) + gate records `1bb3a937`/`bcc91268`/`eaf14e01`+final | **PASS 8/8** | | 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — | | 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | NOT STARTED | — | PENDING | @@ -142,12 +142,12 @@ Checks a gate could not drive, named + carried to the next gate (proportionate-g | Hang-style central failover drill (`docker pause`, not `docker stop` — only a hang discriminates the per-endpoint-deadline fix) | 1B.2 review | unit fact `Each_endpoint_gets_a_FRESH_deadline` | 1B.3 gate (MUST include) | | I-4 measurement: ack burst within ~3 s after a contact-refresh tick (cold PSK cache on the inbound auth path) | 1A.2 review | bounded 3 s store read + `Lazy` stampede suppression | 1B.3 gate (observe) | | `TwoNodeClusterHarness` logs the I-1 null-provider Error at every spawn (harness registers no `IMeshPskProvider`) — EXPECTED, not a rig fault | 1A.2 re-review | production pairing is `hasAdmin`-unconditional in one block | note for 1B.3/Phase 2 readers | -| Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter`; starter-order half now covered by `ControlPlaneStarterOrderTests` | Phase 2 (or 1B.3) — note defect 4 shows the missing pin is not merely role-scoping but **starter-order**: the container-activation test resolves from a collection without `AddZbSecrets` | -| `Resolve`-failure Error wording says "continuing without it" (reads permanent — the memoize heals it at first use) and the line fires on real boots (2/3 in the gate) — reword or whitelist it before the Phase 2 soak error-grep, else a healthy central scores as failing the soak | 1A.3 re-run | heal proven live: singleton moved onto the race-losing node, fan clean, no dialer-missing error | Phase 2 (soak grep design) | -| A repaired `GrpcPort` takes effect only at the next contact refresh (≤60 s): a deploy 5 s post-restore was still skipped and honestly TimedOut — by design, operator-visible; document if it surprises in Phase 2 | 1A.3 re-run | check 7 evidence | Phase 2 (observe) | -| I-4 (cold PSK cache on the inbound auth path after each refresh tick) is structurally unmeasurable on docker-dev — `ClusterPsks` config map answers ahead of the store, so the cache is always empty and `InvalidateAll` clears nothing | 1B.3 gate | every ack landed within the posting second | Phase 2+ (needs a store-backed rig) | -| After a central restart, a node's cached `GrpcChannel` can sit in transient failure ~2 min (client backoff; TCP open, sibling reaches the same endpoint) — no ack lost, the walk absorbs it loudly. Expect a failover-line burst after any central restart in the Phase 2 soak | 1B.3 gate | 3 dispatches: 2 failed over, 3rd direct | Phase 2 (soak expectations) | -| **The PreStart race fires on essentially EVERY admin boot** (4/4 in the 1B gate, 2/3 in 1A) and logs two FALSE Errors ("continuing without it", "gate will NEVER be published") that the memoize immediately heals — the wording fix is now a MUST before the Phase 2 soak error-grep, not a nicety | 1B.3 gate obs (b) | gate published + acks accepted on every one of those boots | Phase 2 opening task | +| Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter`; starter-order half now covered by `ControlPlaneStarterOrderTests` | **CLOSED in Phase 2** (`2e06dfe5`+`027f0403`): source-scan pin in Host.Tests, review-hardened — a Critical phantom-block false pass (braceless `if (hasAdmin)` at Program.cs:146 made the scanner adopt the `if (hasDriver)` block's brace) was found and fixed, both bite-checks proven | +| `Resolve`-failure Error wording says "continuing without it" (reads permanent — the memoize heals it at first use) and the line fires on real boots (2/3 in the gate) — reword or whitelist it before the Phase 2 soak error-grep, else a healthy central scores as failing the soak | 1A.3 re-run | heal proven live: singleton moved onto the race-losing node, fan clean, no dialer-missing error | **CLOSED by Task 2.0** (`a94eb8a3`+`d7c9ead6`): PreStart resolve logs Debug-only; Error moved to first-use sites (per-dispatch for the dialer, once-per-incarnation for the PSK provider); healthy racing boot = 0 Warn/Error, pinned + proven live (soak grep ran clean with NO whitelist entry for this path) | +| A repaired `GrpcPort` takes effect only at the next contact refresh (≤60 s): a deploy 5 s post-restore was still skipped and honestly TimedOut — by design, operator-visible; document if it surprises in Phase 2 | 1A.3 re-run | check 7 evidence | Phase 2: not exercised (no GrpcPort repair drill was in the check table) and nothing surprised; behaviour unchanged by design — note stands for operators | +| I-4 (cold PSK cache on the inbound auth path after each refresh tick) is structurally unmeasurable on docker-dev — `ClusterPsks` config map answers ahead of the store, so the cache is always empty and `InvalidateAll` clears nothing | 1B.3 gate | every ack landed within the posting second | **Re-carried to Phase 4** with the same rationale — docker-dev still configures `ClusterPsks`, no store-backed rig existed during Phase 2 | +| After a central restart, a node's cached `GrpcChannel` can sit in transient failure ~2 min (client backoff; TCP open, sibling reaches the same endpoint) — no ack lost, the walk absorbs it loudly. Expect a failover-line burst after any central restart in the Phase 2 soak | 1B.3 gate | 3 dispatches: 2 failed over, 3rd direct | **Phase 2 OBSERVED, on the command dial**: after the check-3 site-a-1 kill, central-2's cached channel dropped the check-4 dispatch to the already-healthy site-a-1 (loud, single-attempt, honest deadline) ~3 min post-kill; healed by the next deploy which sealed 6/6. Soak itself had no kills, so no burst inside the soak window. Behaviour understood; carry the operator note to Phase 4's drills | +| **The PreStart race fires on essentially EVERY admin boot** (4/4 in the 1B gate, 2/3 in 1A) and logs two FALSE Errors ("continuing without it", "gate will NEVER be published") that the memoize immediately heals — the wording fix is now a MUST before the Phase 2 soak error-grep, not a nicety | 1B.3 gate obs (b) | gate published + acks accepted on every one of those boots | **CLOSED — Task 2.0 shipped as the Phase 2 opening task** (`a94eb8a3`+`d7c9ead6`) and verified live: both centrals booted with zero resolve-path Warn/Error on the cutover rig | | `RevisionHash` moved on byte-identical config after a direct SQL `UPDATE` bumped `Script.RowVersion` (hash covers a column content-identity does not); determinism itself holds (3 no-change dispatches → identical hash) | 1A.3 re-run | incidental, out of program scope | note only | | 1A checks 1/2/7 (deploy fan over gRPC, mixed-transport seal, NULL-`GrpcPort` skip) and the central half of 4 | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — all driven live and PASS | | Check 8 is vacuous for telemetry + config-serve — central never booted, so neither plane carried a call | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — both planes carried authenticated traffic (28 KB artifact fetches from `:4055`, telemetry streams open on all six) with 0 denials | diff --git a/docs/plans/2026-08-06-clusterclient-to-grpc.md.tasks.json b/docs/plans/2026-08-06-clusterclient-to-grpc.md.tasks.json index f5e13fca..63008cbf 100644 --- a/docs/plans/2026-08-06-clusterclient-to-grpc.md.tasks.json +++ b/docs/plans/2026-08-06-clusterclient-to-grpc.md.tasks.json @@ -194,12 +194,14 @@ "branch": "(rig commit)", "classification": "high-risk", "parallelizableWith": [], - "status": "pending", + "status": "completed", "blockedBy": [ 8, 11, 16 ], + "commit": "caf2eec6 (cutover) + gate records 1bb3a937/bcc91268/eaf14e01+final; role-scoping pin 2e06dfe5+027f0403", + "gate": "PASS 8/8 + baseline reproduced + PSK negatives; soak 135min 14/14 cycles denied=0/unauth=0; full suite exit 0 on cutover SHA", "note": "Targeted-test policy applies (see plan Phase 2 header): per-task runs targeted; ONE full-suite run at the phase DoD." }, { @@ -245,11 +247,13 @@ "branch": "(master, post-1B-merge, pre-flip)", "classification": "small", "parallelizableWith": [], - "status": "pending", + "status": "completed", "blockedBy": [ 8, 11 ], + "commit": "a94eb8a3 + review fixes d7c9ead6", + "review": "code ✅ (sonnet; 1 Important fixed: non-vacuous once-per-incarnation pin; re-review APPROVED)", "note": "Both gates: race fires ~every admin boot, two FALSE Errors; soak error-grep impossible without this. Targeted tests only (ControlPlaneStarterOrderTests + CentralCommunicationActorTests)." } ],