From cf229a957901e9d9a7435a72a2ee3a0ca5bbafba Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 6 Aug 2026 09:24:16 -0400 Subject: [PATCH] =?UTF-8?q?docs(plans):=20phases=202-4=20=E2=80=94=20targe?= =?UTF-8?q?ted-test=20policy=20+=20Task=202.0=20(must-fix:=20honest=20PreS?= =?UTF-8?q?tart-race=20logging)=20before=20the=20cutover?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...26-08-06-clusterclient-to-grpc-tracking.md | 2 +- .../plans/2026-08-06-clusterclient-to-grpc.md | 76 +++++++++++++++++-- ...-08-06-clusterclient-to-grpc.md.tasks.json | 20 ++++- 3 files changed, 89 insertions(+), 9 deletions(-) diff --git a/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md b/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md index ee8fa126..5052bf10 100644 --- a/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md +++ b/docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md @@ -25,7 +25,7 @@ posture). | 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **MERGED to `master`** @ `10b6d209` (2026-08-06) — all 6 tasks complete @ `23555ed0`; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed | `10b6d209` | **PASS 4/4** | | 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **COMPLETE @ `e54883a8` — gate re-run PASSED 2026-08-06.** Run 1 @ `fbf7b708` FAILED on defect 4 (no admin node could boot, mode-independent); fixed by `c0475f2d` + `e54883a8` and re-gated. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅. **MERGED to `master`** @ `b360239e` (2026-08-06) | `b360239e` | **PASS 8/8** (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards) | | 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebased on merged 1A | `feat/grpc-central-ack` | **DONE — AWAITING PR/MERGE (user-owned)** @ `a397cdad` (rebased on `a400b415`); 1B.1 ✅ + 1B.2 ✅ (Opus: deadline Critical found+fixed) + gate **PASS 7/7** incl. the 4b hang drill (~10 s failover = exactly one AckTimeoutSeconds — live proof of the fix); at-least-once duplicate absorbed idempotently exactly as traced | — | **PASS 7/7** | -| 2 | Baseline reproduction on pre-flip build, then full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills | (rig commit) | NOT STARTED | — | PENDING | +| 2 | **Opens with Task 2.0 (MUST FIX): honest PreStart-race logging** (both gates: fires ~every admin boot, two false Errors — soak error-grep impossible without it). Then baseline reproduction on pre-flip build, full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills. **Targeted-test policy in force (plan Phase 2 header): per-task unit runs targeted; ONE full-suite run per phase at the DoD** | (rig commit; 2.0 on master) | NOT STARTED | — | PENDING | | 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — | | 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | NOT STARTED | — | PENDING | diff --git a/docs/plans/2026-08-06-clusterclient-to-grpc.md b/docs/plans/2026-08-06-clusterclient-to-grpc.md index bdb0433f..c9ce368b 100644 --- a/docs/plans/2026-08-06-clusterclient-to-grpc.md +++ b/docs/plans/2026-08-06-clusterclient-to-grpc.md @@ -361,6 +361,58 @@ PASS; tracking updated. **STOP.** ## Phase 2 — Full-rig cutover + soak (sequential, after 1A+1B merged) +> **Targeted-test policy (phases 2–4):** the full solution suite is large (~3,000+ tests, +> 40+ min wall). Per-task verification runs are TARGETED — run only the projects/filters the +> task touches (named per task below). ONE full-suite run happens per phase, at the phase DoD +> before the merge/STOP — never per task. The mesh-relevant targeted set, for reference: +> +> ```bash +> # mesh transport core (actor, dialer, provider, starter order) +> dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ --filter "FullyQualifiedName~Communication" +> # node-side actor + ack sender + registration wiring +> dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ --filter "FullyQualifiedName~Communication|FullyQualifiedName~ServiceCollectionExtensions" +> # interceptors + gRPC services (both sides) +> dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/ +> # options/validators +> dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ --filter "FullyQualifiedName~MeshTransport|FullyQualifiedName~SplitTopology" +> # proto contract + mapper goldens +> dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/ --filter "FullyQualifiedName~MeshTransport" +> # central ack ingress +> dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/ --filter "FullyQualifiedName~CentralCommandGrpcService" +> ``` + +### Task 2.0 (MUST FIX, opening task — before the flip and the soak): honest PreStart-race logging + +**Classification:** small +**Estimated implement time:** ~15 min +**Parallelizable with:** none (first in phase) + +**Files:** +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs` +- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/ControlPlaneStarterOrderTests.cs`, `CentralCommunicationActorTests.cs` (string/severity pins only) + +Both live gates proved the `PreStart` resolve race fires on essentially EVERY admin boot +(1A: 2/3; 1B: 4/4) and logs two FALSE Errors — `Resolve`'s "continuing without it" (reads +permanent; the memoize heals it at first use) and the I-1 guard's "known-cluster gate will +NEVER be published" (it is published moments later — the 1B gate accepted acks and refused +`SITE-Z` on every such boot). A healthy central currently fails any error-grep, including +this phase's soak. + +Required behavior after this task: a PreStart-time resolve failure logs at +**Debug/Info** ("not yet resolvable — starter ordering; will retry on first use"); the +**Error** fires only when a FIRST-USE retry also fails (the genuinely-permanent absence — +`Resolve` is already re-invoked by the memoizing properties, so this is a severity/wording +split by call site, not new machinery). The I-1 null-guard Errors in `PreStart` move to the +same first-use semantics or gain "if this persists past the first contact refresh" wording. +Net: **a healthy admin boot produces ZERO Warning/Error lines from this path** (pinned by a +test asserting the boot-time severity), while a truly-absent dependency still produces a +loud Error at first use (existing pins keep this). **Targeted tests only:** +`ControlPlaneStarterOrderTests` + `CentralCommunicationActorTests` filters. Small review +chain (code-reviewer). Commit `fix(mesh): PreStart resolve race logs honestly — quiet on +heal, loud on permanent absence`. Lands on `master` directly (post-1B-merge, pre-flip). + +### The cutover + No feature branch; one commit flipping the rig default: `docker-dev/docker-compose.yml` `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"`. @@ -374,8 +426,11 @@ Then: full rebuild all-Grpc; run the Phase 2 check table (deploy e2e all six nod Restart/Reconnect/AlarmCommand matrix; kill-central-1 ack failover; kill-site-node mid-dispatch = logged per-node failure + deadline backstop, no retry storm; PSK negatives against both services on running containers; **zero ClusterClient log lines under `Grpc`**; -soak ≥ 2 h with 0 `PermissionDenied` noise — PSK drift detector). **Phase 2 DoD:** every -check PASS in the live-gate doc; rig default committed; tracking updated. **STOP.** +soak ≥ 2 h with 0 `PermissionDenied` noise — PSK drift detector; the soak error-grep must be +clean, which Task 2.0 makes possible on a healthy rig). Per-check verification is on the RIG; +unit runs during this phase are targeted (see policy above). **Phase 2 DoD:** every check +PASS in the live-gate doc; ONE full-suite run green on the cutover SHA; rig default +committed; tracking updated. **STOP.** --- @@ -407,6 +462,10 @@ client-settings asserts (keeps frame-size + DPS pins), `CentralCommunicationActo `NodeCommunicationActorTests` / `MeshRouterDispatchTests` reshaped to Dps+Grpc. Grep-gates: `rg -i "clusterclient|receptionist" src tests docker-dev docs` → only historical plans/CHANGELOG; `rg "CentralContactPoints"` → empty. Commit sequence per step. +**Targeted tests per step, NOT the full suite:** steps 1–3 → the ControlPlane + Runtime +`Communication`/`ServiceCollectionExtensions` filters; step 4 → `MeshTransportHoconTests`; +steps 5–6 → the Cluster.Tests `MeshTransport|SplitTopology` filter + Host.Tests; step 7 → +the replaced/reshaped fixtures by name. The FULL suite runs once, at the Phase 3 DoD. ### Task 3.2: docs sweep @@ -418,7 +477,9 @@ plans/CHANGELOG; `rg "CentralContactPoints"` → empty. Commit sequence per step transport table, `docs/Telemetry.md` cross-refs, `OtOpcUa/CLAUDE.md` transport decisions, `deferment.md` if applicable — and the **scadaproj index entry** (`~/Desktop/scadaproj/CLAUDE.md`, committed on scadaproj `main` and pushed, per its index-edit rule). Update tracking doc. -**Phase 3 DoD:** suite green; grep-gates clean; docs merged. **STOP.** +Task 3.2 itself is docs-only — no test runs beyond a build. +**Phase 3 DoD:** ONE full-suite run green on the branch tip (the phase's only full run); +grep-gates clean; docs merged. **STOP.** --- @@ -431,9 +492,12 @@ matrix all six nodes; (3) ack matrix + seal; (4) central-kill ack failover + sit dispatch honesty; (5) mid-deploy central kill → deadline/replay backstops hold, no dupes; (6) **boundary check: no Akka association between any site container and central** (was the baseline gap); (7) restart discipline: full-rig restart clean, **zero -receptionist/ClusterClient log lines anywhere**; (8) `MeshGrpcBoundaryTests` + full suite -green on the merge SHA. Close any carry-forwards named by earlier gates. Tracking doc final -row; program CLOSED. +receptionist/ClusterClient log lines anywhere**; (8) `MeshGrpcBoundaryTests` (targeted) + +the program's TERMINAL full-suite run, green on the merge SHA — this is Phase 4's one full +run and the program's last; everything before it in this phase is rig checks + targeted +fixtures. Close any carry-forwards named by earlier gates (incl. the I-4 cold-cache +measurement if a store-backed rig exists by then — else re-carry with rationale). Tracking +doc final row; program CLOSED. --- diff --git a/docs/plans/2026-08-06-clusterclient-to-grpc.md.tasks.json b/docs/plans/2026-08-06-clusterclient-to-grpc.md.tasks.json index 1b463646..f5e13fca 100644 --- a/docs/plans/2026-08-06-clusterclient-to-grpc.md.tasks.json +++ b/docs/plans/2026-08-06-clusterclient-to-grpc.md.tasks.json @@ -197,8 +197,10 @@ "status": "pending", "blockedBy": [ 8, - 11 - ] + 11, + 16 + ], + "note": "Targeted-test policy applies (see plan Phase 2 header): per-task runs targeted; ONE full-suite run at the phase DoD." }, { "id": 13, @@ -235,6 +237,20 @@ "blockedBy": [ 14 ] + }, + { + "id": 16, + "subject": "Task 2.0 (MUST FIX): PreStart-race logging honest — quiet on heal, loud on permanent absence", + "phase": "2", + "branch": "(master, post-1B-merge, pre-flip)", + "classification": "small", + "parallelizableWith": [], + "status": "pending", + "blockedBy": [ + 8, + 11 + ], + "note": "Both gates: race fires ~every admin boot, two FALSE Errors; soak error-grep impossible without this. Targeted tests only (ControlPlaneStarterOrderTests + CentralCommunicationActorTests)." } ], "lastUpdated": "2026-08-06"