docs(plans): phases 2-4 — targeted-test policy + Task 2.0 (must-fix: honest PreStart-race logging) before the cutover
This commit is contained in:
Joseph Doherty
@@ -25,7 +25,7 @@ posture).
|
||||
| 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **MERGED to `master`** @ `10b6d209` (2026-08-06) — all 6 tasks complete @ `23555ed0`; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed | `10b6d209` | **PASS 4/4** |
|
||||
| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **COMPLETE @ `e54883a8` — gate re-run PASSED 2026-08-06.** Run 1 @ `fbf7b708` FAILED on defect 4 (no admin node could boot, mode-independent); fixed by `c0475f2d` + `e54883a8` and re-gated. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅. **MERGED to `master`** @ `b360239e` (2026-08-06) | `b360239e` | **PASS 8/8** (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards) |
|
||||
| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebased on merged 1A | `feat/grpc-central-ack` | **DONE — AWAITING PR/MERGE (user-owned)** @ `a397cdad` (rebased on `a400b415`); 1B.1 ✅ + 1B.2 ✅ (Opus: deadline Critical found+fixed) + gate **PASS 7/7** incl. the 4b hang drill (~10 s failover = exactly one AckTimeoutSeconds — live proof of the fix); at-least-once duplicate absorbed idempotently exactly as traced | — | **PASS 7/7** |
|
||||
| 2 | Baseline reproduction on pre-flip build, then full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills | (rig commit) | NOT STARTED | — | PENDING |
|
||||
| 2 | **Opens with Task 2.0 (MUST FIX): honest PreStart-race logging** (both gates: fires ~every admin boot, two false Errors — soak error-grep impossible without it). Then baseline reproduction on pre-flip build, full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills. **Targeted-test policy in force (plan Phase 2 header): per-task unit runs targeted; ONE full-suite run per phase at the DoD** | (rig commit; 2.0 on master) | NOT STARTED | — | PENDING |
|
||||
| 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — |
|
||||
| 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | NOT STARTED | — | PENDING |
|
||||
|
||||
|
||||
@@ -361,6 +361,58 @@ PASS; tracking updated. **STOP.**
|
||||
|
||||
## Phase 2 — Full-rig cutover + soak (sequential, after 1A+1B merged)
|
||||
|
||||
> **Targeted-test policy (phases 2–4):** the full solution suite is large (~3,000+ tests,
|
||||
> 40+ min wall). Per-task verification runs are TARGETED — run only the projects/filters the
|
||||
> task touches (named per task below). ONE full-suite run happens per phase, at the phase DoD
|
||||
> before the merge/STOP — never per task. The mesh-relevant targeted set, for reference:
|
||||
>
|
||||
> ```bash
|
||||
> # mesh transport core (actor, dialer, provider, starter order)
|
||||
> dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ --filter "FullyQualifiedName~Communication"
|
||||
> # node-side actor + ack sender + registration wiring
|
||||
> dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ --filter "FullyQualifiedName~Communication|FullyQualifiedName~ServiceCollectionExtensions"
|
||||
> # interceptors + gRPC services (both sides)
|
||||
> dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/
|
||||
> # options/validators
|
||||
> dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ --filter "FullyQualifiedName~MeshTransport|FullyQualifiedName~SplitTopology"
|
||||
> # proto contract + mapper goldens
|
||||
> dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/ --filter "FullyQualifiedName~MeshTransport"
|
||||
> # central ack ingress
|
||||
> dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/ --filter "FullyQualifiedName~CentralCommandGrpcService"
|
||||
> ```
|
||||
|
||||
### Task 2.0 (MUST FIX, opening task — before the flip and the soak): honest PreStart-race logging
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~15 min
|
||||
**Parallelizable with:** none (first in phase)
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs`
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/ControlPlaneStarterOrderTests.cs`, `CentralCommunicationActorTests.cs` (string/severity pins only)
|
||||
|
||||
Both live gates proved the `PreStart` resolve race fires on essentially EVERY admin boot
|
||||
(1A: 2/3; 1B: 4/4) and logs two FALSE Errors — `Resolve`'s "continuing without it" (reads
|
||||
permanent; the memoize heals it at first use) and the I-1 guard's "known-cluster gate will
|
||||
NEVER be published" (it is published moments later — the 1B gate accepted acks and refused
|
||||
`SITE-Z` on every such boot). A healthy central currently fails any error-grep, including
|
||||
this phase's soak.
|
||||
|
||||
Required behavior after this task: a PreStart-time resolve failure logs at
|
||||
**Debug/Info** ("not yet resolvable — starter ordering; will retry on first use"); the
|
||||
**Error** fires only when a FIRST-USE retry also fails (the genuinely-permanent absence —
|
||||
`Resolve` is already re-invoked by the memoizing properties, so this is a severity/wording
|
||||
split by call site, not new machinery). The I-1 null-guard Errors in `PreStart` move to the
|
||||
same first-use semantics or gain "if this persists past the first contact refresh" wording.
|
||||
Net: **a healthy admin boot produces ZERO Warning/Error lines from this path** (pinned by a
|
||||
test asserting the boot-time severity), while a truly-absent dependency still produces a
|
||||
loud Error at first use (existing pins keep this). **Targeted tests only:**
|
||||
`ControlPlaneStarterOrderTests` + `CentralCommunicationActorTests` filters. Small review
|
||||
chain (code-reviewer). Commit `fix(mesh): PreStart resolve race logs honestly — quiet on
|
||||
heal, loud on permanent absence`. Lands on `master` directly (post-1B-merge, pre-flip).
|
||||
|
||||
### The cutover
|
||||
|
||||
No feature branch; one commit flipping the rig default:
|
||||
`docker-dev/docker-compose.yml` `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"`.
|
||||
|
||||
@@ -374,8 +426,11 @@ Then: full rebuild all-Grpc; run the Phase 2 check table (deploy e2e all six nod
|
||||
Restart/Reconnect/AlarmCommand matrix; kill-central-1 ack failover; kill-site-node
|
||||
mid-dispatch = logged per-node failure + deadline backstop, no retry storm; PSK negatives
|
||||
against both services on running containers; **zero ClusterClient log lines under `Grpc`**;
|
||||
soak ≥ 2 h with 0 `PermissionDenied` noise — PSK drift detector). **Phase 2 DoD:** every
|
||||
check PASS in the live-gate doc; rig default committed; tracking updated. **STOP.**
|
||||
soak ≥ 2 h with 0 `PermissionDenied` noise — PSK drift detector; the soak error-grep must be
|
||||
clean, which Task 2.0 makes possible on a healthy rig). Per-check verification is on the RIG;
|
||||
unit runs during this phase are targeted (see policy above). **Phase 2 DoD:** every check
|
||||
PASS in the live-gate doc; ONE full-suite run green on the cutover SHA; rig default
|
||||
committed; tracking updated. **STOP.**
|
||||
|
||||
---
|
||||
|
||||
@@ -407,6 +462,10 @@ client-settings asserts (keeps frame-size + DPS pins), `CentralCommunicationActo
|
||||
`NodeCommunicationActorTests` / `MeshRouterDispatchTests` reshaped to Dps+Grpc. Grep-gates:
|
||||
`rg -i "clusterclient|receptionist" src tests docker-dev docs` → only historical
|
||||
plans/CHANGELOG; `rg "CentralContactPoints"` → empty. Commit sequence per step.
|
||||
**Targeted tests per step, NOT the full suite:** steps 1–3 → the ControlPlane + Runtime
|
||||
`Communication`/`ServiceCollectionExtensions` filters; step 4 → `MeshTransportHoconTests`;
|
||||
steps 5–6 → the Cluster.Tests `MeshTransport|SplitTopology` filter + Host.Tests; step 7 →
|
||||
the replaced/reshaped fixtures by name. The FULL suite runs once, at the Phase 3 DoD.
|
||||
|
||||
### Task 3.2: docs sweep
|
||||
|
||||
@@ -418,7 +477,9 @@ plans/CHANGELOG; `rg "CentralContactPoints"` → empty. Commit sequence per step
|
||||
transport table, `docs/Telemetry.md` cross-refs, `OtOpcUa/CLAUDE.md` transport decisions,
|
||||
`deferment.md` if applicable — and the **scadaproj index entry** (`~/Desktop/scadaproj/CLAUDE.md`,
|
||||
committed on scadaproj `main` and pushed, per its index-edit rule). Update tracking doc.
|
||||
**Phase 3 DoD:** suite green; grep-gates clean; docs merged. **STOP.**
|
||||
Task 3.2 itself is docs-only — no test runs beyond a build.
|
||||
**Phase 3 DoD:** ONE full-suite run green on the branch tip (the phase's only full run);
|
||||
grep-gates clean; docs merged. **STOP.**
|
||||
|
||||
---
|
||||
|
||||
@@ -431,9 +492,12 @@ matrix all six nodes; (3) ack matrix + seal; (4) central-kill ack failover + sit
|
||||
dispatch honesty; (5) mid-deploy central kill → deadline/replay backstops hold, no dupes;
|
||||
(6) **boundary check: no Akka association between any site container and central** (was the
|
||||
baseline gap); (7) restart discipline: full-rig restart clean, **zero
|
||||
receptionist/ClusterClient log lines anywhere**; (8) `MeshGrpcBoundaryTests` + full suite
|
||||
green on the merge SHA. Close any carry-forwards named by earlier gates. Tracking doc final
|
||||
row; program CLOSED.
|
||||
receptionist/ClusterClient log lines anywhere**; (8) `MeshGrpcBoundaryTests` (targeted) +
|
||||
the program's TERMINAL full-suite run, green on the merge SHA — this is Phase 4's one full
|
||||
run and the program's last; everything before it in this phase is rig checks + targeted
|
||||
fixtures. Close any carry-forwards named by earlier gates (incl. the I-4 cold-cache
|
||||
measurement if a store-backed rig exists by then — else re-carry with rationale). Tracking
|
||||
doc final row; program CLOSED.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -197,8 +197,10 @@
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
8,
|
||||
11
|
||||
]
|
||||
11,
|
||||
16
|
||||
],
|
||||
"note": "Targeted-test policy applies (see plan Phase 2 header): per-task runs targeted; ONE full-suite run at the phase DoD."
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
@@ -235,6 +237,20 @@
|
||||
"blockedBy": [
|
||||
14
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 16,
|
||||
"subject": "Task 2.0 (MUST FIX): PreStart-race logging honest — quiet on heal, loud on permanent absence",
|
||||
"phase": "2",
|
||||
"branch": "(master, post-1B-merge, pre-flip)",
|
||||
"classification": "small",
|
||||
"parallelizableWith": [],
|
||||
"status": "pending",
|
||||
"blockedBy": [
|
||||
8,
|
||||
11
|
||||
],
|
||||
"note": "Both gates: race fires ~every admin boot, two FALSE Errors; soak error-grep impossible without this. Targeted tests only (ControlPlaneStarterOrderTests + CentralCommunicationActorTests)."
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-08-06"
|
||||
|
||||
Reference in New Issue
Block a user