From bcc91268ba20943327a83a1cdafd48b5781685a3 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 6 Aug 2026 11:04:36 -0400 Subject: [PATCH] =?UTF-8?q?docs(plans):=20Phase=202=20gate=20=E2=80=94=20c?= =?UTF-8?q?hecks=201-6=20+=20PSK=20negatives=20PASS=20with=20evidence;=20s?= =?UTF-8?q?oak=20+=20full=20suite=20running?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1 --- ...6-08-06-clusterclient-to-grpc-live-gate.md | 39 ++++++++++++++----- 1 file changed, 29 insertions(+), 10 deletions(-) diff --git a/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md b/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md index 784a13ff..e8aeb466 100644 --- a/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md +++ b/docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md @@ -1088,19 +1088,38 @@ restored; revision hash back to `642b9d6f79b9…`). ## Phase 2 — full-rig cutover + soak — **PENDING** -Rig default flipped and committed: `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"`. -Branch/SHA `________`. **Run the Baseline section first on the pre-flip build.** +Rig default flipped and committed: `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"` +(`caf2eec6`, all six node entries). Rig rebuilt from `master @ caf2eec6` — which also carries +Task 2.0 (`a94eb8a3`+`d7c9ead6`, honest PreStart-race logging, review-approved) and the DI +role-scoping pin (`2e06dfe5`+`027f0403`, review-approved) — image rebuilt, node containers +recreated, no `OTOPCUA_MESH_MODE` override in the environment. Baseline was run on the +pre-flip build first (see Baseline section, `1bb3a937`). | # | Check | Result | |---|---|---| -| 1 | Full rebuild all-`Grpc`: six nodes ready, deploy e2e seals, all acks via gRPC | PENDING | -| 2 | Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes | PENDING | -| 3 | Kill site-a-1 mid-dispatch: per-node `Execute` failure **logged with node id + status** (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path) | PENDING | -| 4 | Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action | PENDING | -| 5 | Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; **no duplicate applies** after the pair returns (boot-replay path only) | PENDING | -| 6 | **Zero ClusterClient activity under `Grpc`:** `docker logs \| grep -ci "clusterclient"` = 0 new lines post-boot | PENDING | -| 7 | Soak ≥ 2 h under periodic deploys + command traffic: 0 `PermissionDenied`/`Unauthenticated` across all six nodes (PSK drift detector), 0 unexplained `Unavailable` | PENDING | -| 8 | Full suite green on the cutover SHA | PENDING | +| 1 | Full rebuild all-`Grpc`: six nodes ready, deploy e2e seals, all acks via gRPC | **PASS** — deploy `949d2cc7` sealed (Status 2) 6/6 within ~1 s; every node logged `accepted DispatchDeployment … over gRPC` and `Posted the ApplyAck … to http://central-1:4055`, central logged each `ApplyAck … handed to the central communication actor`; wire-level: central holds established TCP to every node's `:4056`, node `TIME_WAIT` to central `:4055` observed. An earlier identical deploy (`8809464a`, pre-Debug-override) also sealed 6/6 in ~150 ms | +| 2 | Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes | **PASS** — via authenticated grpcurl envelopes (the 1A shape): Restart `SITEA-modbus` → site-a-1 `{}` + `restarting driver … by request of p2-gate` + `spawned` + `connected`; Reconnect + AlarmCommand accepted + handed (correlation ids `c2b1…`/`c2c1…`); site-b-1 delivery proven (`c2d1…` accepted over gRPC). AdminUI origin composed per the 1A carry-forward (no UI control exists; same `DispatchOverGrpc` path driven by check 1) | +| 3 | Kill site-a-1 mid-dispatch: per-node `Execute` failure **logged with node id + status** (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path) | **PASS** — deploy `069b57a4` 1 s after `docker kill`: central-2 WRN `DispatchDeployment … was NOT delivered to site-a-1:4053 (Unavailable: Error connecting to subchannel.). It is dropped and will NOT be retried…` — exactly **1** such line (no retry storm); 5/6 acked; `timed out after 00:02:00 (5/6 acks landed). No ack from: site-a-1:4053` at exactly +2:00; restart → FetchAndCache boot-replay restored last-known-good, timed-out deployment NOT retro-acked (ack count stayed 5) | +| 4 | Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action | **PASS** — deploy `ffc75eae` with central-1 dead: site nodes logged `Posted the ApplyAck … to http://central-2:4055 after 1 endpoint(s) failed (http://central-1:4055 => Unavailable)` (the ordered walk, per node), all live acks landed on central-2; deadline honest at +2:00 naming `central-1, site-a-1` (site-a-1 = the documented ~2-min `GrpcChannel` backoff carry-forward from its own earlier kill — loud, single-attempt). First post-restart probe caught central-1 still booting (5/6, honest TimedOut); second probe `594ecda3` **sealed 6/6** — full convergence, no manual action beyond `docker start` | +| 5 | Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; **no duplicate applies** after the pair returns (boot-replay path only) | **PASS** — deploy `e5a14c5f`: BOTH `NOT delivered to site-b-{1,2}:4053 (Unavailable…)` WRN lines in the SAME SECOND as dispatch (baseline (b): zero correlated lines, sampled-deadLetters INF); `timed out after 00:02:00 (4/6…). No ack from: site-b-1:4053, site-b-2:4053`; pair restart → boot-replay only (`FetchAndCache boot — restored served state`), rows stay 0/0, ack count stays 4 — no dupes, no resurrection | +| 6 | **Zero ClusterClient activity under `Grpc`:** `docker logs \| grep -ci "clusterclient"` = 0 new lines post-boot | **PASS (interim + final at soak end)** — `grep -ic "clusterclient\|receptionist"` = **0 on all six nodes** after checks 1–5 (incl. every kill/restart cycle) | +| 7 | Soak ≥ 2 h under periodic deploys + command traffic: 0 `PermissionDenied`/`Unauthenticated` across all six nodes (PSK drift detector), 0 unexplained `Unavailable` | **RUNNING** — driver started 15:03:44Z, 135-min target: deploy+seal check per 10-min cycle + Restart(site-a)/AlarmCommand(site-b) traffic; whitelist-aware final grep (excludes the standby `/health/active` 503 + `Health check cluster-primary` ERR noise — Health 0.3.0 by-design standby behaviour, pre-existing) | +| 8 | Full suite green on the cutover SHA | **RUNNING** — `dotnet test ZB.MOM.WW.OtOpcUa.slnx` started in parallel with the soak, on `master` with the cutover committed (`caf2eec6` + gate-record commits) | + +**Extra (plan-text requirement): PSK negatives on RUNNING containers, both services** — **PASS**: +`NodeCommandService/Execute` @ site-a-1: no bearer → `PermissionDenied: Node command +authentication failed.`; SITE-B key → same uniform denial; SITE-A key + empty envelope +(positive control) → `InvalidArgument: Command envelope could not be decoded.` (interceptor +ran, payload rejected — auth and decode are distinct). `CentralCommandService/ReportApply` +@ central-1:4055: no bearer / wrong key / SITE-A key claiming `x-otopcua-cluster: SITE-Z` → +all `PermissionDenied: Central command authentication failed.` (uniform text, unknown-cluster +claim refused). + +**Gate observability note:** after check 1's first pass, the six node containers were +recreated once with an UNCOMMITTED scratchpad compose override raising Serilog default to +Debug (the mesh delivery lines are DBG; the base rig logs at Information and the 1A/1B gates +had used the same temporary raise). Same image, same SHA — no rebuild, so no void-run. All +kill drills ran after that recreation. ---