From b9b3cf9a5888ce133e672ecbe3624ed80071e1cc Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 20 Aug 2026 13:44:31 -0400 Subject: [PATCH] =?UTF-8?q?docs(plans):=20R3-04=20live=20gate=20=E2=80=94?= =?UTF-8?q?=20PASS=20on=20docker-dev=20(pre-fix=20control=20at=207c77a8c7:?= =?UTF-8?q?=20Cluster:Roles-only=20site-b-2=20booted=20roles=3D[]=20and=20?= =?UTF-8?q?timed=20out=20the=20fleet=20deploy=205/6,=20miscased=20site-a-2?= =?UTF-8?q?=20ServiceLevel=200;=20post-fix:=20six=20banners=20source=3DCon?= =?UTF-8?q?figured,=20deploy=20Sealed=206/6=20in=20224=20ms,=20one=20Prima?= =?UTF-8?q?ry=20+=20standby=20per=20pair,=20disagreement=20Warning=20fires?= =?UTF-8?q?,=20unknown=20role=20refuses=20to=20start)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/R3-04-one-role-authority-plan.md | 83 +++++++++++++++++++ ...3-04-one-role-authority-plan.md.tasks.json | 4 +- 2 files changed, 85 insertions(+), 2 deletions(-) diff --git a/archreview/plans/R3-04-one-role-authority-plan.md b/archreview/plans/R3-04-one-role-authority-plan.md index c70d614c..978a2cca 100644 --- a/archreview/plans/R3-04-one-role-authority-plan.md +++ b/archreview/plans/R3-04-one-role-authority-plan.md @@ -655,3 +655,86 @@ role's overlay-filename fragment is always lower-case regardless of how it was s `ClusterRoleInfo` is a shared dependency of `RedundancyStateActor` et al. No pushes, no merges. + +## Live gate result (2026-08-20) — **PASS** on docker-dev (pre-fix control at `7c77a8c7` failed first) + +Rig: `docker-dev/docker-compose.yml` on OrbStack (Mac), fresh volumes (`down -v` before the control), the +six host containers + SQL + Traefik, AdminUI on `Security__Auth__DisableLogin=true`. The compose carried the +T11/T12 pins in BOTH runs (site-b-2 `Cluster__Roles__*` only, no `OTOPCUA_ROLES`; site-a-2 +`Cluster__Roles__0="Driver"`). Deploys via `./docker-dev/deploy.sh` (`POST /api/deployments`, +`X-Api-Key: docker-dev-deploy-key`, `202` both times). `Server_ServiceLevel` (`i=2267`) read with Client.CLI on +`:4840`–`:4845`. Image parity asserted ×6 via `docker inspect --format '{{.Image}}'` in both runs. The Modbus +fixture on `10.100.0.35:5020` was left running throughout. Note: master advanced `9242630c → f665e8b8` (R3-05 S18 +follow-up) *while* the post-fix run was in progress; the post-fix image had already been built from the +`9242630c` working tree (image created 13:29:46 local, merge landed 13:29:59) — the only later delta is +`BoundedApiKeyStore.cs` + a compose env wire, neither of which touches roles. + +### Leg 1 — pre-fix control, image `otopcua-host:prefix-r304` = `32a2dcee0285` (scratch worktree at `7c77a8c7`) ×6: **FAILED as required** + +| what the plan predicted | observed | +|---|---| +| site-b-2 boot banner shows **no** roles | `[17:25:22 INF] OtOpcUa.Host starting with roles=[] (admin=False, driver=False)` — every other node `roles=[admin,driver,cluster-main]` / `[driver,cluster-site-a]` / `[driver,cluster-site-b]` (pre-fix banner lower-cases the cluster suffix; no `source=`) | +| site-b-2 gossips `driver` anyway | site-b-1: `Node [akka.tcp://otopcua@site-b-2:4053] is JOINING, roles [cluster-SITE-B,driver]`; site-b-2's own `cluster-primary` health check: `Standby: active node for role 'driver' is akka.tcp://otopcua@site-b-1:4053` — Akka counted it as a driver member while the Host wired nothing | +| fleet deploy seals **TimedOut** naming site-b-2 | Deployment `6c653c61…` (rev `942ad3cc…`): `Status = 4`, `SealedAtUtc = NULL`; `NodeDeploymentState` five rows `Status = 1` (`AppliedAtUtc 17:25:41.077–.107`), **site-b-2 `Status = 0`, `AppliedAtUtc = NULL`**. central-2: `[17:27:41 WRN] Deployment 6c653c61… timed out after 00:02:00 (5/6 acks landed). No ack from: site-b-2:4053. Each is an enabled, non-maintenance ClusterNode row …` | +| site-b-2 does not serve | Client.CLI `read` on `opc.tcp://localhost:4845` → `Remote side closed connection` (no OPC UA server started; no DriverHost / Kestrel lines in its log at all, only Akka remoting + cluster join) | +| site-a (01/C-9): "elects **no** Primary" | **Partly wrong, and the execution notes already said why.** site-a-1 gossips ordinal `driver`, so `ClusterActiveNode.OldestUpMember` still elected it: `:4842` = **250**. What failed was site-a-2: gossiped `roles [cluster-SITE-A,Driver]` (site-a-1's JOINING line), its own `cluster-primary` check `Node carries none of the active roles: admin, driver.`, and `:4843` = **0** — the pair ran with a Primary and **no eligible standby**. A pair whose nodes were BOTH miscased would elect none (pinned by `Primary_election_itself_remains_ordinal_and_still_elects_no_one_for_an_all_miscased_pair`); the rig pins only one | +| control reference | `:4840`=240, `:4841`=250 (MAIN), `:4844`=250 (site-b-1 sole eligible driver) | + +### Leg 2 — post-fix, image `otopcua-host:dev` = `b7cfae1be510` (master `9242630c`) ×6: **PASS** + +1. **Role banners, all six `source=Configured`, config roles win, normalised** — central-1/2 + `starting with roles=[admin,driver,cluster-MAIN] (admin=True, driver=True) source=Configured`; site-a-1/**site-a-2** + `roles=[driver,cluster-SITE-A]` (the `Driver` pin lower-cased, cluster suffix verbatim); site-b-1/**site-b-2** + `roles=[driver,cluster-SITE-B] (admin=False, driver=True)` — the `Cluster:Roles`-only node now wires the driver + stack. Gossip view agrees with the Host view on every node: site-a-1 `Node [site-a-2] is JOINING, roles + [cluster-SITE-A,driver]` (was `Driver`), site-b-1 `Node [site-b-2] is JOINING, roles [cluster-SITE-B,driver]`, + central-2 `Node [central-1] is JOINING, roles [cluster-MAIN,driver,admin]`. (`/hosts` chips were not used as the + cross-check — the page is a Blazor circuit and a `curl` of it renders no member rows; the Akka JOINING lines are the + same gossiped `m.Roles` the page reads.) +2. **Deploy seals Applied 6/6 (01/S-12):** Deployment `a5a7018d…` (same revision `942ad3cc…`): `Status = 2`, + `CreatedAtUtc 17:31:21.914`, `SealedAtUtc 17:31:22.138` (224 ms); six `NodeDeploymentState` rows all `Status = 1`, + `AppliedAtUtc 17:31:22.090–.132` including **site-b-2**. central-1: `Deployment a5a7018d… sealed (acks=6)`. + site-b-2: `Fetched + verified artifact … from http://central-1:4055 (1468 bytes)` → `DriverHost site-b-2:4053: + applied fetched deployment a5a7018d… (rev 942ad3cc…, children=0)` → `cached … as this node's local fallback`. + Client.CLI `browse -u opc.tcp://localhost:4845 -d 2` returns `Server`, `Aliases`, `Locations`, **`OtOpcUa + (ns=2;s=OtOpcUa)`** — site-b-2 serves (was `Remote side closed connection`). +3. **One Primary per pair, with an eligible standby (01/C-9):** `:4840`=240/`:4841`=250 (MAIN), **`:4842`=250 / + `:4843`=240** (SITE-A — site-a-2 was 0 pre-fix; its health check now reads `Standby: active node for role 'driver' + is akka.tcp://otopcua@site-a-1:4053`), `:4844`=250/`:4845`=240 (SITE-B — site-b-2 was unreachable). After the leg-5 + restart of site-b-1 the SITE-B pair re-elected with **site-b-2 as Primary** (`:4845`=250, `:4844`=240) — the + formerly role-less node now holds the Primary-gated data plane. +4. **T6 disagreement Warning fires:** scratch compose override on site-b-1 setting `OTOPCUA_ROLES: "driver"` against its + `Cluster__Roles__0/1 = driver,cluster-SITE-B` → `[17:32:35 INF] … starting with roles=[driver,cluster-SITE-B] … + source=Configured` followed by `[17:32:35 WRN] Cluster:Roles ([driver,cluster-SITE-B]) and OTOPCUA_ROLES ([driver]) + disagree; Cluster:Roles wins outright.` — and the node still joined and served with the config roles. (The rig's + standing pins do NOT fire this Warning by themselves: site-a-2's `Driver` normalises equal to its env value, and + site-b-2 has only one source — choice 5 is explicitly "both set AND differ".) +5. **Unknown role fails host start:** override `Cluster__Roles__0: "operator"` on site-b-1 → the log is exactly six + lines: `Unhandled exception. System.ArgumentException: Cluster:Roles[0] = 'operator' is not a known role. Allowed: + admin, driver, dev, or 'cluster-'.` at `EffectiveRoles.NormalizeAndValidateConfigured` ← + `EffectiveRoles.Resolve(IConfiguration)` ← `Program.cs:67` — no boot banner, no Akka remoting, no Kestrel, no + cluster join. Override removed; site-b-1 recreated from the plain compose → `roles=[driver,cluster-SITE-B] … + source=Configured`, no Warning, `(healthy)`. + ⚠️ **Observation, pre-existing and NOT R3-04's:** the `dotnet OtOpcUa.Host.dll` process did **not exit** after the + unhandled exception — `docker inspect` kept reporting `running exit=0 restarts=0` for > 2 min, the main thread in + state `R` burning ~50 % of a core with 14 Server-GC threads parked. Reproduced identically on a bare `docker run + --rm -e OTOPCUA_ROLES=operator` against **both** `otopcua-host:prefix-r304` (7c77a8c7, pre-fix `RoleParser.Parse` + throw at `Program.cs:61`) and the master image: neither exits within 90 s (`timeout` → 124). So "fail-start" holds + (nothing is wired, nothing listens) but the container's `unless-stopped` restart policy never fires and an orchestrator + watching the exit code would not see a crash; only the compose healthcheck (never `healthy`) shows it. Arm64 OrbStack + `aspnet:10.0` runtime image; root cause not chased here — worth a follow-up (is it the runtime's unhandled-exception + abort path on this host, or something in our `Program.cs` top-level statements?) before relying on a process-exit + signal for any of the boot-time validators (`SplitTopologyTransportValidator`, `AkkaClusterOptionsValidator`, + `ConfigSourceOptionsValidator` fail the same way). + +### Leg 3 — restore + +Six hosts on `b7cfae1be510` ×6 (parity asserted), `GET /health/ready` → 200 via `:9200`, SQL + Traefik up, all six +ServiceLevels one 250 + one 240 per pair; `otopcua-pymodbus-standard` on `10.100.0.35` untouched (`Up 2 weeks`). Scratch +worktree at `7c77a8c7` removed; the control image tag `otopcua-host:prefix-r304` left in the local image cache (same layer +id as R3-03's `prefix-r303` — both are `7c77a8c7`). + +**Verdict: T13 PASS.** Both findings reproduce on the pre-fix build with the rig as pinned (01/S-12 as a real fleet +deploy outage; 01/C-9 as a standby-less pair rather than the plan's "no Primary" — the execution notes' ordinal-election +residual, confirmed live), and neither reproduces on master. One pre-existing process-does-not-exit-after-fail-start +observation recorded above for follow-up. diff --git a/archreview/plans/R3-04-one-role-authority-plan.md.tasks.json b/archreview/plans/R3-04-one-role-authority-plan.md.tasks.json index d901dc01..67b3cef7 100644 --- a/archreview/plans/R3-04-one-role-authority-plan.md.tasks.json +++ b/archreview/plans/R3-04-one-role-authority-plan.md.tasks.json @@ -95,7 +95,7 @@ { "id": "T13", "subject": "LIVE GATE: docker-dev six-node rig \u2014 pre-fix baseline, per-node banner vs /hosts gossip parity, site-a Primary election, deploy seals Applied incl. site-b-2, unknown-role fail-fast", - "status": "pending", + "status": "completed", "blockedBy": [ "T3", "T4", @@ -107,7 +107,7 @@ "T11", "T12" ], - "note": "Skipped by the executing agent per orchestrator instruction \u2014 the live gate runs separately, on the docker-dev rig, by the orchestrator/operator. T1-T12 are implemented, committed, and unit/integration-tested; T13 requires bringing the rig up (docker compose), which this task explicitly excluded." + "note": "PASS 2026-08-20 on docker-dev: pre-fix control at 7c77a8c7 FAILED as required (site-b-2 roles=[], no OPC UA endpoint, deploy TimedOut 5/6; site-a-2 gossiped `Driver`, ServiceLevel 0, \"carries none of the active roles\"); post-fix master image: six banners source=Configured, deploy Sealed 6/6 in 224 ms, one 250/240 split per pair, T6 disagreement Warning fires, Cluster__Roles__0=operator refuses to start. See the plan's \"Live gate result (2026-08-20)\"." } ] }