diff --git a/archreview/plans/R3-01-mesh-grpc-allowanonymous-plan.md b/archreview/plans/R3-01-mesh-grpc-allowanonymous-plan.md index 4f552be4..9ba7f9d4 100644 --- a/archreview/plans/R3-01-mesh-grpc-allowanonymous-plan.md +++ b/archreview/plans/R3-01-mesh-grpc-allowanonymous-plan.md @@ -436,3 +436,128 @@ process that are worth recording: No other deviations from the plan's design or task breakdown. T10 was not run — left `pending` per the run brief, for the orchestrator. + +## Live gate result (2026-08-20) — **PASS** on docker-dev with `DisableLogin=false` (control failed first) + +Rig: `docker-dev/docker-compose.yml` on OrbStack (Mac), fresh volumes (`down -v` first), six host +containers. Both runs used `OTOPCUA_DISABLE_LOGIN=false` on the central pair (verified in-container: +`Security__Auth__DisableLogin=false` on central-1/central-2; `curl -s http://localhost:9200/` → `302 → +/login?ReturnUrl=%2F` in both runs — the cookie FallbackPolicy was live for the UI throughout). +Deploys were triggered with `./docker-dev/deploy.sh` (`POST /api/deployments`, `X-Api-Key: +docker-dev-deploy-key`), which answered `202 Accepted` in both runs — the headless deploy API is +`AllowAnonymous` + key-gated and is NOT behind the cookie fallback, so the API-key path works with +login enabled (one open question from the brief, answered: not a finding). + +### Step 0 — falsification control: **FAILED as required** + +Control image `otopcua-host:prefix` = `53c41fc6bc01`, built from a scratch worktree of master +`f1b976f5` with the five R3-01 opt-outs reverted in `MeshGrpcEndpoints.cs` (`.AllowAnonymous()` removed +from the four `MapGrpcService` maps, the empty-prefix `MapGroup(...).AllowAnonymous()` collapsed back to a +bare `MapZbLocalDbSync()`; `git diff --stat` = 1 file, 5 insertions/5 deletions). Chosen over a checkout of +`851e4cf5` so the control image matched the R3-04 compose (site-b-2 `Cluster:Roles`-only, site-a-2 +miscased pin) — the same compose both runs used. Tagged over `otopcua-host:dev` for the run; image +parity asserted on all six containers (`docker inspect --format '{{.Image}}'` = `53c41fc6bc01…` ×6). + +- Deployment `26077962…` (revision `9e3540a1…`): **`Status = 4` (TimedOut)**, `SealedAtUtc = NULL`, + six `NodeDeploymentState` rows all `Status = 0`, `AppliedAtUtc = NULL`. +- central-1 (seal): `[16:10:04 WRN] Deployment 26077962e53a4ae99538c5aaaa396a85 timed out after + 00:02:00 (0/6 acks landed). No ack from: central-1:4053, central-2:4053, site-a-1:4053, + site-a-2:4053, site-b-1:4053, site-b-2:4053.` +- central-1 (own-pair command dial, the fused-node path): `[16:08:04 WRN] Mesh: DispatchDeployment + (correlationId=9fe30594…) was NOT delivered to central-1:4053 (Cancelled: Bad gRPC response. + Invalid content-type value: text/html; charset=utf-8).` — and identically for central-2:4053. +- site-a-1 (artifact fetch + ack): `[16:08:04 WRN] Artifact fetch for 26077962… from + http://central-1:4055 failed (Cancelled); trying next endpoint.` → `… failed on ALL 2 endpoint(s); + returning no bytes (apply failure — last-known-good is kept).` → `[16:08:04 ERR] DriverHost + site-a-1:4053: FetchAndCache apply of 26077962… FAILED … nothing was applied. Staying on revision + null` → `[16:08:04 WRN] Dropping the ApplyAck for 26077962… from site-a-1:4053: all 4 central + endpoint(s) refused it (http://central-1:4055 => Cancelled; …)`. +- Server-side signature, per central (Kestrel request log, `sort | uniq -c` over the whole control run): + + | endpoint (central-1 / central-2) | responded | + |---|---| + | `POST /deployment_artifact.v1.DeploymentArtifactService/Fetch` | **302** ×4 / ×4 | + | `POST /mesh_transport.v1.CentralCommandService/ReportApply` | **302** ×8 / ×8 | + | `POST /mesh_transport.v1.NodeCommandService/Execute` | **302** ×1 / ×1 | + | `POST /telemetry.v1.TelemetryStreamService/Subscribe` | **302** ×107 / ×105 (central's own-pair dial reconnecting every ~5 s) | + + Site nodes (no auth stack) served `NodeCommandService/Execute` → `200` — the report's "site nodes + unaffected" finding holds live; it is the *central* side of every mesh call that was gated. + +### Step 1 — the fix: **PASS** + +Image `otopcua-host:dev` = `b344565cad3d`, built 12:28 local via `docker compose build` from the master +working tree. ⚠️ Master advanced from `f1b976f5` to `e280db37` (R3-02 merge + one follow-up) *while the +control was running*, and the build ran after that, so the post-fix image is master **`e280db37`** — +which contains `d79d51dd` (R3-01) unchanged; the working tree was clean at build time. Parity asserted +on all six containers (`b344565cad3d` ×6), `DisableLogin=false` still on both centrals, SQL volume kept +(so the TimedOut revision re-dispatched as a new deployment). + +1. Deployment `98ea226a…` (same revision `9e3540a1…`): **`Status = 2` (Sealed)**, `CreatedAtUtc + 16:30:43.743`, `SealedAtUtc 16:30:43.920` (177 ms). Six `NodeDeploymentState` rows, **all + `Status = 1` with `AppliedAtUtc` 16:30:43.836–.910**: central-1, central-2, site-a-1, site-a-2, + site-b-1, site-b-2. A second deploy after recreating the centrals (`49bcf361…`) sealed identically. +2. site-a-1: `[16:30:43 INF] Fetched + verified artifact for 98ea226a… from http://central-1:4055 (1469 + bytes).` → `cached applied deployment 98ea226a… as this node's local fallback`. +3. Centrals' request log, same grep: `DeploymentArtifactService/Fetch` **200** ×4, + `CentralCommandService/ReportApply` **200** ×8, `NodeCommandService/Execute` **200**. Count of + `responded (302|401)|PROTOCOL_ERROR|/login|Invalid content-type` across all six node logs after the + recreate: **0 on every node** except one `HTTP GET / responded 302` on central-2 — the curl probe of + the AdminUI root, i.e. the FallbackPolicy still active for the UI (the intended scope). The four site + nodes: 0. +4. Signed in through the real form at `http://localhost:9200/login` as `multi-role`/`password` (see the + LDAP finding below for why not `alice`); session badge listed all roles. `/hosts` showed the MAIN + cluster's `pymodbus` (`MAIN-modbus`) driver **Healthy, Last read 16:37:23 UTC then 16:37:53 UTC** + across two auto-refreshes — that row is fed only by central's `TelemetryDial` into its *own pair's* + `TelemetryStreamService`, the stream that answered 302 ×212 in the control. `/alerts` pill `live` + (recorded, but per the plan not relied on). The rig seed has **no drivers on SITE-A/SITE-B**, so there + are no per-node site rows to show; site-node telemetry is instead evidenced by central-1's dial log — + one boot-time `Telemetry stream to node … failed (failure streak 1); reconnecting` per node at + recreate (connection refused, peer not yet listening) and **no further failure** for the rest of the + run, and by the `TelemetryStreamService/Subscribe` streams on the centrals never closing (0 + `responded` lines after recreate — a stream logs only when it ends). +5. `/hosts → Controls → Restart → Confirm restart` on `MAIN-modbus`: central-2 `[16:38:19 INF] AdminOps: + RestartDriver dispatched for MAIN-modbus by multi-role`; **both** central-1 and central-2 logged + `HTTP POST /mesh_transport.v1.NodeCommandService/Execute responded 200` followed by `DriverHost + central-N:4053: restarting driver MAIN-modbus by request of multi-role` — the central→own-pair + `NodeCommandService` dial that step 0 broke (`Invalid content-type value: text/html`). + +LocalDb sync (site-a replicating pair, step 2 of the brief): the library logs no success line, so the +evidence is (a) an **ESTABLISHED** TCP session on site-a-2's `:9001` sync listener from site-a-1 +(`/proc/net/tcp6` inside site-a-2: `…FFFF00000761A8C0:2329 ← …FFFF00000861A8C0:E276 state 01`), and +(b) exactly one `Replication session to peer http://site-a-2:9001 faulted; reconnecting.` on site-a-1 +at boot (`16:29:47`, inner `SocketException (111): Connection refused` — peer not yet listening) and +none afterwards. Site nodes carry no auth stack, so this was expected to be unaffected in both runs; +it was (same shape in the control). + +### Step 2 — restore + +`docker compose -f docker-dev/docker-compose.yml up -d central-1 central-2` with no env var → +`Security__Auth__DisableLogin=true` in both containers, `curl -s http://localhost:9200/` → `200` (auto- +authenticated). Rig left up on `otopcua-host:dev` = `b344565cad3d` ×6. Scratch worktree +`/tmp/otopcua-prefix` removed. The `otopcua-host:prefix` image tag is left behind for re-runs. + +### Findings (new, outside R3-01's fix) + +1. **The rig's LDAP login is broken by the SEC-36 rotation, masked until now by `DisableLogin=true`.** + With login enabled, every sign-in failed `/login?error=Unexpected%20authentication%20error` + (= `LdapAuthFailure.ServiceAccountBindFailed`). The shared GLAuth at `10.100.0.35:3893` logged + `invalid credentials binddn=cn=serviceaccount,dc=zb,dc=local src=10.100.0.199` — its + `serviceaccount` `passsha256` (`e9c2967c…`, rotated 2026-08-07, scadaproj `aada53b`) no longer + matches the `serviceaccount123` the compose hardcodes (`sha256 = af29d0e5…`); the directory also has + **no `alice`** (that user is the Host.IntegrationTests harness GLAuth). Same class as the ScadaBridge + rig breakage recorded in the scadaproj memory. The gate proceeded with a gate-only compose override + (`Security__Ldap__Server: host.docker.internal`, pointing at the local `zb-ldap-primary` pair which + keeps `serviceaccount123`) and `multi-role`/`password`; the override was dropped at restore. The + compose's `Security__Ldap__*` block, the `docker-dev/README.md` "Auth" section and the plan's + `alice`/`password` recipe all need correcting — filed as a follow-up, not fixed here. +2. **The live cookie-challenge signature differs from the unit-test one.** Execution note 4 recorded + `StatusCode.Internal … PROTOCOL_ERROR` from the in-process fixture. On the rig the server side is a + plain `302` (Kestrel log), and the gRPC *client* side surfaces as **`Cancelled: Bad gRPC response. + Invalid content-type value: text/html; charset=utf-8`** (the `DispatchDeployment` warning) or a bare + `(Cancelled)` (the fetcher's per-endpoint warning). Grep for `responded 302` on the serving side and + `Invalid content-type value: text/html` / `Cancelled` on the dialling side; `PROTOCOL_ERROR` never + appeared in any container log. +3. **The seed gives the site clusters no drivers**, so `/hosts` cannot show per-node driver health for + site nodes — plan step-1 item 4 ("all six nodes") is only satisfiable for the central pair on this + rig as seeded. Not a defect; noted so the next gate does not read "No drivers." as a telemetry break. diff --git a/archreview/plans/R3-01-mesh-grpc-allowanonymous-plan.md.tasks.json b/archreview/plans/R3-01-mesh-grpc-allowanonymous-plan.md.tasks.json index 9464b7e2..75b3fcb7 100644 --- a/archreview/plans/R3-01-mesh-grpc-allowanonymous-plan.md.tasks.json +++ b/archreview/plans/R3-01-mesh-grpc-allowanonymous-plan.md.tasks.json @@ -71,7 +71,7 @@ { "id": "T10", "subject": "LIVE GATE: docker-dev with DisableLogin=false — step-0 falsification control MUST fail on the pre-fix build (Deployment.Status=4), then the fixed build seals Applied on all six nodes; restore DisableLogin=true", - "status": "pending", + "status": "done", "blockedBy": [ "T3", "T4",