docs(plans): R3-01 live gate — PASS on docker-dev with DisableLogin=false (control failed first)
This commit is contained in:
Joseph Doherty
@@ -436,3 +436,128 @@ process that are worth recording:
|
||||
|
||||
No other deviations from the plan's design or task breakdown. T10 was not run — left `pending`
|
||||
per the run brief, for the orchestrator.
|
||||
|
||||
## Live gate result (2026-08-20) — **PASS** on docker-dev with `DisableLogin=false` (control failed first)
|
||||
|
||||
Rig: `docker-dev/docker-compose.yml` on OrbStack (Mac), fresh volumes (`down -v` first), six host
|
||||
containers. Both runs used `OTOPCUA_DISABLE_LOGIN=false` on the central pair (verified in-container:
|
||||
`Security__Auth__DisableLogin=false` on central-1/central-2; `curl -s http://localhost:9200/` → `302 →
|
||||
/login?ReturnUrl=%2F` in both runs — the cookie FallbackPolicy was live for the UI throughout).
|
||||
Deploys were triggered with `./docker-dev/deploy.sh` (`POST /api/deployments`, `X-Api-Key:
|
||||
docker-dev-deploy-key`), which answered `202 Accepted` in both runs — the headless deploy API is
|
||||
`AllowAnonymous` + key-gated and is NOT behind the cookie fallback, so the API-key path works with
|
||||
login enabled (one open question from the brief, answered: not a finding).
|
||||
|
||||
### Step 0 — falsification control: **FAILED as required**
|
||||
|
||||
Control image `otopcua-host:prefix` = `53c41fc6bc01`, built from a scratch worktree of master
|
||||
`f1b976f5` with the five R3-01 opt-outs reverted in `MeshGrpcEndpoints.cs` (`.AllowAnonymous()` removed
|
||||
from the four `MapGrpcService` maps, the empty-prefix `MapGroup(...).AllowAnonymous()` collapsed back to a
|
||||
bare `MapZbLocalDbSync()`; `git diff --stat` = 1 file, 5 insertions/5 deletions). Chosen over a checkout of
|
||||
`851e4cf5` so the control image matched the R3-04 compose (site-b-2 `Cluster:Roles`-only, site-a-2
|
||||
miscased pin) — the same compose both runs used. Tagged over `otopcua-host:dev` for the run; image
|
||||
parity asserted on all six containers (`docker inspect --format '{{.Image}}'` = `53c41fc6bc01…` ×6).
|
||||
|
||||
- Deployment `26077962…` (revision `9e3540a1…`): **`Status = 4` (TimedOut)**, `SealedAtUtc = NULL`,
|
||||
six `NodeDeploymentState` rows all `Status = 0`, `AppliedAtUtc = NULL`.
|
||||
- central-1 (seal): `[16:10:04 WRN] Deployment 26077962e53a4ae99538c5aaaa396a85 timed out after
|
||||
00:02:00 (0/6 acks landed). No ack from: central-1:4053, central-2:4053, site-a-1:4053,
|
||||
site-a-2:4053, site-b-1:4053, site-b-2:4053.`
|
||||
- central-1 (own-pair command dial, the fused-node path): `[16:08:04 WRN] Mesh: DispatchDeployment
|
||||
(correlationId=9fe30594…) was NOT delivered to central-1:4053 (Cancelled: Bad gRPC response.
|
||||
Invalid content-type value: text/html; charset=utf-8).` — and identically for central-2:4053.
|
||||
- site-a-1 (artifact fetch + ack): `[16:08:04 WRN] Artifact fetch for 26077962… from
|
||||
http://central-1:4055 failed (Cancelled); trying next endpoint.` → `… failed on ALL 2 endpoint(s);
|
||||
returning no bytes (apply failure — last-known-good is kept).` → `[16:08:04 ERR] DriverHost
|
||||
site-a-1:4053: FetchAndCache apply of 26077962… FAILED … nothing was applied. Staying on revision
|
||||
null` → `[16:08:04 WRN] Dropping the ApplyAck for 26077962… from site-a-1:4053: all 4 central
|
||||
endpoint(s) refused it (http://central-1:4055 => Cancelled; …)`.
|
||||
- Server-side signature, per central (Kestrel request log, `sort | uniq -c` over the whole control run):
|
||||
|
||||
| endpoint (central-1 / central-2) | responded |
|
||||
|---|---|
|
||||
| `POST /deployment_artifact.v1.DeploymentArtifactService/Fetch` | **302** ×4 / ×4 |
|
||||
| `POST /mesh_transport.v1.CentralCommandService/ReportApply` | **302** ×8 / ×8 |
|
||||
| `POST /mesh_transport.v1.NodeCommandService/Execute` | **302** ×1 / ×1 |
|
||||
| `POST /telemetry.v1.TelemetryStreamService/Subscribe` | **302** ×107 / ×105 (central's own-pair dial reconnecting every ~5 s) |
|
||||
|
||||
Site nodes (no auth stack) served `NodeCommandService/Execute` → `200` — the report's "site nodes
|
||||
unaffected" finding holds live; it is the *central* side of every mesh call that was gated.
|
||||
|
||||
### Step 1 — the fix: **PASS**
|
||||
|
||||
Image `otopcua-host:dev` = `b344565cad3d`, built 12:28 local via `docker compose build` from the master
|
||||
working tree. ⚠️ Master advanced from `f1b976f5` to `e280db37` (R3-02 merge + one follow-up) *while the
|
||||
control was running*, and the build ran after that, so the post-fix image is master **`e280db37`** —
|
||||
which contains `d79d51dd` (R3-01) unchanged; the working tree was clean at build time. Parity asserted
|
||||
on all six containers (`b344565cad3d` ×6), `DisableLogin=false` still on both centrals, SQL volume kept
|
||||
(so the TimedOut revision re-dispatched as a new deployment).
|
||||
|
||||
1. Deployment `98ea226a…` (same revision `9e3540a1…`): **`Status = 2` (Sealed)**, `CreatedAtUtc
|
||||
16:30:43.743`, `SealedAtUtc 16:30:43.920` (177 ms). Six `NodeDeploymentState` rows, **all
|
||||
`Status = 1` with `AppliedAtUtc` 16:30:43.836–.910**: central-1, central-2, site-a-1, site-a-2,
|
||||
site-b-1, site-b-2. A second deploy after recreating the centrals (`49bcf361…`) sealed identically.
|
||||
2. site-a-1: `[16:30:43 INF] Fetched + verified artifact for 98ea226a… from http://central-1:4055 (1469
|
||||
bytes).` → `cached applied deployment 98ea226a… as this node's local fallback`.
|
||||
3. Centrals' request log, same grep: `DeploymentArtifactService/Fetch` **200** ×4,
|
||||
`CentralCommandService/ReportApply` **200** ×8, `NodeCommandService/Execute` **200**. Count of
|
||||
`responded (302|401)|PROTOCOL_ERROR|/login|Invalid content-type` across all six node logs after the
|
||||
recreate: **0 on every node** except one `HTTP GET / responded 302` on central-2 — the curl probe of
|
||||
the AdminUI root, i.e. the FallbackPolicy still active for the UI (the intended scope). The four site
|
||||
nodes: 0.
|
||||
4. Signed in through the real form at `http://localhost:9200/login` as `multi-role`/`password` (see the
|
||||
LDAP finding below for why not `alice`); session badge listed all roles. `/hosts` showed the MAIN
|
||||
cluster's `pymodbus` (`MAIN-modbus`) driver **Healthy, Last read 16:37:23 UTC then 16:37:53 UTC**
|
||||
across two auto-refreshes — that row is fed only by central's `TelemetryDial` into its *own pair's*
|
||||
`TelemetryStreamService`, the stream that answered 302 ×212 in the control. `/alerts` pill `live`
|
||||
(recorded, but per the plan not relied on). The rig seed has **no drivers on SITE-A/SITE-B**, so there
|
||||
are no per-node site rows to show; site-node telemetry is instead evidenced by central-1's dial log —
|
||||
one boot-time `Telemetry stream to node … failed (failure streak 1); reconnecting` per node at
|
||||
recreate (connection refused, peer not yet listening) and **no further failure** for the rest of the
|
||||
run, and by the `TelemetryStreamService/Subscribe` streams on the centrals never closing (0
|
||||
`responded` lines after recreate — a stream logs only when it ends).
|
||||
5. `/hosts → Controls → Restart → Confirm restart` on `MAIN-modbus`: central-2 `[16:38:19 INF] AdminOps:
|
||||
RestartDriver dispatched for MAIN-modbus by multi-role`; **both** central-1 and central-2 logged
|
||||
`HTTP POST /mesh_transport.v1.NodeCommandService/Execute responded 200` followed by `DriverHost
|
||||
central-N:4053: restarting driver MAIN-modbus by request of multi-role` — the central→own-pair
|
||||
`NodeCommandService` dial that step 0 broke (`Invalid content-type value: text/html`).
|
||||
|
||||
LocalDb sync (site-a replicating pair, step 2 of the brief): the library logs no success line, so the
|
||||
evidence is (a) an **ESTABLISHED** TCP session on site-a-2's `:9001` sync listener from site-a-1
|
||||
(`/proc/net/tcp6` inside site-a-2: `…FFFF00000761A8C0:2329 ← …FFFF00000861A8C0:E276 state 01`), and
|
||||
(b) exactly one `Replication session to peer http://site-a-2:9001 faulted; reconnecting.` on site-a-1
|
||||
at boot (`16:29:47`, inner `SocketException (111): Connection refused` — peer not yet listening) and
|
||||
none afterwards. Site nodes carry no auth stack, so this was expected to be unaffected in both runs;
|
||||
it was (same shape in the control).
|
||||
|
||||
### Step 2 — restore
|
||||
|
||||
`docker compose -f docker-dev/docker-compose.yml up -d central-1 central-2` with no env var →
|
||||
`Security__Auth__DisableLogin=true` in both containers, `curl -s http://localhost:9200/` → `200` (auto-
|
||||
authenticated). Rig left up on `otopcua-host:dev` = `b344565cad3d` ×6. Scratch worktree
|
||||
`/tmp/otopcua-prefix` removed. The `otopcua-host:prefix` image tag is left behind for re-runs.
|
||||
|
||||
### Findings (new, outside R3-01's fix)
|
||||
|
||||
1. **The rig's LDAP login is broken by the SEC-36 rotation, masked until now by `DisableLogin=true`.**
|
||||
With login enabled, every sign-in failed `/login?error=Unexpected%20authentication%20error`
|
||||
(= `LdapAuthFailure.ServiceAccountBindFailed`). The shared GLAuth at `10.100.0.35:3893` logged
|
||||
`invalid credentials binddn=cn=serviceaccount,dc=zb,dc=local src=10.100.0.199` — its
|
||||
`serviceaccount` `passsha256` (`e9c2967c…`, rotated 2026-08-07, scadaproj `aada53b`) no longer
|
||||
matches the `serviceaccount123` the compose hardcodes (`sha256 = af29d0e5…`); the directory also has
|
||||
**no `alice`** (that user is the Host.IntegrationTests harness GLAuth). Same class as the ScadaBridge
|
||||
rig breakage recorded in the scadaproj memory. The gate proceeded with a gate-only compose override
|
||||
(`Security__Ldap__Server: host.docker.internal`, pointing at the local `zb-ldap-primary` pair which
|
||||
keeps `serviceaccount123`) and `multi-role`/`password`; the override was dropped at restore. The
|
||||
compose's `Security__Ldap__*` block, the `docker-dev/README.md` "Auth" section and the plan's
|
||||
`alice`/`password` recipe all need correcting — filed as a follow-up, not fixed here.
|
||||
2. **The live cookie-challenge signature differs from the unit-test one.** Execution note 4 recorded
|
||||
`StatusCode.Internal … PROTOCOL_ERROR` from the in-process fixture. On the rig the server side is a
|
||||
plain `302` (Kestrel log), and the gRPC *client* side surfaces as **`Cancelled: Bad gRPC response.
|
||||
Invalid content-type value: text/html; charset=utf-8`** (the `DispatchDeployment` warning) or a bare
|
||||
`(Cancelled)` (the fetcher's per-endpoint warning). Grep for `responded 302` on the serving side and
|
||||
`Invalid content-type value: text/html` / `Cancelled` on the dialling side; `PROTOCOL_ERROR` never
|
||||
appeared in any container log.
|
||||
3. **The seed gives the site clusters no drivers**, so `/hosts` cannot show per-node driver health for
|
||||
site nodes — plan step-1 item 4 ("all six nodes") is only satisfiable for the central pair on this
|
||||
rig as seeded. Not a defect; noted so the next gate does not read "No drivers." as a telemetry break.
|
||||
|
||||
@@ -71,7 +71,7 @@
|
||||
{
|
||||
"id": "T10",
|
||||
"subject": "LIVE GATE: docker-dev with DisableLogin=false — step-0 falsification control MUST fail on the pre-fix build (Deployment.Status=4), then the fixed build seals Applied on all six nodes; restore DisableLogin=true",
|
||||
"status": "pending",
|
||||
"status": "done",
|
||||
"blockedBy": [
|
||||
"T3",
|
||||
"T4",
|
||||
|
||||
Reference in New Issue
Block a user