diff --git a/docs/reviews/2026-10-21/gates.md b/docs/reviews/2026-10-21/gates.md index 1379f9ef8..f8108b295 100644 --- a/docs/reviews/2026-10-21/gates.md +++ b/docs/reviews/2026-10-21/gates.md @@ -397,3 +397,190 @@ therefore not observed.** Re-check `mxgateway_sessions_open` once the VPN is bac `Deployment` rows carry only the `env:` reference, no secret. - All host nodes were recreated without the exported key, so `GALAXY_MXGW_API_KEY` is back to the committed docker-dev default. The rig runs `otopcua-host:dev` `sha256:47ca0da6eaf7` (`5ac55205`). + +## Re-run on current master — all runnable gates (2026-11-12) + +| | | +|---|---| +| Commit tested | `54c4357f` = `master` `412da27c` + the deploy-API fix below (landed on master as `fa8c3d1a` after a rebase over two unrelated commits; same diff). Image `otopcua-host:g761` `sha256:24432d7fbc2d`, built from that commit, on all six nodes. | +| Rig | docker-dev as its **own compose project** `otopcua-g761` (an overlay remapping every host port and image tag), so the shared `otopcua-dev` rig that someone else was running was not touched. The ConfigDb was restored from a `COPY_ONLY` backup of the `otopcua-dev` ConfigDb (the same MAIN-modbus / MAIN-opcua / equipment `EQ-32296cc68964` + `okvt` config the 11-09 run used), then migrated (no pending migrations). Host ports: OPC UA 24840–24845, AdminUI 29200, per-node HTTP 29100–29105. | +| Clock | Rig UTC 2026-11-13 03:05–03:44 (= 2026-11-12 evening US Eastern). | +| Load | The 48 `int3b-mb-*` and the `int3b-blackhole` drivers were **disabled** after gate A. With them on, two rigs' worth of Modbus drivers saturated the shared pymodbus sim on 10.100.0.35 (about 117–136 % CPU) and `HR100` read `BadCommunicationError`. Gate D's 50-driver checks were not re-run (see D). | + +**Summary: A, B, C, D (check 4), E, H1 (live), the F removal check and every fixture lane PASS. One product defect +found and fixed (deploy API). Hardware userGates still open.** + +### Defect found and fixed: a bodyless `POST /api/deployments` answered 404 — FIXED (`fa8c3d1a`) + +The documented headless deploy (`docker-dev/deploy.sh`, README "Headless deploy") returned **404** on both centrals, +while the same POST with a JSON body was accepted. The optional body was an inferred `DeployRequest?` parameter, +which stamps `Accepts(application/json)` metadata on the endpoint. For a request with no `Content-Type` the routing +`AcceptsMatcherPolicy` prefers endpoints that declare no consumes metadata. On a fused admin+driver host +`MapOtOpcUaMeshGrpcServices` has mapped grpc-dotnet's `{unimplementedService}/{unimplementedMethod}` catch-all, so +the request was routed away from the deploy handler and got a 404. The deploy API on its own does not reproduce +it. The handler now reads the optional body itself: no body is the headless form, a non-JSON body answers 415 and +malformed JSON 400. `DeployApiBodylessPostTests` (Host.IntegrationTests) composes the deploy API with the mesh gRPC +services and drives real HTTP; with the fix reverted both bodyless cases fail with 404. Live on the rebuilt rig, a +bodyless POST through Traefik answered **202** (every deploy below used it). + +### Gate A (#731) — PASS, now with `dotnet-counters` + +Method as on 11-09 (each iteration edits `MAIN-modbus` `TimeoutMs` and `MAIN-opcua` `SessionName`, deploys, waits for +the seal + 8 s), 2 warm-ups + 20 iterations, every deploy sealed in 1–2 s. This time `dotnet-counters collect -p 1 +--counters System.Runtime --refresh-interval 5` (the linux-arm64 single-file tool copied into each central) ran for +the whole loop, alongside `/proc/1` threads / fds / RSS. + +| | central-1 | central-2 | +|---|---|---| +| `disposed Modbus driver on stop` / `disposed OpcUaClient driver on stop` | 20 / 20 | 20 / 20 | +| `spawned … MAIN-modbus` / `MAIN-opcua` | 20 / 20 | 20 / 20 | +| dispose/shutdown timeout or throw | 0 | 0 | +| threads (baseline → range over 20 → settled 75 s later) | 54 → 49–57 → 48 | 53 → 48–56 → 46 | +| open fds | 739 → 691–739 → 715 | 739 → 704–740 → 719 | +| `dotnet.assembly.count` (57 / 54 samples) | 312, constant | 314, constant | +| `dotnet.timer.count` first-quarter vs last-quarter mean | 139 → 137 | 144 → 145 | +| `dotnet.thread_pool.thread.count` (UpDownCounter deltas) | net ≈ 0 | net ≈ 0 | +| working set first-quarter vs last-quarter mean | 520 → 505 MB | 522 → 502 MB | + +No upward trend in any series; nothing (assemblies, timers, threads, fds) accumulates per respawn. Raw CSVs: +`counters-central-{1,2}.csv` in the run's evidence (not committed). + +### Gate B (#732) — PASS + +Same procedure as 11-09 on `opc.tcp://localhost:24840`, session 1 held 120 s on `ns=2;s=pymodbus/plc/HR100` and +`ns=3;s=EQ-32296cc68964/okvt`. + +- **Surgical add** (`HR101`, 03:14:27): both centrals logged `kind=PureAdd, added=1, … rebuild=False`. Each + subscription took one `BadNoCommunication` sample at 03:14:27 and was Good again in the same second (the accepted + driver respawn), then 21 / 22 Good updates until the rebuild. `HR101` itself reads `BadOutOfRange` (the fixture + refuses address 101, as on 11-09). +- **Rebuild** (rename `e1` → `e1r`, 03:14:49): `kind=Rebuild, … rebuild=True`; session 1's items got + `BadNodeIdUnknown` and nothing after. +- **Re-created after the rebuild** (session 2, 20 s): HR100 17 updates, okvt 16, **all Good**, none Bad. + +### Gate C (#733) — PASS + +- **C-a, kill `central-1` mid-dispatch.** `site-b-1` paused, deployment `56e5ede1` at `AwaitingApplyAcks` 5/6, `docker + kill central-1` at 03:16:37. A second POST while in flight: **409**. After unpausing `site-b-1` the held + deployment **sealed 6/6** (on 11-09 it timed out). After `central-1` restarted, the next deploy `6df32531` + **sealed 6/6**. +- **C-b, kill the singleton host.** `central-2` (the only `/health/active` 200 on MAIN) killed at 03:17:30 with + `548ccbb9` in flight (5/6). `central-1` auto-downed it at 03:17:58, the singletons moved + (`Younger -> BecomingOldest`), the AdminUI was back through Traefik at 03:18:02, and at 03:18:14 `central-1` + logged `Coordinator recovered in-flight deployment 548ccbb9… (5/6 acks landed)`. It closed `TimedOut` with + `failureReason` = `Timed out waiting for acks (5/6 landed). No ack from: site-b-1:4053. …` (the #784 fix, + verified live). After `central-2` restarted, the next deploy `246594e0` **sealed 6/6**. +- **C-port.** As on 11-09, a replacement container from the compose service with a `perl` listener holding + `0.0.0.0:4840` before `dotnet OtOpcUa.Host.dll` starts (the 4840 socket inode belongs to perl, none to dotnet): + - fused admin+driver (`central-2`): `GET /health/ready` **503**, `opcua-endpoint: Unhealthy — OPC UA SDK failed to + start: Failed to establish tcp listener sockets on port 4840 for Ipv4 and IPv6.`; `configdb`/`akka` Healthy. + - driver-only (`site-a-2`): `FTL OtOpcUaServerHostedService: SDK start failed on a driver-only node …`, **exit 1**. + - So the C3 behaviour holds on OPC UA SDK 2.0.1. (A first attempt was invalid: the Perl one-liner did not keep the + socket in a variable, so Perl closed it at once and the SDK bound 4840 normally. Re-run with the socket held.) +- **Side effect worth knowing:** the 39 s `docker pause` of `site-b-1` in C-b outlived the 15 s auto-down, so + `site-b-2` downed it and the SITE-B pair came back as **two 1-member clusters, both Active**, until `site-b-2` + was restarted. That is the documented auto-down trade (`docs/Redundancy.md`, restart one side after the link + heals), not a defect, but every C-style drill that pauses a node past 15 s leaves this behind. + +### Gate D (#745) — check 4 re-run after #785: PASS + +Checks 1, 2, 3 and 5 passed on 11-09 and their code is unchanged. Check 4, the one that failed, was re-run with +Playwright against `http://localhost:29200` (login disabled), with a native-`confirm` listener armed (it never +fired): + +| page | Delete click | Cancel | Confirm | +|---|---|---|---| +| `/role-grants` (throwaway grant `cn=g761-gateD,…`) | modal "Delete role grant" naming the group; row still in DB | row kept | row deleted (0) | +| `/clusters/SITE-B/acls/g761-acl` (throwaway ACL) | modal "Delete ACL"; row still in DB | row kept | row deleted (0) | +| `/clusters/SITE-B/edit` | modal "Delete cluster 'SITE-B'? This cannot be undone." | modal closed, cluster kept | not confirmed (real row) | +| `/clusters/SITE-B/nodes/site-b-2:4053` | modal "Delete node 'site-b-2:4053' from cluster 'SITE-B'? …" | modal closed, node kept | not confirmed (real row) | + +### Gate E (#734) — PASS + +`gateE-spin` (`var v = ctx.GetTag("pymodbus/plc/HR100").Value; while (true) { }`) as change-triggered virtual tag +`spinvt` on `EQ-32296cc68964`, deployed 03:27:21, sampled every ~4 s for 1 min 50 s: + +- **Quarantined** on both centrals at 03:27:42 (21.4 s / 22.3 s, over 10 × the 2 s budget); `/metrics` + `otopcua_script_quarantined_total{reason="stuck",script="gateE-vt-spin"} 1`, `run_refused_total` 8 / 11. The + main-log mirror now names the script (`[Script] gateE-vt-spin emitted Error …`, the #784 fix). +- **Responsive:** `/health/ready` 200 in 2–6 ms throughout (2.9–3.5 ms before); `okvt` kept reading Good, connect + + read 0.40–0.48 s (0.41–0.52 s before); `spinvt` `BadInternalError`. +- **Threads flat:** central-1 42–44, central-2 41–45 (41–42 before). +- CPU: both centrals went to ~1 core (≈100 ticks/s) until restarted after the gate — the known in-process limit + (#750; the out-of-process runner is the fix). + +### H1 (#736) — live end-to-end over OPC UA: PASS + +This closes the 11-09 scope caveat (that run drove the backend directly). `ServerHistorian:Backend=SqlServer` on +the centrals (DB `OtOpcUaHistorian` on the rig SQL), 52,560 samples (every 10 min, 2025-11-01 → 2026-10-31) seeded +for `HR300` (`lg9.long`). A probe client (OPC UA SDK 2.0.1) sent `HistoryRead` raw with **`NumValuesPerNode = 0`** +for the whole year on `opc.tcp://localhost:24840` and followed every continuation point: + +| page | values | continuation point | +|---|---|---| +| 1 | 9,999 | 16 bytes | +| 2–5 | 9,998 each | 16 bytes | +| 6 | 2,569 | none | + +52,560 values in total (= the seeded set: first 0, last 52559), timestamps strictly ascending, no duplicates, +0.2 s. central-1 GC heap after the read: gen2 48 MB + LOH 56 MB + gen1 7 MB (≈ 112 MB), committed 170 MB — under the +500 MB bound. QuestDb was not re-run live: the paging is the backend-agnostic node-manager `RawPageLimit` plan, and +the QuestDb backend half passed on 11-09. + +### Gate F (#735) — the removal drop, now observed: PASS + +The 11-11 run could not see `mxgateway_sessions_open` return to baseline after removing the driver (the VPN +dropped). The VD03 gateway has since restarted (its counters reset), so this was re-checked directly: a read-only +`GalaxyMxGateway` driver `g761-gal` (one `Read` tag, `DevWinPlatform.ScanState`; key only as +`env:GALAXY_MXGW_API_KEY`, sourced into the compose shell from the owner's env file and never written anywhere) was +added to MAIN and then removed. Gateway `/metrics` (anonymous, scraped from the Mac): + +| when (UTC) | open | opened_total | closed_total | workers | +|---|---|---|---|---| +| baseline 03:40:49–03:41:19 | 2 | 2 | – | 2 | +| driver deployed 03:41:47 (both centrals `connected`, ScanState read Good) | **4** | 4 | – | 4 | +| driver removed 03:42:58 → 03:43:13 … 03:43:59 | **2** | 4 | 2 | 2 | + +Both centrals logged `stopped driver child g761-gal` + `disposed GalaxyMxGateway driver on stop`; the gateway +closed both sessions within 15 s. No Galaxy writes, no alarm subscriptions. + +### Driver fixture lanes (10.100.0.35) at `412da27c` — PASS + +| Suite | Result | Notes | +|---|---|---| +| Modbus standard (:5020) | 12 / 0 / 32 | Two earlier attempts failed 8 then 1 tests on connect timeouts: the shared sim was saturated by this gate's own rig plus `otopcua-dev` (196 connections from this Mac). Clean once the load dropped. Skips are the other profiles. | +| S7 s7_1500 (`S7_SIM_ALLOW_WRITES=1`) | 23 / 0 / 3 | Same 3 skips as 11-09 (TM/CT span limit, the two destructive CMD tests). | +| AbCip controllogix, `AB_SERVER_ALLOW_WRITES=1 AB_DIFF_ENDPOINT=…` | 34 / 0 / 22 | **No write test runs in this combination**: `IntegrationWriteGate` refuses writes while `AB_DIFF_ENDPOINT` is set (read-only differential run). That was also true of the 11-09 "with writes" run. | +| AbCip controllogix, `AB_SERVER_ALLOW_WRITES=1` only | 38 / 0 / 18 | The real write pass: all 10 write tests pass, incl. `AbCipRefusalLiveTests.A_whole_array_write_round_trips` (DINT[16]) and `AbCipUdtMemberWriteTests.A_member_array_element_write_leaves_the_other_elements_alone`, Native + libplctag. | +| OpcUaClient (opc-plc :50000) | 9 / 0 / 0 | One flake on the first try (`ServerHostNameValidationTests…(validate: False)`, `BadRequestInterrupted`); clean rerun. | +| OpcUaClient.Browser (+ `OPCUA_NAT_ENDPOINT=…:50010`) | 12 / 0 / 0 | The NAT fixture is up, so the #806 NAT tests ran too. | +| MQTT + Sparkplug | 30 / 0 / 0 | | +| Sql (SQL Server, PG, Oracle, blackhole) | 58 / 0 / 25 | 21 MySQL + 4 ODBC skips (no fixture). | +| **MTConnect (G1, :5000)** | 21 / 0 / 0 | First run of the G1 MTConnect lane against the shared agent. | + +### Hardware userGates — simulator evidence only; real hardware still required + +| userGate | Simulator evidence (2026-11-12, `412da27c`) | Real hardware | +|---|---|---| +| AbCip array write | ab_server ControlLogix: whole-array DINT[16] round-trip + UDT member-array element write pass (above) | not run | +| AbLegacy array write | ab_server SLC500 / MicroLogix / PLC-5: `N_file_array_write_round_trips_every_element` passes on all 3 families × 2 transports; AbLegacy suite 51 / 0 / 0 | not run (#752) | +| TwinCAT array + STRING write | **none** — no ADS simulator exists (`elma-ads-tcp/udp` on 10.100.0.35 are socat relays to an absent ADS box); suite 0 / 0 / 18 skipped | not run (#755) | +| FOCAS `MACRO:` > 32767 | `otopcua-focas-fixture` mock (byte-identical to the repo's `focas_mock/server.py`): `WireBackendCoverageTests.Macro_and_parameter_numbers_above_short_range_read_the_authored_variable` (`MACRO:99999`, `PARAM:40000`, with a decoy at the 16-bit-wrapped address) passes; FOCAS suite 18 / 0 / 0 | not run | +| AbCip probe-typo recovery | `AbCipRefusalLiveTests.A_typod_probe_tag_does_not_black_out_a_reachable_device` passes on ab_server, Native + libplctag | not run | + +Plan INT-4 marks these "USER-ORDERED — NON-SKIPPABLE … real ControlLogix / SLC / TwinCAT hardware", so the simulator +passes do not close them. They need bench endpoints from the owner, or an explicit owner decision to accept the +simulator evidence for the four that have it (TwinCAT has none). + +### Observations + +1. The bodyless-deploy 404 (fixed above) had been masked because every earlier gate run sent a JSON body. +2. Two docker-dev rigs on one Mac saturate the shared pymodbus sim; disable the 48 `int3b-mb-*` drivers (they exist + only for gate D's 50-driver checks) when not running those. + +### Rig state left behind + +The `otopcua-g761` project (containers, its volumes, the `otopcua-host:g761` / `otopcua-migrator:g761` images), +the `g761-probe` network and the temporary `OtOpcUaHistorian` database (inside the project's SQL volume) were all +removed. The shared `otopcua-dev` rig was only read (one `COPY_ONLY` backup to its container's `/tmp`, deleted +straight after). Nothing was started or stopped on 10.100.0.35.