dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(review): #761 Wave-4 gates re-run on current master (2026-11-12)
v2-ci / unit-tests (push) Has been cancelled
Details

Browse Source 
Re-ran every gate docker-dev and the fixtures can run, on master 412da27c plus the
deploy-API fix: A (now with dotnet-counters), B, C, D check 4 (after #785), E, H1
live end-to-end over OPC UA (N=0 year window: 6 pages <= 10k, continuation points,
all 52,560 values), the gate-F removal drop on the VD03 gateway (4 -> 2 within 15 s),
and all fixture lanes incl. the first G1 MTConnect run. All pass.

Recorded: the AbCip "writes" combination ran no writes (AB_DIFF_ENDPOINT gates them
off), a C-style pause past 15 s leaves a split SITE-B pair, and simulator evidence for
four of the five hardware userGates (TwinCAT has no simulator). The hardware
userGates stay open.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 22:46:25 -05:00
parent fa8c3d1a1a
commit e5196ef79a
1 changed files with 187 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reviews/2026-10-21/gates.md
+187
View File
@@ -397,3 +397,190 @@ therefore not observed.** Re-check `mxgateway_sessions_open` once the VPN is bac
`Deployment` rows carry only the `env:` reference, no secret.
- All host nodes were recreated without the exported key, so `GALAXY_MXGW_API_KEY` is back to the committed
docker-dev default. The rig runs `otopcua-host:dev` `sha256:47ca0da6eaf7` (`5ac55205`).
## Re-run on current master — all runnable gates (2026-11-12)
| | |
|---|---|
| Commit tested | `54c4357f` = `master` `412da27c` + the deploy-API fix below (landed on master as `fa8c3d1a` after a rebase over two unrelated commits; same diff). Image `otopcua-host:g761` `sha256:24432d7fbc2d`, built from that commit, on all six nodes. |
| Rig | docker-dev as its **own compose project** `otopcua-g761` (an overlay remapping every host port and image tag), so the shared `otopcua-dev` rig that someone else was running was not touched. The ConfigDb was restored from a `COPY_ONLY` backup of the `otopcua-dev` ConfigDb (the same MAIN-modbus / MAIN-opcua / equipment `EQ-32296cc68964` + `okvt` config the 11-09 run used), then migrated (no pending migrations). Host ports: OPC UA 24840–24845, AdminUI 29200, per-node HTTP 29100–29105. |
| Clock | Rig UTC 2026-11-13 03:05–03:44 (= 2026-11-12 evening US Eastern). |
| Load | The 48 `int3b-mb-*` and the `int3b-blackhole` drivers were **disabled** after gate A. With them on, two rigs' worth of Modbus drivers saturated the shared pymodbus sim on 10.100.0.35 (about 117–136 % CPU) and `HR100` read `BadCommunicationError`. Gate D's 50-driver checks were not re-run (see D). |
**Summary: A, B, C, D (check 4), E, H1 (live), the F removal check and every fixture lane PASS. One product defect
found and fixed (deploy API). Hardware userGates still open.**
### Defect found and fixed: a bodyless `POST /api/deployments` answered 404 — FIXED (`fa8c3d1a`)
The documented headless deploy (`docker-dev/deploy.sh`, README "Headless deploy") returned **404** on both centrals,
while the same POST with a JSON body was accepted. The optional body was an inferred `DeployRequest?` parameter,
which stamps `Accepts(application/json)` metadata on the endpoint. For a request with no `Content-Type` the routing
`AcceptsMatcherPolicy` prefers endpoints that declare no consumes metadata. On a fused admin+driver host
`MapOtOpcUaMeshGrpcServices` has mapped grpc-dotnet's `{unimplementedService}/{unimplementedMethod}` catch-all, so
the request was routed away from the deploy handler and got a 404. The deploy API on its own does not reproduce
it. The handler now reads the optional body itself: no body is the headless form, a non-JSON body answers 415 and
malformed JSON 400. `DeployApiBodylessPostTests` (Host.IntegrationTests) composes the deploy API with the mesh gRPC
services and drives real HTTP; with the fix reverted both bodyless cases fail with 404. Live on the rebuilt rig, a
bodyless POST through Traefik answered **202** (every deploy below used it).
### Gate A (#731) — PASS, now with `dotnet-counters`
Method as on 11-09 (each iteration edits `MAIN-modbus` `TimeoutMs` and `MAIN-opcua` `SessionName`, deploys, waits for
the seal + 8 s), 2 warm-ups + 20 iterations, every deploy sealed in 1–2 s. This time `dotnet-counters collect -p 1
--counters System.Runtime --refresh-interval 5` (the linux-arm64 single-file tool copied into each central) ran for
the whole loop, alongside `/proc/1` threads / fds / RSS.
| | central-1 | central-2 |
|---|---|---|
| `disposed Modbus driver on stop` / `disposed OpcUaClient driver on stop` | 20 / 20 | 20 / 20 |
| `spawned … MAIN-modbus` / `MAIN-opcua` | 20 / 20 | 20 / 20 |
| dispose/shutdown timeout or throw | 0 | 0 |
| threads (baseline → range over 20 → settled 75 s later) | 54 → 49–57 → 48 | 53 → 48–56 → 46 |
| open fds | 739 → 691–739 → 715 | 739 → 704–740 → 719 |
| `dotnet.assembly.count` (57 / 54 samples) | 312, constant | 314, constant |
| `dotnet.timer.count` first-quarter vs last-quarter mean | 139 → 137 | 144 → 145 |
| `dotnet.thread_pool.thread.count` (UpDownCounter deltas) | net ≈ 0 | net ≈ 0 |
| working set first-quarter vs last-quarter mean | 520 → 505 MB | 522 → 502 MB |
No upward trend in any series; nothing (assemblies, timers, threads, fds) accumulates per respawn. Raw CSVs:
`counters-central-{1,2}.csv` in the run's evidence (not committed).
### Gate B (#732) — PASS
Same procedure as 11-09 on `opc.tcp://localhost:24840`, session 1 held 120 s on `ns=2;s=pymodbus/plc/HR100` and
`ns=3;s=EQ-32296cc68964/okvt`.
- **Surgical add** (`HR101`, 03:14:27): both centrals logged `kind=PureAdd, added=1, … rebuild=False`. Each
subscription took one `BadNoCommunication` sample at 03:14:27 and was Good again in the same second (the accepted
driver respawn), then 21 / 22 Good updates until the rebuild. `HR101` itself reads `BadOutOfRange` (the fixture
refuses address 101, as on 11-09).
- **Rebuild** (rename `e1` → `e1r`, 03:14:49): `kind=Rebuild, … rebuild=True`; session 1's items got
`BadNodeIdUnknown` and nothing after.
- **Re-created after the rebuild** (session 2, 20 s): HR100 17 updates, okvt 16, **all Good**, none Bad.
### Gate C (#733) — PASS
- **C-a, kill `central-1` mid-dispatch.** `site-b-1` paused, deployment `56e5ede1` at `AwaitingApplyAcks` 5/6, `docker
kill central-1` at 03:16:37. A second POST while in flight: **409**. After unpausing `site-b-1` the held
deployment **sealed 6/6** (on 11-09 it timed out). After `central-1` restarted, the next deploy `6df32531`
**sealed 6/6**.
- **C-b, kill the singleton host.** `central-2` (the only `/health/active` 200 on MAIN) killed at 03:17:30 with
`548ccbb9` in flight (5/6). `central-1` auto-downed it at 03:17:58, the singletons moved
(`Younger -> BecomingOldest`), the AdminUI was back through Traefik at 03:18:02, and at 03:18:14 `central-1`
logged `Coordinator recovered in-flight deployment 548ccbb9… (5/6 acks landed)`. It closed `TimedOut` with
`failureReason` = `Timed out waiting for acks (5/6 landed). No ack from: site-b-1:4053. …` (the #784 fix,
verified live). After `central-2` restarted, the next deploy `246594e0` **sealed 6/6**.
- **C-port.** As on 11-09, a replacement container from the compose service with a `perl` listener holding
`0.0.0.0:4840` before `dotnet OtOpcUa.Host.dll` starts (the 4840 socket inode belongs to perl, none to dotnet):
- fused admin+driver (`central-2`): `GET /health/ready` **503**, `opcua-endpoint: Unhealthy — OPC UA SDK failed to
start: Failed to establish tcp listener sockets on port 4840 for Ipv4 and IPv6.`; `configdb`/`akka` Healthy.
- driver-only (`site-a-2`): `FTL OtOpcUaServerHostedService: SDK start failed on a driver-only node …`, **exit 1**.
- So the C3 behaviour holds on OPC UA SDK 2.0.1. (A first attempt was invalid: the Perl one-liner did not keep the
socket in a variable, so Perl closed it at once and the SDK bound 4840 normally. Re-run with the socket held.)
- **Side effect worth knowing:** the 39 s `docker pause` of `site-b-1` in C-b outlived the 15 s auto-down, so
`site-b-2` downed it and the SITE-B pair came back as **two 1-member clusters, both Active**, until `site-b-2`
was restarted. That is the documented auto-down trade (`docs/Redundancy.md`, restart one side after the link
heals), not a defect, but every C-style drill that pauses a node past 15 s leaves this behind.
### Gate D (#745) — check 4 re-run after #785: PASS
Checks 1, 2, 3 and 5 passed on 11-09 and their code is unchanged. Check 4, the one that failed, was re-run with
Playwright against `http://localhost:29200` (login disabled), with a native-`confirm` listener armed (it never
fired):
| page | Delete click | Cancel | Confirm |
|---|---|---|---|
| `/role-grants` (throwaway grant `cn=g761-gateD,…`) | modal "Delete role grant" naming the group; row still in DB | row kept | row deleted (0) |
| `/clusters/SITE-B/acls/g761-acl` (throwaway ACL) | modal "Delete ACL"; row still in DB | row kept | row deleted (0) |
| `/clusters/SITE-B/edit` | modal "Delete cluster 'SITE-B'? This cannot be undone." | modal closed, cluster kept | not confirmed (real row) |
| `/clusters/SITE-B/nodes/site-b-2:4053` | modal "Delete node 'site-b-2:4053' from cluster 'SITE-B'? …" | modal closed, node kept | not confirmed (real row) |
### Gate E (#734) — PASS
`gateE-spin` (`var v = ctx.GetTag("pymodbus/plc/HR100").Value; while (true) { }`) as change-triggered virtual tag
`spinvt` on `EQ-32296cc68964`, deployed 03:27:21, sampled every ~4 s for 1 min 50 s:
- **Quarantined** on both centrals at 03:27:42 (21.4 s / 22.3 s, over 10 × the 2 s budget); `/metrics`
`otopcua_script_quarantined_total{reason="stuck",script="gateE-vt-spin"} 1`, `run_refused_total` 8 / 11. The
main-log mirror now names the script (`[Script] gateE-vt-spin emitted Error …`, the #784 fix).
- **Responsive:** `/health/ready` 200 in 2–6 ms throughout (2.9–3.5 ms before); `okvt` kept reading Good, connect +
read 0.40–0.48 s (0.41–0.52 s before); `spinvt` `BadInternalError`.
- **Threads flat:** central-1 42–44, central-2 41–45 (41–42 before).
- CPU: both centrals went to ~1 core (≈100 ticks/s) until restarted after the gate — the known in-process limit
(#750; the out-of-process runner is the fix).
### H1 (#736) — live end-to-end over OPC UA: PASS
This closes the 11-09 scope caveat (that run drove the backend directly). `ServerHistorian:Backend=SqlServer` on
the centrals (DB `OtOpcUaHistorian` on the rig SQL), 52,560 samples (every 10 min, 2025-11-01 → 2026-10-31) seeded
for `HR300` (`lg9.long`). A probe client (OPC UA SDK 2.0.1) sent `HistoryRead` raw with **`NumValuesPerNode = 0`**
for the whole year on `opc.tcp://localhost:24840` and followed every continuation point:
| page | values | continuation point |
|---|---|---|
| 1 | 9,999 | 16 bytes |
| 2–5 | 9,998 each | 16 bytes |
| 6 | 2,569 | none |
52,560 values in total (= the seeded set: first 0, last 52559), timestamps strictly ascending, no duplicates,
0.2 s. central-1 GC heap after the read: gen2 48 MB + LOH 56 MB + gen1 7 MB (≈ 112 MB), committed 170 MB — under the
500 MB bound. QuestDb was not re-run live: the paging is the backend-agnostic node-manager `RawPageLimit` plan, and
the QuestDb backend half passed on 11-09.
### Gate F (#735) — the removal drop, now observed: PASS
The 11-11 run could not see `mxgateway_sessions_open` return to baseline after removing the driver (the VPN
dropped). The VD03 gateway has since restarted (its counters reset), so this was re-checked directly: a read-only
`GalaxyMxGateway` driver `g761-gal` (one `Read` tag, `DevWinPlatform.ScanState`; key only as
`env:GALAXY_MXGW_API_KEY`, sourced into the compose shell from the owner's env file and never written anywhere) was
added to MAIN and then removed. Gateway `/metrics` (anonymous, scraped from the Mac):
| when (UTC) | open | opened_total | closed_total | workers |
|---|---|---|---|---|
| baseline 03:40:49–03:41:19 | 2 | 2 | – | 2 |
| driver deployed 03:41:47 (both centrals `connected`, ScanState read Good) | **4** | 4 | – | 4 |
| driver removed 03:42:58 → 03:43:13 … 03:43:59 | **2** | 4 | 2 | 2 |
Both centrals logged `stopped driver child g761-gal` + `disposed GalaxyMxGateway driver on stop`; the gateway
closed both sessions within 15 s. No Galaxy writes, no alarm subscriptions.
### Driver fixture lanes (10.100.0.35) at `412da27c` — PASS
| Suite | Result | Notes |
|---|---|---|
| Modbus standard (:5020) | 12 / 0 / 32 | Two earlier attempts failed 8 then 1 tests on connect timeouts: the shared sim was saturated by this gate's own rig plus `otopcua-dev` (196 connections from this Mac). Clean once the load dropped. Skips are the other profiles. |
| S7 s7_1500 (`S7_SIM_ALLOW_WRITES=1`) | 23 / 0 / 3 | Same 3 skips as 11-09 (TM/CT span limit, the two destructive CMD tests). |
| AbCip controllogix, `AB_SERVER_ALLOW_WRITES=1 AB_DIFF_ENDPOINT=…` | 34 / 0 / 22 | **No write test runs in this combination**: `IntegrationWriteGate` refuses writes while `AB_DIFF_ENDPOINT` is set (read-only differential run). That was also true of the 11-09 "with writes" run. |
| AbCip controllogix, `AB_SERVER_ALLOW_WRITES=1` only | 38 / 0 / 18 | The real write pass: all 10 write tests pass, incl. `AbCipRefusalLiveTests.A_whole_array_write_round_trips` (DINT[16]) and `AbCipUdtMemberWriteTests.A_member_array_element_write_leaves_the_other_elements_alone`, Native + libplctag. |
| OpcUaClient (opc-plc :50000) | 9 / 0 / 0 | One flake on the first try (`ServerHostNameValidationTests…(validate: False)`, `BadRequestInterrupted`); clean rerun. |
| OpcUaClient.Browser (+ `OPCUA_NAT_ENDPOINT=…:50010`) | 12 / 0 / 0 | The NAT fixture is up, so the #806 NAT tests ran too. |
| MQTT + Sparkplug | 30 / 0 / 0 | |
| Sql (SQL Server, PG, Oracle, blackhole) | 58 / 0 / 25 | 21 MySQL + 4 ODBC skips (no fixture). |
| **MTConnect (G1, :5000)** | 21 / 0 / 0 | First run of the G1 MTConnect lane against the shared agent. |
### Hardware userGates — simulator evidence only; real hardware still required
| userGate | Simulator evidence (2026-11-12, `412da27c`) | Real hardware |
|---|---|---|
| AbCip array write | ab_server ControlLogix: whole-array DINT[16] round-trip + UDT member-array element write pass (above) | not run |
| AbLegacy array write | ab_server SLC500 / MicroLogix / PLC-5: `N_file_array_write_round_trips_every_element` passes on all 3 families × 2 transports; AbLegacy suite 51 / 0 / 0 | not run (#752) |
| TwinCAT array + STRING write | **none** — no ADS simulator exists (`elma-ads-tcp/udp` on 10.100.0.35 are socat relays to an absent ADS box); suite 0 / 0 / 18 skipped | not run (#755) |
| FOCAS `MACRO:` > 32767 | `otopcua-focas-fixture` mock (byte-identical to the repo's `focas_mock/server.py`): `WireBackendCoverageTests.Macro_and_parameter_numbers_above_short_range_read_the_authored_variable` (`MACRO:99999`, `PARAM:40000`, with a decoy at the 16-bit-wrapped address) passes; FOCAS suite 18 / 0 / 0 | not run |
| AbCip probe-typo recovery | `AbCipRefusalLiveTests.A_typod_probe_tag_does_not_black_out_a_reachable_device` passes on ab_server, Native + libplctag | not run |
Plan INT-4 marks these "USER-ORDERED — NON-SKIPPABLE … real ControlLogix / SLC / TwinCAT hardware", so the simulator
passes do not close them. They need bench endpoints from the owner, or an explicit owner decision to accept the
simulator evidence for the four that have it (TwinCAT has none).
### Observations
1. The bodyless-deploy 404 (fixed above) had been masked because every earlier gate run sent a JSON body.
2. Two docker-dev rigs on one Mac saturate the shared pymodbus sim; disable the 48 `int3b-mb-*` drivers (they exist
only for gate D's 50-driver checks) when not running those.
### Rig state left behind
The `otopcua-g761` project (containers, its volumes, the `otopcua-host:g761` / `otopcua-migrator:g761` images),
the `g761-probe` network and the temporary `OtOpcUaHistorian` database (inside the project's SQL volume) were all
removed. The shared `otopcua-dev` rig was only read (one `COPY_ONLY` backup to its container's `/tmp`, deleted
straight after). Nothing was started or stopped on 10.100.0.35.