Joseph Doherty
00f39c849e
The 2026-07 integration sweep flagged the Modbus non-standard profiles as 8F/5F/3F/8F and guessed 'pymodbus 4.x drift'. Re-investigation shows that was wrong on both counts: - The image runs the PINNED pymodbus 3.13.0 (verified via import pymodbus; __version__). The '...removed in v4' log is 3.13.0's forward-deprecation warning, misread as running 4.x. - The failures were a fixture-cycling artifact: each profile is a separate profile-gated compose service sharing :5020, and a plain 'docker compose down' (no --profile) does NOT stop the running profile container. It kept holding :5020; the next '--profile up' container silently failed to bind (compose still says 'Started'), so every later profile run hit the STALE sim. Proven by a raw Modbus probe: the 'mitsubishi' sim served DL205's 0xCAFE at HR0. After force-removing all modbus containers and bringing up each profile cleanly (verifying it actually published :5020), all four pass: dl205 16/16, mitsubishi 13/13, s7_1500 10/10, exception_injection 17/17. No OtOpcUa or test code change needed. Added a profile-cycling gotcha + reliable-cycle recipe to Docker/README.md and corrected the sweep record. Integration-sweep follow-up #2.
167 lines
12 KiB
Markdown
167 lines
12 KiB
Markdown
# Integration-test sweep — status + follow-ups (2026-07-14)
|
||
|
||
**Context:** ran the deferred `*.IntegrationTests` sweep (the round-2 STATUS.md "Full integration-suite
|
||
sweep" item that discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24).
|
||
**How:** tests run **on the Mac** (repo + .NET 10 SDK), pointing at **driver fixtures brought up on the
|
||
docker host `10.100.0.35`**, one driver at a time (serial — the suites have a ~16 GB/run memory leak).
|
||
Master at run time: `30672888` (branch `master`, clean).
|
||
|
||
**Docker host `10.100.0.35` (`DOCKER`) note:** has `dotnet 10.0.300`, 22 cores, but only ~7 GB free RAM
|
||
(the rest is shared SQL/GLAuth/fixtures) and **no repo checkout** — so the tests can't run *on* it (OOM risk
|
||
to shared infra). Run on the Mac against its fixtures. Fixture dirs live at `/opt/otopcua-<driver>/` (owned,
|
||
need sudo to create new ones — I deployed FOCAS to `~/otopcua-focas` on the host instead).
|
||
|
||
---
|
||
|
||
## Results
|
||
|
||
| Suite | Result | Root cause of failures |
|
||
|---|---|---|
|
||
| **OpcUaClient** | ✅ 4/4 | — (opc-plc `:50000`) |
|
||
| **OpcUaClient.Browser** | ✅ 1/1 | — |
|
||
| **S7** | ✅ 3/3 (2 skip) | skips = outage tests needing `S7_TIMEOUT_OUTAGE_*` env |
|
||
| **Modbus** (profile=standard) | ✅ 6/6 | — |
|
||
| **Host** | ⚠️ **105/117** (7F, 5 skip) | 7 heavy 2-node deploy/failover E2E time out (20–45 s) under amd64-emulated SQL; 5 LDAP-backed skip |
|
||
| **TwinCAT** | ⏭️ 13 skip | no TC3 XAR fixture exists (expected clean-skip) |
|
||
| **Modbus** (dl205 / mitsubishi / s7_1500 / exception_injection) | ✅ 16/13/10/17 (was 8F/5F/3F/8F) | **NOT drift** — fixture-cycling artifact, re-verified GREEN 2026-07-15 — see below |
|
||
| **AbCip** | ✅ 10P/3skip (was 6F/1P/3skip) | static-init-order bug FIXED 2026-07-15 (`326b22a7`) — see below |
|
||
| **FOCAS** | ❌ 9F/1P | tests assume a localhost-colocated mock — see below |
|
||
| **AbLegacy** | ❌ 4F | no AbLegacy sim fixture deployed to the host |
|
||
| **OpcUaServer** | ✅ 4/4 (was 4F) | cert-store gap + SDK read-throws drift FIXED 2026-07-15 — see below |
|
||
|
||
### Headline conclusion
|
||
**No OtOpcUa production regression.** Every failure is **pre-existing test/fixture infrastructure rot** in
|
||
long-deferred suites — or, in the Modbus case, a **fixture-cycling false alarm** (see below). The drivers are
|
||
fine: 4 suites fully green, **all Modbus profiles green** (standard + dl205/mitsubishi/s7_1500/exception once
|
||
correctly bound — the sweep's non-standard "failures" were the stale-container artifact), Host 105/117 green,
|
||
and all driver **unit** suites + the live `/run` gates pass. These integration suites were DEFERRED in round 2
|
||
and never run, so they'd bit-rotted in several distinct ways.
|
||
|
||
**Follow-up progress (2026-07-15):** #1 AbCip (PR #446), #5 OpcUaServer (PR #447), and #2 Modbus (PR #448 — false
|
||
alarm, no code) all closed. Remaining: #3 FOCAS, #4 AbLegacy, #6 Host — each needs a design/infra decision.
|
||
|
||
---
|
||
|
||
## Per-suite failure detail (evidence)
|
||
|
||
- **Modbus non-standard profiles** — ✅ **RE-VERIFIED GREEN 2026-07-15 (PR #448). The sweep's 8F/5F/3F/8F was a
|
||
false alarm — a fixture-cycling artifact, NOT pymodbus drift and NOT an OtOpcUa regression.** Correcting the
|
||
sweep's original diagnosis:
|
||
- The image genuinely runs **pymodbus 3.13.0** (`docker run --entrypoint python ... -c 'import pymodbus;
|
||
print(pymodbus.__version__)'` → `3.13.0`; the Dockerfile already pins `pymodbus[simulator]==3.13.0`). The
|
||
*"…removed in v4"* log line is 3.13.0's **forward-deprecation warning**, not evidence it runs 4.x — the
|
||
original sweep note misread it.
|
||
- Root cause: each Modbus profile is a **separate profile-gated compose service** sharing `:5020`. A plain
|
||
`docker compose down` (no `--profile`) does **not** stop the running profile container, so it keeps holding
|
||
`:5020`; the next `--profile … up` container silently fails to bind (compose still says "Started"). Every
|
||
later profile run then hit the **stale** container. Proven by a raw Modbus probe: the "mitsubishi" sim
|
||
served DL205's `0xCAFE` at HR0 (DL205's V0 marker) and Illegal-Data-Address for mitsubishi-only addresses,
|
||
while `docker ps` showed the mitsubishi container up with **no** port mapping and DL205 still holding `:5020`.
|
||
- After force-removing all modbus containers and bringing up each profile cleanly (verifying it actually
|
||
published `:5020`), all four pass: **dl205 16/16, mitsubishi 13/13, s7_1500 10/10, exception_injection
|
||
17/17** (the injector needs `--force-recreate` — it uses a different container name). **No code change** —
|
||
added a profile-cycling gotcha + reliable-cycle recipe to `Docker/README.md`.
|
||
|
||
- **AbCip** — **static-init-order bug in the test harness** (always-fail, blocks the whole suite).
|
||
`tests/.../AbCip.IntegrationTests/AbServerProfile.cs`: `KnownProfiles.All { get; } = [ControlLogix, …]` is
|
||
declared at **line 30**, *before* the `static readonly` profile fields (line 34+). C# runs static
|
||
initializers in textual order → `All` captures `[null,null,null,null]` → `ForFamily` NREs at `p.Family`
|
||
(line 61) and `AbServerFixture..ctor` gets null. **Fix (trivial, safe):** make `All` an expression-bodied
|
||
property: `public static IReadOnlyList<AbServerProfile> All => [ControlLogix, CompactLogix, Micro800,
|
||
GuardLogix];` (evaluated on access, after the fields are set). Unblocks the suite; then re-run to see if the
|
||
actual sim tests pass.
|
||
|
||
- **FOCAS** — tests key on `focas://127.0.0.1:8193/Axes/X/AbsolutePosition` (KeyNotFound) — they assume a
|
||
**localhost-colocated mock**, not a remote docker fixture at `10.100.0.35:8193`. The repo fixture
|
||
(`tests/.../FOCAS.IntegrationTests/Docker/`, publishes `8193:8193`) was NOT deployed to the host; I deployed
|
||
it to `~/otopcua-focas` and it serves, but the tests' node-key/topology assumption (127.0.0.1) doesn't match
|
||
a remote sim. **Fix:** reconcile the mock topology — either run the FOCAS mock on localhost, or make the test
|
||
node-keys honor `OTOPCUA_FOCAS_SIM_ENDPOINT`/`FOCAS_MOCK_HOST`.
|
||
|
||
- **AbLegacy** — 4F on `StatusCode` because **no AbLegacy sim exists on the host** (no `/opt/otopcua-ablegacy`,
|
||
none deployed). Reads return Bad. **Fix:** deploy an AbLegacy fixture (check `tests/.../AbLegacy.IntegrationTests/Docker/`).
|
||
|
||
- **OpcUaServer** — ✅ **FIXED 2026-07-15** (PR #447). Was 4F on `Opc.Ua.ServiceResultException :
|
||
TrustedIssuerCertificates StorePath must be specified` — the in-test *client* `SecurityConfiguration`
|
||
omitted the trusted-issuer/peer/rejected store paths that `ValidateAsync` requires. Fix: `TestClientSecurity`
|
||
helper rooting Directory stores at a throwaway PKI dir (mirrors `DefaultApplicationConfigurationFactory`).
|
||
Fixing it unmasked a second drift — the 1.5.378 node-cache read path throws
|
||
`ServiceResultException(BadNodeIdUnknown)` for a removed node rather than returning a bad `DataValue`;
|
||
the two subscription-survival assertions now use `Should.ThrowAsync`. Suite **4F → 4/4 green**.
|
||
|
||
- **Host** — largely healthy (105/117). 7 failures are the heavy 2-node deploy/failover/reconnect E2E tests
|
||
(`DeployHappyPathTests`, `FailoverDuringDeployTests`, `DriverReconnectE2eTests`,
|
||
`FleetDiagnosticsRoundTripTests`, `EquipmentNamespaceMaterializationTests`,
|
||
`RoslynVirtualTagEvaluatorTests`) taking 20–45 s — likely **timing under the amd64-emulated SQL** on the
|
||
arm64 Mac (mssql:2022 has no arm64 image). 5 LDAP-backed tests skip because **`bitnami/openldap:2.6` is gone
|
||
from Bitnami's catalog** (manifest unknown — Bitnami deprecated their free image catalog in 2025).
|
||
**Fix:** replace the openldap image with a maintained one; run Host E2E against real (non-emulated) SQL or
|
||
raise deadlines.
|
||
|
||
- **TwinCAT** — 13 skip cleanly (no TC3 XAR fixture exists; unit-only per the driver's design). Expected.
|
||
|
||
---
|
||
|
||
## Follow-up backlog (distinct, not one fix)
|
||
|
||
1. ~~**AbCip static-init bug** — trivial always-fail test bug. Expression-body `KnownProfiles.All`.~~
|
||
✅ **DONE 2026-07-15** (commit `326b22a7`, PR #446). Live-verified against the controllogix fixture:
|
||
suite went **6F/1P/3skip → 10 pass / 0 fail / 3 skip** (skips = emulator-mode tests needing a different
|
||
fixture mode).
|
||
2. ~~**Modbus non-standard profiles** — pin pymodbus in the fixture Dockerfile.~~
|
||
✅ **RESOLVED 2026-07-15 (PR #448) — false alarm, no code change.** Image already runs the pinned
|
||
pymodbus 3.13.0; the 8F/5F/3F/8F was a fixture-cycling artifact (stale profile container holding `:5020`).
|
||
Re-verified all four profiles GREEN (16/13/10/17). Added a profile-cycling gotcha + reliable-cycle recipe
|
||
to `Docker/README.md`.
|
||
3. **FOCAS** — reconcile the 127.0.0.1-mock topology with the remote-fixture model.
|
||
4. **AbLegacy** — deploy an AbLegacy sim fixture (none on host).
|
||
5. ~~**OpcUaServer** — supply cert-store StorePath in the harness config.~~
|
||
✅ **DONE 2026-07-15** (PR #447). Added `TestClientSecurity` helper building a validated client
|
||
`SecurityConfiguration` with Directory own/issuer/trusted/rejected stores under a throwaway PKI dir
|
||
(mirrors `DefaultApplicationConfigurationFactory`); wired into both `DualEndpointTests` +
|
||
`SubscriptionSurvivalTests`. That unmasked a **second** drift: the 1.5.378 node-cache read path
|
||
*throws* `ServiceResultException(BadNodeIdUnknown)` for a removed node instead of returning a bad
|
||
`DataValue` — updated the two subscription-survival assertions to `Should.ThrowAsync`. Suite **4F → 4/4**.
|
||
6. **Host** — replace `bitnami/openldap:2.6` (image gone); run E2E on real SQL or raise deadlines.
|
||
7. **Fixture deployment gap** — FOCAS + AbLegacy fixtures aren't on the host at `/opt/otopcua-*`; the sweep
|
||
requires deploying them (and `/opt` needs sudo — used `~/otopcua-<driver>` for FOCAS).
|
||
|
||
---
|
||
|
||
## Fixture setup recipes (to re-run the sweep)
|
||
|
||
Bring up on the docker host (distinct ports, can coexist except Modbus profiles share `:5020`):
|
||
|
||
```bash
|
||
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-modbus && docker compose --profile standard up -d' # :5020
|
||
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-abcip && docker compose --profile controllogix up -d' # :44818
|
||
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d' # :1102
|
||
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-opcuaclient && docker compose up -d' # :50000
|
||
# FOCAS/AbLegacy not on host — rsync tests/.../Docker/ to ~/otopcua-<driver> then `docker compose up -d --build`
|
||
```
|
||
|
||
Run per suite (env → project):
|
||
|
||
| Suite | Env | Project |
|
||
|---|---|---|
|
||
| OpcUaClient(+Browser) | `OPCUA_SIM_ENDPOINT=opc.tcp://10.100.0.35:50000` | `tests/Drivers/…OpcUaClient(.Browser).IntegrationTests` |
|
||
| S7 | `S7_SIM_ENDPOINT=10.100.0.35:1102` | `tests/Drivers/…S7.IntegrationTests` |
|
||
| Modbus | `MODBUS_SIM_ENDPOINT=10.100.0.35:5020` `MODBUS_SIM_PROFILE=<standard\|dl205\|mitsubishi\|s7_1500\|exception_injection>` (swap the host sim to match) | `tests/Drivers/…Modbus.IntegrationTests` |
|
||
| AbCip | `AB_SERVER_ENDPOINT=10.100.0.35:44818` `AB_SERVER_PROFILE=controllogix` | `tests/Drivers/…AbCip.IntegrationTests` |
|
||
| FOCAS | `OTOPCUA_FOCAS_SIM_ENDPOINT=10.100.0.35:8193` (but tests assume 127.0.0.1 — see above) | `tests/Drivers/…FOCAS.IntegrationTests` |
|
||
| Host | `OTOPCUA_HARNESS_USE_SQL=1` (+ SQL `:14331` via `tests/Server/…Host.IntegrationTests/docker-compose.yml up sql`) `OTOPCUA_HARNESS_USE_LDAP=0` | `tests/Server/…Host.IntegrationTests` |
|
||
|
||
**Modbus profile cycling:** the sim services share `:5020`, so to run a profile:
|
||
`ssh … 'cd /opt/otopcua-modbus && docker compose down && docker compose --profile <p> up -d'`, then run with
|
||
`MODBUS_SIM_PROFILE=<p>` (each class skips unless its profile is active). Wait for `:5020` before running —
|
||
TCP opens before the register map fully loads.
|
||
|
||
---
|
||
|
||
## Current state (post-sweep)
|
||
|
||
- **All fixtures torn down** — modbus/abcip/s7/opcuaclient/focas on the host removed (ports 5020/44818/1102/
|
||
50000/8193 all closed); the local Host SQL fixture `down -v`'d. Shared host is clean.
|
||
- **Repo:** `master @ 30672888`, clean. No sweep changes committed (this file is the only artifact).
|
||
- **Nothing merged for the sweep** — it's a read-only test run; the follow-ups above are unstarted.
|