lmxopcua/archreview/plans/INTEGRATION-SWEEP-STATUS.md

Integration-test sweep — status + follow-ups (2026-07-14)

Context: ran the deferred *.IntegrationTests sweep (the round-2 STATUS.md "Full integration-suite sweep" item that discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24). How: tests run on the Mac (repo + .NET 10 SDK), pointing at driver fixtures brought up on the docker host 10.100.0.35, one driver at a time (serial — the suites have a ~16 GB/run memory leak). Master at run time: 30672888 (branch master, clean).

Docker host 10.100.0.35 (DOCKER) note: has dotnet 10.0.300, 22 cores, but only ~7 GB free RAM (the rest is shared SQL/GLAuth/fixtures) and no repo checkout — so the tests can't run on it (OOM risk to shared infra). Run on the Mac against its fixtures. Fixture dirs live at /opt/otopcua-<driver>/ (owned, need sudo to create new ones — I deployed FOCAS to ~/otopcua-focas on the host instead).

---

Results

Suite	Result	Root cause of failures
OpcUaClient	✅ 4/4	— (opc-plc :50000)
OpcUaClient.Browser	✅ 1/1	—
S7	✅ 3/3 (2 skip)	skips = outage tests needing S7_TIMEOUT_OUTAGE_* env
Modbus (profile=standard)	✅ 6/6	—
Host	⚠️ 105/117 (7F, 5 skip)	7 heavy 2-node deploy/failover E2E time out (20–45 s) under amd64-emulated SQL; 5 LDAP-backed skip
TwinCAT	⏭️ 13 skip	no TC3 XAR fixture exists (expected clean-skip)
Modbus (dl205 / mitsubishi / s7_1500 / exception_injection)	✅ 16/13/10/17 (was 8F/5F/3F/8F)	NOT drift — fixture-cycling artifact, re-verified GREEN 2026-07-15 — see below
AbCip	✅ 10P/3skip (was 6F/1P/3skip)	static-init-order bug FIXED 2026-07-15 (326b22a7) — see below
FOCAS	❌ 9F/1P	tests assume a localhost-colocated mock — see below
AbLegacy	❌ 4F	no AbLegacy sim fixture deployed to the host
OpcUaServer	✅ 4/4 (was 4F)	cert-store gap + SDK read-throws drift FIXED 2026-07-15 — see below

Headline conclusion

No OtOpcUa production regression. Every failure is pre-existing test/fixture infrastructure rot in long-deferred suites — or, in the Modbus case, a fixture-cycling false alarm (see below). The drivers are fine: 4 suites fully green, all Modbus profiles green (standard + dl205/mitsubishi/s7_1500/exception once correctly bound — the sweep's non-standard "failures" were the stale-container artifact), Host 105/117 green, and all driver unit suites + the live /run gates pass. These integration suites were DEFERRED in round 2 and never run, so they'd bit-rotted in several distinct ways.

Follow-up progress (2026-07-15): #1 AbCip (PR #446), #5 OpcUaServer (PR #447), and #2 Modbus (PR #448 — false alarm, no code) all closed. Remaining: #3 FOCAS, #4 AbLegacy, #6 Host — each needs a design/infra decision.

---

Per-suite failure detail (evidence)

• Modbus non-standard profiles — ✅ RE-VERIFIED GREEN 2026-07-15 (PR #448). The sweep's 8F/5F/3F/8F was a false alarm — a fixture-cycling artifact, NOT pymodbus drift and NOT an OtOpcUa regression. Correcting the sweep's original diagnosis:

  • The image genuinely runs pymodbus 3.13.0 (docker run --entrypoint python ... -c 'import pymodbus; print(pymodbus.__version__)' → 3.13.0; the Dockerfile already pins pymodbus[simulator]==3.13.0). The "…removed in v4" log line is 3.13.0's forward-deprecation warning, not evidence it runs 4.x — the original sweep note misread it.
  • Root cause: each Modbus profile is a separate profile-gated compose service sharing :5020. A plain docker compose down (no --profile) does not stop the running profile container, so it keeps holding :5020; the next --profile … up container silently fails to bind (compose still says "Started"). Every later profile run then hit the stale container. Proven by a raw Modbus probe: the "mitsubishi" sim served DL205's 0xCAFE at HR0 (DL205's V0 marker) and Illegal-Data-Address for mitsubishi-only addresses, while docker ps showed the mitsubishi container up with no port mapping and DL205 still holding :5020.
  • After force-removing all modbus containers and bringing up each profile cleanly (verifying it actually published :5020), all four pass: dl205 16/16, mitsubishi 13/13, s7_1500 10/10, exception_injection 17/17 (the injector needs --force-recreate — it uses a different container name). No code change — added a profile-cycling gotcha + reliable-cycle recipe to Docker/README.md.

• AbCip — static-init-order bug in the test harness (always-fail, blocks the whole suite). tests/.../AbCip.IntegrationTests/AbServerProfile.cs: KnownProfiles.All { get; } = [ControlLogix, …] is declared at line 30, before the static readonly profile fields (line 34+). C# runs static initializers in textual order → All captures [null,null,null,null] → ForFamily NREs at p.Family (line 61) and AbServerFixture..ctor gets null. Fix (trivial, safe): make All an expression-bodied property: public static IReadOnlyList<AbServerProfile> All => [ControlLogix, CompactLogix, Micro800, GuardLogix]; (evaluated on access, after the fields are set). Unblocks the suite; then re-run to see if the actual sim tests pass.

• FOCAS — tests key on focas://127.0.0.1:8193/Axes/X/AbsolutePosition (KeyNotFound) — they assume a localhost-colocated mock, not a remote docker fixture at 10.100.0.35:8193. The repo fixture (tests/.../FOCAS.IntegrationTests/Docker/, publishes 8193:8193) was NOT deployed to the host; I deployed it to ~/otopcua-focas and it serves, but the tests' node-key/topology assumption (127.0.0.1) doesn't match a remote sim. Fix: reconcile the mock topology — either run the FOCAS mock on localhost, or make the test node-keys honor OTOPCUA_FOCAS_SIM_ENDPOINT/FOCAS_MOCK_HOST.

• AbLegacy — 4F on StatusCode because no AbLegacy sim exists on the host (no /opt/otopcua-ablegacy, none deployed). Reads return Bad. Fix: deploy an AbLegacy fixture (check tests/.../AbLegacy.IntegrationTests/Docker/).

• OpcUaServer — ✅ FIXED 2026-07-15 (PR #447). Was 4F on Opc.Ua.ServiceResultException : TrustedIssuerCertificates StorePath must be specified — the in-test client SecurityConfiguration omitted the trusted-issuer/peer/rejected store paths that ValidateAsync requires. Fix: TestClientSecurity helper rooting Directory stores at a throwaway PKI dir (mirrors DefaultApplicationConfigurationFactory). Fixing it unmasked a second drift — the 1.5.378 node-cache read path throws ServiceResultException(BadNodeIdUnknown) for a removed node rather than returning a bad DataValue; the two subscription-survival assertions now use Should.ThrowAsync. Suite 4F → 4/4 green.

• Host — largely healthy (105/117). 7 failures are the heavy 2-node deploy/failover/reconnect E2E tests (DeployHappyPathTests, FailoverDuringDeployTests, DriverReconnectE2eTests, FleetDiagnosticsRoundTripTests, EquipmentNamespaceMaterializationTests, RoslynVirtualTagEvaluatorTests) taking 20–45 s — likely timing under the amd64-emulated SQL on the arm64 Mac (mssql:2022 has no arm64 image). 5 LDAP-backed tests skip because bitnami/openldap:2.6 is gone from Bitnami's catalog (manifest unknown — Bitnami deprecated their free image catalog in 2025). Fix: replace the openldap image with a maintained one; run Host E2E against real (non-emulated) SQL or raise deadlines.

• TwinCAT — 13 skip cleanly (no TC3 XAR fixture exists; unit-only per the driver's design). Expected.

---

Follow-up backlog (distinct, not one fix)

1. AbCip static-init bug — trivial always-fail test bug. Expression-body KnownProfiles.All. ✅ DONE 2026-07-15 (commit 326b22a7, PR #446). Live-verified against the controllogix fixture: suite went 6F/1P/3skip → 10 pass / 0 fail / 3 skip (skips = emulator-mode tests needing a different fixture mode).
2. Modbus non-standard profiles — pin pymodbus in the fixture Dockerfile. ✅ RESOLVED 2026-07-15 (PR #448) — false alarm, no code change. Image already runs the pinned pymodbus 3.13.0; the 8F/5F/3F/8F was a fixture-cycling artifact (stale profile container holding :5020). Re-verified all four profiles GREEN (16/13/10/17). Added a profile-cycling gotcha + reliable-cycle recipe to Docker/README.md.
3. FOCAS — reconcile the 127.0.0.1-mock topology with the remote-fixture model.
4. AbLegacy — deploy an AbLegacy sim fixture (none on host).
5. OpcUaServer — supply cert-store StorePath in the harness config. ✅ DONE 2026-07-15 (PR #447). Added TestClientSecurity helper building a validated client SecurityConfiguration with Directory own/issuer/trusted/rejected stores under a throwaway PKI dir (mirrors DefaultApplicationConfigurationFactory); wired into both DualEndpointTests + SubscriptionSurvivalTests. That unmasked a second drift: the 1.5.378 node-cache read path throws ServiceResultException(BadNodeIdUnknown) for a removed node instead of returning a bad DataValue — updated the two subscription-survival assertions to Should.ThrowAsync. Suite 4F → 4/4.
6. Host — replace bitnami/openldap:2.6 (image gone); run E2E on real SQL or raise deadlines.
7. Fixture deployment gap — FOCAS + AbLegacy fixtures aren't on the host at /opt/otopcua-*; the sweep requires deploying them (and /opt needs sudo — used ~/otopcua-<driver> for FOCAS).

---

Fixture setup recipes (to re-run the sweep)

Bring up on the docker host (distinct ports, can coexist except Modbus profiles share :5020):

ssh dohertj2@10.100.0.35 'cd /opt/otopcua-modbus     && docker compose --profile standard up -d'   # :5020
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-abcip      && docker compose --profile controllogix up -d' # :44818
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7         && docker compose --profile s7_1500 up -d'     # :1102
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-opcuaclient && docker compose up -d'                       # :50000
# FOCAS/AbLegacy not on host — rsync tests/.../Docker/ to ~/otopcua-<driver> then `docker compose up -d --build`

Run per suite (env → project):

Suite	Env	Project
OpcUaClient(+Browser)	OPCUA_SIM_ENDPOINT=opc.tcp://10.100.0.35:50000	tests/Drivers/…OpcUaClient(.Browser).IntegrationTests
S7	S7_SIM_ENDPOINT=10.100.0.35:1102	tests/Drivers/…S7.IntegrationTests
Modbus	MODBUS_SIM_ENDPOINT=10.100.0.35:5020 MODBUS_SIM_PROFILE=<standard|dl205|mitsubishi|s7_1500|exception_injection> (swap the host sim to match)	tests/Drivers/…Modbus.IntegrationTests
AbCip	AB_SERVER_ENDPOINT=10.100.0.35:44818 AB_SERVER_PROFILE=controllogix	tests/Drivers/…AbCip.IntegrationTests
FOCAS	OTOPCUA_FOCAS_SIM_ENDPOINT=10.100.0.35:8193 (but tests assume 127.0.0.1 — see above)	tests/Drivers/…FOCAS.IntegrationTests
Host	OTOPCUA_HARNESS_USE_SQL=1 (+ SQL :14331 via tests/Server/…Host.IntegrationTests/docker-compose.yml up sql) OTOPCUA_HARNESS_USE_LDAP=0	tests/Server/…Host.IntegrationTests

Modbus profile cycling: the sim services share :5020, so to run a profile: ssh … 'cd /opt/otopcua-modbus && docker compose down && docker compose --profile <p> up -d', then run with MODBUS_SIM_PROFILE=<p> (each class skips unless its profile is active). Wait for :5020 before running — TCP opens before the register map fully loads.

---

Current state (post-sweep)

• All fixtures torn down — modbus/abcip/s7/opcuaclient/focas on the host removed (ports 5020/44818/1102/ 50000/8193 all closed); the local Host SQL fixture down -v'd. Shared host is clean.
• Repo: master @ 30672888, clean. No sweep changes committed (this file is the only artifact).
• Nothing merged for the sweep — it's a read-only test run; the follow-ups above are unstarted.