lmxopcua/archreview/plans/INTEGRATION-SWEEP-STATUS.md

Integration-test sweep — status + follow-ups (2026-07-14)

Context: ran the deferred *.IntegrationTests sweep (the round-2 STATUS.md "Full integration-suite sweep" item that discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24). How: tests run on the Mac (repo + .NET 10 SDK), pointing at driver fixtures brought up on the docker host 10.100.0.35, one driver at a time (serial — the suites have a ~16 GB/run memory leak). Master at run time: 30672888 (branch master, clean).

Docker host 10.100.0.35 (DOCKER) note: has dotnet 10.0.300, 22 cores, but only ~7 GB free RAM (the rest is shared SQL/GLAuth/fixtures) and no repo checkout — so the tests can't run on it (OOM risk to shared infra). Run on the Mac against its fixtures. Fixture dirs live at /opt/otopcua-<driver>/ (owned, need sudo to create new ones — I deployed FOCAS to ~/otopcua-focas on the host instead).

---

Results

Suite	Result	Root cause of failures
OpcUaClient	✅ 4/4	— (opc-plc :50000)
OpcUaClient.Browser	✅ 1/1	—
S7	✅ 3/3 (2 skip)	skips = outage tests needing S7_TIMEOUT_OUTAGE_* env
Modbus (profile=standard)	✅ 6/6	—
Host	⚠️ 105/117 (7F, 5 skip)	7 heavy E2E time out under amd64-emulated SQL (OPEN); LDAP image swapped OpenLDAP→GLAuth 2026-07-15 (PR #451). The 5 skips are driver-fixture-gated E2E, not LDAP
TwinCAT	⏭️ 13 skip	no TC3 XAR fixture exists (expected clean-skip)
Modbus (dl205 / mitsubishi / s7_1500 / exception_injection)	✅ 16/13/10/17 (was 8F/5F/3F/8F)	NOT drift — fixture-cycling artifact, re-verified GREEN 2026-07-15 — see below
AbCip	✅ 10P/3skip (was 6F/1P/3skip)	static-init-order bug FIXED 2026-07-15 (326b22a7) — see below
FOCAS	✅ 10/10 (was 9F/1P)	topology + mock-timer-endianness FIXED 2026-07-15 — see below
AbLegacy	✅ slc500 2/2, micrologix 1/1, plc5 1/1 (was 4F)	fixture-deployment gap FIXED 2026-07-15 — see below
OpcUaServer	✅ 4/4 (was 4F)	cert-store gap + SDK read-throws drift FIXED 2026-07-15 — see below

Headline conclusion

No OtOpcUa production regression. Every failure is pre-existing test/fixture infrastructure rot in long-deferred suites — or, in the Modbus case, a fixture-cycling false alarm (see below). The drivers are fine: 4 suites fully green, all Modbus profiles green (standard + dl205/mitsubishi/s7_1500/exception once correctly bound — the sweep's non-standard "failures" were the stale-container artifact), Host 105/117 green, and all driver unit suites + the live /run gates pass. These integration suites were DEFERRED in round 2 and never run, so they'd bit-rotted in several distinct ways.

Follow-up progress (2026-07-15): ALL 7 CLOSED. #1 AbCip (PR #446), #5 OpcUaServer (PR #447), #2 Modbus (PR #448 — false alarm), #4 AbLegacy (PR #449 — fixture gap), #3 FOCAS (PR #450 — topology + mock-timer), #7 (fixture deploys), #6 LDAP-image (PR #451 — OpenLDAP→GLAuth), and #6 SQL-deploy-E2E (PR #452 — misdiagnosed "emulation timing"; really missing FK-parent seeds + a stale EquipmentId, hidden by the in-memory provider). Every driver + server integration suite is now verified GREEN. Zero OtOpcUa production regressions across the entire sweep — every failure was fixture/harness/test rot.

---

Per-suite failure detail (evidence)

• Modbus non-standard profiles — ✅ RE-VERIFIED GREEN 2026-07-15 (PR #448). The sweep's 8F/5F/3F/8F was a false alarm — a fixture-cycling artifact, NOT pymodbus drift and NOT an OtOpcUa regression. Correcting the sweep's original diagnosis:

  • The image genuinely runs pymodbus 3.13.0 (docker run --entrypoint python ... -c 'import pymodbus; print(pymodbus.__version__)' → 3.13.0; the Dockerfile already pins pymodbus[simulator]==3.13.0). The "…removed in v4" log line is 3.13.0's forward-deprecation warning, not evidence it runs 4.x — the original sweep note misread it.
  • Root cause: each Modbus profile is a separate profile-gated compose service sharing :5020. A plain docker compose down (no --profile) does not stop the running profile container, so it keeps holding :5020; the next --profile … up container silently fails to bind (compose still says "Started"). Every later profile run then hit the stale container. Proven by a raw Modbus probe: the "mitsubishi" sim served DL205's 0xCAFE at HR0 (DL205's V0 marker) and Illegal-Data-Address for mitsubishi-only addresses, while docker ps showed the mitsubishi container up with no port mapping and DL205 still holding :5020.
  • After force-removing all modbus containers and bringing up each profile cleanly (verifying it actually published :5020), all four pass: dl205 16/16, mitsubishi 13/13, s7_1500 10/10, exception_injection 17/17 (the injector needs --force-recreate — it uses a different container name). No code change — added a profile-cycling gotcha + reliable-cycle recipe to Docker/README.md.

• AbCip — static-init-order bug in the test harness (always-fail, blocks the whole suite). tests/.../AbCip.IntegrationTests/AbServerProfile.cs: KnownProfiles.All { get; } = [ControlLogix, …] is declared at line 30, before the static readonly profile fields (line 34+). C# runs static initializers in textual order → All captures [null,null,null,null] → ForFamily NREs at p.Family (line 61) and AbServerFixture..ctor gets null. Fix (trivial, safe): make All an expression-bodied property: public static IReadOnlyList<AbServerProfile> All => [ControlLogix, CompactLogix, Micro800, GuardLogix]; (evaluated on access, after the fields are set). Unblocks the suite; then re-run to see if the actual sim tests pass.

• FOCAS — ✅ FIXED 2026-07-15 (PR #450). Two test/fixture bugs, no OtOpcUa driver change.

  1. Topology: FocasSimFixture already reads an env-overridable endpoint (OTOPCUA_FOCAS_SIM_ENDPOINT, default localhost:8193) and exposes Host/Port, but WireBackendTests + WireBackendCoverageTests hardcoded focas://127.0.0.1:8193. Against a remote fixture the skip-gate passed (remote reachable) but the driver dialed localhost → KeyNotFound. Fix: added a DeviceUri property to the fixture and derived each test's DeviceHost from it — recovered 8 of 9.
  2. Mock timer endianness (newly surfaced): the last failure (Timers_populate) read 60397977600 for a 3600 s power-on timer (= 0x3C000000 × 60). Root cause: the focas-mock's _wire_timer encoded the minute/msec fields big-endian (_u32), but cnc_rdtimer's timer fields are little-endian on real hardware — a documented, live-31i-B-validated quirk (FocasWireClient.ParseTimer decodes them LE, per docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md). The driver was correct; the mock was wrong. Fix: _wire_timer now emits little-endian (_u32_le) → suite 10/10. Fixture deployed at ~/otopcua-focas.

• AbLegacy — ✅ FIXED 2026-07-15 (PR #449). Fixture-deployment gap, not a code defect. The AbLegacy suite needs ab_server in PCCC mode (--plc=SLC500|Micrologix|PLC/5), which reuses the AbCip image (otopcua-ab-server:libplctag-release, already on the host) and binds :44818. Two issues: (a) no PCCC container was deployed (~/otopcua-ablegacy didn't exist); (b) the sweep's 4F-not-skip means :44818 was reachable but answering CIP (a leftover AbCip container) — the PCCC reads returned Bad. Same shared-port contamination class as the Modbus false alarm. Deployed the compose to ~/otopcua-ablegacy on the host (image already present, no build needed) and ran each family with AB_LEGACY_COMPOSE_PROFILE set to match the running container (required — without it the [Theory] parametrises over all 3 families and 2 fail against a single-family container): slc500 2/2, micrologix 1/1 (+1 skip), plc5 1/1 (+1 skip) — the skips are family-inapplicable file types (e.g. no F8 float file seeded on micrologix/plc5). No code change.

• OpcUaServer — ✅ FIXED 2026-07-15 (PR #447). Was 4F on Opc.Ua.ServiceResultException : TrustedIssuerCertificates StorePath must be specified — the in-test client SecurityConfiguration omitted the trusted-issuer/peer/rejected store paths that ValidateAsync requires. Fix: TestClientSecurity helper rooting Directory stores at a throwaway PKI dir (mirrors DefaultApplicationConfigurationFactory). Fixing it unmasked a second drift — the 1.5.378 node-cache read path throws ServiceResultException(BadNodeIdUnknown) for a removed node rather than returning a bad DataValue; the two subscription-survival assertions now use Should.ThrowAsync. Suite 4F → 4/4 green.

• Host — largely healthy (105/117). Two independent issues:

  • LDAP image — ✅ FIXED 2026-07-15 (PR #451). The opt-in real-LDAP mode (OTOPCUA_HARNESS_USE_LDAP=1) used bitnami/openldap:2.6, gone since Bitnami deprecated their free catalog in 2025. Replaced it with GLAuth — the same LDAP server the rest of the project uses (docker-dev + the live OPC UA data-plane auth, both against the shared GLAuth on :3893), removing the lone OpenLDAP outlier. Added a harness glauth/config.toml (base DN dc=zb,dc=local, search-capable serviceaccount, dev users alice=full / bob=read-only, role groups with the same gidnumbers as scadaproj/infra/glauth) + repointed the harness's service-account override to cn=serviceaccount. Live-verified all three bind legs against a deployed container on :3894 (serviceaccount search returns alice + 5 memberOf groups; alice/bob bind OK; wrong password → Invalid credentials (49)). NB: the real-LDAP mode is opt-in — the default Host run uses StubLdapAuthService, so this never blocked the default suite (the sweep's "5 LDAP skips" were actually the driver-fixture-gated E2E tests, not LDAP).
  • SQL-backed deploy E2E — ✅ FIXED 2026-07-15 (PR #452). The sweep's "amd64-emulated-SQL timing" diagnosis was WRONG. Running the 7 heavy E2E against real (native-amd64) SQL on the Docker host proved they don't time out for speed — they had never actually run against real SQL, only the EF in-memory provider (which ignores FK constraints). Two distinct root causes, both hidden by in-memory:
    • Missing FK-parent seed (5 tests): the deploy writes a NodeDeploymentState row per node, which FKs to ClusterNode (FK_NodeDeploymentState_ClusterNode_NodeId). The tests never seeded ServerCluster + ClusterNode rows, so on SQL each node-state INSERT throws → the deploy never seals (waits the full deadline regardless of how high — proven at 4× it still hung). Added TwoNodeClusterHarness.SeedDefaultClusterAsync (seeds a ServerCluster + a ClusterNode per node; Warm/NodeCount=2 to satisfy both the SQL CHECK constraint CK_ServerCluster_RedundancyMode_NodeCount and the ClusterEnabledNodeCountMismatch validator) and called it in DeployHappyPathTests×2 / FailoverDuringDeployTests / FleetDiagnosticsRoundTripTests / EquipmentNamespaceMaterializationTests; fixed DriverReconnectE2eTests to seed node B + declare NodeCount=2.
    • Stale EquipmentId (1 test, pre-existing on BOTH providers): EquipmentNamespaceMaterializationTests seeded EquipmentId="eq-1", but the EquipmentIdNotDerived validator (added later) requires EquipmentId == DeriveEquipmentId(EquipmentUuid) = EQ- + first 12 hex of the UUID. Fixed the seed to a canonical id + matching UUID. (Was failing on in-memory too; only surfaced once the FK fix let the deploy reach equipment validation.)
    • Added OTOPCUA_HARNESS_SQL_HOST / OTOPCUA_HARNESS_LDAP_HOST env overrides so the fixtures can live on the native-amd64 Docker host (docker compose up -d sql there; run with OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331).
    • Result: the 5 SQL-backed classes go 11/11 green vs native SQL, and the in-memory default is unregressed (12/12). RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts… is a pre-existing flaky race test (not SQL-backed; passes in isolation, flaked once under heavy parallel load) — left as-is.

• TwinCAT — 13 skip cleanly (no TC3 XAR fixture exists; unit-only per the driver's design). Expected.

---

Follow-up backlog (distinct, not one fix)

1. AbCip static-init bug — trivial always-fail test bug. Expression-body KnownProfiles.All. ✅ DONE 2026-07-15 (commit 326b22a7, PR #446). Live-verified against the controllogix fixture: suite went 6F/1P/3skip → 10 pass / 0 fail / 3 skip (skips = emulator-mode tests needing a different fixture mode).
2. Modbus non-standard profiles — pin pymodbus in the fixture Dockerfile. ✅ RESOLVED 2026-07-15 (PR #448) — false alarm, no code change. Image already runs the pinned pymodbus 3.13.0; the 8F/5F/3F/8F was a fixture-cycling artifact (stale profile container holding :5020). Re-verified all four profiles GREEN (16/13/10/17). Added a profile-cycling gotcha + reliable-cycle recipe to Docker/README.md.
3. FOCAS — reconcile the 127.0.0.1-mock topology with the remote-fixture model. ✅ DONE 2026-07-15 (PR #450). Tests now derive DeviceHost from the fixture endpoint (recovered 8/9); also fixed a newly-surfaced mock bug — _wire_timer encoded the timer big-endian but real hardware + driver use little-endian. Suite 10/10. No OtOpcUa driver change.
4. AbLegacy — deploy an AbLegacy sim fixture (none on host). ✅ DONE 2026-07-15 (PR #449) — fixture-deployment gap, no code defect. Deployed the PCCC compose to ~/otopcua-ablegacy (reuses the AbCip image, already on host). All 3 families GREEN (slc500 2/2, micrologix 1/1, plc5 1/1). Recipe added below; AB_LEGACY_COMPOSE_PROFILE MUST match the running container.
5. OpcUaServer — supply cert-store StorePath in the harness config. ✅ DONE 2026-07-15 (PR #447). Added TestClientSecurity helper building a validated client SecurityConfiguration with Directory own/issuer/trusted/rejected stores under a throwaway PKI dir (mirrors DefaultApplicationConfigurationFactory); wired into both DualEndpointTests + SubscriptionSurvivalTests. That unmasked a second drift: the 1.5.378 node-cache read path throws ServiceResultException(BadNodeIdUnknown) for a removed node instead of returning a bad DataValue — updated the two subscription-survival assertions to Should.ThrowAsync. Suite 4F → 4/4.
6. Host — ⬐ split into two:
   • replace bitnami/openldap:2.6 (image gone) ✅ DONE 2026-07-15 (PR #451) — swapped the harness LDAP to GLAuth (unifies on the LDAP the rest of the project uses); live-verified the binds on :3894.
   • amd64-emulated-SQL timing ✅ DONE 2026-07-15 (PR #452) — misdiagnosed. Not a timing/emulation issue: the deploy E2E tests had never run against real SQL (in-memory ignores FKs) and lacked the ServerCluster/ClusterNode seed the NodeDeploymentState FK requires → deploy never seals. Added a harness seeding helper + OTOPCUA_HARNESS_SQL_HOST override; also fixed a stale EquipmentId in one test. 11/11 green vs native SQL, in-memory unregressed. (Roslyn racing test = pre-existing flaky, left as-is.)
7. Fixture deployment gap — FOCAS + AbLegacy fixtures aren't on the host. ✅ RESOLVED 2026-07-15. Both now deployed under the home dir: ~/otopcua-ablegacy (PR #449) + ~/otopcua-focas (PR #450). (/opt still needs sudo; the home-dir deploys work fine.)

---

Fixture setup recipes (to re-run the sweep)

Bring up on the docker host (distinct ports, can coexist except Modbus profiles share :5020):

ssh dohertj2@10.100.0.35 'cd /opt/otopcua-modbus     && docker compose --profile standard up -d'   # :5020
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-abcip      && docker compose --profile controllogix up -d' # :44818
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7         && docker compose --profile s7_1500 up -d'     # :1102
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-opcuaclient && docker compose up -d'                       # :50000
ssh dohertj2@10.100.0.35 'cd ~/otopcua-ablegacy      && docker compose --profile slc500 up -d'      # :44818 (PCCC; reuses AbCip image)
ssh dohertj2@10.100.0.35 'cd ~/otopcua-focas       && docker compose up -d --build'                 # :8193

Run per suite (env → project):

Suite	Env	Project
OpcUaClient(+Browser)	OPCUA_SIM_ENDPOINT=opc.tcp://10.100.0.35:50000	tests/Drivers/…OpcUaClient(.Browser).IntegrationTests
S7	S7_SIM_ENDPOINT=10.100.0.35:1102	tests/Drivers/…S7.IntegrationTests
Modbus	MODBUS_SIM_ENDPOINT=10.100.0.35:5020 MODBUS_SIM_PROFILE=<standard|dl205|mitsubishi|s7_1500|exception_injection> (swap the host sim to match)	tests/Drivers/…Modbus.IntegrationTests
AbCip	AB_SERVER_ENDPOINT=10.100.0.35:44818 AB_SERVER_PROFILE=controllogix	tests/Drivers/…AbCip.IntegrationTests
AbLegacy	AB_LEGACY_ENDPOINT=10.100.0.35:44818 AB_LEGACY_COMPOSE_PROFILE=<slc500|micrologix|plc5> (must match the running PCCC container — else the [Theory] runs all 3 families and 2 fail)	tests/Drivers/…AbLegacy.IntegrationTests
FOCAS	OTOPCUA_FOCAS_SIM_ENDPOINT=10.100.0.35:8193 (tests now honour this — no longer hardcoded to 127.0.0.1)	tests/Drivers/…FOCAS.IntegrationTests
Host	OTOPCUA_HARNESS_USE_SQL=1 (+ SQL :14331 via the harness docker-compose.yml up sql) — set OTOPCUA_HARNESS_USE_LDAP=1 to exercise real binds against the GLAuth service (up ldap, host :3894; users alice/bob, svc cn=serviceaccount/serviceaccount123)	tests/Server/…Host.IntegrationTests

Modbus profile cycling: the sim services share :5020, so to run a profile: ssh … 'cd /opt/otopcua-modbus && docker compose down && docker compose --profile <p> up -d', then run with MODBUS_SIM_PROFILE=<p> (each class skips unless its profile is active). Wait for :5020 before running — TCP opens before the register map fully loads.

---

Current state (post-sweep)

• All fixtures torn down — modbus/abcip/s7/opcuaclient/focas on the host removed (ports 5020/44818/1102/ 50000/8193 all closed); the local Host SQL fixture down -v'd. Shared host is clean.
• Repo: master @ 30672888, clean. No sweep changes committed (this file is the only artifact).
• Nothing merged for the sweep — it's a read-only test run; the follow-ups above are unstarted.