dohertj2/lmxopcua
1
0
Fork 0
Code Issues 3 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
00d8422cef94da43b9cf5dc45f70309f6624fcff
lmxopcua/archreview/plans/INTEGRATION-SWEEP-STATUS.md
T
Joseph Doherty f381b471f9
v2-ci / build (pull_request) Successful in 4m26s
Details
v2-ci / unit-tests (pull_request) Failing after 3h11m47s
Details
docs(sweep): AbLegacy verified green — fixture-deployment gap, not a code defect (#4) 
The sweep's AbLegacy 4F was a fixture-deployment gap, not a driver/test defect.
The suite needs ab_server in PCCC mode (--plc=SLC500|Micrologix|PLC/5), which
reuses the AbCip image (already on the host) and binds :44818. No PCCC container
was deployed, and the 4F-not-skip means :44818 was answering CIP (a leftover
AbCip container) so the PCCC reads returned Bad — same shared-port contamination
class as the Modbus false alarm.

Deployed the PCCC compose to ~/otopcua-ablegacy and ran each family with
AB_LEGACY_COMPOSE_PROFILE set to match the running container (required — without
it the [Theory] parametrises over all 3 families and 2 fail against a single-
family container): slc500 2/2, micrologix 1/1 (+1 skip), plc5 1/1 (+1 skip).
Skips are family-inapplicable file types. No code change.

Corrected the sweep record + added the AbLegacy bring-up + run recipe.
Integration-sweep follow-up #4.
2026-07-15 05:38:18 -04:00

14 KiB
Raw Blame History

Integration-test sweep — status + follow-ups (2026-07-14)

Context: ran the deferred *.IntegrationTests sweep (the round-2 STATUS.md "Full integration-suite sweep" item that discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24). How: tests run on the Mac (repo + .NET 10 SDK), pointing at driver fixtures brought up on the docker host 10.100.0.35, one driver at a time (serial — the suites have a ~16 GB/run memory leak). Master at run time: 30672888 (branch master, clean).

Docker host 10.100.0.35 (DOCKER) note: has dotnet 10.0.300, 22 cores, but only ~7 GB free RAM (the rest is shared SQL/GLAuth/fixtures) and no repo checkout — so the tests can't run on it (OOM risk to shared infra). Run on the Mac against its fixtures. Fixture dirs live at /opt/otopcua-<driver>/ (owned, need sudo to create new ones — I deployed FOCAS to ~/otopcua-focas on the host instead).

Results

Suite Result Root cause of failures
OpcUaClient ✅ 4/4 — (opc-plc :50000)
OpcUaClient.Browser ✅ 1/1 —
S7 ✅ 3/3 (2 skip) skips = outage tests needing S7_TIMEOUT_OUTAGE_* env
Modbus (profile=standard) ✅ 6/6 —
Host ⚠️ 105/117 (7F, 5 skip) 7 heavy 2-node deploy/failover E2E time out (20–45 s) under amd64-emulated SQL; 5 LDAP-backed skip
TwinCAT ⏭️ 13 skip no TC3 XAR fixture exists (expected clean-skip)
Modbus (dl205 / mitsubishi / s7_1500 / exception_injection) ✅ 16/13/10/17 (was 8F/5F/3F/8F) NOT drift — fixture-cycling artifact, re-verified GREEN 2026-07-15 — see below
AbCip ✅ 10P/3skip (was 6F/1P/3skip) static-init-order bug FIXED 2026-07-15 (326b22a7) — see below
FOCAS ❌ 9F/1P tests assume a localhost-colocated mock — see below
AbLegacy ✅ slc500 2/2, micrologix 1/1, plc5 1/1 (was 4F) fixture-deployment gap FIXED 2026-07-15 — see below
OpcUaServer ✅ 4/4 (was 4F) cert-store gap + SDK read-throws drift FIXED 2026-07-15 — see below

Headline conclusion

No OtOpcUa production regression. Every failure is pre-existing test/fixture infrastructure rot in long-deferred suites — or, in the Modbus case, a fixture-cycling false alarm (see below). The drivers are fine: 4 suites fully green, all Modbus profiles green (standard + dl205/mitsubishi/s7_1500/exception once correctly bound — the sweep's non-standard "failures" were the stale-container artifact), Host 105/117 green, and all driver unit suites + the live /run gates pass. These integration suites were DEFERRED in round 2 and never run, so they'd bit-rotted in several distinct ways.

Follow-up progress (2026-07-15): #1 AbCip (PR #446), #5 OpcUaServer (PR #447), #2 Modbus (PR #448 — false alarm, no code), and #4 AbLegacy (PR #449 — fixture-deployment gap, no code) all closed. 4 of 7 done; all driver suites now verified GREEN. Remaining: #3 FOCAS + #6 Host (each needs a design/infra decision) and #7 (FOCAS fixture deploy).

Per-suite failure detail (evidence)

  • Modbus non-standard profiles — ✅ RE-VERIFIED GREEN 2026-07-15 (PR #448). The sweep's 8F/5F/3F/8F was a false alarm — a fixture-cycling artifact, NOT pymodbus drift and NOT an OtOpcUa regression. Correcting the sweep's original diagnosis:

    • The image genuinely runs pymodbus 3.13.0 (docker run --entrypoint python ... -c 'import pymodbus; print(pymodbus.__version__)' → 3.13.0; the Dockerfile already pins pymodbus[simulator]==3.13.0). The "…removed in v4" log line is 3.13.0's forward-deprecation warning, not evidence it runs 4.x — the original sweep note misread it.
    • Root cause: each Modbus profile is a separate profile-gated compose service sharing :5020. A plain docker compose down (no --profile) does not stop the running profile container, so it keeps holding :5020; the next --profile … up container silently fails to bind (compose still says "Started"). Every later profile run then hit the stale container. Proven by a raw Modbus probe: the "mitsubishi" sim served DL205's 0xCAFE at HR0 (DL205's V0 marker) and Illegal-Data-Address for mitsubishi-only addresses, while docker ps showed the mitsubishi container up with no port mapping and DL205 still holding :5020.
    • After force-removing all modbus containers and bringing up each profile cleanly (verifying it actually published :5020), all four pass: dl205 16/16, mitsubishi 13/13, s7_1500 10/10, exception_injection 17/17 (the injector needs --force-recreate — it uses a different container name). No code change — added a profile-cycling gotcha + reliable-cycle recipe to Docker/README.md.

  • AbCip — static-init-order bug in the test harness (always-fail, blocks the whole suite). tests/.../AbCip.IntegrationTests/AbServerProfile.cs: KnownProfiles.All { get; } = [ControlLogix, …] is declared at line 30, before the static readonly profile fields (line 34+). C# runs static initializers in textual order → All captures [null,null,null,null] → ForFamily NREs at p.Family (line 61) and AbServerFixture..ctor gets null. Fix (trivial, safe): make All an expression-bodied property: public static IReadOnlyList<AbServerProfile> All => [ControlLogix, CompactLogix, Micro800, GuardLogix]; (evaluated on access, after the fields are set). Unblocks the suite; then re-run to see if the actual sim tests pass.

  • FOCAS — tests key on focas://127.0.0.1:8193/Axes/X/AbsolutePosition (KeyNotFound) — they assume a localhost-colocated mock, not a remote docker fixture at 10.100.0.35:8193. The repo fixture (tests/.../FOCAS.IntegrationTests/Docker/, publishes 8193:8193) was NOT deployed to the host; I deployed it to ~/otopcua-focas and it serves, but the tests' node-key/topology assumption (127.0.0.1) doesn't match a remote sim. Fix: reconcile the mock topology — either run the FOCAS mock on localhost, or make the test node-keys honor OTOPCUA_FOCAS_SIM_ENDPOINT/FOCAS_MOCK_HOST.

  • AbLegacy — ✅ FIXED 2026-07-15 (PR #449). Fixture-deployment gap, not a code defect. The AbLegacy suite needs ab_server in PCCC mode (--plc=SLC500|Micrologix|PLC/5), which reuses the AbCip image (otopcua-ab-server:libplctag-release, already on the host) and binds :44818. Two issues: (a) no PCCC container was deployed (~/otopcua-ablegacy didn't exist); (b) the sweep's 4F-not-skip means :44818 was reachable but answering CIP (a leftover AbCip container) — the PCCC reads returned Bad. Same shared-port contamination class as the Modbus false alarm. Deployed the compose to ~/otopcua-ablegacy on the host (image already present, no build needed) and ran each family with AB_LEGACY_COMPOSE_PROFILE set to match the running container (required — without it the [Theory] parametrises over all 3 families and 2 fail against a single-family container): slc500 2/2, micrologix 1/1 (+1 skip), plc5 1/1 (+1 skip) — the skips are family-inapplicable file types (e.g. no F8 float file seeded on micrologix/plc5). No code change.

  • OpcUaServer — ✅ FIXED 2026-07-15 (PR #447). Was 4F on Opc.Ua.ServiceResultException : TrustedIssuerCertificates StorePath must be specified — the in-test client SecurityConfiguration omitted the trusted-issuer/peer/rejected store paths that ValidateAsync requires. Fix: TestClientSecurity helper rooting Directory stores at a throwaway PKI dir (mirrors DefaultApplicationConfigurationFactory). Fixing it unmasked a second drift — the 1.5.378 node-cache read path throws ServiceResultException(BadNodeIdUnknown) for a removed node rather than returning a bad DataValue; the two subscription-survival assertions now use Should.ThrowAsync. Suite 4F → 4/4 green.

  • Host — largely healthy (105/117). 7 failures are the heavy 2-node deploy/failover/reconnect E2E tests (DeployHappyPathTests, FailoverDuringDeployTests, DriverReconnectE2eTests, FleetDiagnosticsRoundTripTests, EquipmentNamespaceMaterializationTests, RoslynVirtualTagEvaluatorTests) taking 20–45 s — likely timing under the amd64-emulated SQL on the arm64 Mac (mssql:2022 has no arm64 image). 5 LDAP-backed tests skip because bitnami/openldap:2.6 is gone from Bitnami's catalog (manifest unknown — Bitnami deprecated their free image catalog in 2025). Fix: replace the openldap image with a maintained one; run Host E2E against real (non-emulated) SQL or raise deadlines.

  • TwinCAT — 13 skip cleanly (no TC3 XAR fixture exists; unit-only per the driver's design). Expected.

Follow-up backlog (distinct, not one fix)

  1. AbCip static-init bug — trivial always-fail test bug. Expression-body KnownProfiles.All. ✅ DONE 2026-07-15 (commit 326b22a7, PR #446). Live-verified against the controllogix fixture: suite went 6F/1P/3skip → 10 pass / 0 fail / 3 skip (skips = emulator-mode tests needing a different fixture mode).
  2. Modbus non-standard profiles — pin pymodbus in the fixture Dockerfile. ✅ RESOLVED 2026-07-15 (PR #448) — false alarm, no code change. Image already runs the pinned pymodbus 3.13.0; the 8F/5F/3F/8F was a fixture-cycling artifact (stale profile container holding :5020). Re-verified all four profiles GREEN (16/13/10/17). Added a profile-cycling gotcha + reliable-cycle recipe to Docker/README.md.
  3. FOCAS — reconcile the 127.0.0.1-mock topology with the remote-fixture model.
  4. AbLegacy — deploy an AbLegacy sim fixture (none on host). ✅ DONE 2026-07-15 (PR #449) — fixture-deployment gap, no code defect. Deployed the PCCC compose to ~/otopcua-ablegacy (reuses the AbCip image, already on host). All 3 families GREEN (slc500 2/2, micrologix 1/1, plc5 1/1). Recipe added below; AB_LEGACY_COMPOSE_PROFILE MUST match the running container.
  5. OpcUaServer — supply cert-store StorePath in the harness config. ✅ DONE 2026-07-15 (PR #447). Added TestClientSecurity helper building a validated client SecurityConfiguration with Directory own/issuer/trusted/rejected stores under a throwaway PKI dir (mirrors DefaultApplicationConfigurationFactory); wired into both DualEndpointTests + SubscriptionSurvivalTests. That unmasked a second drift: the 1.5.378 node-cache read path throws ServiceResultException(BadNodeIdUnknown) for a removed node instead of returning a bad DataValue — updated the two subscription-survival assertions to Should.ThrowAsync. Suite 4F → 4/4.
  6. Host — replace bitnami/openldap:2.6 (image gone); run E2E on real SQL or raise deadlines.
  7. Fixture deployment gap — FOCAS fixture isn't on the host at /opt/otopcua-* (and /opt needs sudo — use ~/otopcua-<driver>). AbLegacy is now deployed at ~/otopcua-ablegacy (2026-07-15, PR #449).

Fixture setup recipes (to re-run the sweep)

Bring up on the docker host (distinct ports, can coexist except Modbus profiles share :5020):

ssh dohertj2@10.100.0.35 'cd /opt/otopcua-modbus     && docker compose --profile standard up -d'   # :5020
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-abcip      && docker compose --profile controllogix up -d' # :44818
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7         && docker compose --profile s7_1500 up -d'     # :1102
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-opcuaclient && docker compose up -d'                       # :50000
ssh dohertj2@10.100.0.35 'cd ~/otopcua-ablegacy      && docker compose --profile slc500 up -d'      # :44818 (PCCC; reuses AbCip image)
# FOCAS not on host — rsync tests/.../Docker/ to ~/otopcua-focas then `docker compose up -d --build`

Run per suite (env → project):

Suite Env Project
OpcUaClient(+Browser) OPCUA_SIM_ENDPOINT=opc.tcp://10.100.0.35:50000 tests/Drivers/…OpcUaClient(.Browser).IntegrationTests
S7 S7_SIM_ENDPOINT=10.100.0.35:1102 tests/Drivers/…S7.IntegrationTests
Modbus MODBUS_SIM_ENDPOINT=10.100.0.35:5020 MODBUS_SIM_PROFILE=<standard|dl205|mitsubishi|s7_1500|exception_injection> (swap the host sim to match) tests/Drivers/…Modbus.IntegrationTests
AbCip AB_SERVER_ENDPOINT=10.100.0.35:44818 AB_SERVER_PROFILE=controllogix tests/Drivers/…AbCip.IntegrationTests
AbLegacy AB_LEGACY_ENDPOINT=10.100.0.35:44818 AB_LEGACY_COMPOSE_PROFILE=<slc500|micrologix|plc5> (must match the running PCCC container — else the [Theory] runs all 3 families and 2 fail) tests/Drivers/…AbLegacy.IntegrationTests
FOCAS OTOPCUA_FOCAS_SIM_ENDPOINT=10.100.0.35:8193 (but tests assume 127.0.0.1 — see above) tests/Drivers/…FOCAS.IntegrationTests
Host OTOPCUA_HARNESS_USE_SQL=1 (+ SQL :14331 via tests/Server/…Host.IntegrationTests/docker-compose.yml up sql) OTOPCUA_HARNESS_USE_LDAP=0 tests/Server/…Host.IntegrationTests

Modbus profile cycling: the sim services share :5020, so to run a profile: ssh … 'cd /opt/otopcua-modbus && docker compose down && docker compose --profile <p> up -d', then run with MODBUS_SIM_PROFILE=<p> (each class skips unless its profile is active). Wait for :5020 before running — TCP opens before the register map fully loads.

Current state (post-sweep)

  • All fixtures torn down — modbus/abcip/s7/opcuaclient/focas on the host removed (ports 5020/44818/1102/ 50000/8193 all closed); the local Host SQL fixture down -v'd. Shared host is clean.
  • Repo: master @ 30672888, clean. No sweep changes committed (this file is the only artifact).
  • Nothing merged for the sweep — it's a read-only test run; the follow-ups above are unstarted.
Reference in New Issue View Git Blame Copy Permalink