lmxopcua/archreview/plans/INTEGRATION-SWEEP-STATUS.md

# Integration-test sweep — status + follow-ups (2026-07-14)

**Context:** ran the deferred `*.IntegrationTests` sweep (the round-2 STATUS.md "Full integration-suite
sweep" item that discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24).
**How:** tests run **on the Mac** (repo + .NET 10 SDK), pointing at **driver fixtures brought up on the
docker host `10.100.0.35`**, one driver at a time (serial — the suites have a ~16 GB/run memory leak).
Master at run time: `30672888` (branch `master`, clean).

**Docker host `10.100.0.35` (`DOCKER`) note:** has `dotnet 10.0.300`, 22 cores, but only ~7 GB free RAM
(the rest is shared SQL/GLAuth/fixtures) and **no repo checkout** — so the tests can't run *on* it (OOM risk
to shared infra). Run on the Mac against its fixtures. Fixture dirs live at `/opt/otopcua-<driver>/` (owned,
need sudo to create new ones — I deployed FOCAS to `~/otopcua-focas` on the host instead).

---

## Results

| Suite | Result | Root cause of failures |
|---|---|---|
| **OpcUaClient** | ✅ 4/4 | — (opc-plc `:50000`) |
| **OpcUaClient.Browser** | ✅ 1/1 | — |
| **S7** | ✅ 3/3 (2 skip) | skips = outage tests needing `S7_TIMEOUT_OUTAGE_*` env |
| **Modbus** (profile=standard) | ✅ 6/6 | — |
| **Host** | ⚠️ **105/117** (7F, 5 skip) | 7 heavy E2E time out under amd64-emulated SQL (OPEN); LDAP image swapped OpenLDAP→GLAuth 2026-07-15 (PR #451). The 5 skips are driver-fixture-gated E2E, not LDAP |
| **TwinCAT** | ⏭️ 13 skip | no TC3 XAR fixture exists (expected clean-skip) |
| **Modbus** (dl205 / mitsubishi / s7_1500 / exception_injection) | ✅ 16/13/10/17 (was 8F/5F/3F/8F) | **NOT drift** — fixture-cycling artifact, re-verified GREEN 2026-07-15 — see below |
| **AbCip** | ✅ 10P/3skip (was 6F/1P/3skip) | static-init-order bug FIXED 2026-07-15 (`326b22a7`) — see below |
| **FOCAS** | ✅ 10/10 (was 9F/1P) | topology + mock-timer-endianness FIXED 2026-07-15 — see below |
| **AbLegacy** | ✅ slc500 2/2, micrologix 1/1, plc5 1/1 (was 4F) | fixture-deployment gap FIXED 2026-07-15 — see below |
| **OpcUaServer** | ✅ 4/4 (was 4F) | cert-store gap + SDK read-throws drift FIXED 2026-07-15 — see below |

### Headline conclusion
**No OtOpcUa production regression.** Every failure is **pre-existing test/fixture infrastructure rot** in
long-deferred suites — or, in the Modbus case, a **fixture-cycling false alarm** (see below). The drivers are
fine: 4 suites fully green, **all Modbus profiles green** (standard + dl205/mitsubishi/s7_1500/exception once
correctly bound — the sweep's non-standard "failures" were the stale-container artifact), Host 105/117 green,
and all driver **unit** suites + the live `/run` gates pass. These integration suites were DEFERRED in round 2
and never run, so they'd bit-rotted in several distinct ways.

**Follow-up progress (2026-07-15): ALL 7 CLOSED.** #1 AbCip (PR #446), #5 OpcUaServer (PR #447), #2 Modbus
(PR #448 — false alarm), #4 AbLegacy (PR #449 — fixture gap), #3 FOCAS (PR #450 — topology + mock-timer), #7
(fixture deploys), #6 LDAP-image (PR #451 — OpenLDAP→GLAuth), and #6 SQL-deploy-E2E (PR #452 — misdiagnosed
"emulation timing"; really missing FK-parent seeds + a stale EquipmentId, hidden by the in-memory provider).
**Every driver + server integration suite is now verified GREEN. Zero OtOpcUa production regressions across the
entire sweep** — every failure was fixture/harness/test rot.

---

## Per-suite failure detail (evidence)

- **Modbus non-standard profiles** — ✅ **RE-VERIFIED GREEN 2026-07-15 (PR #448). The sweep's 8F/5F/3F/8F was a
  false alarm — a fixture-cycling artifact, NOT pymodbus drift and NOT an OtOpcUa regression.** Correcting the
  sweep's original diagnosis:
  - The image genuinely runs **pymodbus 3.13.0** (`docker run --entrypoint python ... -c 'import pymodbus;
    print(pymodbus.__version__)'` → `3.13.0`; the Dockerfile already pins `pymodbus[simulator]==3.13.0`). The
    *"…removed in v4"* log line is 3.13.0's **forward-deprecation warning**, not evidence it runs 4.x — the
    original sweep note misread it.
  - Root cause: each Modbus profile is a **separate profile-gated compose service** sharing `:5020`. A plain
    `docker compose down` (no `--profile`) does **not** stop the running profile container, so it keeps holding
    `:5020`; the next `--profile … up` container silently fails to bind (compose still says "Started"). Every
    later profile run then hit the **stale** container. Proven by a raw Modbus probe: the "mitsubishi" sim
    served DL205's `0xCAFE` at HR0 (DL205's V0 marker) and Illegal-Data-Address for mitsubishi-only addresses,
    while `docker ps` showed the mitsubishi container up with **no** port mapping and DL205 still holding `:5020`.
  - After force-removing all modbus containers and bringing up each profile cleanly (verifying it actually
    published `:5020`), all four pass: **dl205 16/16, mitsubishi 13/13, s7_1500 10/10, exception_injection
    17/17** (the injector needs `--force-recreate` — it uses a different container name). **No code change** —
    added a profile-cycling gotcha + reliable-cycle recipe to `Docker/README.md`.

- **AbCip** — **static-init-order bug in the test harness** (always-fail, blocks the whole suite).
  `tests/.../AbCip.IntegrationTests/AbServerProfile.cs`: `KnownProfiles.All { get; } = [ControlLogix, …]` is
  declared at **line 30**, *before* the `static readonly` profile fields (line 34+). C# runs static
  initializers in textual order → `All` captures `[null,null,null,null]` → `ForFamily` NREs at `p.Family`
  (line 61) and `AbServerFixture..ctor` gets null. **Fix (trivial, safe):** make `All` an expression-bodied
  property: `public static IReadOnlyList<AbServerProfile> All => [ControlLogix, CompactLogix, Micro800,
  GuardLogix];` (evaluated on access, after the fields are set). Unblocks the suite; then re-run to see if the
  actual sim tests pass.

- **FOCAS** — ✅ **FIXED 2026-07-15 (PR #450). Two test/fixture bugs, no OtOpcUa driver change.**
  1. *Topology:* `FocasSimFixture` already reads an env-overridable endpoint (`OTOPCUA_FOCAS_SIM_ENDPOINT`,
     default `localhost:8193`) and exposes `Host`/`Port`, but `WireBackendTests` + `WireBackendCoverageTests`
     **hardcoded** `focas://127.0.0.1:8193`. Against a remote fixture the skip-gate passed (remote reachable) but
     the driver dialed localhost → KeyNotFound. Fix: added a `DeviceUri` property to the fixture and derived each
     test's `DeviceHost` from it — recovered **8 of 9**.
  2. *Mock timer endianness (newly surfaced):* the last failure (`Timers_populate`) read `60397977600` for a
     `3600 s` power-on timer (= `0x3C000000 × 60`). Root cause: the focas-mock's `_wire_timer` encoded the
     `minute`/`msec` fields **big-endian** (`_u32`), but `cnc_rdtimer`'s timer fields are **little-endian** on
     real hardware — a documented, live-31i-B-validated quirk (`FocasWireClient.ParseTimer` decodes them LE, per
     `docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md`). **The driver was correct; the mock was wrong.** Fix:
     `_wire_timer` now emits little-endian (`_u32_le`) → suite **10/10**. Fixture deployed at `~/otopcua-focas`.

- **AbLegacy** — ✅ **FIXED 2026-07-15 (PR #449). Fixture-deployment gap, not a code defect.** The AbLegacy suite
  needs `ab_server` in **PCCC mode** (`--plc=SLC500|Micrologix|PLC/5`), which reuses the **AbCip image**
  (`otopcua-ab-server:libplctag-release`, already on the host) and binds `:44818`. Two issues: (a) no PCCC
  container was deployed (`~/otopcua-ablegacy` didn't exist); (b) the sweep's *4F-not-skip* means `:44818` was
  reachable but answering **CIP** (a leftover AbCip container) — the PCCC reads returned Bad. Same shared-port
  contamination class as the Modbus false alarm. Deployed the compose to `~/otopcua-ablegacy` on the host (image
  already present, no build needed) and ran each family with `AB_LEGACY_COMPOSE_PROFILE` set to match the running
  container (**required** — without it the `[Theory]` parametrises over all 3 families and 2 fail against a
  single-family container): **slc500 2/2, micrologix 1/1 (+1 skip), plc5 1/1 (+1 skip)** — the skips are
  family-inapplicable file types (e.g. no `F8` float file seeded on micrologix/plc5). **No code change.**

- **OpcUaServer** — ✅ **FIXED 2026-07-15** (PR #447). Was 4F on `Opc.Ua.ServiceResultException :
  TrustedIssuerCertificates StorePath must be specified` — the in-test *client* `SecurityConfiguration`
  omitted the trusted-issuer/peer/rejected store paths that `ValidateAsync` requires. Fix: `TestClientSecurity`
  helper rooting Directory stores at a throwaway PKI dir (mirrors `DefaultApplicationConfigurationFactory`).
  Fixing it unmasked a second drift — the 1.5.378 node-cache read path throws
  `ServiceResultException(BadNodeIdUnknown)` for a removed node rather than returning a bad `DataValue`;
  the two subscription-survival assertions now use `Should.ThrowAsync`. Suite **4F → 4/4 green**.

- **Host** — largely healthy (105/117). Two independent issues:
  - **LDAP image — ✅ FIXED 2026-07-15 (PR #451).** The opt-in real-LDAP mode (`OTOPCUA_HARNESS_USE_LDAP=1`)
    used `bitnami/openldap:2.6`, gone since Bitnami deprecated their free catalog in 2025. **Replaced it with
    GLAuth** — the same LDAP server the rest of the project uses (docker-dev + the live OPC UA data-plane auth,
    both against the shared GLAuth on `:3893`), removing the lone OpenLDAP outlier. Added a harness
    `glauth/config.toml` (base DN `dc=zb,dc=local`, search-capable `serviceaccount`, dev users alice=full /
    bob=read-only, role groups with the same gidnumbers as `scadaproj/infra/glauth`) + repointed the harness's
    service-account override to `cn=serviceaccount`. Live-verified all three bind legs against a deployed
    container on `:3894` (serviceaccount search returns alice + 5 memberOf groups; alice/bob bind OK; wrong
    password → `Invalid credentials (49)`). NB: the real-LDAP mode is **opt-in** — the default Host run uses
    `StubLdapAuthService`, so this never blocked the default suite (the sweep's "5 LDAP skips" were actually the
    driver-fixture-gated E2E tests, not LDAP).
  - **SQL-backed deploy E2E — ✅ FIXED 2026-07-15 (PR #452). The sweep's "amd64-emulated-SQL timing" diagnosis
    was WRONG.** Running the 7 heavy E2E against real (native-amd64) SQL on the Docker host proved they don't
    time out for speed — they had **never actually run against real SQL**, only the EF in-memory provider (which
    ignores FK constraints). Two distinct root causes, both hidden by in-memory:
    - *Missing FK-parent seed (5 tests):* the deploy writes a `NodeDeploymentState` row per node, which FKs to
      `ClusterNode` (`FK_NodeDeploymentState_ClusterNode_NodeId`). The tests never seeded `ServerCluster` +
      `ClusterNode` rows, so on SQL each node-state INSERT throws → the deploy never seals (waits the full
      deadline regardless of how high — proven at 4× it still hung). Added `TwoNodeClusterHarness.SeedDefaultClusterAsync`
      (seeds a `ServerCluster` + a `ClusterNode` per node; `Warm`/NodeCount=2 to satisfy both the SQL CHECK
      constraint `CK_ServerCluster_RedundancyMode_NodeCount` and the `ClusterEnabledNodeCountMismatch` validator)
      and called it in `DeployHappyPathTests`×2 / `FailoverDuringDeployTests` / `FleetDiagnosticsRoundTripTests` /
      `EquipmentNamespaceMaterializationTests`; fixed `DriverReconnectE2eTests` to seed node B + declare NodeCount=2.
    - *Stale `EquipmentId` (1 test, pre-existing on BOTH providers):* `EquipmentNamespaceMaterializationTests`
      seeded `EquipmentId="eq-1"`, but the `EquipmentIdNotDerived` validator (added later) requires
      `EquipmentId == DeriveEquipmentId(EquipmentUuid)` = `EQ-` + first 12 hex of the UUID. Fixed the seed to a
      canonical id + matching UUID. (Was failing on in-memory too; only surfaced once the FK fix let the deploy
      reach equipment validation.)
    - Added `OTOPCUA_HARNESS_SQL_HOST` / `OTOPCUA_HARNESS_LDAP_HOST` env overrides so the fixtures can live on the
      native-amd64 Docker host (`docker compose up -d sql` there; run with `OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331`).
    - Result: the 5 SQL-backed classes go **11/11 green vs native SQL**, and the in-memory default is unregressed
      (12/12). `RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts…` is a **pre-existing flaky
      race test** (not SQL-backed; passes in isolation, flaked once under heavy parallel load) — left as-is.

- **TwinCAT** — 13 skip cleanly (no TC3 XAR fixture exists; unit-only per the driver's design). Expected.

---

## Follow-up backlog (distinct, not one fix)

1. ~~**AbCip static-init bug** — trivial always-fail test bug. Expression-body `KnownProfiles.All`.~~
   ✅ **DONE 2026-07-15** (commit `326b22a7`, PR #446). Live-verified against the controllogix fixture:
   suite went **6F/1P/3skip → 10 pass / 0 fail / 3 skip** (skips = emulator-mode tests needing a different
   fixture mode).
2. ~~**Modbus non-standard profiles** — pin pymodbus in the fixture Dockerfile.~~
   ✅ **RESOLVED 2026-07-15 (PR #448) — false alarm, no code change.** Image already runs the pinned
   pymodbus 3.13.0; the 8F/5F/3F/8F was a fixture-cycling artifact (stale profile container holding `:5020`).
   Re-verified all four profiles GREEN (16/13/10/17). Added a profile-cycling gotcha + reliable-cycle recipe
   to `Docker/README.md`.
3. ~~**FOCAS** — reconcile the 127.0.0.1-mock topology with the remote-fixture model.~~
   ✅ **DONE 2026-07-15 (PR #450).** Tests now derive `DeviceHost` from the fixture endpoint (recovered 8/9);
   also fixed a newly-surfaced mock bug — `_wire_timer` encoded the timer big-endian but real hardware +
   driver use little-endian. Suite **10/10**. No OtOpcUa driver change.
4. ~~**AbLegacy** — deploy an AbLegacy sim fixture (none on host).~~
   ✅ **DONE 2026-07-15 (PR #449) — fixture-deployment gap, no code defect.** Deployed the PCCC compose to
   `~/otopcua-ablegacy` (reuses the AbCip image, already on host). All 3 families GREEN (slc500 2/2, micrologix
   1/1, plc5 1/1). Recipe added below; `AB_LEGACY_COMPOSE_PROFILE` MUST match the running container.
5. ~~**OpcUaServer** — supply cert-store StorePath in the harness config.~~
   ✅ **DONE 2026-07-15** (PR #447). Added `TestClientSecurity` helper building a validated client
   `SecurityConfiguration` with Directory own/issuer/trusted/rejected stores under a throwaway PKI dir
   (mirrors `DefaultApplicationConfigurationFactory`); wired into both `DualEndpointTests` +
   `SubscriptionSurvivalTests`. That unmasked a **second** drift: the 1.5.378 node-cache read path
   *throws* `ServiceResultException(BadNodeIdUnknown)` for a removed node instead of returning a bad
   `DataValue` — updated the two subscription-survival assertions to `Should.ThrowAsync`. Suite **4F → 4/4**.
6. **Host** — ⬐ split into two:
   - ~~replace `bitnami/openldap:2.6` (image gone)~~ ✅ **DONE 2026-07-15 (PR #451)** — swapped the harness LDAP
     to **GLAuth** (unifies on the LDAP the rest of the project uses); live-verified the binds on `:3894`.
   - ~~**amd64-emulated-SQL timing**~~ ✅ **DONE 2026-07-15 (PR #452) — misdiagnosed.** Not a timing/emulation
     issue: the deploy E2E tests had never run against real SQL (in-memory ignores FKs) and lacked the
     `ServerCluster`/`ClusterNode` seed the `NodeDeploymentState` FK requires → deploy never seals. Added a
     harness seeding helper + `OTOPCUA_HARNESS_SQL_HOST` override; also fixed a stale `EquipmentId` in one test.
     11/11 green vs native SQL, in-memory unregressed. (Roslyn racing test = pre-existing flaky, left as-is.)
7. ~~**Fixture deployment gap** — FOCAS + AbLegacy fixtures aren't on the host.~~
   ✅ **RESOLVED 2026-07-15.** Both now deployed under the home dir: `~/otopcua-ablegacy` (PR #449) +
   `~/otopcua-focas` (PR #450). (`/opt` still needs sudo; the home-dir deploys work fine.)

---

## Fixture setup recipes (to re-run the sweep)

Bring up on the docker host (distinct ports, can coexist except Modbus profiles share `:5020`):

```bash
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-modbus     && docker compose --profile standard up -d'   # :5020
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-abcip      && docker compose --profile controllogix up -d' # :44818
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7         && docker compose --profile s7_1500 up -d'     # :1102
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-opcuaclient && docker compose up -d'                       # :50000
ssh dohertj2@10.100.0.35 'cd ~/otopcua-ablegacy      && docker compose --profile slc500 up -d'      # :44818 (PCCC; reuses AbCip image)
ssh dohertj2@10.100.0.35 'cd ~/otopcua-focas       && docker compose up -d --build'                 # :8193
```

Run per suite (env → project):

| Suite | Env | Project |
|---|---|---|
| OpcUaClient(+Browser) | `OPCUA_SIM_ENDPOINT=opc.tcp://10.100.0.35:50000` | `tests/Drivers/…OpcUaClient(.Browser).IntegrationTests` |
| S7 | `S7_SIM_ENDPOINT=10.100.0.35:1102` | `tests/Drivers/…S7.IntegrationTests` |
| Modbus | `MODBUS_SIM_ENDPOINT=10.100.0.35:5020` `MODBUS_SIM_PROFILE=<standard\|dl205\|mitsubishi\|s7_1500\|exception_injection>` (swap the host sim to match) | `tests/Drivers/…Modbus.IntegrationTests` |
| AbCip | `AB_SERVER_ENDPOINT=10.100.0.35:44818` `AB_SERVER_PROFILE=controllogix` | `tests/Drivers/…AbCip.IntegrationTests` |
| AbLegacy | `AB_LEGACY_ENDPOINT=10.100.0.35:44818` `AB_LEGACY_COMPOSE_PROFILE=<slc500\|micrologix\|plc5>` (**must** match the running PCCC container — else the `[Theory]` runs all 3 families and 2 fail) | `tests/Drivers/…AbLegacy.IntegrationTests` |
| FOCAS | `OTOPCUA_FOCAS_SIM_ENDPOINT=10.100.0.35:8193` (tests now honour this — no longer hardcoded to 127.0.0.1) | `tests/Drivers/…FOCAS.IntegrationTests` |
| Host | `OTOPCUA_HARNESS_USE_SQL=1` (+ SQL `:14331` via the harness `docker-compose.yml up sql`) — set `OTOPCUA_HARNESS_USE_LDAP=1` to exercise real binds against the **GLAuth** service (`up ldap`, host `:3894`; users alice/bob, svc `cn=serviceaccount`/`serviceaccount123`) | `tests/Server/…Host.IntegrationTests` |

**Modbus profile cycling:** the sim services share `:5020`, so to run a profile:
`ssh … 'cd /opt/otopcua-modbus && docker compose down && docker compose --profile <p> up -d'`, then run with
`MODBUS_SIM_PROFILE=<p>` (each class skips unless its profile is active). Wait for `:5020` before running —
TCP opens before the register map fully loads.

---

## Current state (post-sweep)

- **All fixtures torn down** — modbus/abcip/s7/opcuaclient/focas on the host removed (ports 5020/44818/1102/
  50000/8193 all closed); the local Host SQL fixture `down -v`'d. Shared host is clean.
- **Repo:** `master @ 30672888`, clean. No sweep changes committed (this file is the only artifact).
- **Nothing merged for the sweep** — it's a read-only test run. ~~The follow-ups above are unstarted.~~
  **Update 2026-08-07 (truth sweep):** stale on arrival — all 7 follow-ups shipped 2026-07-15 as PRs
  **#446–#452** (see the "Follow-up progress: ALL 7 CLOSED" note at the top of this file, which this
  line contradicted).