fix(harness): make deploy E2E tests pass against real SQL (seed cluster FKs)
v2-ci / build (pull_request) Successful in 4m23s
v2-ci / unit-tests (pull_request) Failing after 10m14s

Integration-sweep follow-up #6 (SQL leg). The sweep flagged 7 heavy 2-node
deploy/failover/reconnect E2E tests as 'time out under amd64-emulated SQL' and
suggested raising deadlines. That diagnosis was WRONG: run against real
(native-amd64) SQL on the Docker host, they still hang — even at 4x deadline.
Root cause is that these tests had never actually run against real SQL, only the
EF in-memory provider, which does NOT enforce foreign keys.

Two distinct defects, both hidden by in-memory:

1. Missing FK-parent seed (5 tests). The deploy records a NodeDeploymentState row
   per node, FK'd to ClusterNode (FK_NodeDeploymentState_ClusterNode_NodeId). The
   tests never seeded ServerCluster + ClusterNode, so on SQL each node-state INSERT
   throws and the deploy never seals (waits the full deadline regardless of size).
   Added TwoNodeClusterHarness.SeedDefaultClusterAsync (seeds a ServerCluster +
   a ClusterNode per node; Warm/NodeCount=2 to satisfy the SQL CHECK constraint
   CK_ServerCluster_RedundancyMode_NodeCount AND the ClusterEnabledNodeCountMismatch
   validator). Called it in DeployHappyPath x2 / Failover / FleetDiagnostics /
   EquipmentNamespace; fixed DriverReconnect to seed node B + NodeCount=2.

2. Stale EquipmentId (EquipmentNamespace test; pre-existing on BOTH providers).
   Seeded EquipmentId='eq-1', but the later-added EquipmentIdNotDerived validator
   requires EquipmentId == DeriveEquipmentId(EquipmentUuid) ('EQ-'+first 12 hex).
   Fixed the seed to a canonical id + matching UUID. Only surfaced once the FK fix
   let the deploy reach equipment validation.

Also added OTOPCUA_HARNESS_SQL_HOST / OTOPCUA_HARNESS_LDAP_HOST overrides so the
harness fixtures can run on the native-amd64 Docker host (avoids arm64 mssql
emulation entirely).

Verified: the 5 SQL-backed classes go 11/11 green vs native SQL; in-memory default
unregressed (12/12). RoslynVirtualTagEvaluatorTests racing test is a pre-existing
flaky race (not SQL-backed; passes in isolation) — left as-is.
This commit is contained in:
Joseph Doherty
2026-07-15 06:43:15 -04:00
parent 779afbb66f
commit 859d63178a
7 changed files with 129 additions and 24 deletions
+33 -14
View File
@@ -37,12 +37,12 @@ correctly bound — the sweep's non-standard "failures" were the stale-container
and all driver **unit** suites + the live `/run` gates pass. These integration suites were DEFERRED in round 2
and never run, so they'd bit-rotted in several distinct ways.
**Follow-up progress (2026-07-15):** #1 AbCip (PR #446), #5 OpcUaServer (PR #447), #2 Modbus (PR #448 — false
alarm), #4 AbLegacy (PR #449 — fixture gap), #3 FOCAS (PR #450 — topology + mock-timer), #7 (fixture deploys),
and the #6 **LDAP-image** leg (PR #451 swapped OpenLDAP→GLAuth) all closed. **The only thing left is the #6
amd64-emulated-SQL timing leg** (7 heavy E2E tests time out under emulated `mssql:2022` on arm64 — run against
real SQL or raise deadlines; a local-emulation artifact, not a product defect). Every driver + server suite is
verified GREEN.
**Follow-up progress (2026-07-15): ALL 7 CLOSED.** #1 AbCip (PR #446), #5 OpcUaServer (PR #447), #2 Modbus
(PR #448 — false alarm), #4 AbLegacy (PR #449 — fixture gap), #3 FOCAS (PR #450 — topology + mock-timer), #7
(fixture deploys), #6 LDAP-image (PR #451 — OpenLDAP→GLAuth), and #6 SQL-deploy-E2E (PR #452 — misdiagnosed
"emulation timing"; really missing FK-parent seeds + a stale EquipmentId, hidden by the in-memory provider).
**Every driver + server integration suite is now verified GREEN. Zero OtOpcUa production regressions across the
entire sweep** — every failure was fixture/harness/test rot.
---
@@ -119,12 +119,28 @@ verified GREEN.
password → `Invalid credentials (49)`). NB: the real-LDAP mode is **opt-in** — the default Host run uses
`StubLdapAuthService`, so this never blocked the default suite (the sweep's "5 LDAP skips" were actually the
driver-fixture-gated E2E tests, not LDAP).
- **amd64-emulated-SQL timing (still open).** 7 heavy 2-node deploy/failover/reconnect E2E tests
(`DeployHappyPathTests`, `FailoverDuringDeployTests`, `DriverReconnectE2eTests`,
`FleetDiagnosticsRoundTripTests`, `EquipmentNamespaceMaterializationTests`, `RoslynVirtualTagEvaluatorTests`)
take 2045 s and time out under the amd64-emulated `mssql:2022` (no arm64 image) on the arm64 Mac. **Fix:**
run Host E2E against real (non-emulated) SQL — e.g. the shared SQL on the Linux Docker host — or raise the
per-test deadlines. Not a product defect; a local-emulation timing artifact.
- **SQL-backed deploy E2E — ✅ FIXED 2026-07-15 (PR #452). The sweep's "amd64-emulated-SQL timing" diagnosis
was WRONG.** Running the 7 heavy E2E against real (native-amd64) SQL on the Docker host proved they don't
time out for speed — they had **never actually run against real SQL**, only the EF in-memory provider (which
ignores FK constraints). Two distinct root causes, both hidden by in-memory:
- *Missing FK-parent seed (5 tests):* the deploy writes a `NodeDeploymentState` row per node, which FKs to
`ClusterNode` (`FK_NodeDeploymentState_ClusterNode_NodeId`). The tests never seeded `ServerCluster` +
`ClusterNode` rows, so on SQL each node-state INSERT throws → the deploy never seals (waits the full
deadline regardless of how high — proven at 4× it still hung). Added `TwoNodeClusterHarness.SeedDefaultClusterAsync`
(seeds a `ServerCluster` + a `ClusterNode` per node; `Warm`/NodeCount=2 to satisfy both the SQL CHECK
constraint `CK_ServerCluster_RedundancyMode_NodeCount` and the `ClusterEnabledNodeCountMismatch` validator)
and called it in `DeployHappyPathTests`×2 / `FailoverDuringDeployTests` / `FleetDiagnosticsRoundTripTests` /
`EquipmentNamespaceMaterializationTests`; fixed `DriverReconnectE2eTests` to seed node B + declare NodeCount=2.
- *Stale `EquipmentId` (1 test, pre-existing on BOTH providers):* `EquipmentNamespaceMaterializationTests`
seeded `EquipmentId="eq-1"`, but the `EquipmentIdNotDerived` validator (added later) requires
`EquipmentId == DeriveEquipmentId(EquipmentUuid)` = `EQ-` + first 12 hex of the UUID. Fixed the seed to a
canonical id + matching UUID. (Was failing on in-memory too; only surfaced once the FK fix let the deploy
reach equipment validation.)
- Added `OTOPCUA_HARNESS_SQL_HOST` / `OTOPCUA_HARNESS_LDAP_HOST` env overrides so the fixtures can live on the
native-amd64 Docker host (`docker compose up -d sql` there; run with `OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331`).
- Result: the 5 SQL-backed classes go **11/11 green vs native SQL**, and the in-memory default is unregressed
(12/12). `RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts…` is a **pre-existing flaky
race test** (not SQL-backed; passes in isolation, flaked once under heavy parallel load) — left as-is.
- **TwinCAT** — 13 skip cleanly (no TC3 XAR fixture exists; unit-only per the driver's design). Expected.
@@ -159,8 +175,11 @@ verified GREEN.
6. **Host** — ⬐ split into two:
- ~~replace `bitnami/openldap:2.6` (image gone)~~ ✅ **DONE 2026-07-15 (PR #451)** — swapped the harness LDAP
to **GLAuth** (unifies on the LDAP the rest of the project uses); live-verified the binds on `:3894`.
- **[OPEN] amd64-emulated-SQL timing** — the 7 heavy E2E tests time out under emulated `mssql:2022` on arm64.
Run them against real (non-emulated) SQL or raise deadlines. Local-emulation artifact, not a product defect.
- ~~**amd64-emulated-SQL timing**~~ ✅ **DONE 2026-07-15 (PR #452) — misdiagnosed.** Not a timing/emulation
issue: the deploy E2E tests had never run against real SQL (in-memory ignores FKs) and lacked the
`ServerCluster`/`ClusterNode` seed the `NodeDeploymentState` FK requires → deploy never seals. Added a
harness seeding helper + `OTOPCUA_HARNESS_SQL_HOST` override; also fixed a stale `EquipmentId` in one test.
11/11 green vs native SQL, in-memory unregressed. (Roslyn racing test = pre-existing flaky, left as-is.)
7. ~~**Fixture deployment gap** — FOCAS + AbLegacy fixtures aren't on the host.~~
✅ **RESOLVED 2026-07-15.** Both now deployed under the home dir: `~/otopcua-ablegacy` (PR #449) +
`~/otopcua-focas` (PR #450). (`/opt` still needs sudo; the home-dir deploys work fine.)