Integration-sweep follow-up #6 (SQL leg). The sweep flagged 7 heavy 2-node
deploy/failover/reconnect E2E tests as 'time out under amd64-emulated SQL' and
suggested raising deadlines. That diagnosis was WRONG: run against real
(native-amd64) SQL on the Docker host, they still hang — even at 4x deadline.
Root cause is that these tests had never actually run against real SQL, only the
EF in-memory provider, which does NOT enforce foreign keys.
Two distinct defects, both hidden by in-memory:
1. Missing FK-parent seed (5 tests). The deploy records a NodeDeploymentState row
per node, FK'd to ClusterNode (FK_NodeDeploymentState_ClusterNode_NodeId). The
tests never seeded ServerCluster + ClusterNode, so on SQL each node-state INSERT
throws and the deploy never seals (waits the full deadline regardless of size).
Added TwoNodeClusterHarness.SeedDefaultClusterAsync (seeds a ServerCluster +
a ClusterNode per node; Warm/NodeCount=2 to satisfy the SQL CHECK constraint
CK_ServerCluster_RedundancyMode_NodeCount AND the ClusterEnabledNodeCountMismatch
validator). Called it in DeployHappyPath x2 / Failover / FleetDiagnostics /
EquipmentNamespace; fixed DriverReconnect to seed node B + NodeCount=2.
2. Stale EquipmentId (EquipmentNamespace test; pre-existing on BOTH providers).
Seeded EquipmentId='eq-1', but the later-added EquipmentIdNotDerived validator
requires EquipmentId == DeriveEquipmentId(EquipmentUuid) ('EQ-'+first 12 hex).
Fixed the seed to a canonical id + matching UUID. Only surfaced once the FK fix
let the deploy reach equipment validation.
Also added OTOPCUA_HARNESS_SQL_HOST / OTOPCUA_HARNESS_LDAP_HOST overrides so the
harness fixtures can run on the native-amd64 Docker host (avoids arm64 mssql
emulation entirely).
Verified: the 5 SQL-backed classes go 11/11 green vs native SQL; in-memory default
unregressed (12/12). RoslynVirtualTagEvaluatorTests racing test is a pre-existing
flaky race (not SQL-backed; passes in isolation) — left as-is.
The Host.IntegrationTests opt-in real-LDAP mode (OTOPCUA_HARNESS_USE_LDAP=1)
used bitnami/openldap:2.6, gone since Bitnami deprecated their free image
catalog in 2025. Replaced it with GLAuth — the same LDAP server the rest of
the project already uses (docker-dev + the live OPC UA data-plane auth, both
against the shared GLAuth on :3893) — removing the lone OpenLDAP outlier and
the gone-image breakage.
- Added tests/.../Host.IntegrationTests/glauth/config.toml: baseDN dc=zb,dc=local,
a search-capable serviceaccount, dev users alice (full access) / bob (read-only),
and role groups with the same gidnumbers as scadaproj/infra/glauth so GroupToRole
maps identically.
- Compose ldap service -> glauth/glauth:latest, mounting the config, host :3894 -> :3893.
- Repointed the harness real-LDAP override from the OpenLDAP cn=admin/ldapadmin to
GLAuth's cn=serviceaccount/serviceaccount123.
Live-verified against a deployed container on :3894: the serviceaccount bind
searches and returns alice + her 5 memberOf groups; alice/bob bind with the
correct password; a wrong password yields Invalid credentials (49) — exactly the
OtOpcUaLdapAuthService flow (proven identical to the data-plane's shared-GLAuth path).
NB: real-LDAP mode is opt-in; the default Host run uses StubLdapAuthService, so
this never blocked the default suite. Integration-sweep follow-up #6 (LDAP leg);
the amd64-emulated-SQL deadline leg remains open.
Two test/fixture bugs; NO OtOpcUa driver change.
1. Topology: FocasSimFixture reads an env-overridable endpoint
(OTOPCUA_FOCAS_SIM_ENDPOINT, default localhost:8193) and exposes Host/Port,
but WireBackendTests + WireBackendCoverageTests hardcoded
focas://127.0.0.1:8193. Against a remote fixture the skip-gate passed
(remote reachable) but the driver dialed localhost -> KeyNotFound. Added a
DeviceUri property to the fixture; each test now derives DeviceHost from it.
Recovered 8 of 9.
2. Mock timer endianness (surfaced once #1 let the tests reach the mock): the
last failure read 60397977600 for a 3600 s power-on timer (= 0x3C000000 x 60).
The focas-mock's _wire_timer encoded the minute/msec fields big-endian (_u32),
but cnc_rdtimer's timer fields are LITTLE-endian on real hardware — a
documented, live-31i-B-validated quirk that FocasWireClient.ParseTimer decodes
LE (docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md). The driver was
correct; the mock was wrong. _wire_timer now emits little-endian (_u32_le).
Suite 9F/1P -> 10/10 against the remote fixture at ~/otopcua-focas.
Integration-sweep follow-up #3.
The sweep's AbLegacy 4F was a fixture-deployment gap, not a driver/test defect.
The suite needs ab_server in PCCC mode (--plc=SLC500|Micrologix|PLC/5), which
reuses the AbCip image (already on the host) and binds :44818. No PCCC container
was deployed, and the 4F-not-skip means :44818 was answering CIP (a leftover
AbCip container) so the PCCC reads returned Bad — same shared-port contamination
class as the Modbus false alarm.
Deployed the PCCC compose to ~/otopcua-ablegacy and ran each family with
AB_LEGACY_COMPOSE_PROFILE set to match the running container (required — without
it the [Theory] parametrises over all 3 families and 2 fail against a single-
family container): slc500 2/2, micrologix 1/1 (+1 skip), plc5 1/1 (+1 skip).
Skips are family-inapplicable file types. No code change.
Corrected the sweep record + added the AbLegacy bring-up + run recipe.
Integration-sweep follow-up #4.
The 2026-07 integration sweep flagged the Modbus non-standard profiles as
8F/5F/3F/8F and guessed 'pymodbus 4.x drift'. Re-investigation shows that was
wrong on both counts:
- The image runs the PINNED pymodbus 3.13.0 (verified via import pymodbus;
__version__). The '...removed in v4' log is 3.13.0's forward-deprecation
warning, misread as running 4.x.
- The failures were a fixture-cycling artifact: each profile is a separate
profile-gated compose service sharing :5020, and a plain 'docker compose down'
(no --profile) does NOT stop the running profile container. It kept holding
:5020; the next '--profile up' container silently failed to bind (compose
still says 'Started'), so every later profile run hit the STALE sim. Proven by
a raw Modbus probe: the 'mitsubishi' sim served DL205's 0xCAFE at HR0.
After force-removing all modbus containers and bringing up each profile cleanly
(verifying it actually published :5020), all four pass: dl205 16/16,
mitsubishi 13/13, s7_1500 10/10, exception_injection 17/17. No OtOpcUa or test
code change needed.
Added a profile-cycling gotcha + reliable-cycle recipe to Docker/README.md and
corrected the sweep record. Integration-sweep follow-up #2.
OpcUaServer.IntegrationTests was 4F. Two distinct, previously-masked issues:
1. Cert-store gap: the in-test client SecurityConfiguration used a bare
new SecurityConfiguration()/CertificateIdentifier(), so ValidateAsync threw
'TrustedIssuerCertificates StorePath must be specified' before any connect.
Added TestClientSecurity helper building Directory own/issuer/trusted/rejected
stores under a throwaway temp PKI dir (mirrors DefaultApplicationConfigurationFactory);
wired into DualEndpointTests + SubscriptionSurvivalTests.
2. Fixing #1 unmasked an SDK-version drift: the 1.5.378 node-cache read path
throws ServiceResultException(BadNodeIdUnknown) for a removed node instead of
returning a bad DataValue. The two subscription-survival assertions now expect
the throw via Should.ThrowAsync. Behavior under test (removed node -> unknown)
is unchanged; only the delivery mechanism differs.
Suite 4F -> 4/4 green. Integration-sweep follow-up #5.
Joseph Doherty