Closes the R2-08 S2 live leg. The offline LdapAuthResilienceTests prove the
directory-outage circuit with fakes + an unroutable-host blackhole; this drives
the SAME circuit through the production auth path (OtOpcUaLdapAuthService +
LdapOpcUaUserAuthenticator) against a REAL GLAuth that is paused mid-run and
unpaused — and doubles as the first end-to-end verification that the PR #451
GLAuth swap (retired bitnami/openldap) binds correctly.
Triple-gated (GLAUTH_LIVE_HOST + GLAUTH_OUTAGE_START_CMD/STOP_CMD), skips clean
offline (verified: 1 skipped, 8 ms). try/finally always unpauses GLAuth.
Verified GREEN against 10.100.0.35:3894 (12 s): healthy binds (alice+bob, mapped
roles) -> docker pause -> 3 bounded failures -> circuit opens -> sub-second
"unavailable" fast-deny -> docker unpause -> half-open probe closes -> binds
succeed again. A boundary AuthTimeout feeds the circuit (authenticator :123-130),
so the docker-pause timeout shape is a faithful outage.
Follow-ups from the fleet-wide read-timeout audit that the S7 R2-01 read-leg
fix (PR #453) prompted. The audit confirmed S7 was the ONLY driver with the
async-read-ignores-socket-timeout hang; these are the two adjacent (non-hang)
findings it surfaced.
1. FOCAS TimeoutMs:0 footgun (the risky one): a non-positive Timeout made
SynchronizedFocasClient DISABLE its per-call wall-clock ceiling, reverting to
the caller's long-lived poll token — reintroducing exactly the frozen-peer
wedge S7 just eliminated, under misconfig. Clamp non-positive TimeoutMs to the
2s default at the config boundary so the deadline can never be authored away.
2. TimeoutMs validation symmetry: apply the same clamp in the AbCip + AbLegacy
factories. libplctag's Tag.Timeout setter throws on <=0, faulting tag creation
on every read/write; clamping keeps a misconfigured TimeoutMs:0 running on the
default bound instead. (Shared PositiveTimeoutOrDefault helper per factory.)
3. AbLegacy reconnect parity with AbCip: AbLegacy evicted the cached libplctag
runtime on neither the non-zero-status nor transport-exception read/write path
(AbCip evicts on both), so a data-path fault recovered only via the probe loop
/ libplctag internals. Added EvictRuntime + wired it into both read and write
failure paths so a fresh handle is created on the next call.
Tests: FOCAS 265->269 (clamp theory + positive), AbCip 336->339 (clamp theory +
positive), AbLegacy 209->212 (read-nonzero / read-exception / write-nonzero evict).
No production regressions; all three driver suites green.
The R2-01 live gate (S7_1500ConnectTimeoutOutageTests, docker-pause blackhole)
surfaced a real gap the offline fakes couldn't: S7.Net's ReadTimeout/WriteTimeout
map to the TcpClient's ReceiveTimeout/SendTimeout, which govern only SYNCHRONOUS
socket calls — the async read/write paths S7.Net uses ignore them. On an
established-but-frozen peer (frozen PLC / firewall DROP / cable pulled mid-flow,
TCP session still open) a read blocked until the OS TCP stack gave up (minutes),
silently wedging the poll loop: no Bad tick, no reconnect. STAB-14 fixed only the
CONNECT leg (EnsureConnectedAsync CancelAfter); this is its READ-leg sibling.
- New S7OperationDeadline: bounds every data-plane wire op with a wall-clock
ceiling (= _options.Timeout), surfacing an overrun as TimeoutException. Applied
in S7PlcAdapter to Read/ReadBytes/Write/WriteBytes/ReadStatus. OpenAsync is left
to EnsureConnectedAsync's own CancelAfter (not double-bounded).
- IsS7ConnectionFatal now classifies TimeoutException fatal → handle marked dead →
next EnsureConnectedAsync reopens (connect-timeout fix takes over from there).
- Tests: 5 S7OperationDeadline unit tests (deadline / token-honouring / caller-
cancel-passthrough / resultless), 1 driver-reaction test (read TimeoutException
→ reopen). Driver.S7.Tests 260/260.
- Live gate S7_1500ConnectTimeoutOutageTests now GREEN against the real snap7 sim
(8s: baseline Good -> pause blackhole -> Bad tick -> unpause -> recovered Good).
Integration-sweep follow-up #6 (SQL leg). The sweep flagged 7 heavy 2-node
deploy/failover/reconnect E2E tests as 'time out under amd64-emulated SQL' and
suggested raising deadlines. That diagnosis was WRONG: run against real
(native-amd64) SQL on the Docker host, they still hang — even at 4x deadline.
Root cause is that these tests had never actually run against real SQL, only the
EF in-memory provider, which does NOT enforce foreign keys.
Two distinct defects, both hidden by in-memory:
1. Missing FK-parent seed (5 tests). The deploy records a NodeDeploymentState row
per node, FK'd to ClusterNode (FK_NodeDeploymentState_ClusterNode_NodeId). The
tests never seeded ServerCluster + ClusterNode, so on SQL each node-state INSERT
throws and the deploy never seals (waits the full deadline regardless of size).
Added TwoNodeClusterHarness.SeedDefaultClusterAsync (seeds a ServerCluster +
a ClusterNode per node; Warm/NodeCount=2 to satisfy the SQL CHECK constraint
CK_ServerCluster_RedundancyMode_NodeCount AND the ClusterEnabledNodeCountMismatch
validator). Called it in DeployHappyPath x2 / Failover / FleetDiagnostics /
EquipmentNamespace; fixed DriverReconnect to seed node B + NodeCount=2.
2. Stale EquipmentId (EquipmentNamespace test; pre-existing on BOTH providers).
Seeded EquipmentId='eq-1', but the later-added EquipmentIdNotDerived validator
requires EquipmentId == DeriveEquipmentId(EquipmentUuid) ('EQ-'+first 12 hex).
Fixed the seed to a canonical id + matching UUID. Only surfaced once the FK fix
let the deploy reach equipment validation.
Also added OTOPCUA_HARNESS_SQL_HOST / OTOPCUA_HARNESS_LDAP_HOST overrides so the
harness fixtures can run on the native-amd64 Docker host (avoids arm64 mssql
emulation entirely).
Verified: the 5 SQL-backed classes go 11/11 green vs native SQL; in-memory default
unregressed (12/12). RoslynVirtualTagEvaluatorTests racing test is a pre-existing
flaky race (not SQL-backed; passes in isolation) — left as-is.
The Host.IntegrationTests opt-in real-LDAP mode (OTOPCUA_HARNESS_USE_LDAP=1)
used bitnami/openldap:2.6, gone since Bitnami deprecated their free image
catalog in 2025. Replaced it with GLAuth — the same LDAP server the rest of
the project already uses (docker-dev + the live OPC UA data-plane auth, both
against the shared GLAuth on :3893) — removing the lone OpenLDAP outlier and
the gone-image breakage.
- Added tests/.../Host.IntegrationTests/glauth/config.toml: baseDN dc=zb,dc=local,
a search-capable serviceaccount, dev users alice (full access) / bob (read-only),
and role groups with the same gidnumbers as scadaproj/infra/glauth so GroupToRole
maps identically.
- Compose ldap service -> glauth/glauth:latest, mounting the config, host :3894 -> :3893.
- Repointed the harness real-LDAP override from the OpenLDAP cn=admin/ldapadmin to
GLAuth's cn=serviceaccount/serviceaccount123.
Live-verified against a deployed container on :3894: the serviceaccount bind
searches and returns alice + her 5 memberOf groups; alice/bob bind with the
correct password; a wrong password yields Invalid credentials (49) — exactly the
OtOpcUaLdapAuthService flow (proven identical to the data-plane's shared-GLAuth path).
NB: real-LDAP mode is opt-in; the default Host run uses StubLdapAuthService, so
this never blocked the default suite. Integration-sweep follow-up #6 (LDAP leg);
the amd64-emulated-SQL deadline leg remains open.
Two test/fixture bugs; NO OtOpcUa driver change.
1. Topology: FocasSimFixture reads an env-overridable endpoint
(OTOPCUA_FOCAS_SIM_ENDPOINT, default localhost:8193) and exposes Host/Port,
but WireBackendTests + WireBackendCoverageTests hardcoded
focas://127.0.0.1:8193. Against a remote fixture the skip-gate passed
(remote reachable) but the driver dialed localhost -> KeyNotFound. Added a
DeviceUri property to the fixture; each test now derives DeviceHost from it.
Recovered 8 of 9.
2. Mock timer endianness (surfaced once #1 let the tests reach the mock): the
last failure read 60397977600 for a 3600 s power-on timer (= 0x3C000000 x 60).
The focas-mock's _wire_timer encoded the minute/msec fields big-endian (_u32),
but cnc_rdtimer's timer fields are LITTLE-endian on real hardware — a
documented, live-31i-B-validated quirk that FocasWireClient.ParseTimer decodes
LE (docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md). The driver was
correct; the mock was wrong. _wire_timer now emits little-endian (_u32_le).
Suite 9F/1P -> 10/10 against the remote fixture at ~/otopcua-focas.
Integration-sweep follow-up #3.
The 2026-07 integration sweep flagged the Modbus non-standard profiles as
8F/5F/3F/8F and guessed 'pymodbus 4.x drift'. Re-investigation shows that was
wrong on both counts:
- The image runs the PINNED pymodbus 3.13.0 (verified via import pymodbus;
__version__). The '...removed in v4' log is 3.13.0's forward-deprecation
warning, misread as running 4.x.
- The failures were a fixture-cycling artifact: each profile is a separate
profile-gated compose service sharing :5020, and a plain 'docker compose down'
(no --profile) does NOT stop the running profile container. It kept holding
:5020; the next '--profile up' container silently failed to bind (compose
still says 'Started'), so every later profile run hit the STALE sim. Proven by
a raw Modbus probe: the 'mitsubishi' sim served DL205's 0xCAFE at HR0.
After force-removing all modbus containers and bringing up each profile cleanly
(verifying it actually published :5020), all four pass: dl205 16/16,
mitsubishi 13/13, s7_1500 10/10, exception_injection 17/17. No OtOpcUa or test
code change needed.
Added a profile-cycling gotcha + reliable-cycle recipe to Docker/README.md and
corrected the sweep record. Integration-sweep follow-up #2.
OpcUaServer.IntegrationTests was 4F. Two distinct, previously-masked issues:
1. Cert-store gap: the in-test client SecurityConfiguration used a bare
new SecurityConfiguration()/CertificateIdentifier(), so ValidateAsync threw
'TrustedIssuerCertificates StorePath must be specified' before any connect.
Added TestClientSecurity helper building Directory own/issuer/trusted/rejected
stores under a throwaway temp PKI dir (mirrors DefaultApplicationConfigurationFactory);
wired into DualEndpointTests + SubscriptionSurvivalTests.
2. Fixing #1 unmasked an SDK-version drift: the 1.5.378 node-cache read path
throws ServiceResultException(BadNodeIdUnknown) for a removed node instead of
returning a bad DataValue. The two subscription-survival assertions now expect
the throw via Should.ThrowAsync. Behavior under test (removed node -> unknown)
is unchanged; only the delivery mechanism differs.
Suite 4F -> 4/4 green. Integration-sweep follow-up #5.
KnownProfiles.All used a { get; } = [...] initialiser declared textually
before the static readonly profile fields it references, so C# static-init
order captured an all-null list. ForFamily() then NRE'd on p.Family, faulting
AbServerFixture..ctor and blocking the whole AbCip.IntegrationTests suite.
Expression-body it so it evaluates on access, after the fields are set.
Live-verified against the controllogix fixture: suite goes 6F/1P/3skip ->
10 pass / 0 fail / 3 skip (skips = emulator-mode tests).
Integration-sweep follow-up #1 (archreview/plans/INTEGRATION-SWEEP-STATUS.md).
The historian's Int1 analog-tag creation path is server-degenerate:
EnsureTags(Int1) stores an unusable stub (live-probed on the deployed
2023 R2 historian; root-caused + documented gateway-side in
HistorianGateway #11 / PR #16). HistorianTypeMapper mapped
DriverDataType.Boolean -> HistorianDataType.Int1, so Boolean historized
tags failed auto-provisioning.
Map Boolean -> Int2 instead (store 0/1). Int2 is a proven-creatable
analog type, and the value-write path already sends every value as a
double on WriteLiveValues regardless of tag type, so Boolean 0.0/1.0
round-trips cleanly. Both Float and Int2 are supported, so a
pre-existing Float Boolean tag retypes cleanly on its next deploy.
- HistorianTypeMapper: Boolean => Int2 (+ remarks)
- Unit pins: HistorianTypeMapperTests, GatewayTagProvisionerTests
(Boolean_definition_carries_explicit_Int2_presence)
- Live gate: EnsureTags_boolean_sandbox_provisions_as_Int2 (was _Int1);
retype probe reframed Float->Int2 (drops the dead Int1 leg)
- Docs: docs/Historian.md 'Boolean historization maps to Int2', CLAUDE.md,
archreview STATUS.md (R2-06 live gate -> 6/6)
Resolves#441.
GatewayTagProvisioner built HistorianTagDefinition without a StorageRateMs, so
the proto uint32 defaulted to 0. The gateway's EnsureTags maps StorageRateMs
straight to the AVEVA SDK's storageRateMs, which requires a positive value —
0 makes the SDK throw ArgumentOutOfRangeException ('Storage rate must be > 0 ms'),
so EVERY historized-tag auto-provision (the deploy-time IHistorianProvisioning
hook) silently failed at the gateway with failed=N.
Stamp DefaultStorageRateMs=1000 (matches the gateway's own live provisioner
convention; quantized 1000/5000/10000). Add a regression unit test asserting the
definition carries a positive rate, and set StorageRateMs on the live-suite's
inline definitions so the retype probe reaches a real EnsureTags.
Found via the R2-06 live gate against a real 0.2.0 gateway + AVEVA historian:
with this fixed, Float/Int2/UInt2/Int4/UInt4/Double all provision and the live
suite goes 3/6 -> 5/6. (Int1/Boolean remains blocked by the gateway histsdk's
ProtocolEvidenceMissingException for Int1 — a gateway-side type-support gap, not
this bug.)
Gateway driver unit 103/103.
Four continuous-historization types (IHistorizationOutbox, HistorizationOutboxEntry,
IHistorianValueWriter/HistorizationValue, HistorizationCommitMode) drifted into a
ZB.MOM.WW.OtOpcUa.Core.Abstractions.Historian sub-namespace while the other six files
in the same folder correctly use the root namespace. This violated decision #59's
flat-public-surface rule and left InterfaceIndependenceTests.AllPublicTypes_LiveInRootNamespace
red on master (pre-existing, predates round-2 remediation).
Move all four to the root namespace and drop/retarget the now-redundant
'using ...Core.Abstractions.Historian;' in the ~11 consumers. No behavioral change.
Verified: Core.Abstractions.Tests 129/129 (was 128/129), Runtime.Tests Historian+DI 72/72,
Gateway driver unit 102/102, full solution build 0 errors.
Joseph Doherty