Compare commits

...

43 Commits

Author SHA1 Message Date
dohertj2 b980eeb148 Merge pull request 'docs(archreview): live finding — keep-oldest downs the survivor on oldest-crash (#459)' (#462) from docs/459-oldest-crash-live-finding into master
v2-ci / build (push) Successful in 4m33s
v2-ci / unit-tests (push) Has started running
2026-07-15 10:03:49 -04:00
Joseph Doherty d7470c7960 docs(archreview): live finding — keep-oldest downs the survivor on oldest-crash (#459)
v2-ci / build (pull_request) Successful in 4m27s
v2-ci / unit-tests (pull_request) Has started running
2-container docker-dev rig: hard-killing the oldest node makes the younger
survivor down ITSELF (SBR DownReachable including myself -> CoordinatedShutdown)
-> no failover, total outage. Reproduced twice. Confirms #459 is REAL and
disproves Option A (keep keep-oldest). Baseline ServiceLevel 250(leader)/240(follower)
captured before the kill. Documents why (textbook 2-node keep-oldest limitation;
down-if-alone doesn't rescue a crashed oldest) and why the in-process
HardKillFailoverTests gave false confidence (Transport.Shutdown keeps the
ActorSystem alive, not a real crash). Fix needs Option B (witness+keep-majority)
or Option C (DB-lease arbiter).
2026-07-15 10:03:31 -04:00
dohertj2 b364a499e5 Merge pull request 'test+docs(resilience): prove authored ResilienceConfig reaches the runtime pipeline (#456)' (#461) from fix/456-resilience-config-artifact-roundtrip into master
v2-ci / build (push) Successful in 4m22s
v2-ci / unit-tests (push) Failing after 10m28s
2026-07-15 08:49:51 -04:00
Joseph Doherty 0c005c9a72 docs(archreview): live proof authored ResilienceConfig reaches the runtime pipeline (#456)
v2-ci / build (pull_request) Successful in 4m17s
v2-ci / unit-tests (pull_request) Failing after 10m27s
Captures the docker-dev live verification for follow-up #10 acceptance item 2:
a non-default per-instance ResilienceConfig (Subscribe.retryCount:999) authored on
MAIN rode the deploy artifact into BOTH central-1/central-2 runtimes, which parsed
and clamped it — the authored value appears verbatim in the spawn-time diagnostic.
Confirms the artifact bytes carry ResilienceConfig and the runtime pipeline applies
authored (not tier-default) policy. Item 1 (raw retry/breaker-on-fault log line)
remains, with the connect-then-idempotent-write recipe documented.
2026-07-15 08:49:17 -04:00
Joseph Doherty 809e7886d8 test(resilience): prove ResilienceConfig survives the real ConfigComposer round-trip (#456)
The arch-review #10 sub-gap worried that per-instance ResilienceConfig never
reaches the runtime pipeline. The threading is in fact wired end-to-end (task #13):
ConfigComposer serialises the whole DriverInstance entity, so ResilienceConfig
rides the artifact into DriverInstanceSpec, which DriverHostActor layers onto the
driver's Polly pipeline. The only leg with no test was the real composer->artifact
serialization — the existing DeploymentArtifactTests hand-build the JSON.

Adds ResilienceConfig_survives_ConfigComposer_to_ParseDriverInstances_round_trip
(mirrors the DeviceHost round-trip guard): seeds a DriverInstance WITH a non-default
override + one WITHOUT, runs the real SnapshotAndFlattenAsync, and asserts
ParseDriverInstances recovers the override byte-for-byte (and null stays null).
Guards against a future projection / [JsonIgnore] silently reverting authored
resilience policy to tier defaults while hand-built artifact tests stay green.

ControlPlane.Tests ConfigComposerTests 6/6 green.
2026-07-15 08:38:13 -04:00
dohertj2 5908339d67 Merge pull request 'fix(deps): clear CVE-2025-6965 — pin SQLitePCLRaw bundle to 2.1.12, drop audit suppression' (#460) from fix/458-sqlitepclraw-cve-unsuppress into master
v2-ci / build (push) Successful in 4m28s
v2-ci / unit-tests (push) Failing after 10m45s
2026-07-15 08:17:55 -04:00
Joseph Doherty b461b2bcf8 fix(deps): clear CVE-2025-6965 — pin SQLitePCLRaw bundle to 2.1.12, drop audit suppression
v2-ci / build (pull_request) Successful in 5m41s
v2-ci / unit-tests (pull_request) Failing after 10m49s
The patched native SQLite bundle shipped: SQLitePCLRaw.bundle_e_sqlite3 2.1.12
is the first version outside the CVE-2025-6965 / GHSA-2m69-gcr7-jv3q vulnerable
range (<= 2.1.11), embedding the SQLite 3.50.2+ fix.

- Bump Microsoft.Data.Sqlite 9.0.0 -> 10.0.7 (aligns with the EF Core 10.0.7 family).
  This alone still pulls the native bundle 2.1.11, so:
- Add a surgical direct pin of SQLitePCLRaw.bundle_e_sqlite3 2.1.12 in Core.AlarmHistorian
  (the sole consumer), overriding the transitive 2.1.11 without enabling
  CentralPackageTransitivePinningEnabled (which breaks the Roslyn 5.0/4.12 split).
- Remove the temporary NuGetAuditSuppress from Directory.Build.props.

Verified: dotnet restore/build clean with audit active (no GHSA-2m69), every
in-solution project resolves lib.e_sqlite3 2.1.12, AlarmHistorian tests 29/29 green
against the real native bundle.

Closes #458
2026-07-15 08:17:19 -04:00
Joseph Doherty e0ff908d26 docs(infra): fix memory pointers rendering as broken shortcut-links
v2-ci / build (push) Successful in 5m4s
v2-ci / unit-tests (push) Failing after 12m19s
The [reference_*] pointers target agent memory (not repo files) and rendered as
undefined markdown shortcut-reference links; converted to plain code spans. All
repo-relative links + paths in infra/README.md verified to resolve.
2026-07-15 08:06:54 -04:00
Joseph Doherty 09066eb88f docs(archreview): file residual follow-ups as Gitea issues #456-#459
v2-ci / build (push) Successful in 4m29s
v2-ci / unit-tests (push) Failing after 13m3s
The 4 non-blocking residuals (#10 live-gate/ResilienceConfig sub-gap, R2-11 Phase C
strict parsers, CVE-2025-6965 suppression removal, SBR oldest-node failover gap) are
now tracked as Gitea issues, cross-referenced in the STATUS.md capstone.
2026-07-15 08:02:24 -04:00
Joseph Doherty 1d3b6d1264 docs(infra): add infra/ folder — canonical testing + deployment infra index
v2-ci / build (push) Successful in 4m4s
v2-ci / unit-tests (push) Failing after 12m22s
Consolidates the scattered infra facts (docker host layout, driver fixtures, the
three env-gated live-gate recipes, the Host.IntegrationTests harness, the HistGW
local deploy, docker-dev rig, shared GLAuth, deploy) into infra/README.md as the
single entry point, cross-linked to dev-environment.md + the deep docs. CLAUDE.md
points at it from the Docker Workflow section.
2026-07-15 08:00:37 -04:00
Joseph Doherty c9cf5015e5 docs(archreview): R2-08 S3 gateway LiveIntegration soak GREEN — all Round-2 live gates closed
v2-ci / build (push) Successful in 5m15s
v2-ci / unit-tests (push) Failing after 11m9s
Ran Category=LiveIntegration 6/6 against the running local 0.2.0 HistorianGateway
(localhost:5221 h2c) + the real AVEVA historian over VPN. Confirms R2-08 S3's
GatewayHistorianDataSource read path and re-confirms R2-06. With R2-01 + R2-08
GLAuth already self-serviced, every Round-2 live gate is now GREEN.
2026-07-15 07:50:56 -04:00
Joseph Doherty 7bca0eeca0 docs(archreview): R2-08 GLAuth outage/recovery live gate CLOSED (PR #455)
v2-ci / build (push) Successful in 5m38s
v2-ci / unit-tests (push) Failing after 12m34s
LdapGlauthOutageLiveTests drives the S2 directory-outage circuit through the
production auth path against a real GLAuth (docker pause/unpause) on 10.100.0.35.
Only R2-06 VPN + R2-08 S3 LiveIntegration gateway soak remain (both need VPN).
2026-07-15 07:46:39 -04:00
dohertj2 5f12636cbc Merge pull request 'test(ldap): live GLAuth outage/recovery gate (R2-08 03/S2)' (#455) from test/r2-08-glauth-outage-live-gate into master
v2-ci / build (push) Successful in 5m33s
v2-ci / unit-tests (push) Failing after 11m32s
2026-07-15 07:45:24 -04:00
Joseph Doherty 4d2dcff5df test(ldap): live GLAuth outage/recovery gate (R2-08 03/S2)
v2-ci / build (pull_request) Successful in 3m59s
v2-ci / unit-tests (pull_request) Failing after 11m55s
Closes the R2-08 S2 live leg. The offline LdapAuthResilienceTests prove the
directory-outage circuit with fakes + an unroutable-host blackhole; this drives
the SAME circuit through the production auth path (OtOpcUaLdapAuthService +
LdapOpcUaUserAuthenticator) against a REAL GLAuth that is paused mid-run and
unpaused — and doubles as the first end-to-end verification that the PR #451
GLAuth swap (retired bitnami/openldap) binds correctly.

Triple-gated (GLAUTH_LIVE_HOST + GLAUTH_OUTAGE_START_CMD/STOP_CMD), skips clean
offline (verified: 1 skipped, 8 ms). try/finally always unpauses GLAuth.

Verified GREEN against 10.100.0.35:3894 (12 s): healthy binds (alice+bob, mapped
roles) -> docker pause -> 3 bounded failures -> circuit opens -> sub-second
"unavailable" fast-deny -> docker unpause -> half-open probe closes -> binds
succeed again. A boundary AuthTimeout feeds the circuit (authenticator :123-130),
so the docker-pause timeout shape is a faithful outage.
2026-07-15 07:45:07 -04:00
dohertj2 a19c6b987c Merge pull request 'fix(drivers): harden operator TimeoutMs handling + AbLegacy evict parity' (#454) from fix/driver-timeout-hardening into master
v2-ci / build (push) Successful in 4m12s
v2-ci / unit-tests (push) Failing after 10m27s
2026-07-15 07:35:37 -04:00
Joseph Doherty a5a0f96d49 fix(drivers): harden operator TimeoutMs handling + AbLegacy evict parity
v2-ci / build (pull_request) Successful in 4m5s
v2-ci / unit-tests (pull_request) Failing after 10m45s
Follow-ups from the fleet-wide read-timeout audit that the S7 R2-01 read-leg
fix (PR #453) prompted. The audit confirmed S7 was the ONLY driver with the
async-read-ignores-socket-timeout hang; these are the two adjacent (non-hang)
findings it surfaced.

1. FOCAS TimeoutMs:0 footgun (the risky one): a non-positive Timeout made
   SynchronizedFocasClient DISABLE its per-call wall-clock ceiling, reverting to
   the caller's long-lived poll token — reintroducing exactly the frozen-peer
   wedge S7 just eliminated, under misconfig. Clamp non-positive TimeoutMs to the
   2s default at the config boundary so the deadline can never be authored away.

2. TimeoutMs validation symmetry: apply the same clamp in the AbCip + AbLegacy
   factories. libplctag's Tag.Timeout setter throws on <=0, faulting tag creation
   on every read/write; clamping keeps a misconfigured TimeoutMs:0 running on the
   default bound instead. (Shared PositiveTimeoutOrDefault helper per factory.)

3. AbLegacy reconnect parity with AbCip: AbLegacy evicted the cached libplctag
   runtime on neither the non-zero-status nor transport-exception read/write path
   (AbCip evicts on both), so a data-path fault recovered only via the probe loop
   / libplctag internals. Added EvictRuntime + wired it into both read and write
   failure paths so a fresh handle is created on the next call.

Tests: FOCAS 265->269 (clamp theory + positive), AbCip 336->339 (clamp theory +
positive), AbLegacy 209->212 (read-nonzero / read-exception / write-nonzero evict).
No production regressions; all three driver suites green.
2026-07-15 07:35:17 -04:00
Joseph Doherty c21e21423f docs(archreview): R2-01 S7 live gate CLOSED — read-leg wall-clock fix (PR #453)
v2-ci / build (push) Successful in 4m42s
v2-ci / unit-tests (push) Failing after 12m57s
Running S7_1500ConnectTimeoutOutageTests (docker-pause blackhole, self-serviceable
over SSH) uncovered + fixed the read-leg wall-clock gap STAB-14 left. Updates the
R2-01 STATUS row, the needs-user/still-open lists, and the STAB-14 remediation note.
2026-07-15 07:13:46 -04:00
dohertj2 4dc937fcb8 Merge pull request 'fix(s7): bound async wire ops with a wall-clock deadline (R2-01 read leg)' (#453) from fix/r2-01-s7-read-wallclock-timeout into master
v2-ci / build (push) Successful in 5m10s
v2-ci / unit-tests (push) Failing after 14m39s
2026-07-15 07:12:14 -04:00
Joseph Doherty 88e0977ae6 fix(s7): bound async wire ops with a wall-clock deadline (R2-01 read leg)
v2-ci / build (pull_request) Successful in 5m5s
v2-ci / unit-tests (pull_request) Failing after 13m50s
The R2-01 live gate (S7_1500ConnectTimeoutOutageTests, docker-pause blackhole)
surfaced a real gap the offline fakes couldn't: S7.Net's ReadTimeout/WriteTimeout
map to the TcpClient's ReceiveTimeout/SendTimeout, which govern only SYNCHRONOUS
socket calls — the async read/write paths S7.Net uses ignore them. On an
established-but-frozen peer (frozen PLC / firewall DROP / cable pulled mid-flow,
TCP session still open) a read blocked until the OS TCP stack gave up (minutes),
silently wedging the poll loop: no Bad tick, no reconnect. STAB-14 fixed only the
CONNECT leg (EnsureConnectedAsync CancelAfter); this is its READ-leg sibling.

- New S7OperationDeadline: bounds every data-plane wire op with a wall-clock
  ceiling (= _options.Timeout), surfacing an overrun as TimeoutException. Applied
  in S7PlcAdapter to Read/ReadBytes/Write/WriteBytes/ReadStatus. OpenAsync is left
  to EnsureConnectedAsync's own CancelAfter (not double-bounded).
- IsS7ConnectionFatal now classifies TimeoutException fatal → handle marked dead →
  next EnsureConnectedAsync reopens (connect-timeout fix takes over from there).
- Tests: 5 S7OperationDeadline unit tests (deadline / token-honouring / caller-
  cancel-passthrough / resultless), 1 driver-reaction test (read TimeoutException
  → reopen). Driver.S7.Tests 260/260.
- Live gate S7_1500ConnectTimeoutOutageTests now GREEN against the real snap7 sim
  (8s: baseline Good -> pause blackhole -> Bad tick -> unpause -> recovered Good).
2026-07-15 07:11:42 -04:00
dohertj2 152a5645ad fix(harness): deploy E2E tests pass against real SQL (#452)
v2-ci / build (push) Successful in 4m28s
v2-ci / unit-tests (push) Failing after 10m2s
2026-07-15 06:43:37 -04:00
Joseph Doherty 859d63178a fix(harness): make deploy E2E tests pass against real SQL (seed cluster FKs)
v2-ci / build (pull_request) Successful in 4m23s
v2-ci / unit-tests (pull_request) Failing after 10m14s
Integration-sweep follow-up #6 (SQL leg). The sweep flagged 7 heavy 2-node
deploy/failover/reconnect E2E tests as 'time out under amd64-emulated SQL' and
suggested raising deadlines. That diagnosis was WRONG: run against real
(native-amd64) SQL on the Docker host, they still hang — even at 4x deadline.
Root cause is that these tests had never actually run against real SQL, only the
EF in-memory provider, which does NOT enforce foreign keys.

Two distinct defects, both hidden by in-memory:

1. Missing FK-parent seed (5 tests). The deploy records a NodeDeploymentState row
   per node, FK'd to ClusterNode (FK_NodeDeploymentState_ClusterNode_NodeId). The
   tests never seeded ServerCluster + ClusterNode, so on SQL each node-state INSERT
   throws and the deploy never seals (waits the full deadline regardless of size).
   Added TwoNodeClusterHarness.SeedDefaultClusterAsync (seeds a ServerCluster +
   a ClusterNode per node; Warm/NodeCount=2 to satisfy the SQL CHECK constraint
   CK_ServerCluster_RedundancyMode_NodeCount AND the ClusterEnabledNodeCountMismatch
   validator). Called it in DeployHappyPath x2 / Failover / FleetDiagnostics /
   EquipmentNamespace; fixed DriverReconnect to seed node B + NodeCount=2.

2. Stale EquipmentId (EquipmentNamespace test; pre-existing on BOTH providers).
   Seeded EquipmentId='eq-1', but the later-added EquipmentIdNotDerived validator
   requires EquipmentId == DeriveEquipmentId(EquipmentUuid) ('EQ-'+first 12 hex).
   Fixed the seed to a canonical id + matching UUID. Only surfaced once the FK fix
   let the deploy reach equipment validation.

Also added OTOPCUA_HARNESS_SQL_HOST / OTOPCUA_HARNESS_LDAP_HOST overrides so the
harness fixtures can run on the native-amd64 Docker host (avoids arm64 mssql
emulation entirely).

Verified: the 5 SQL-backed classes go 11/11 green vs native SQL; in-memory default
unregressed (12/12). RoslynVirtualTagEvaluatorTests racing test is a pre-existing
flaky race (not SQL-backed; passes in isolation) — left as-is.
2026-07-15 06:43:15 -04:00
dohertj2 779afbb66f fix(harness): swap retired bitnami/openldap for GLAuth (#451)
v2-ci / build (push) Successful in 4m0s
v2-ci / unit-tests (push) Failing after 9m29s
2026-07-15 06:04:17 -04:00
Joseph Doherty 3a48eb0c70 fix(harness): swap retired bitnami/openldap for GLAuth in the Host real-LDAP mode
v2-ci / build (pull_request) Successful in 4m13s
v2-ci / unit-tests (pull_request) Failing after 10m18s
The Host.IntegrationTests opt-in real-LDAP mode (OTOPCUA_HARNESS_USE_LDAP=1)
used bitnami/openldap:2.6, gone since Bitnami deprecated their free image
catalog in 2025. Replaced it with GLAuth — the same LDAP server the rest of
the project already uses (docker-dev + the live OPC UA data-plane auth, both
against the shared GLAuth on :3893) — removing the lone OpenLDAP outlier and
the gone-image breakage.

- Added tests/.../Host.IntegrationTests/glauth/config.toml: baseDN dc=zb,dc=local,
  a search-capable serviceaccount, dev users alice (full access) / bob (read-only),
  and role groups with the same gidnumbers as scadaproj/infra/glauth so GroupToRole
  maps identically.
- Compose ldap service -> glauth/glauth:latest, mounting the config, host :3894 -> :3893.
- Repointed the harness real-LDAP override from the OpenLDAP cn=admin/ldapadmin to
  GLAuth's cn=serviceaccount/serviceaccount123.

Live-verified against a deployed container on :3894: the serviceaccount bind
searches and returns alice + her 5 memberOf groups; alice/bob bind with the
correct password; a wrong password yields Invalid credentials (49) — exactly the
OtOpcUaLdapAuthService flow (proven identical to the data-plane's shared-GLAuth path).

NB: real-LDAP mode is opt-in; the default Host run uses StubLdapAuthService, so
this never blocked the default suite. Integration-sweep follow-up #6 (LDAP leg);
the amd64-emulated-SQL deadline leg remains open.
2026-07-15 06:03:50 -04:00
dohertj2 18481f3d11 fix(focas-tests): honour fixture endpoint + fix mock timer endianness (#450)
v2-ci / build (push) Successful in 4m34s
v2-ci / unit-tests (push) Failing after 10m37s
2026-07-15 05:52:03 -04:00
Joseph Doherty 7982043673 fix(focas-tests): honour fixture endpoint + fix mock timer endianness (9F/1P -> 10/10)
v2-ci / build (pull_request) Successful in 4m31s
v2-ci / unit-tests (pull_request) Failing after 10m26s
Two test/fixture bugs; NO OtOpcUa driver change.

1. Topology: FocasSimFixture reads an env-overridable endpoint
   (OTOPCUA_FOCAS_SIM_ENDPOINT, default localhost:8193) and exposes Host/Port,
   but WireBackendTests + WireBackendCoverageTests hardcoded
   focas://127.0.0.1:8193. Against a remote fixture the skip-gate passed
   (remote reachable) but the driver dialed localhost -> KeyNotFound. Added a
   DeviceUri property to the fixture; each test now derives DeviceHost from it.
   Recovered 8 of 9.

2. Mock timer endianness (surfaced once #1 let the tests reach the mock): the
   last failure read 60397977600 for a 3600 s power-on timer (= 0x3C000000 x 60).
   The focas-mock's _wire_timer encoded the minute/msec fields big-endian (_u32),
   but cnc_rdtimer's timer fields are LITTLE-endian on real hardware — a
   documented, live-31i-B-validated quirk that FocasWireClient.ParseTimer decodes
   LE (docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md). The driver was
   correct; the mock was wrong. _wire_timer now emits little-endian (_u32_le).

Suite 9F/1P -> 10/10 against the remote fixture at ~/otopcua-focas.
Integration-sweep follow-up #3.
2026-07-15 05:51:47 -04:00
dohertj2 00d8422cef docs(sweep): AbLegacy verified green — fixture-deployment gap, no code (#449)
v2-ci / build (push) Successful in 4m28s
v2-ci / unit-tests (push) Failing after 11m5s
2026-07-15 05:38:37 -04:00
Joseph Doherty f381b471f9 docs(sweep): AbLegacy verified green — fixture-deployment gap, not a code defect (#4)
v2-ci / build (pull_request) Successful in 4m26s
v2-ci / unit-tests (pull_request) Failing after 3h11m47s
The sweep's AbLegacy 4F was a fixture-deployment gap, not a driver/test defect.
The suite needs ab_server in PCCC mode (--plc=SLC500|Micrologix|PLC/5), which
reuses the AbCip image (already on the host) and binds :44818. No PCCC container
was deployed, and the 4F-not-skip means :44818 was answering CIP (a leftover
AbCip container) so the PCCC reads returned Bad — same shared-port contamination
class as the Modbus false alarm.

Deployed the PCCC compose to ~/otopcua-ablegacy and ran each family with
AB_LEGACY_COMPOSE_PROFILE set to match the running container (required — without
it the [Theory] parametrises over all 3 families and 2 fail against a single-
family container): slc500 2/2, micrologix 1/1 (+1 skip), plc5 1/1 (+1 skip).
Skips are family-inapplicable file types. No code change.

Corrected the sweep record + added the AbLegacy bring-up + run recipe.
Integration-sweep follow-up #4.
2026-07-15 05:38:18 -04:00
dohertj2 ab1644daa4 docs(modbus-tests): sweep #2 was a fixture-cycling false alarm (#448)
v2-ci / build (push) Successful in 4m19s
v2-ci / unit-tests (push) Failing after 10m47s
2026-07-15 05:13:12 -04:00
Joseph Doherty 00f39c849e docs(modbus-tests): document profile-cycling gotcha; sweep #2 was a false alarm
v2-ci / build (pull_request) Successful in 4m13s
v2-ci / unit-tests (pull_request) Failing after 9m41s
The 2026-07 integration sweep flagged the Modbus non-standard profiles as
8F/5F/3F/8F and guessed 'pymodbus 4.x drift'. Re-investigation shows that was
wrong on both counts:

- The image runs the PINNED pymodbus 3.13.0 (verified via import pymodbus;
  __version__). The '...removed in v4' log is 3.13.0's forward-deprecation
  warning, misread as running 4.x.
- The failures were a fixture-cycling artifact: each profile is a separate
  profile-gated compose service sharing :5020, and a plain 'docker compose down'
  (no --profile) does NOT stop the running profile container. It kept holding
  :5020; the next '--profile up' container silently failed to bind (compose
  still says 'Started'), so every later profile run hit the STALE sim. Proven by
  a raw Modbus probe: the 'mitsubishi' sim served DL205's 0xCAFE at HR0.

After force-removing all modbus containers and bringing up each profile cleanly
(verifying it actually published :5020), all four pass: dl205 16/16,
mitsubishi 13/13, s7_1500 10/10, exception_injection 17/17. No OtOpcUa or test
code change needed.

Added a profile-cycling gotcha + reliable-cycle recipe to Docker/README.md and
corrected the sweep record. Integration-sweep follow-up #2.
2026-07-15 05:12:52 -04:00
dohertj2 271562e00e fix(opcuaserver-tests): fix cert-store gap + SDK read-throws drift (#447)
v2-ci / build (push) Successful in 7m17s
v2-ci / unit-tests (push) Failing after 12m55s
2026-07-15 03:34:35 -04:00
Joseph Doherty de9d9697fd fix(opcuaserver-tests): supply client cert-store paths + adapt to SDK read-throws
v2-ci / build (pull_request) Successful in 7m23s
v2-ci / unit-tests (pull_request) Failing after 15m33s
OpcUaServer.IntegrationTests was 4F. Two distinct, previously-masked issues:

1. Cert-store gap: the in-test client SecurityConfiguration used a bare
   new SecurityConfiguration()/CertificateIdentifier(), so ValidateAsync threw
   'TrustedIssuerCertificates StorePath must be specified' before any connect.
   Added TestClientSecurity helper building Directory own/issuer/trusted/rejected
   stores under a throwaway temp PKI dir (mirrors DefaultApplicationConfigurationFactory);
   wired into DualEndpointTests + SubscriptionSurvivalTests.

2. Fixing #1 unmasked an SDK-version drift: the 1.5.378 node-cache read path
   throws ServiceResultException(BadNodeIdUnknown) for a removed node instead of
   returning a bad DataValue. The two subscription-survival assertions now expect
   the throw via Should.ThrowAsync. Behavior under test (removed node -> unknown)
   is unchanged; only the delivery mechanism differs.

Suite 4F -> 4/4 green. Integration-sweep follow-up #5.
2026-07-15 03:34:14 -04:00
dohertj2 c1482802d4 fix(abcip-tests): fix static-init NRE blocking AbCip.IntegrationTests (#446)
v2-ci / build (push) Successful in 4m15s
v2-ci / unit-tests (push) Failing after 18m20s
2026-07-15 03:29:03 -04:00
Joseph Doherty 2ba867b0f8 docs(archreview): mark integration-sweep follow-up #1 (AbCip static-init) done
v2-ci / build (pull_request) Successful in 4m19s
v2-ci / unit-tests (pull_request) Failing after 17m55s
2026-07-15 03:28:24 -04:00
Joseph Doherty 326b22a7ba fix(abcip-tests): expression-body KnownProfiles.All to fix static-init NRE
KnownProfiles.All used a { get; } = [...] initialiser declared textually
before the static readonly profile fields it references, so C# static-init
order captured an all-null list. ForFamily() then NRE'd on p.Family, faulting
AbServerFixture..ctor and blocking the whole AbCip.IntegrationTests suite.

Expression-body it so it evaluates on access, after the fields are set.
Live-verified against the controllogix fixture: suite goes 6F/1P/3skip ->
10 pass / 0 fail / 3 skip (skips = emulator-mode tests).

Integration-sweep follow-up #1 (archreview/plans/INTEGRATION-SWEEP-STATUS.md).
2026-07-15 03:28:04 -04:00
dohertj2 b94ea3d412 Merge pull request 'docs(archreview): integration-suite sweep results + follow-ups' (#445)
v2-ci / build (push) Successful in 4m50s
v2-ci / unit-tests (push) Failing after 9m2s
2026-07-15 01:26:14 -04:00
Joseph Doherty 1ee73d76d6 docs(archreview): record integration-suite sweep results + follow-ups (2026-07-14)
v2-ci / build (pull_request) Successful in 4m46s
v2-ci / unit-tests (pull_request) Failing after 10m56s
Ran the deferred *.IntegrationTests sweep (Mac + docker-host fixtures,
serial). No OtOpcUa regression: OpcUaClient(+Browser)/S7/Modbus-standard
green, Host 105/117, TwinCAT clean-skip. The rest fail on pre-existing
test/fixture bit-rot of distinct kinds (AbCip static-init bug, Modbus
pymodbus-4.x sim drift, FOCAS 127.0.0.1-mock topology, AbLegacy no
fixture, OpcUaServer cert-store config, Host openldap:2.6 image gone).

New INTEGRATION-SWEEP-STATUS.md has the full table, per-suite evidence,
7 follow-ups, and re-run fixture recipes; STATUS.md links it.
2026-07-15 01:25:55 -04:00
dohertj2 306728885d Merge pull request 'docs(archreview): R2-04 T13/T15 live-passed' (#444)
v2-ci / build (push) Successful in 4m13s
v2-ci / unit-tests (push) Failing after 10m54s
2026-07-14 15:14:05 -04:00
Joseph Doherty eccb15bdab docs(archreview): R2-04 T13/T15 live-passed 2026-07-14
v2-ci / build (pull_request) Successful in 4m8s
v2-ci / unit-tests (pull_request) Failing after 11m0s
Both R2-04 verification gates for the S4 primary-gate default-deny run:
- T15 (docker-dev 2-node rig, rebuilt from master 365a90e6 + Modbus sim):
  ServiceLevel 250/240; boot-window write rejected 'not primary (role
  unknown)' (meter reason=role-unknown,site=write); steady-state
  secondary rejected 'not primary' + node reverted (read-back 1234 not
  9999, meter reason=secondary); primary write reaches device; clean
  apply (no apply.failed). The behavior-affecting S4 slice is trusted.
- T13 (in-process PrimaryGateFailoverTests): 1/1, ran without SQL; the
  real DPS-delivered snapshot drives the gate; negative control covered.

STATUS.md + R2-04 tasks.json updated.
2026-07-14 15:13:43 -04:00
dohertj2 365a90e63d Merge pull request 'fix(s7-cli): restore OnDataChange CliFx-console rationale comment' (#443)
v2-ci / build (push) Successful in 3m17s
v2-ci / unit-tests (push) Failing after 8m2s
2026-07-14 14:49:26 -04:00
Joseph Doherty d327243fd3 fix(s7-cli): restore OnDataChange CliFx-console rationale comment
v2-ci / build (pull_request) Failing after 15s
v2-ci / unit-tests (pull_request) Has been skipped
The 9cad9ed0 'strip tracking-ID comments' XML-doc sweep collaterally
removed the comment explaining why the S7 subscribe command routes
OnDataChange through the CliFx IConsole (not System.Console), leaving
SubscribeCommandConsoleHandlerCommentTests red. The Modbus copy still
carries the comment verbatim and the test guards S7-mirrors-Modbus
parity, so restore it (matching the Modbus wording) rather than weaken
the guard.

S7.Cli tests 49/49.
2026-07-14 14:49:10 -04:00
dohertj2 38e21df296 Merge pull request 'fix(historian): map Boolean historization to Int2, not degenerate Int1 (#441)' (#442)
v2-ci / build (push) Successful in 3m24s
v2-ci / unit-tests (push) Failing after 8m26s
2026-07-14 12:38:04 -04:00
Joseph Doherty 9f62310b49 fix(historian): map Boolean historization to Int2, not degenerate Int1 (#441)
v2-ci / build (pull_request) Successful in 3m24s
v2-ci / unit-tests (pull_request) Failing after 8m19s
The historian's Int1 analog-tag creation path is server-degenerate:
EnsureTags(Int1) stores an unusable stub (live-probed on the deployed
2023 R2 historian; root-caused + documented gateway-side in
HistorianGateway #11 / PR #16). HistorianTypeMapper mapped
DriverDataType.Boolean -> HistorianDataType.Int1, so Boolean historized
tags failed auto-provisioning.

Map Boolean -> Int2 instead (store 0/1). Int2 is a proven-creatable
analog type, and the value-write path already sends every value as a
double on WriteLiveValues regardless of tag type, so Boolean 0.0/1.0
round-trips cleanly. Both Float and Int2 are supported, so a
pre-existing Float Boolean tag retypes cleanly on its next deploy.

- HistorianTypeMapper: Boolean => Int2 (+ remarks)
- Unit pins: HistorianTypeMapperTests, GatewayTagProvisionerTests
  (Boolean_definition_carries_explicit_Int2_presence)
- Live gate: EnsureTags_boolean_sandbox_provisions_as_Int2 (was _Int1);
  retype probe reframed Float->Int2 (drops the dead Int1 leg)
- Docs: docs/Historian.md 'Boolean historization maps to Int2', CLAUDE.md,
  archreview STATUS.md (R2-06 live gate -> 6/6)

Resolves #441.
2026-07-14 11:57:27 -04:00
dohertj2 7f79cd5941 Merge pull request 'deps: bump HistorianGateway Client+Contracts to 0.3.0' (#440) from bump-historiangateway-0.3.0 into master
v2-ci / build (push) Successful in 4m17s
v2-ci / unit-tests (push) Failing after 12m24s
2026-07-14 08:18:07 -04:00
48 changed files with 1928 additions and 230 deletions
+19 -11
View File
@@ -111,10 +111,12 @@ dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests # a single tes
dotnet test --filter "FullyQualifiedName~MyTestClass.MyMethod" # a single test
```
`Directory.Build.props` carries a temporary `NuGetAuditSuppress` for
CVE-2025-6965 / GHSA-2m69-gcr7-jv3q (transitive native `SQLitePCLRaw.lib.e_sqlite3`,
no upstream fix yet) — so restore/build not flagging it is expected; remove the
suppression once a patched bundle ships.
CVE-2025-6965 / GHSA-2m69-gcr7-jv3q (transitive native `SQLitePCLRaw.lib.e_sqlite3`)
is **resolved** — the temporary `NuGetAuditSuppress` was removed once the patched bundle
shipped. `Core.AlarmHistorian` now carries a surgical direct pin of
`SQLitePCLRaw.bundle_e_sqlite3` to `2.1.12` (the first fixed native bundle, embedding the
SQLite 3.50.2+ patch), overriding the `2.1.11` that `Microsoft.Data.Sqlite` pulls
transitively. `dotnet restore`/`build` is clean with the audit active.
Test projects live under `tests/<module>/` (Core, Server, Drivers,
Drivers/Cli, Client, Tooling) — there is no single unit-test project.
@@ -125,6 +127,11 @@ central SQL Server.
## Docker Workflow (driver fixtures + central SQL Server)
> **Infra index:** [`infra/README.md`](infra/README.md) is the canonical index of all testing +
> deployment infrastructure — the Docker host layout, driver fixtures, the env-gated **live-gate**
> recipes (S7 blackhole, GLAuth outage, HistorianGateway LiveIntegration), the integration-test
> harness, the docker-dev rig, and the deploy setup. Start there; the sections below + `docs/v2/dev-environment.md` carry the detail.
> **Migrated 2026-04-28**: Docker config + host moved off this dev VM (DESKTOP-6JL3KKO) onto the shared Linux Docker host (`DOCKER`, 10.100.0.35) so the dev VM could shed WSL2/Hyper-V and have its GPU re-attached via ESXi passthrough. Docker Desktop is no longer installed here. All checked-in `appsettings.json` defaults, fixture-class default endpoints, and `e2e-config.sample.json` were rewritten to target `10.100.0.35`. The driver fixture compose files under `tests/.../Docker/docker-compose.yml` now carry a `project: lmxopcua` label on every service. See `docs/v2/dev-environment.md` for the full rewrite (header dated 2026-04-28).
Docker workloads run on a shared Linux host at **`10.100.0.35`** — not on this VM. Stacks live at `/opt/otopcua-<driver>/` on the host and carry the `project=lmxopcua` label so they're discoverable via `docker ps --filter label=project=lmxopcua`.
@@ -180,7 +187,7 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
The server uses LDAP-based user authentication via the `Security:Ldap` section in `appsettings.json`. When enabled, credentials are validated by LDAP bind against a GLAuth server, and LDAP group membership maps to OPC UA permissions: `ReadOnly` (browse/read), `WriteOperate` (write FreeAccess/Operate attributes), `WriteTune` (write Tune attributes), `WriteConfigure` (write Configure attributes), `AlarmAck` (alarm acknowledgment). `LdapOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`) implements `IOpcUaUserAuthenticator`, delegating the LDAP bind + group lookup to `OtOpcUaLdapAuthService` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`, an `ILdapAuthService`). See `docs/security.md` for the full guide.
Dev/test LDAP is the **shared GLAuth** running on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`, plaintext/`Transport=None`). It is managed via `scadaproj/infra/glauth/` (source of truth + deploy runbook). Single bind account `cn=serviceaccount,dc=zb,dc=local` / `serviceaccount123`; all test users password `password`. The docker-dev compose binds this shared instance directly — `DevStubMode` is no longer used. The per-VM NSSM GLAuth at `C:\publish\glauth\` and the old base DNs `dc=lmxopcua,dc=local` / `dc=otopcua,dc=local` are obsolete. (The integration-test harness under `tests/.../Host.IntegrationTests/` uses a separate ephemeral bitnami/openldap on port 3894 for automated tests — that is distinct from the shared dev GLAuth.)
Dev/test LDAP is the **shared GLAuth** running on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`, plaintext/`Transport=None`). It is managed via `scadaproj/infra/glauth/` (source of truth + deploy runbook). Single bind account `cn=serviceaccount,dc=zb,dc=local` / `serviceaccount123`; all test users password `password`. The docker-dev compose binds this shared instance directly — `DevStubMode` is no longer used. The per-VM NSSM GLAuth at `C:\publish\glauth\` and the old base DNs `dc=lmxopcua,dc=local` / `dc=otopcua,dc=local` are obsolete. (The integration-test harness under `tests/.../Host.IntegrationTests/` runs its own ephemeral **GLAuth** container on port 3894 — seeded by `tests/.../Host.IntegrationTests/glauth/config.toml` with dev users alice/bob + the `cn=serviceaccount` bind account — for the opt-in `OTOPCUA_HARNESS_USE_LDAP=1` real-bind mode; distinct from the shared dev GLAuth on 3893. It replaced the retired `bitnami/openldap:2.6` in PR #451, 2026-07-15.)
## Library Preferences
@@ -319,11 +326,12 @@ When the section is disabled the applier binds the no-op `NullHistorianProvision
run. Every dispatch logs a tally (`dispatched/requested/ensured/skipped/failed`); `dispatched=N` with
`requested=0` flags the dormant no-op. The API key must carry `historian:tags:write` for `EnsureTags`. (PR
#423 shipped `GatewayTagProvisioner` but left this wiring out, so provisioning was dormant; restored 2026-06-27.)
**0.2.0 DataType delta (archreview 06/U-7):** `EnsureTags` is an **upsert** (create-or-update, not
skip-if-exists), and the 0.2.0 `HistorianTagDefinition.DataType` proto3-optional change means Boolean tags now
provision as **Int1** (0.1.0 silently defaulted them to Float). A pre-bump Float Boolean is asked to retype
on its next deploy; the AVEVA-side retype policy is pinned by the live retype probe — see the "0.2.0 DataType
provisioning delta" section in `docs/Historian.md`.
**DataType mapping (issue #441):** `EnsureTags` is an **upsert** (create-or-update, not skip-if-exists).
`HistorianTypeMapper` maps **`Boolean → Int2`** (Boolean historizes as a small 0/1 integer) because the
historian's `Int1` analog-tag creation path is **server-degenerate**`EnsureTags(Int1)` stores an unusable
stub (live-probed; gateway-side resolution in HistorianGateway #11 / PR #16). Both Float and Int2 are
supported analog types, so a pre-existing Float Boolean tag retypes cleanly to Int2 on its next deploy — see
the "Boolean historization maps to Int2" section in `docs/Historian.md`.
### Gateway-side prerequisites
@@ -358,7 +366,7 @@ export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222 # absolute gateway
export HISTGW_GATEWAY_APIKEY=histgw_<id>_<secret> # must carry historian:read + historian:write (+ tags:write) scopes
export HISTGW_TEST_TAG=<existing-tag> # read round-trip
export HISTGW_WRITE_SANDBOX_TAG=<writable-float-tag> # e.g. HistGW.LiveTest.Sandbox — write round-trip (EnsureTags + write)
export HISTGW_BOOL_SANDBOX_TAG=<writable-bool-tag> # e.g. HistGW.LiveTest.BoolSandbox — 0.2.0 Boolean EnsureTags leg + retype probe (06/U-7)
export HISTGW_BOOL_SANDBOX_TAG=<writable-bool-tag> # e.g. HistGW.LiveTest.BoolSandbox — Boolean→Int2 EnsureTags leg + retype probe (issue #441)
export HISTGW_ALARM_SOURCE=<source-name> # alarm SendEvent → ReadEvents round-trip
dotnet test --filter "Category=LiveIntegration"
```
-15
View File
@@ -16,21 +16,6 @@
<LangVersion>latest</LangVersion>
</PropertyGroup>
<!--
NuGet audit suppression: CVE-2025-6965 / GHSA-2m69-gcr7-jv3q affects the transitive native
bundle SQLitePCLRaw.lib.e_sqlite3 (advisory range <= 2.1.11), pulled in via
Microsoft.Data.Sqlite 9.0.0 -> SQLitePCLRaw.bundle_e_sqlite3 by Core.AlarmHistorian's
store-and-forward sink. As of 2026-06-20 the maintainer has NOT shipped a patched bundle
(SQLite fixed the underlying flaw upstream in 3.50.2 but no SQLitePCLRaw release embeds it
yet), so there is no package version to bump to. The flaw requires attacker-controlled
pathological aggregate SQL; the sink only issues internal parameterised statements, so the
exposure here is negligible. REMOVE this suppression once SQLitePCLRaw ships a patched native
bundle and bump Microsoft.Data.Sqlite to a release that references it.
-->
<ItemGroup>
<NuGetAuditSuppress Include="https://github.com/advisories/GHSA-2m69-gcr7-jv3q" />
</ItemGroup>
<!--
Wire the custom OTOPCUA0001 analyzer (UnwrappedCapabilityCallAnalyzer) into every project so
the CapabilityInvoker-wrapping rule is enforced tree-wide, not just against the analyzer's own
+9 -1
View File
@@ -51,7 +51,7 @@
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp.Scripting" Version="4.12.0" />
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="5.0.0" />
<PackageVersion Include="Microsoft.Data.SqlClient" Version="6.1.1" />
<PackageVersion Include="Microsoft.Data.Sqlite" Version="9.0.0" />
<PackageVersion Include="Microsoft.Data.Sqlite" Version="10.0.7" />
<PackageVersion Include="Microsoft.EntityFrameworkCore" Version="10.0.7" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.Design" Version="10.0.7" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.InMemory" Version="10.0.7" />
@@ -95,6 +95,14 @@
<PackageVersion Include="Serilog.Sinks.Console" Version="6.0.0" />
<PackageVersion Include="Serilog.Sinks.File" Version="7.0.0" />
<PackageVersion Include="Shouldly" Version="4.3.0" />
<!--
Surgical transitive pin: Microsoft.Data.Sqlite pulls SQLitePCLRaw.bundle_e_sqlite3 2.1.11
(native lib.e_sqlite3 2.1.11 — the top of the CVE-2025-6965 / GHSA-2m69-gcr7-jv3q vulnerable
range, <= 2.1.11). 2.1.12 is the first fixed native bundle (embeds the SQLite 3.50.2+ patch).
A direct PackageReference in Core.AlarmHistorian promotes this to the resolved version,
clearing the advisory without CentralPackageTransitivePinningEnabled (which breaks the Roslyn split).
-->
<PackageVersion Include="SQLitePCLRaw.bundle_e_sqlite3" Version="2.1.12" />
<PackageVersion Include="System.CommandLine" Version="2.0.5" />
<PackageVersion Include="System.Data.SqlClient" Version="4.9.0" />
<PackageVersion Include="System.IdentityModel.Tokens.Jwt" Version="8.11.0" />
+7 -1
View File
@@ -172,7 +172,13 @@ are the new STAB-14..17 below.
> `when (token.IsCancellationRequested)` filters on the read/write ensure-wrappers (`:488`/`:1000`) and all
> three poll-loop OCE catches — so a non-teardown OCE degrades/backs-off instead of killing the loop. Unit
> guards T1T4 (`S7DriverReconnectTests`, incl. the poll-loop-survival regression). Live gate
> `S7_1500ConnectTimeoutOutageTests` (env-gated SYN-blackhole, `deferred-live`).
> `S7_1500ConnectTimeoutOutageTests` **CLOSED 2026-07-15** — running it (blackhole via `docker pause`)
> exposed that the fix above covered only the *connect* leg: S7.Net's socket `ReadTimeout`/`WriteTimeout`
> are ignored by its async read/write paths, so a read on an established-but-frozen peer blocked for
> minutes and wedged the poll loop. Fixed (PR #453, master `4dc937fc`) by a new `S7OperationDeadline`
> wall-clock ceiling on every data-plane wire op (`S7PlcAdapter` → `TimeoutException`) + `IsS7ConnectionFatal`
> now treating `TimeoutException` as fatal → mark-dead → reopen (the READ-leg sibling of STAB-14). Live gate
> now GREEN; Driver.S7.Tests 246→260.
**STAB-14 — High — NEW — S7's reconnect wrapper rethrows connect-timeout
cancellation, and the poll loop's bare OCE catch then permanently kills the
@@ -0,0 +1,226 @@
# Integration-test sweep — status + follow-ups (2026-07-14)
**Context:** ran the deferred `*.IntegrationTests` sweep (the round-2 STATUS.md "Full integration-suite
sweep" item that discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24).
**How:** tests run **on the Mac** (repo + .NET 10 SDK), pointing at **driver fixtures brought up on the
docker host `10.100.0.35`**, one driver at a time (serial — the suites have a ~16 GB/run memory leak).
Master at run time: `30672888` (branch `master`, clean).
**Docker host `10.100.0.35` (`DOCKER`) note:** has `dotnet 10.0.300`, 22 cores, but only ~7 GB free RAM
(the rest is shared SQL/GLAuth/fixtures) and **no repo checkout** — so the tests can't run *on* it (OOM risk
to shared infra). Run on the Mac against its fixtures. Fixture dirs live at `/opt/otopcua-<driver>/` (owned,
need sudo to create new ones — I deployed FOCAS to `~/otopcua-focas` on the host instead).
---
## Results
| Suite | Result | Root cause of failures |
|---|---|---|
| **OpcUaClient** | ✅ 4/4 | — (opc-plc `:50000`) |
| **OpcUaClient.Browser** | ✅ 1/1 | — |
| **S7** | ✅ 3/3 (2 skip) | skips = outage tests needing `S7_TIMEOUT_OUTAGE_*` env |
| **Modbus** (profile=standard) | ✅ 6/6 | — |
| **Host** | ⚠️ **105/117** (7F, 5 skip) | 7 heavy E2E time out under amd64-emulated SQL (OPEN); LDAP image swapped OpenLDAP→GLAuth 2026-07-15 (PR #451). The 5 skips are driver-fixture-gated E2E, not LDAP |
| **TwinCAT** | ⏭️ 13 skip | no TC3 XAR fixture exists (expected clean-skip) |
| **Modbus** (dl205 / mitsubishi / s7_1500 / exception_injection) | ✅ 16/13/10/17 (was 8F/5F/3F/8F) | **NOT drift** — fixture-cycling artifact, re-verified GREEN 2026-07-15 — see below |
| **AbCip** | ✅ 10P/3skip (was 6F/1P/3skip) | static-init-order bug FIXED 2026-07-15 (`326b22a7`) — see below |
| **FOCAS** | ✅ 10/10 (was 9F/1P) | topology + mock-timer-endianness FIXED 2026-07-15 — see below |
| **AbLegacy** | ✅ slc500 2/2, micrologix 1/1, plc5 1/1 (was 4F) | fixture-deployment gap FIXED 2026-07-15 — see below |
| **OpcUaServer** | ✅ 4/4 (was 4F) | cert-store gap + SDK read-throws drift FIXED 2026-07-15 — see below |
### Headline conclusion
**No OtOpcUa production regression.** Every failure is **pre-existing test/fixture infrastructure rot** in
long-deferred suites — or, in the Modbus case, a **fixture-cycling false alarm** (see below). The drivers are
fine: 4 suites fully green, **all Modbus profiles green** (standard + dl205/mitsubishi/s7_1500/exception once
correctly bound — the sweep's non-standard "failures" were the stale-container artifact), Host 105/117 green,
and all driver **unit** suites + the live `/run` gates pass. These integration suites were DEFERRED in round 2
and never run, so they'd bit-rotted in several distinct ways.
**Follow-up progress (2026-07-15): ALL 7 CLOSED.** #1 AbCip (PR #446), #5 OpcUaServer (PR #447), #2 Modbus
(PR #448 — false alarm), #4 AbLegacy (PR #449 — fixture gap), #3 FOCAS (PR #450 — topology + mock-timer), #7
(fixture deploys), #6 LDAP-image (PR #451 — OpenLDAP→GLAuth), and #6 SQL-deploy-E2E (PR #452 — misdiagnosed
"emulation timing"; really missing FK-parent seeds + a stale EquipmentId, hidden by the in-memory provider).
**Every driver + server integration suite is now verified GREEN. Zero OtOpcUa production regressions across the
entire sweep** — every failure was fixture/harness/test rot.
---
## Per-suite failure detail (evidence)
- **Modbus non-standard profiles** — ✅ **RE-VERIFIED GREEN 2026-07-15 (PR #448). The sweep's 8F/5F/3F/8F was a
false alarm — a fixture-cycling artifact, NOT pymodbus drift and NOT an OtOpcUa regression.** Correcting the
sweep's original diagnosis:
- The image genuinely runs **pymodbus 3.13.0** (`docker run --entrypoint python ... -c 'import pymodbus;
print(pymodbus.__version__)'` → `3.13.0`; the Dockerfile already pins `pymodbus[simulator]==3.13.0`). The
*"…removed in v4"* log line is 3.13.0's **forward-deprecation warning**, not evidence it runs 4.x — the
original sweep note misread it.
- Root cause: each Modbus profile is a **separate profile-gated compose service** sharing `:5020`. A plain
`docker compose down` (no `--profile`) does **not** stop the running profile container, so it keeps holding
`:5020`; the next `--profile … up` container silently fails to bind (compose still says "Started"). Every
later profile run then hit the **stale** container. Proven by a raw Modbus probe: the "mitsubishi" sim
served DL205's `0xCAFE` at HR0 (DL205's V0 marker) and Illegal-Data-Address for mitsubishi-only addresses,
while `docker ps` showed the mitsubishi container up with **no** port mapping and DL205 still holding `:5020`.
- After force-removing all modbus containers and bringing up each profile cleanly (verifying it actually
published `:5020`), all four pass: **dl205 16/16, mitsubishi 13/13, s7_1500 10/10, exception_injection
17/17** (the injector needs `--force-recreate` — it uses a different container name). **No code change** —
added a profile-cycling gotcha + reliable-cycle recipe to `Docker/README.md`.
- **AbCip** — **static-init-order bug in the test harness** (always-fail, blocks the whole suite).
`tests/.../AbCip.IntegrationTests/AbServerProfile.cs`: `KnownProfiles.All { get; } = [ControlLogix, …]` is
declared at **line 30**, *before* the `static readonly` profile fields (line 34+). C# runs static
initializers in textual order → `All` captures `[null,null,null,null]` → `ForFamily` NREs at `p.Family`
(line 61) and `AbServerFixture..ctor` gets null. **Fix (trivial, safe):** make `All` an expression-bodied
property: `public static IReadOnlyList<AbServerProfile> All => [ControlLogix, CompactLogix, Micro800,
GuardLogix];` (evaluated on access, after the fields are set). Unblocks the suite; then re-run to see if the
actual sim tests pass.
- **FOCAS** — ✅ **FIXED 2026-07-15 (PR #450). Two test/fixture bugs, no OtOpcUa driver change.**
1. *Topology:* `FocasSimFixture` already reads an env-overridable endpoint (`OTOPCUA_FOCAS_SIM_ENDPOINT`,
default `localhost:8193`) and exposes `Host`/`Port`, but `WireBackendTests` + `WireBackendCoverageTests`
**hardcoded** `focas://127.0.0.1:8193`. Against a remote fixture the skip-gate passed (remote reachable) but
the driver dialed localhost → KeyNotFound. Fix: added a `DeviceUri` property to the fixture and derived each
test's `DeviceHost` from it — recovered **8 of 9**.
2. *Mock timer endianness (newly surfaced):* the last failure (`Timers_populate`) read `60397977600` for a
`3600 s` power-on timer (= `0x3C000000 × 60`). Root cause: the focas-mock's `_wire_timer` encoded the
`minute`/`msec` fields **big-endian** (`_u32`), but `cnc_rdtimer`'s timer fields are **little-endian** on
real hardware — a documented, live-31i-B-validated quirk (`FocasWireClient.ParseTimer` decodes them LE, per
`docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md`). **The driver was correct; the mock was wrong.** Fix:
`_wire_timer` now emits little-endian (`_u32_le`) → suite **10/10**. Fixture deployed at `~/otopcua-focas`.
- **AbLegacy** — ✅ **FIXED 2026-07-15 (PR #449). Fixture-deployment gap, not a code defect.** The AbLegacy suite
needs `ab_server` in **PCCC mode** (`--plc=SLC500|Micrologix|PLC/5`), which reuses the **AbCip image**
(`otopcua-ab-server:libplctag-release`, already on the host) and binds `:44818`. Two issues: (a) no PCCC
container was deployed (`~/otopcua-ablegacy` didn't exist); (b) the sweep's *4F-not-skip* means `:44818` was
reachable but answering **CIP** (a leftover AbCip container) — the PCCC reads returned Bad. Same shared-port
contamination class as the Modbus false alarm. Deployed the compose to `~/otopcua-ablegacy` on the host (image
already present, no build needed) and ran each family with `AB_LEGACY_COMPOSE_PROFILE` set to match the running
container (**required** — without it the `[Theory]` parametrises over all 3 families and 2 fail against a
single-family container): **slc500 2/2, micrologix 1/1 (+1 skip), plc5 1/1 (+1 skip)** — the skips are
family-inapplicable file types (e.g. no `F8` float file seeded on micrologix/plc5). **No code change.**
- **OpcUaServer** — ✅ **FIXED 2026-07-15** (PR #447). Was 4F on `Opc.Ua.ServiceResultException :
TrustedIssuerCertificates StorePath must be specified` — the in-test *client* `SecurityConfiguration`
omitted the trusted-issuer/peer/rejected store paths that `ValidateAsync` requires. Fix: `TestClientSecurity`
helper rooting Directory stores at a throwaway PKI dir (mirrors `DefaultApplicationConfigurationFactory`).
Fixing it unmasked a second drift — the 1.5.378 node-cache read path throws
`ServiceResultException(BadNodeIdUnknown)` for a removed node rather than returning a bad `DataValue`;
the two subscription-survival assertions now use `Should.ThrowAsync`. Suite **4F → 4/4 green**.
- **Host** — largely healthy (105/117). Two independent issues:
- **LDAP image — ✅ FIXED 2026-07-15 (PR #451).** The opt-in real-LDAP mode (`OTOPCUA_HARNESS_USE_LDAP=1`)
used `bitnami/openldap:2.6`, gone since Bitnami deprecated their free catalog in 2025. **Replaced it with
GLAuth** — the same LDAP server the rest of the project uses (docker-dev + the live OPC UA data-plane auth,
both against the shared GLAuth on `:3893`), removing the lone OpenLDAP outlier. Added a harness
`glauth/config.toml` (base DN `dc=zb,dc=local`, search-capable `serviceaccount`, dev users alice=full /
bob=read-only, role groups with the same gidnumbers as `scadaproj/infra/glauth`) + repointed the harness's
service-account override to `cn=serviceaccount`. Live-verified all three bind legs against a deployed
container on `:3894` (serviceaccount search returns alice + 5 memberOf groups; alice/bob bind OK; wrong
password → `Invalid credentials (49)`). NB: the real-LDAP mode is **opt-in** — the default Host run uses
`StubLdapAuthService`, so this never blocked the default suite (the sweep's "5 LDAP skips" were actually the
driver-fixture-gated E2E tests, not LDAP).
- **SQL-backed deploy E2E — ✅ FIXED 2026-07-15 (PR #452). The sweep's "amd64-emulated-SQL timing" diagnosis
was WRONG.** Running the 7 heavy E2E against real (native-amd64) SQL on the Docker host proved they don't
time out for speed — they had **never actually run against real SQL**, only the EF in-memory provider (which
ignores FK constraints). Two distinct root causes, both hidden by in-memory:
- *Missing FK-parent seed (5 tests):* the deploy writes a `NodeDeploymentState` row per node, which FKs to
`ClusterNode` (`FK_NodeDeploymentState_ClusterNode_NodeId`). The tests never seeded `ServerCluster` +
`ClusterNode` rows, so on SQL each node-state INSERT throws → the deploy never seals (waits the full
deadline regardless of how high — proven at 4× it still hung). Added `TwoNodeClusterHarness.SeedDefaultClusterAsync`
(seeds a `ServerCluster` + a `ClusterNode` per node; `Warm`/NodeCount=2 to satisfy both the SQL CHECK
constraint `CK_ServerCluster_RedundancyMode_NodeCount` and the `ClusterEnabledNodeCountMismatch` validator)
and called it in `DeployHappyPathTests`×2 / `FailoverDuringDeployTests` / `FleetDiagnosticsRoundTripTests` /
`EquipmentNamespaceMaterializationTests`; fixed `DriverReconnectE2eTests` to seed node B + declare NodeCount=2.
- *Stale `EquipmentId` (1 test, pre-existing on BOTH providers):* `EquipmentNamespaceMaterializationTests`
seeded `EquipmentId="eq-1"`, but the `EquipmentIdNotDerived` validator (added later) requires
`EquipmentId == DeriveEquipmentId(EquipmentUuid)` = `EQ-` + first 12 hex of the UUID. Fixed the seed to a
canonical id + matching UUID. (Was failing on in-memory too; only surfaced once the FK fix let the deploy
reach equipment validation.)
- Added `OTOPCUA_HARNESS_SQL_HOST` / `OTOPCUA_HARNESS_LDAP_HOST` env overrides so the fixtures can live on the
native-amd64 Docker host (`docker compose up -d sql` there; run with `OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331`).
- Result: the 5 SQL-backed classes go **11/11 green vs native SQL**, and the in-memory default is unregressed
(12/12). `RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts…` is a **pre-existing flaky
race test** (not SQL-backed; passes in isolation, flaked once under heavy parallel load) — left as-is.
- **TwinCAT** — 13 skip cleanly (no TC3 XAR fixture exists; unit-only per the driver's design). Expected.
---
## Follow-up backlog (distinct, not one fix)
1. ~~**AbCip static-init bug** — trivial always-fail test bug. Expression-body `KnownProfiles.All`.~~
✅ **DONE 2026-07-15** (commit `326b22a7`, PR #446). Live-verified against the controllogix fixture:
suite went **6F/1P/3skip → 10 pass / 0 fail / 3 skip** (skips = emulator-mode tests needing a different
fixture mode).
2. ~~**Modbus non-standard profiles** — pin pymodbus in the fixture Dockerfile.~~
✅ **RESOLVED 2026-07-15 (PR #448) — false alarm, no code change.** Image already runs the pinned
pymodbus 3.13.0; the 8F/5F/3F/8F was a fixture-cycling artifact (stale profile container holding `:5020`).
Re-verified all four profiles GREEN (16/13/10/17). Added a profile-cycling gotcha + reliable-cycle recipe
to `Docker/README.md`.
3. ~~**FOCAS** — reconcile the 127.0.0.1-mock topology with the remote-fixture model.~~
✅ **DONE 2026-07-15 (PR #450).** Tests now derive `DeviceHost` from the fixture endpoint (recovered 8/9);
also fixed a newly-surfaced mock bug — `_wire_timer` encoded the timer big-endian but real hardware +
driver use little-endian. Suite **10/10**. No OtOpcUa driver change.
4. ~~**AbLegacy** — deploy an AbLegacy sim fixture (none on host).~~
✅ **DONE 2026-07-15 (PR #449) — fixture-deployment gap, no code defect.** Deployed the PCCC compose to
`~/otopcua-ablegacy` (reuses the AbCip image, already on host). All 3 families GREEN (slc500 2/2, micrologix
1/1, plc5 1/1). Recipe added below; `AB_LEGACY_COMPOSE_PROFILE` MUST match the running container.
5. ~~**OpcUaServer** — supply cert-store StorePath in the harness config.~~
✅ **DONE 2026-07-15** (PR #447). Added `TestClientSecurity` helper building a validated client
`SecurityConfiguration` with Directory own/issuer/trusted/rejected stores under a throwaway PKI dir
(mirrors `DefaultApplicationConfigurationFactory`); wired into both `DualEndpointTests` +
`SubscriptionSurvivalTests`. That unmasked a **second** drift: the 1.5.378 node-cache read path
*throws* `ServiceResultException(BadNodeIdUnknown)` for a removed node instead of returning a bad
`DataValue` — updated the two subscription-survival assertions to `Should.ThrowAsync`. Suite **4F → 4/4**.
6. **Host** — ⬐ split into two:
- ~~replace `bitnami/openldap:2.6` (image gone)~~ ✅ **DONE 2026-07-15 (PR #451)** — swapped the harness LDAP
to **GLAuth** (unifies on the LDAP the rest of the project uses); live-verified the binds on `:3894`.
- ~~**amd64-emulated-SQL timing**~~ ✅ **DONE 2026-07-15 (PR #452) — misdiagnosed.** Not a timing/emulation
issue: the deploy E2E tests had never run against real SQL (in-memory ignores FKs) and lacked the
`ServerCluster`/`ClusterNode` seed the `NodeDeploymentState` FK requires → deploy never seals. Added a
harness seeding helper + `OTOPCUA_HARNESS_SQL_HOST` override; also fixed a stale `EquipmentId` in one test.
11/11 green vs native SQL, in-memory unregressed. (Roslyn racing test = pre-existing flaky, left as-is.)
7. ~~**Fixture deployment gap** — FOCAS + AbLegacy fixtures aren't on the host.~~
✅ **RESOLVED 2026-07-15.** Both now deployed under the home dir: `~/otopcua-ablegacy` (PR #449) +
`~/otopcua-focas` (PR #450). (`/opt` still needs sudo; the home-dir deploys work fine.)
---
## Fixture setup recipes (to re-run the sweep)
Bring up on the docker host (distinct ports, can coexist except Modbus profiles share `:5020`):
```bash
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-modbus && docker compose --profile standard up -d' # :5020
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-abcip && docker compose --profile controllogix up -d' # :44818
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d' # :1102
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-opcuaclient && docker compose up -d' # :50000
ssh dohertj2@10.100.0.35 'cd ~/otopcua-ablegacy && docker compose --profile slc500 up -d' # :44818 (PCCC; reuses AbCip image)
ssh dohertj2@10.100.0.35 'cd ~/otopcua-focas && docker compose up -d --build' # :8193
```
Run per suite (env → project):
| Suite | Env | Project |
|---|---|---|
| OpcUaClient(+Browser) | `OPCUA_SIM_ENDPOINT=opc.tcp://10.100.0.35:50000` | `tests/Drivers/…OpcUaClient(.Browser).IntegrationTests` |
| S7 | `S7_SIM_ENDPOINT=10.100.0.35:1102` | `tests/Drivers/…S7.IntegrationTests` |
| Modbus | `MODBUS_SIM_ENDPOINT=10.100.0.35:5020` `MODBUS_SIM_PROFILE=<standard\|dl205\|mitsubishi\|s7_1500\|exception_injection>` (swap the host sim to match) | `tests/Drivers/…Modbus.IntegrationTests` |
| AbCip | `AB_SERVER_ENDPOINT=10.100.0.35:44818` `AB_SERVER_PROFILE=controllogix` | `tests/Drivers/…AbCip.IntegrationTests` |
| AbLegacy | `AB_LEGACY_ENDPOINT=10.100.0.35:44818` `AB_LEGACY_COMPOSE_PROFILE=<slc500\|micrologix\|plc5>` (**must** match the running PCCC container — else the `[Theory]` runs all 3 families and 2 fail) | `tests/Drivers/…AbLegacy.IntegrationTests` |
| FOCAS | `OTOPCUA_FOCAS_SIM_ENDPOINT=10.100.0.35:8193` (tests now honour this — no longer hardcoded to 127.0.0.1) | `tests/Drivers/…FOCAS.IntegrationTests` |
| Host | `OTOPCUA_HARNESS_USE_SQL=1` (+ SQL `:14331` via the harness `docker-compose.yml up sql`) — set `OTOPCUA_HARNESS_USE_LDAP=1` to exercise real binds against the **GLAuth** service (`up ldap`, host `:3894`; users alice/bob, svc `cn=serviceaccount`/`serviceaccount123`) | `tests/Server/…Host.IntegrationTests` |
**Modbus profile cycling:** the sim services share `:5020`, so to run a profile:
`ssh … 'cd /opt/otopcua-modbus && docker compose down && docker compose --profile <p> up -d'`, then run with
`MODBUS_SIM_PROFILE=<p>` (each class skips unless its profile is active). Wait for `:5020` before running —
TCP opens before the register map fully loads.
---
## Current state (post-sweep)
- **All fixtures torn down** — modbus/abcip/s7/opcuaclient/focas on the host removed (ports 5020/44818/1102/
50000/8193 all closed); the local Host SQL fixture `down -v`'d. Shared host is clean.
- **Repo:** `master @ 30672888`, clean. No sweep changes committed (this file is the only artifact).
- **Nothing merged for the sweep** — it's a read-only test run; the follow-ups above are unstarted.
@@ -2,20 +2,122 @@
"planPath": "archreview/plans/R2-04-failure-visibility-plan.md",
"lastUpdated": "2026-07-12",
"tasks": [
{ "id": "T1", "subject": "Extend AddressSpaceApplyOutcome with RebuildFailed/FailedNodes + SafeRebuild returns bool (01/S-1)", "status": "completed", "blockedBy": [] },
{ "id": "T2", "subject": "Safe* helpers return bool; Apply tallies removal-pass failures into FailedNodes", "status": "completed", "blockedBy": ["T1"] },
{ "id": "T3", "subject": "Materialise* passes return swallowed-failure counts (void -> int, five methods)", "status": "completed", "blockedBy": ["T2"] },
{ "id": "T4", "subject": "OpcUaApplyFailed counter + OpcUaPublishActor Error/meter surfacing of degraded applies", "status": "completed", "blockedBy": ["T3"] },
{ "id": "T5", "subject": "Galaxy internal IMxWriteOps seam under GatewayGalaxyDataWriter (sealed SDK session untestable) (06/S-1)", "status": "completed", "blockedBy": [] },
{ "id": "T6", "subject": "Galaxy fail-closed write on AdviseSupervisory failure -> BadCommunicationError, no raw write", "status": "completed", "blockedBy": ["T5"] },
{ "id": "T7", "subject": "Galaxy counters galaxy.writes.advise_failed + galaxy.writes.unconfirmed (empty-statuses Good metered)", "status": "completed", "blockedBy": ["T6"] },
{ "id": "T8", "subject": "Galaxy skip-gated live smoke: normal advised write still returns Good post-change", "status": "completed", "blockedBy": ["T6"] },
{ "id": "T9", "subject": "PrimaryGatePolicy pure class + truth-table tests + otopcua.redundancy.primary_gate_denied counter (03/S4)", "status": "completed", "blockedBy": [] },
{ "id": "T10", "subject": "Wire DriverHostActor gates (:977/:1039/:1095) through PrimaryGatePolicy with member-count provider + S5 log-once", "status": "completed", "blockedBy": ["T9"] },
{ "id": "T11", "subject": "DriverHostActorPrimaryGateTests: deny-on-unknown multi-node, allow single-node, ack/emit variants, meter tags", "status": "completed", "blockedBy": ["T10"] },
{ "id": "T12", "subject": "ScriptedAlarmHostActor alerts-emit gate adopts PrimaryGatePolicy (consistency extension)", "status": "completed", "blockedBy": ["T9"] },
{ "id": "T13", "subject": "In-process 2-node delivery-path test (TwoNodeClusterHarness): DPS-delivered snapshot drives the gate", "status": "deferred-live", "note": "rig/live; Host.IntegrationTests (~16GB/run leak + needs SQL/2-node rig) — implemented + build-verified, controller runs it in the serialized heavy pass", "blockedBy": ["T11"] },
{ "id": "T14", "subject": "docs/Redundancy.md: gate policy table, boot-window client experience, new meters, Galaxy fail-closed semantics", "status": "completed", "blockedBy": [] },
{ "id": "T15", "subject": "LIVE GATE: 2-node docker-dev rig verify (ServiceLevel 240/100, boot-window write rejected+reverted, single /alerts row)", "status": "deferred-live", "note": "rig/live; 2-node docker-dev rig session (rebuild both centrals) — merge-blocking gate the controller runs in the serialized heavy pass", "blockedBy": ["T10", "T11", "T12", "T13"] }
{
"id": "T1",
"subject": "Extend AddressSpaceApplyOutcome with RebuildFailed/FailedNodes + SafeRebuild returns bool (01/S-1)",
"status": "completed",
"blockedBy": []
},
{
"id": "T2",
"subject": "Safe* helpers return bool; Apply tallies removal-pass failures into FailedNodes",
"status": "completed",
"blockedBy": [
"T1"
]
},
{
"id": "T3",
"subject": "Materialise* passes return swallowed-failure counts (void -> int, five methods)",
"status": "completed",
"blockedBy": [
"T2"
]
},
{
"id": "T4",
"subject": "OpcUaApplyFailed counter + OpcUaPublishActor Error/meter surfacing of degraded applies",
"status": "completed",
"blockedBy": [
"T3"
]
},
{
"id": "T5",
"subject": "Galaxy internal IMxWriteOps seam under GatewayGalaxyDataWriter (sealed SDK session untestable) (06/S-1)",
"status": "completed",
"blockedBy": []
},
{
"id": "T6",
"subject": "Galaxy fail-closed write on AdviseSupervisory failure -> BadCommunicationError, no raw write",
"status": "completed",
"blockedBy": [
"T5"
]
},
{
"id": "T7",
"subject": "Galaxy counters galaxy.writes.advise_failed + galaxy.writes.unconfirmed (empty-statuses Good metered)",
"status": "completed",
"blockedBy": [
"T6"
]
},
{
"id": "T8",
"subject": "Galaxy skip-gated live smoke: normal advised write still returns Good post-change",
"status": "completed",
"blockedBy": [
"T6"
]
},
{
"id": "T9",
"subject": "PrimaryGatePolicy pure class + truth-table tests + otopcua.redundancy.primary_gate_denied counter (03/S4)",
"status": "completed",
"blockedBy": []
},
{
"id": "T10",
"subject": "Wire DriverHostActor gates (:977/:1039/:1095) through PrimaryGatePolicy with member-count provider + S5 log-once",
"status": "completed",
"blockedBy": [
"T9"
]
},
{
"id": "T11",
"subject": "DriverHostActorPrimaryGateTests: deny-on-unknown multi-node, allow single-node, ack/emit variants, meter tags",
"status": "completed",
"blockedBy": [
"T10"
]
},
{
"id": "T12",
"subject": "ScriptedAlarmHostActor alerts-emit gate adopts PrimaryGatePolicy (consistency extension)",
"status": "completed",
"blockedBy": [
"T9"
]
},
{
"id": "T13",
"subject": "In-process 2-node delivery-path test (TwoNodeClusterHarness): DPS-delivered snapshot drives the gate",
"status": "completed",
"note": "DONE 2026-07-14: PrimaryGateFailoverTests PASS 1/1 (3s), ran without SQL (gate test needs only the in-process 2-node cluster + DPS snapshot). Delivered snapshot promotes exactly one Primary; negative-control (broken subscribe) times out. See STATUS.md.",
"blockedBy": [
"T11"
]
},
{
"id": "T14",
"subject": "docs/Redundancy.md: gate policy table, boot-window client experience, new meters, Galaxy fail-closed semantics",
"status": "completed",
"blockedBy": []
},
{
"id": "T15",
"subject": "LIVE GATE: 2-node docker-dev rig verify (ServiceLevel 240/100, boot-window write rejected+reverted, single /alerts row)",
"status": "completed",
"note": "DONE 2026-07-14: docker-dev 2-node rig rebuilt from master 365a90e6 + Modbus sim up. ServiceLevel 250/240; boot-window write rejected \"not primary (role unknown)\" (meter reason=role-unknown,site=write); steady-state secondary rejected \"not primary\" + node reverted (read-back 1234 not 9999, meter reason=secondary); primary write reaches device; clean apply (no apply.failed). Full record in STATUS.md.",
"blockedBy": [
"T10",
"T11",
"T12",
"T13"
]
}
]
}
}
+93 -24
View File
@@ -171,12 +171,12 @@ what the unit leg cannot; (c) migrate the 3 xunit v2 holdouts once Akka ships an
|---|---|---|---|---|
| **R2-06** | 06/S-11 (rank 7), 06/U-7 (rank 8), 06/C-7 | `r2/06-serverhistorian-failfast` (off `1676c8f4`) | `3a049a13``4640ecb1` (10 task commits) | **Code-complete, offline-verified.** S-11: fail-fast `ServerHistorianOptionsValidator` (consumer-gated on `ServerHistorian:Enabled` OR `AlarmHistorian:Enabled`, fails empty/non-http(s) Endpoint at startup) + `AddValidatedOptions` wiring + adapter defense-in-depth (`TryCreate`→named `InvalidOperationException`, no more raw `UriFormatException` crash-loop) + `Validate()` warning parity + alarm-only-mode startup warning. U-7: pinned Boolean→Int1 explicit-presence (`HasDataType`) + `HISTGW_BOOL_SANDBOX_TAG` live Boolean EnsureTags leg + retype probe (skip-clean offline) + upsert/retype migration note (`docs/Historian.md`). C-7: keep-populating decision + dead-`Quality` doc comments (0.2.0 C-002). **Live leg operator-gated (T12) — pending a VPN run on a 0.2.0 gateway** (records the observed in-place retype policy back into `docs/Historian.md`; also discharges 06/U-2's full documented run). |
| **R2-05** | 04/C-1 (High) | `r2/05-adminui-authz` (off `1676c8f4`) | `183b72b7``6c1f05cb` (5 commits) | **Code-complete, offline-verified.** All 38 routable AdminUI pages carry an explicit named-policy `@attribute` via new `AdminUiPolicies` constants (`Security/Auth/`). New `ConfigEditor` (Administrator+Designer) gates the 20-page config-authoring surface incl. the 8 driver pages that author live `ResilienceConfig` + `/api/script-analysis/*`; new `AuthenticatedRead` on the 16 read pages; `FleetAdmin`/`DriverOperator` unchanged (literals→constants). Anti-drift guard `PageAuthorizationGuardTests` (bans bare `[Authorize]`/`Roles=`/unknown-policy/unclassified) + `AdminUiPoliciesTests` semantics matrix. Corrections folded in: Reservations + ClusterRedundancy read-only (→`AuthenticatedRead`); Home had no attribute (now `AuthenticatedRead`). AdminUI.Tests 517/517, Security.Tests 80/80. **Live `/run` positive pass deferred (T15)** — docker-dev `DisableLogin=true` auto-admin can't observe deny; negative proof is CI-side (both guard tests). |
| **R2-01** | 05/STAB-14 (High regression, rank 1), 05/STAB-15 (rank 3), 05/STAB-8 seam | `r2/01-s7-fault-hardening` (off `1676c8f4`) | `fcdcf0e2``bdb7ed41` (13 task commits) | **Code-complete, offline-verified.** All prod changes in `S7Driver.cs`. STAB-14: connect-timeout surfaces as `TimeoutException` (not an escaping OCE) at the `EnsureConnectedAsync`/`InitializeAsync` source + `when(token.IsCancellationRequested)` filters on the read/write ensure-wrappers and all three poll-loop OCE catches — subscription poll loops survive an unreachable-host outage and recover (T2 regression RED→GREEN). STAB-15: `IsS7ConnectionFatal` broadened to the ISO-on-TCP framing surface (TPKT/TPDU/WrongNumberOfBytes/`PlcException{WrongNumberReceivedBytes}`, NOT `InvalidDataException`); mid-PDU cancel marks handle dead; probe classifies+marks-dead under the gate. STAB-8 (S7 part): `EnsureConnectedAsync` documented as the R2-09 connect-throttle seam (note-only). Driver.S7.Tests 246/246. **Live outage gate deferred (T11)**`S7_1500ConnectTimeoutOutageTests` (env `S7_TIMEOUT_OUTAGE_START_CMD`/`STOP_CMD`, SYN-blackhole) skips clean offline; the connect-*timeout* class the `docker restart` bounce can't produce. (Pre-existing unrelated CLI docs-scan test fails at base — out of scope.) |
| **R2-01** | 05/STAB-14 (High regression, rank 1), 05/STAB-15 (rank 3), 05/STAB-8 seam | `r2/01-s7-fault-hardening` (off `1676c8f4`) | `fcdcf0e2``bdb7ed41` (13 task commits) | **Code-complete, offline-verified.** All prod changes in `S7Driver.cs`. STAB-14: connect-timeout surfaces as `TimeoutException` (not an escaping OCE) at the `EnsureConnectedAsync`/`InitializeAsync` source + `when(token.IsCancellationRequested)` filters on the read/write ensure-wrappers and all three poll-loop OCE catches — subscription poll loops survive an unreachable-host outage and recover (T2 regression RED→GREEN). STAB-15: `IsS7ConnectionFatal` broadened to the ISO-on-TCP framing surface (TPKT/TPDU/WrongNumberOfBytes/`PlcException{WrongNumberReceivedBytes}`, NOT `InvalidDataException`); mid-PDU cancel marks handle dead; probe classifies+marks-dead under the gate. STAB-8 (S7 part): `EnsureConnectedAsync` documented as the R2-09 connect-throttle seam (note-only). Driver.S7.Tests 246/246. **Live outage gate CLOSED 2026-07-15 (T11) — and it uncovered + fixed a real prod gap (PR #453, master `4dc937fc`).** Running `S7_1500ConnectTimeoutOutageTests` against the live s7_1500 sim (blackholed via `docker pause`, the ACK-less-established shape) exposed that STAB-14 fixed only the *connect* leg: S7.Net's `ReadTimeout`/`WriteTimeout` map to the TcpClient's *synchronous* receive/send timeout, which async reads ignore, so a read on an established-but-frozen peer blocked for minutes — the poll loop wedged (no Bad tick, no reconnect), which offline fakes (open-hang only) couldn't surface. Fix = new `S7OperationDeadline` wall-clock ceiling on every data-plane wire op in `S7PlcAdapter` (→ `TimeoutException`) + `IsS7ConnectionFatal` now classifies `TimeoutException` fatal → mark-dead → reopen (the read-leg sibling of the connect-timeout fix). Driver.S7.Tests 246→260 (6 new: 5 deadline + 1 driver-reaction); **live gate now GREEN** (8s: baseline Good → pause blackhole → Bad tick → unpause → recovered Good). (Pre-existing unrelated CLI docs-scan test fails at base — out of scope.) |
| **R2-03** | 02/S13 (rank 5), 02/S12, 02/P7 | `r2/03-vt-failure-quality` (off `1676c8f4`) | `b9775485``6bb91e8d` (12 task commits) | **Code-complete, offline-verified.** S13: VT script failure/timeout degrades the node to `OpcUaQuality.Bad` (carrying last-known value) instead of freezing on last-Good forever — additive `EvaluationResult.Quality`, bridge historizes `0x80020000u` BadInternalError, published once per Good→Bad transition gated on all deps arrived (inputs-ready, no cold-start flash), recovery force-publishes next Good. S12: apply-boundary `ClearCompiledScripts` racing an in-flight eval into ObjectDisposedException retried once in `RoslynVirtualTagEvaluator` (re-fetch→recompile into current generation); stress guard reproduces pre-fix, deterministically green post-fix. P7: apply-boundary clear gated on a changed deployed expression set (`HashSet.SetEquals`, ordinal) — identical redeploy / rename-with-same-expressions no longer forces full VT recompile. Runtime.Tests 372/372, RoslynVirtualTagEvaluatorTests 20/20. **Deferred (T11/T12):** full cross-suite sweep + docker-dev live-`/run` (fail→Bad→recover + P7 no-recompile spot-check). |
| **R2-02** | 01/S-6 (High, rank 2), 01/U-6, 01/S-8=03/S12 (rank 4), 04/C-7, 01/S-7=03/S13 | `r2/02-resilience-config-hardening` (off `1676c8f4`) | `4be13429``dd82e740` (17 task commits) | **Code-complete, offline-verified.** **S-6:** parser clamps timeout/retry/breaker overrides (timeout≤0→tier default, breaker==1→2, caps) with diagnostics + belt-and-braces `Build` clamp so pipeline construction can never throw Polly `ValidationException` (new `ResiliencePolicyRanges` in Core.Abstractions); brick-repro + `Build_NeverThrows_ForHostileDirectOptions` + factory-seam wiring proof; AdminUI `Validate()` range warnings. **S-8:** parser forces Write/AlarmAck retry→0 (spec invariant) **and** dispatch flips to `isIdempotent:false` (RED-first on the Runtime wiring test); non-idempotent arm caches the no-retry snapshot once/invoker (01/P-4) + tracker in-flight accounting; `WriteIdempotentAttribute` false claim corrected; retry cells disabled. **U-6:** bulkhead knob deleted end-to-end (migration-free; stored keys ignored) + `Options_properties_are_exactly_the_pipeline_wired_set` inertness guard. **C-7:** `ResilienceFormModel` JsonObject-bag round-trip preserves unknown keys; malformed JSON → `ParseFailed` lock + verbatim `ToJson` + discard button. **S-7:** per-`Create` `Interlocked` options-generation in the pipeline-cache key kills the respawn stale-pipeline race. Core resilience 121/121, AdminUI form 10/10, Analyzers 32/32, Runtime 364/364; 0 production OTOPCUA0001. **Deferred (T15/T18):** docker-dev `:9200` `/run` (warning banner, input lockout, unknown-key survival) + whole-solution sweep. |
| **R2-10** | 03/U10=04/U-5 (Medium/Low, action #10) | `r2/10-resilience-observability` (off `46fedda3`) | `205d3980``<docs>` (11 task commits) | **Code-complete, offline-verified.** The resilience status tracker now has a production reader, mirroring the driver-health flow. Core truthfulness: `RecordBreakerClose` + derived `IsBreakerOpen` (breaker-open state derivable), `OnClosed` records the close, and `RecordSuccess` wired onto the invoker success paths (`ConsecutiveFailures` is a true consecutive counter, not stuck-forever). Transport: `DriverResilienceStatusChanged` Commons contract → periodic 5 s full-snapshot publisher `DriverResilienceStatusPublisherService` (driver nodes, registered in `AddOtOpcUaDriverFactories` with wiring guard `ResilienceStatusReaderRegistrationTests` — closes the built-but-never-wired class) → `driver-resilience-status` DPS topic → per-admin-node `DriverResilienceStatusBridge``IDriverResilienceStatusStore` → resilience chips (breaker-OPEN / consecutive-failures / in-flight / staleness) on `DriverStatusPanel.razor`, read in-process (self-HubConnection ban honored). Core.Tests resilience 31/31 (incl. deterministic breaker-close via a controllable TimeProvider), Host.IntegrationTests publisher+guard 6/6, AdminUI.Tests store 5/5; full-solution build clean. **Deferred (T10):** docker-dev live-`/run` — S7 dead-endpoint breaker-open observed on BOTH centrals + recovery-clear + staleness-dim (also the first live sighting of the #10 `circuit-breaker OPENED` log line, closing FOLLOWUP-10's behavioral gate). |
| **R2-08** | 03/S2 (High, action #8), 03/S3 (High) | `r2/08-ldap-historyread-async` (off `1a698cbb`) | `ce6d9e21``<docs>` (14 task commits) | **Code-complete, offline-verified.** **S2 (LDAP async/timeout):** the SDK raises the impersonation callback synchronously under a `SessionManager`-wide `m_eventLock` (verified vs. 1.5.378 — a hung bind serializes ALL activations), so the fix is entirely at the `LdapOpcUaUserAuthenticator` boundary: five new `Security:Ldap` keys (`ConnectionTimeoutMs`=10000 projected into the shared lib, `AuthTimeoutMs`=15000 hard whole-flow `WaitAsync` bound → fail-closed "timed out", `MaxConcurrentAuthentications`=8 `SemaphoreSlim` with release-on-core-completion so orphaned binds still count, `OutageFailureThreshold`=3/`OutageCooldownSeconds`=15 directory-outage circuit → fast-deny "unavailable", half-open probe) + `LdapAuthResult.IsSystemFailure` seam (mapped from `ServiceAccountBindFailed`) distinguishing outage from credential-deny + `LdapOptionsValidator` rules + `HandleImpersonation` contract doc + `docs/security.md`. Stall-repro RED→GREEN; offline outage-sim through the real `HandleImpersonation` against an unroutable host. **S3 (HistoryRead fan-out):** all four HistoryRead arms async-ified (`ServeNodeAsync`/`ServeRawPagedAsync`/`ServeEventsAsync`, OCE→`BadTimeout`) then bounded per-node fan-out (`RunBounded` semaphore + single arm-boundary bridge) + per-request deadline CTS (`HistoryReadDeadline`=60s) + process-wide lazy batch limiter (`MaxConcurrentHistoryReadBatches`=16 → flood degrades to `BadTooManyOperations`); 16×4=64 in-flight cap; three new `ServerHistorian` keys + node-manager props + hosted-service wiring + `docs/Historian.md`/CLAUDE.md tables. Sequential-serving RED→GREEN. Host.IntegrationTests (LDAP resilience) 13/13, Security.Tests 11/11, Runtime.Tests options 11/11, OpcUaServer.Tests HistoryRead 30/30. **Deferred (live):** S2 docker-dev unreachable-LDAP + GLAuth outage/recovery cycle; S3 `Category=LiveIntegration` gateway soak. |
| **R2-08** | 03/S2 (High, action #8), 03/S3 (High) | `r2/08-ldap-historyread-async` (off `1a698cbb`) | `ce6d9e21``<docs>` (14 task commits) | **Code-complete, offline-verified.** **S2 (LDAP async/timeout):** the SDK raises the impersonation callback synchronously under a `SessionManager`-wide `m_eventLock` (verified vs. 1.5.378 — a hung bind serializes ALL activations), so the fix is entirely at the `LdapOpcUaUserAuthenticator` boundary: five new `Security:Ldap` keys (`ConnectionTimeoutMs`=10000 projected into the shared lib, `AuthTimeoutMs`=15000 hard whole-flow `WaitAsync` bound → fail-closed "timed out", `MaxConcurrentAuthentications`=8 `SemaphoreSlim` with release-on-core-completion so orphaned binds still count, `OutageFailureThreshold`=3/`OutageCooldownSeconds`=15 directory-outage circuit → fast-deny "unavailable", half-open probe) + `LdapAuthResult.IsSystemFailure` seam (mapped from `ServiceAccountBindFailed`) distinguishing outage from credential-deny + `LdapOptionsValidator` rules + `HandleImpersonation` contract doc + `docs/security.md`. Stall-repro RED→GREEN; offline outage-sim through the real `HandleImpersonation` against an unroutable host. **S3 (HistoryRead fan-out):** all four HistoryRead arms async-ified (`ServeNodeAsync`/`ServeRawPagedAsync`/`ServeEventsAsync`, OCE→`BadTimeout`) then bounded per-node fan-out (`RunBounded` semaphore + single arm-boundary bridge) + per-request deadline CTS (`HistoryReadDeadline`=60s) + process-wide lazy batch limiter (`MaxConcurrentHistoryReadBatches`=16 → flood degrades to `BadTooManyOperations`); 16×4=64 in-flight cap; three new `ServerHistorian` keys + node-manager props + hosted-service wiring + `docs/Historian.md`/CLAUDE.md tables. Sequential-serving RED→GREEN. Host.IntegrationTests (LDAP resilience) 13/13, Security.Tests 11/11, Runtime.Tests options 11/11, OpcUaServer.Tests HistoryRead 30/30. **S2 GLAuth outage/recovery LIVE gate CLOSED 2026-07-15 (PR #455, master `5f12636c`)** — new env-gated `LdapGlauthOutageLiveTests` drives the circuit through the production path (`OtOpcUaLdapAuthService`+`LdapOpcUaUserAuthenticator`) against a REAL GLAuth on `10.100.0.35:3894` (paused/unpaused via `docker pause`); GREEN (12s): healthy alice/bob binds w/ mapped roles → pause → 3 bounded failures → circuit opens → sub-second "unavailable" → unpause → half-open close → recover. Also the FIRST end-to-end proof the PR #451 GLAuth swap binds correctly. Skips clean offline (1 skipped, 8ms). **S3 `Category=LiveIntegration` gateway soak — GREEN 2026-07-15 (6/6)** against the running local 0.2.0 gateway (`http://localhost:5221` h2c) + the real AVEVA historian over VPN; the Galaxy read round-trip exercises the `GatewayHistorianDataSource` read path S3 hardened. **R2-08 fully live-verified.** |
| **R2-11** | 01/C-1, 01/P-1 + 05/CONV-2, UNDER-1, UNDER-6 | `r2/11-tagconfig-consolidation` (off `46fedda3`) | first `2e14fe1f` … (24 task commits) | **Code-complete, offline-verified.** **Part 1 (C-1/P-1):** single `TagConfigIntent.Parse` + `DeviceConfigIntent` in Commons replace the four byte-parity TagConfig parse copies (composer/artifact/walker/DraftValidator) — composer now parses once per tag (P-1); a permanent `Runtime.Tests/TagConfigCorpusParityTests` (30-blob corpus) guards compose≡decode; `grep "MUST parse identically" src/` = 0. **Part 2 (CONV-2):** shared strict-capable `TagConfigJson` readers in Core.Abstractions replace the six copy-pasted `ReadEnum`s; all six driver parsers gain `Inspect()` warnings + honour the `writable` key (FOCAS **forced read-only**, 05/UNDER-1); FOCAS equipment tags run the capability-matrix pre-flight (05/UNDER-6); Modbus probe parses the factory DTO shape (`timeoutMs` no longer dropped). New `EquipmentTagConfigInspector` in ControlPlane + `AdminOperationsActor` deploy-gate wiring with `Deployment:TagConfigValidationMode` (**Warn default | Error opt-in**), proven through the actor (F10b consuming test). AdminUI: `writable` checkbox in the six typed editors. **Phase-C follow-up (deferred, recorded here):** flip the runtime parsers to strict (typo'd enum ⇒ `BadNodeIdUnknown` instead of wrong-width `Good`) once fleets have run `Error` mode clean — behavior change deliberately not shipped in this plan. **Deferred (T22/T24):** docker-dev `/run` of one typed editor's Writable checkbox + whole-solution serial sweep. |
| **R2-07 (Phases 1+2+3)** | 03/P1 (action #7, surgical pure-adds/removes/mixed) | `r2/07-surgical-pure-adds` (off `1a698cbb`) | `bb226f45``e2b47638` (T1T14) | **Phases 1+2 code-complete, offline-verified.** New pure `AddressSpaceChangeClassifier` (Empty/AttributeOnly/PureAdd/PureRemove/AddRemoveMix/Rebuild; first-match-wins, default-closed Rebuild; the F10b tag/vtag surgical-eligibility predicates moved here as `internal static`). `AddressSpaceApplier.Apply` routes by kind. **Phase 1 (PureAdd/AttributeOnly):** skip `RebuildAddressSpace` (the idempotent Materialise passes create exactly the added nodes; existing subscriptions survive); adds announced via `ComputeAddAnnouncements`+`AnnounceAddedNodes` (dedup affected-parent → existing guard-covered `RaiseNodesAddedModelChange`), fired by `OpcUaPublishActor` after the 4 passes gated on `!outcome.RebuildCalled`; `MaterialiseAlarmCondition` made idempotent (skip-if-present-and-same-kind keeps the condition NodeState alive). **NO new sink members in Phase 1.** **Phase 2 (PureRemove):** 3 new `ISurgicalAddressSpaceSink` members (`RemoveVariableNode`/`RemoveAlarmConditionNode`/`RemoveEquipmentSubtree`) — guard-first with the **observed negative control** (`DeferredSinkForwardingReflectionTests` fails member-named until `DeferredAddressSpaceSink` forwards) + `SdkAddressSpaceSink` forwards + `OtOpcUaNodeManager` in-place teardown (scoped map cleanup, notifier demotion via `RemoveRootNotifier`, `BuildNodesRemovedModelChange` verb=NodeDeleted reported outside `Lock`). Applier `PureRemove` arm: terminal Bad / `RemovedConditionState` writes (closing the removed-alarm-tag today-gap) → subsumption partition (removed-equipment children folded into the subtree remove) → in-place removes → any false/throw ⇒ full-rebuild ratchet; non-surgical sink ⇒ rebuild. **Phase 3 (AddRemoveMix):** removes-then-adds within one apply — the remove pass runs in `Apply` (no rebuild), then the publish actor's Materialise passes create the adds + `AnnounceAddedNodes` announces them; an id reused across the remove+add sets is remove-then-recreated as a fresh NodeState (correct — a different tag). **Guard stays green (Commons.Tests 17/17), OpcUaServer.Tests 336/336, Runtime.Tests 398/398.** **03/P1 marked REMEDIATED in `archreview/03-server-runtime.md`.** **Deferred-live (T5, T6, T12, T14):** `SubscriptionSurvivalTests` add + remove + mixed cases (authored + compile-green; NOT run — `*.IntegrationTests` memory constraint) and the docker-dev live-`/run` add/remove/mixed gates (`kind=PureAdd/PureRemove/AddRemoveMix, rebuild=False`; survivor subscription uninterrupted; removed node → `BadNodeIdUnknown`) — MUST run in the serial heavy pass (F10b inertness risk). |
@@ -221,10 +221,10 @@ Client.UI 126, Core.AlarmHistorian 29, Core.ScriptedAlarms 71, and all drivers (
Every plan's integration-suite and live-`/run` gates were deferred (not skipped) so nothing OOMs the box and the
shared docker-dev rig / VPN infra isn't contended. To run one-at-a-time:
- **Full integration-suite sweep** (10 `*.IntegrationTests` projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24.
- **Full integration-suite sweep** (10 `*.IntegrationTests` projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24. **RAN 2026-07-14 — see [`INTEGRATION-SWEEP-STATUS.md`](INTEGRATION-SWEEP-STATUS.md).** Outcome: OpcUaClient(+Browser)/S7/Modbus-standard green, Host 105/117, TwinCAT clean-skip; the rest fail on **pre-existing test/fixture bit-rot** (AbCip static-init bug, Modbus pymodbus-4.x sim drift, FOCAS 127.0.0.1-mock topology, AbLegacy no fixture, OpcUaServer cert-store config, Host openldap:2.6 image gone). **No OtOpcUa regression.** 7 follow-ups in that file.
- **docker-dev `:9200` live-`/run`** (rebuild BOTH centrals): R2-02 T15 (ResilienceConfig warning banner / input lockout / unknown-key survival), R2-03 T12 (VT fail→Bad→recover + P7 no-recompile), R2-10 T10 (breaker-open on both centrals + recovery + FOLLOWUP-10 log line), R2-11 T22 (typed-editor Writable checkbox), **R2-07 T12 (surgical add/remove/mixed — F10b inertness risk, MUST run)**, R2-05 T15 (over-gating regression only — deny unobservable under `DisableLogin`).
- **2-node redundancy rig:** **R2-04 T13/T15 — behavior-affecting S4 primary-gate default-deny; MUST run before trusting the failover-role change.**
- **Genuinely external / needs-user:** R2-06 T12 (VPN to a 0.2.0 HistorianGateway for the live EnsureTags retype probe), R2-08 live legs (real GLAuth outage/recovery cycle; `Category=LiveIntegration` gateway soak), R2-01 T11 (S7 SYN-blackhole outage on the `10.100.0.35` s7_1500 fixture).
- **2-node redundancy rig:** ~~R2-04 T13/T15~~ **BOTH ✅ PASSED 2026-07-14** — T15 live docker-dev gate + T13 in-process delivery-path test (see below).
- **Genuinely external / needs-user:** ~~ALL CLOSED as of 2026-07-15.~~ ~~R2-06 T12~~ (6/6 on the VPN, 2026-07-13). ~~R2-08 S3 `Category=LiveIntegration` gateway soak~~ **CLOSED 2026-07-15** (6/6 vs. the running local 0.2.0 gateway + real historian over VPN). ~~R2-01 T11~~ **CLOSED 2026-07-15** (S7 blackhole via `docker pause`; found + fixed the read-leg wall-clock gap, PR #453). ~~R2-08 GLAuth outage cycle~~ **CLOSED 2026-07-15** (`LdapGlauthOutageLiveTests` vs. the harness GLAuth paused via `docker pause`, PR #455). **Every Round-2 live gate is now GREEN.**
### Deferred behavior-change follow-up (deliberately not shipped)
**R2-11 Phase C** — flip the runtime driver parsers to strict (typo'd enum ⇒ `BadNodeIdUnknown` instead of a
@@ -234,9 +234,22 @@ wrong-width `Good`). Land only after fleets have run `Deployment:TagConfigValida
**Bottom line:** R2-02, R2-11, R2-07 (both surgical phases), R2-04 (achievable failover), and R2-10 (resilience pipeline in
dispatch) verified live on the docker-dev central cluster. R2-05 partial-by-design; R2-03 unit-verified (rig has no driver
data). Everything still open needs infra this box lacks: the `*.IntegrationTests` sweep needs the `10.100.0.35` driver
fixtures + ephemeral LDAP up; R2-06 needs VPN to a 0.2.0 gateway; R2-08 a live GLAuth outage cycle; R2-01 the S7 blackhole
fixture. Those are the only Round-2 items not yet exercised.
data). R2-01's S7 blackhole gate is **CLOSED (2026-07-15)** — run via `docker pause` on the `10.100.0.35` s7_1500 fixture;
it found + fixed the read-leg wall-clock gap (PR #453). R2-08's GLAuth outage-cycle leg is **CLOSED (2026-07-15, PR #455)**
`LdapGlauthOutageLiveTests` vs. the harness GLAuth on `10.100.0.35`. R2-08 S3 `Category=LiveIntegration` gateway soak is
**CLOSED (2026-07-15, 6/6)** and R2-06 (6/6, 2026-07-13) — both vs. the local 0.2.0 gateway + real historian over VPN.
**Every Round-2 live gate is now GREEN — no Round-2 items remain unexercised.**
### Residual follow-ups — tracked in Gitea (2026-07-15)
The remaining non-blocking items are filed as issues (none gate the remediation, which is complete):
| Issue | Item | Nature |
|---|---|---|
| **#456** | #10 — observe retry/breaker firing in live dispatch; thread `ResilienceConfig` into the deploy artifact (tier-defaults only today) | niche / sub-gap |
| **#457** | R2-11 Phase C — flip runtime parsers to strict, after fleets run `TagConfigValidationMode=Error` clean | deliberately deferred (operational rollout) |
| **#458** | Remove the `NuGetAuditSuppress` for CVE-2025-6965 once a patched SQLitePCLRaw bundle ships | upstream-gated |
| **#459** | SBR topology — crashing the OLDEST Akka node does not fail over | pre-registered user-deferred design gap |
Brought up the 2-node central cluster (`central-1`+`central-2`, SQL, traefik) from source at merged master `6aeb53a9`
(single-tenant rig; drove the `:9200` AdminUI via browser + Client.CLI on `:4840`/`:4841`).
@@ -275,8 +288,57 @@ Brought up the 2-node central cluster (`central-1`+`central-2`, SQL, traefik) fr
**Rig note:** the pass mutated the persisted ConfigDb (added a `drv-modbus-line3-01` Modbus driver on MAIN; removed
`capper-03`'s `production-count` virtual tag for the PureRemove test). Reset with
`docker compose -f docker-dev/docker-compose.yml down -v`. The remaining integration-suite sweep + genuinely-external
gates (R2-06 VPN, R2-08 live LDAP, R2-01 S7 blackhole) still need their infra.
`docker compose -f docker-dev/docker-compose.yml down -v`. R2-01's S7 blackhole gate is now CLOSED (2026-07-15, PR #453);
**all Round-2 live gates are now GREEN** (R2-06 + R2-08 S3 verified on the VPN 2026-07-15; R2-01 + R2-08 GLAuth self-serviced on `10.100.0.35`).
---
## R2-04 T15 — 2-node primary-gate LIVE GATE ✅ PASSED (2026-07-14)
Ran the S4 primary-gate live recipe on the docker-dev 2-node central cluster **rebuilt from master `365a90e6`**
(`up -d --build central-1 central-2`). Brought up the **Modbus sim** (`otopcua-pymodbus-standard` on
`10.100.0.35:5020`) so `MAIN`'s Modbus equipment tags read **Good** (a data-less rig materialises every node
`Bad_WaitingForInitialData`, and the CLI write reads-first — so a live driver is required to have a writable-Good
node to drive the gate). Made `EQ-55297329838d/ModbusHR200` writable (`TagConfig.writable=true` via SQL +
`POST :9200/api/deployments`). Data-plane writes bound `multi-role`/`password` (carries the `WriteOperate` LDAP role).
**All core assertions PASS:**
- **Step 1 — ServiceLevel split:** `Server_ServiceLevel` (`i=2267`) = **250 on `:4840` (primary) / 240 on `:4841`
(secondary)** — differentiated, no split-brain. (Plan said 240/100; the deployment's healthy values are 250/240 —
the invariant "primary > secondary" holds.)
- **Step 2 — boot-window default-DENY (the NEW S4 behavior):** restarting **both** centrals (to widen the
role-null-with-≥2-members window — a single-node restart's window is sub-second because cluster-join and the first
redundancy snapshot land near-simultaneously) caught, ~5 s into boot:
`WRN Operator write to EQ-55297329838d/ModbusHR200 rejected: not primary (role unknown)`. Meter
`otopcua_redundancy_primary_gate_denied_total{reason="role-unknown",site="write"} = 1`.
- **Step 3 — steady-state secondary DENY + #5 revert:** write `9999` to `:4841` → CLI reports optimistic
`Write successful` (synchronous, by design) but read-back = **1234 (the primary's value, reverted — NOT 9999)**;
central-2 logs `rejected: not primary`; meter `reason="secondary",site="write" = 1`.
- **Step 3b — primary write reaches the device:** write `1234` to `:4840` → `Write successful` and read-back on
`:4841` = `1234` (the Modbus driver wrote through).
- **T4 sanity — clean apply:** both centrals logged `AddressSpaceApplier: applied plan (kind=PureAdd … failed=0)` /
`materialised (… failed=0)`; **no `[ERR]` / `apply.failed` / degraded-apply line**; the `apply_failed` meter is
absent (never incremented).
- **Step 4 — `/alerts`:** endpoint 200, **0 rows** — no active alarm on the data-less rig, so trivially no
dual-primary double-emission (the double-row regression is only observable with a firing alarm, which needs live
driver data this rig lacks; the alerts-emit gate shares the same `PrimaryGatePolicy` and is unit-covered).
**Net:** the R2-04 `PrimaryGatePolicy` membership-aware **default-DENY-on-unknown** is confirmed live end-to-end —
both denial reasons (`role-unknown`, `secondary`) fire the log + the new meter, the #5 optimistic-write revert
corrects the node, and the primary path is unaffected. The behavior-affecting S4 slice is trusted.
**R2-04 T13 — in-process delivery-path test ✅ PASSED (2026-07-14).**
`dotnet test …Host.IntegrationTests --filter FullyQualifiedName~PrimaryGateFailoverTests` → **1/1, 3 s**. Ran
**without SQL** (`OTOPCUA_HARNESS_USE_SQL` unset) — the gate test needs only the in-process 2-node Akka cluster +
the DPS-delivered redundancy snapshot, not a ConfigDb deploy. The `TwoNodeClusterHarness` formed the cluster and
the **real DistributedPubSub-delivered** snapshot promoted exactly one node to Primary (Secondary's
`DriverHostActor.RouteNodeWrite` → `not primary`; Primary passed the gate, rejecting only with a mapping reason).
Its built-in negative control (a broken redundancy-topic subscribe leaves both stuck `role unknown` → 45 s timeout)
means the PASS proves genuine snapshot delivery drives the gate — the half the pure-TestKit unit guards can't cover.
**Rig mutations (this pass):** `ModbusHR200.TagConfig.writable=true`; Modbus sim brought up on the shared docker host
(`10.100.0.35`). Reset the central rig with `docker compose -f docker-dev/docker-compose.yml down -v`; the sim with
`ssh dohertj2@10.100.0.35 'cd /opt/otopcua-modbus && docker compose --profile standard down'`.
---
@@ -327,24 +389,31 @@ convention) + a regression unit test + `StorageRateMs` on the live-suite inline
- ✅ `Write_then_read_on_sandbox_tag` — `EnsureTags(Float)` + `WriteLiveValues` (SQL) + read-back.
- ✅ `Alarm_SendEvent_is_acked` + ✅ `Alarm_SendEvent_then_ReadEvents` — alarm `SendEvent` + SQL `ReadEvents`.
- ✅ `EnsureTags_type_change_outcome_is_observable` — now observes a **real** retype outcome (not a storage-rate error).
- `EnsureTags_boolean_sandbox_provisions_as_Int1` — **Int1 (Boolean) tag creation is not supported by the
gateway's histsdk on this historian**: `EnsureTags(Int1)` returns `ProtocolEvidenceMissingException` (the vendored
client's "no capture evidence for this type" guard → gRPC `Unimplemented`), both fresh and as a retype target.
A **type sweep** confirmed it is Int1-specific: **Int2/UInt2/Int4/UInt4/Float/Double all provision `success=True`;
only Int1 fails.** Root cause: `histsdk` `HistorianTagWriteProtocol.GetAnalogDataTypeCode` has no captured
CTagMetadata wire code for Int1. This is a **gateway/histsdk type-support gap**, not an OtOpcUa defect — **filed as
[historiangw#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11)**.
- ⚠️ `EnsureTags_boolean_sandbox_provisions_as_Int1` (as originally written) — **Int1 (Boolean) tag creation is not
supported by the gateway's histsdk on this historian**: `EnsureTags(Int1)` returns `ProtocolEvidenceMissingException`
(the vendored client's "no capture evidence for this type" guard → gRPC `Unimplemented`), both fresh and as a retype
target. A **type sweep** confirmed it is Int1-specific: **Int2/UInt2/Int4/UInt4/Float/Double all provision
`success=True`; only Int1 fails.** Root cause: `histsdk` `HistorianTagWriteProtocol.GetAnalogDataTypeCode` has no
captured CTagMetadata wire code for Int1 a **gateway/histsdk type-support gap**, not an OtOpcUa defect
(**filed as [historiangw#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11)**, which routed the
decision back as **[issue #441](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/441)**).
**✅ RESOLVED (issue #441, 2026-07-14):** `HistorianTypeMapper` now maps **`Boolean → Int2`** (0/1 integer) instead
of the degenerate `Int1`. Int2 is a proven-creatable analog type on this historian, and the value-write path already
sends every value as a `double` on `WriteLiveValues` regardless of tag type, so Boolean `0.0/1.0` round-trips
cleanly. The live gate test is renamed `EnsureTags_boolean_sandbox_provisions_as_Int2` and now passes → **6/6**. Unit
pins updated (`HistorianTypeMapperTests`, `GatewayTagProvisionerTests.Boolean_definition_carries_explicit_Int2_presence`);
gateway driver unit `103/103`.
**✅ U-7 retype policy — ANSWERED (via `grpcurl` against the real historian):**
- `EnsureTags` is a **real create-or-update upsert** — re-`EnsureTags` of an existing tag at the same type → `success`.
- An **in-place retype between SUPPORTED types is ACCEPTED** — `Float→Double` on an existing tag returns `success`
(of the three 06/U-7 hypotheses: **accepted-retyped**, not rejected).
- **But the specific 0.2.0 Boolean case (Float→Int1) can't retype** — Int1 isn't a creatable type here, so a
pre-0.2.0 **Float Boolean tag asked to become Int1 fails the provision** with `ProtocolEvidenceMissingException`
— it is **neither silently retyped nor data-lost**; the provision just fails (and is metered `failed=N`). That is
the practically-relevant U-7 answer for Boolean tags on this deployment.
- **The specific `Int1` case can't retype** — Int1 isn't a creatable type here. Rather than accept a broken Boolean
path, OtOpcUa **remapped `Boolean → Int2`** (issue #441, above), which retypes cleanly from a pre-existing Float
Boolean tag (both are supported analog types) and never hits the Int1 gap.
The R2-06 **code** deliverables (S-11 fail-fast + `ServerHistorianOptionsValidator`; U-7 Boolean→Int1 mapping +
The R2-06 **code** deliverables (S-11 fail-fast + `ServerHistorianOptionsValidator`; Boolean provisioning mapping +
upsert; C-7 doc) were already unit-covered + merged; this live gate additionally **found + fixed the StorageRateMs
provisioning bug** and **answered the U-7 retype policy**. The one remaining red test is a documented gateway
histsdk Int1-support gap.
provisioning bug**, **answered the U-7 retype policy**, and surfaced the Int1 histsdk gap that drove the
**Boolean→Int2 remap (issue #441)** — closing the last red test to reach **6/6**.
@@ -0,0 +1,62 @@
# Follow-up #10 / issue #456 — live proof that authored ResilienceConfig reaches the runtime pipeline
**Date:** 2026-07-15 · **Rig:** docker-dev (central-1 + central-2, MAIN cluster, Warm/2-node) built from source.
## What this proves
The #456 "sub-gap" worried that per-instance `DriverInstance.ResilienceConfig` never reaches the
runtime resilience pipeline (so it runs tier-defaults only). This is **disproven** — both by a new
regression test and by this live capture.
## Setup
Seeded on MAIN (SQL): an Equipment namespace `ns-res` + an `OpcUaClient` driver `drv-res` carrying a
**non-default** override:
```json
"ResilienceConfig": {"capabilityPolicies":{"Subscribe":{"retryCount":999}}}
```
`retryCount:999` is deliberately over the cap (100) so the parser emits an unmistakable clamp
diagnostic that echoes the authored value. Deployed headless: `POST /api/deployments` → Accepted
(deploymentId 80b4be3c…).
## Evidence
**1. The persisted deployment artifact bytes carry it** (`dbo.Deployment.ArtifactBlob`, 4055 bytes):
`CHARINDEX('ResilienceConfig') > 0` = YES, `CHARINDEX('999') > 0` = YES; the `drv-res` object slice
shows `…"DriverType":"OpcUaClient"…"DriverConfig":"…"…"Resil…"`.
**2. The runtime parsed + applied it — on BOTH redundancy nodes:**
```
central-1 [WRN] Driver resilience config for instance=drv-res type=OpcUaClient: Subscribe.retryCount 999 exceeds the cap (100); clamped.
central-1 [INF] DriverHost central-1:4053: spawned OpcUaClient driver drv-res (stub=False)
central-2 [WRN] Driver resilience config for instance=drv-res type=OpcUaClient: Subscribe.retryCount 999 exceeds the cap (100); clamped.
central-2 [INF] DriverHost central-2:4053: spawned OpcUaClient driver drv-res (stub=False)
```
The authored value `999` appears verbatim in the runtime log. If the artifact did not carry
`ResilienceConfig` (the feared sub-gap), `spec.ResilienceConfig` would be null, the parser would take
pure tier-defaults, and **no clamp diagnostic would fire**. The diagnostic firing = the authored
policy rode DB → ConfigComposer → artifact → DriverHostActor → DriverCapabilityInvokerFactory.Create →
DriverResilienceOptionsParser. This is arch-review follow-up #10 acceptance item 2 ("re-verify the
behavioral gate with a non-default policy"), verified live on both nodes.
## Regression guard
`ConfigComposerTests.ResilienceConfig_survives_ConfigComposer_to_ParseDriverInstances_round_trip`
(master 809e7886) proves the same composer→artifact→parse leg deterministically, guarding against a
future projection / `[JsonIgnore]` silently reverting authored policy to tier defaults.
## Still open (acceptance item 1)
The raw retry/breaker log line **on an actual driver fault** was NOT captured. It requires a
connect-then-fault-mid-flight trigger: the wrapped capability calls (`Subscribe`/`Discover`/`Write`)
only fire after a driver reaches **Connected** (`DriverInstanceActor` `ResubscribeDesired` runs on the
`Become(Connected)` transition), so a driver at a dead endpoint stays in Reconnecting and never
exercises the wrapped path. The reliable trigger is an **idempotent Write to a Connected-then-killed
driver** (LDAP-gated Client.CLI write) or an S7 dead-socket reconnect. The retry/breaker **behaviour**
is already proven deterministically (`CapabilityInvokerTests` retry, the pipeline-builder logging test,
`DriverCapabilityInvokerFactoryTests` override-applies-to-behaviour). This remaining line is
operator-confidence only.
@@ -0,0 +1,44 @@
# #459 — LIVE finding: keep-oldest downs the survivor when the oldest crashes (2-container rig)
**Date:** 2026-07-15 · **Rig:** docker-dev, central-1 (seed/oldest, :4840) + central-2 (:4841), MAIN Warm/2-node.
## Headline
On a **real 2-container rig**, hard-killing the **oldest** node makes the **younger survivor down ITSELF**
→ both processes exit → **no failover, total outage**. This CONFIRMS #459 (the premise is real, not stale)
and **disproves Option A** (keep `keep-oldest` is insufficient). Reproduced twice, deterministically.
## Baseline (both nodes healthy, converged)
- `Server.ServiceLevel` (NodeId `i=2267`): central-1 (oldest) = **250** (healthy leader), central-2 = **240** (healthy follower). Matches the documented tiering.
## The kill
`docker kill otopcua-dev-central-1-1` (true process death, not a graceful Leave). central-2'"'"'s SBR:
```
[WRN] SBR took decision Akka.Cluster.SBR.DownReachable and is downing
[akka.tcp://otopcua@central-2:4053] including myself, [1] unreachable of [2] members
[INF] Cluster Node - Marking node [central-2:4053] as [Down]
[INF] Self downed, stopping ClusterSingletonManager
[INF] CoordinatedShutdown invoked due to [ClusterDowningReason]. Exiting with [ExitCode:3].
```
Result: central-2 Exited (255); `opc.tcp://localhost:4841` → Connection refused. No survivor.
## Why (this is textbook keep-oldest, not a misconfig)
`keep-oldest` keeps the partition CONTAINING the oldest and downs the other side. When the oldest CRASHES,
the survivor (younger) is "the side without the oldest" → it downs itself. `down-if-alone = on` does NOT
rescue it: that knob only makes the OLDEST down itself when the oldest is isolated-but-alive; a crashed
oldest can'"'"'t execute it, and the younger applies strict keep-oldest → `DownReachable` (self-down). This is
the documented 2-node keep-oldest limitation; Akka recommends 3+ members / a real quorum.
## Why the in-process test gave false confidence
`HardKillFailoverTests.Hard_kill_of_oldest_node_fails_over_to_survivor` PASSES, but `HardKillNodeAAsync`
only calls `provider.Transport.Shutdown()` — node A'"'"'s **ActorSystem stays alive** (a transport partition
with the oldest still running), NOT a process crash. So the in-process test exercises a different scenario
than a real `docker kill`. It should be treated as non-representative for the oldest-CRASH case
(test-quality follow-up: make it kill the process, or reframe what it proves).
## Consequence
Option A (keep `keep-oldest`) does not provide oldest-crash failover on a real 2-node deployment. The fix
requires **Option B** (3rd witness/lighthouse member + `keep-majority` — survives any single crash) or
**Option C** (DB-lease arbiter decoupling Primary from Akka oldest). A 2-node cluster fundamentally cannot
auto-survive an arbitrary single-node crash under any SBR strategy without a tiebreaker.
+36 -40
View File
@@ -141,48 +141,44 @@ for this node), which is the signal to check that `ServerHistorian:Enabled=true`
> so a HistoryRead on it runs to the full `CallTimeout` (~30 s) before erroring. Auto-provisioning on deploy
> keeps freshly-historized tags from hitting that slow path. See the `CallTimeout` row above.
### 0.2.0 DataType provisioning delta (archreview 06/U-7)
### Boolean historization maps to Int2 (issue #441)
`EnsureTags` is contract-documented (HistorianGateway 0.2.0) as an **upsert** — *"creates or updates
`EnsureTags` is contract-documented (HistorianGateway 0.2.0+) as an **upsert** — *"creates or updates
historian tags according to the given definitions"* — that maps per-tag onto AVEVA's
`HistorianClient.EnsureTagAsync(definition) -> bool`. It is **not idempotent** (not routed through the
idempotent retry pipeline) and **not skip-if-exists**.
The 0.1.0→0.2.0 client bump made `HistorianTagDefinition.DataType` **proto3-optional** (explicit presence).
Under 0.1.0, the provisioner's `Boolean → Int1` mapping sent wire value `0`, indistinguishable from unset, so
the gateway defaulted such tags to the SDK-default **Float**. Under 0.2.0 the same call carries explicit
presence, so Boolean tags now provision as **Int1** (the pre-existing intent of `HistorianTypeMapper` — no
mapper code change is needed or wanted). Consequences:
`HistorianTypeMapper` maps **`DriverDataType.Boolean → HistorianDataType.Int2`** and Boolean values
historize as a small **0/1 integer**. Int2 is used rather than the semantically-exact `Int1` because the
historian's **`Int1` analog-tag creation path is server-degenerate** — `EnsureTags(Int1)` stores an
unusable stub, so a Boolean tag mapped to `Int1` never becomes historizable. This is an evidence-based
decision from a live probe against the deployed 2023 R2 historian; the root cause and gateway-side
resolution are tracked in [issue #441](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/441) and
HistorianGateway [#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11) / PR #16 (which
documents `Int1` as an explicit, evidence-based exclusion). The value-write path is unaffected: the
recorder sends every value as a `double` on `WriteLiveValues` regardless of tag type, so a Boolean
`0.0/1.0` round-trips cleanly into the Int2 tag.
1. **New Boolean tags provision correctly as Int1.** This is the fix arriving as a silent, zero-diff
semantic change from the bump.
2. **Pre-bump Boolean tags exist historian-side as Float.** On the next deploy that dispatches provisioning
for such a tag, the gateway is asked (via the upsert) to update it to Int1.
**Observed live (2026-07-13, real 0.2.0 gateway → real AVEVA historian on `wonder-sql-vd03`, probed via
`grpcurl` — archreview R2-06 T12):**
- `EnsureTags` is a **real create-or-update upsert** — re-`EnsureTags` of an existing tag at the same type
returns `Success=true`.
- An **in-place retype between SUPPORTED types is ACCEPTED**`Float→Double` on an existing tag returns
`Success=true`.
- A type sweep confirmed **`Int2/UInt2/Int4/UInt4/Float/Double` all provision `Success=true`; only `Int1`
fails** (`Success=false, Error="ProtocolEvidenceMissingException"` — the vendored client's "no capture
evidence for this type" guard → gRPC `Unimplemented`), both for a fresh tag and as a retype target. Mapping
Boolean to `Int2` sidesteps that gap entirely.
**Observed live (2026-07-13, real 0.2.0 gateway → real AVEVA historian on `wonder-sql-vd03`, probed via
`grpcurl` — archreview R2-06 T12):**
- `EnsureTags` is a **real create-or-update upsert** — re-`EnsureTags` of an existing tag at the same type
returns `Success=true`.
- An **in-place retype between SUPPORTED types is ACCEPTED**`Float→Double` on an existing tag returns
`Success=true` (hypothesis **(ii) accepted as a plain retype**, not rejected).
- **BUT `Int1` — the 0.2.0 Boolean target — is *not a creatable type* on this historian's histsdk:**
`EnsureTags(Int1)` returns `Success=false, Error="ProtocolEvidenceMissingException"` (the vendored client's
"no capture evidence for this type" guard → gRPC `Unimplemented`), both for a fresh tag and as a retype
target. A type sweep confirmed it is **Int1-specific**: `Int2/UInt2/Int4/UInt4/Float/Double` all provision
`Success=true`; only `Int1` fails.
- **Net for a pre-bump Float Boolean tag asked to become Int1:** the provision **fails** (per-tag
`Success=false``failed=N` tally). It is **neither silently retyped nor data-lost** — the tag stays Float
and remains readable; the Int1 provision simply doesn't land until the gateway's histsdk gains `Int1`
create-evidence — filed as
[historiangw#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11). (Prerequisite for reaching this
outcome at all: the provisioner must stamp a **positive
`StorageRateMs`** — a `0` made `EnsureTags` throw `ArgumentOutOfRangeException` and fail *every* provision;
fixed in `GatewayTagProvisioner`, PR #439.)
3. **The failure mode is bounded and observable either way.** Provisioning is fire-and-forget and never
blocks/fails a deploy; a per-tag rejection surfaces only as the tally's `failed` count. Values written to
a still-Float tag by the recorder remain readable (Float supersets Int1's 0/1 range); nothing is lost
pending a reconcile. A bulk retype sweep is **not** done in code — assess via the probe first, then decide
whether reconciliation is a one-time operator action.
> **Prerequisite for provisioning to land at all:** the provisioner must stamp a **positive `StorageRateMs`**
> — a `0` made `EnsureTags` throw `ArgumentOutOfRangeException` and fail *every* provision (fixed in
> `GatewayTagProvisioner`, PR #439).
**Migration note for pre-existing Boolean tags.** A Boolean tag historian-side as Float (e.g. from the
0.1.0-era default) is asked, on its next deploy, to upsert to Int2. Both are supported analog types, so the
retype is accepted. Provisioning is fire-and-forget and never blocks/fails a deploy; a per-tag rejection (if
any) surfaces only as the tally's `failed` count, and values written to a still-Float tag remain readable
(Float supersets the 0/1 range) pending reconcile. A bulk retype sweep is **not** done in code.
---
@@ -407,11 +403,11 @@ The gateway-backed driver also ships an env-gated live suite (`Category=LiveInte
`HISTGW_GATEWAY_ENDPOINT` / `HISTGW_GATEWAY_APIKEY` / `HISTGW_TEST_TAG` / `HISTGW_WRITE_SANDBOX_TAG` /
`HISTGW_BOOL_SANDBOX_TAG` / `HISTGW_ALARM_SOURCE` env vars (it skips cleanly when they are absent).
`HISTGW_BOOL_SANDBOX_TAG` (e.g. `HistGW.LiveTest.BoolSandbox`, a **dedicated** writable Boolean tag —
never the shared Float write sandbox) drives the 0.2.0 Boolean `EnsureTags` leg
(`EnsureTags_boolean_sandbox_provisions_as_Int1`) and the retype probe
(`EnsureTags_type_change_outcome_is_observable`) that resolve the
[0.2.0 DataType provisioning delta](#020-datatype-provisioning-delta-archreview-06u-7) open question. Running
the full suite against a **0.2.0** gateway (all six tests, none skipped) also discharges 06/U-2's
never the shared Float write sandbox) drives the Boolean historization leg
(`EnsureTags_boolean_sandbox_provisions_as_Int2`) and the retype probe
(`EnsureTags_type_change_outcome_is_observable`) that back the
[Boolean historization maps to Int2](#boolean-historization-maps-to-int2-issue-441) decision. Running
the full suite against the gateway (all six tests, none skipped) also discharges 06/U-2's
"full documented run" for the historian suite.
See [AlarmHistorian.md](AlarmHistorian.md) for the alarm write path and
+193
View File
@@ -0,0 +1,193 @@
# Infrastructure — testing & deployment
Canonical index of every piece of infrastructure OtOpcUa's tests and deployments
depend on: the shared Docker host, the driver integration fixtures, the env-gated
**live-gate** recipes, the integration-test harness, the local dev rig, and the
gateway/deploy setups. This file is the single entry point; deeper history lives in
the linked docs.
> **Credentials below are dev-only** (per decision #137) — never production values.
> Production uses Integrated Security / gMSA (SQL) and a real directory (LDAP).
**Related deep docs**
- [`docs/v2/dev-environment.md`](../docs/v2/dev-environment.md) — full historical inventory + Windows bootstrap.
- [`docs/v2/Galaxy.ParityRig.md`](../docs/v2/Galaxy.ParityRig.md) — mxaccessgw / Galaxy parity-rig runbook.
- `docs/drivers/*-Test-Fixture.md` — per-driver fixture coverage maps.
- [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/) — the shared GLAuth source-of-truth + deploy runbook.
- `reference_histgw_local_deploy`, `reference_ww_historian_vm` (memory) — HistorianGateway deploy detail.
---
## 1. Shared Linux Docker host — `10.100.0.35`
All Docker workloads (central SQL, driver fixtures, integration-harness deps) run on a
shared Debian host, **not** on the dev machine. Discover everything with the
`project=lmxopcua` label:
```bash
ssh dohertj2@10.100.0.35 'docker ps --filter label=project=lmxopcua --format "{{.Names}}\t{{.Status}}"'
```
- **From this Mac**, drive it over passwordless SSH (`ssh dohertj2@10.100.0.35 'docker …'`).
`docker -H ssh://…` does **not** work from the Windows VM (stdio bridge hangs) — the
Windows VM uses the `lmxopcua-fix.ps1` helper (`~/bin`) instead; the Mac uses SSH directly.
- **Stacks** live at `/opt/otopcua-<driver>/` (root-owned) or `~/otopcua-<driver>/` (user-owned,
where `/opt` needs sudo). Current layout:
| Stack dir | Purpose |
|---|---|
| `/opt/otopcua-mssql` | central SQL (always-on) |
| `/opt/otopcua-modbus` · `/opt/otopcua-abcip` · `/opt/otopcua-s7` · `/opt/otopcua-opcuaclient` | driver fixtures |
| `~/otopcua-ablegacy` · `~/otopcua-focas` | driver fixtures (user-owned) |
| `~/otopcua-harness` | Host.IntegrationTests real-mode deps (SQL + GLAuth) — see §4 |
- The repo's `tests/.../Docker/` trees are the **source of truth**; the host dirs are mirrored
deployments. Push changes with `rsync`/`lmxopcua-fix sync` before `docker compose up`.
## 2. Central SQL Server (always-on)
| | |
|---|---|
| Container | `otopcua-mssql` (`mcr.microsoft.com/mssql/server:2022-latest`), stack `/opt/otopcua-mssql` |
| Endpoint | `10.100.0.35:14330``1433` |
| Creds (dev) | `sa` / `OtOpcUaDev_2026!` |
| Used by | `appsettings.json` `ConfigDb`; DB-backed test suites (`*.Configuration.Tests`, `*.Admin.Tests`, `*.Server.Tests`) |
Nuclear reset: `ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mssql && docker compose down -v && docker compose up -d'`.
## 3. Driver integration fixtures
Each driver's integration suite TCP-probes its fixture at collection init and **skips cleanly**
when it isn't running. Bring one up, `dotnet test`, tear down. Repo compose lives under
`tests/Drivers/…​.IntegrationTests/Docker/`.
| Driver | Fixture image | Endpoint | Bring up (on the host) |
|---|---|---|---|
| Modbus | `otopcua-pymodbus:3.13.0` | `10.100.0.35:5020` | `docker compose --profile <standard\|dl205\|mitsubishi\|s7_1500> up -d` |
| AB CIP | `otopcua-ab-server:libplctag-release` | `10.100.0.35:44818` | `docker compose --profile <controllogix\|compactlogix\|micro800\|guardlogix> up -d` |
| AB Legacy (PCCC) | `otopcua-ab-server` (PCCC mode) | `10.100.0.35:44818` | `AB_LEGACY_COMPOSE_PROFILE=<…> docker compose up -d` (stack `~/otopcua-ablegacy`) |
| S7 | `otopcua-python-snap7` | `10.100.0.35:1102` | `docker compose --profile s7_1500 up -d` |
| OpcUaClient | `mcr.microsoft.com/iotedge/opc-plc:2.14.10` | `opc.tcp://10.100.0.35:50000` | `docker compose up -d` |
| FOCAS | `otopcua-focas-sim` (vendored focas-mock) | `10.100.0.35:8193` | `docker compose up -d` (stack `~/otopcua-focas`) |
**Endpoint overrides** point a suite at a real PLC instead of the sim:
`MODBUS_SIM_ENDPOINT` · `AB_SERVER_ENDPOINT` · `S7_SIM_ENDPOINT` · `OPCUA_SIM_ENDPOINT`.
> **Fixture-cycling gotcha:** profile-gated services can share a port; a plain `docker compose down`
> may leave a stale container bound. Force-remove before switching profiles:
> `docker rm -f $(docker ps -aq --filter name=otopcua-<driver>)` then `up --force-recreate`.
## 4. Host.IntegrationTests real-mode harness
`tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests` runs on EF-InMemory + a stub LDAP by
default (zero infra). Real-mode swaps in **SQL Server** + a real **GLAuth** bind. Stack:
`~/otopcua-harness` (repo: `tests/Server/.../Host.IntegrationTests/docker-compose.yml` + `glauth/config.toml`).
| Dep | Container | Endpoint | Notes |
|---|---|---|---|
| SQL | `otopcua-harness-sql-1` | `10.100.0.35:14331` → 1433 | port 14331 avoids the always-on 14330 |
| GLAuth | `otopcua-harness-ldap-1` | `10.100.0.35:3894` → 3893 | GLAuth (swapped in for retired bitnami/openldap, PR #451); users below |
Harness GLAuth dev users (`glauth/config.toml`, baseDN `dc=zb,dc=local`):
`serviceaccount`/`serviceaccount123` (search), `alice`/`alice123` (full-access), `bob`/`bob123` (read-only).
```bash
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d'
export OTOPCUA_HARNESS_USE_SQL=1 OTOPCUA_HARNESS_USE_LDAP=1
export OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331
export OTOPCUA_HARNESS_LDAP_HOST=10.100.0.35 # harness uses port 3894
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose down'
```
## 5. Live-gate recipes (env-gated destructive tests)
These prove failure/recovery behavior against **real** infra and skip cleanly offline. Each is
gated by env vars and always restores the shared resource in a `finally`.
### 5a. S7 connect-timeout blackhole (`S7_1500ConnectTimeoutOutageTests`)
Proves an S7 subscription survives a SYN-blackhole outage (frozen PLC), which a `docker restart`
can't reproduce. `docker pause` gives the ACK-less-established shape — no sudo/VPN.
```bash
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d'
export S7_TIMEOUT_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-python-snap7-s7_1500"'
export S7_TIMEOUT_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-python-snap7-s7_1500"'
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.IntegrationTests --filter "FullyQualifiedName~ConnectTimeoutOutage"
```
### 5b. LDAP GLAuth outage/recovery (`LdapGlauthOutageLiveTests`)
Drives the directory-outage circuit through the production auth path against the harness GLAuth
(§4), paused/unpaused. Also verifies the GLAuth swap binds correctly.
```bash
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d ldap'
export GLAUTH_LIVE_HOST=10.100.0.35 # port defaults to 3894 (GLAUTH_LIVE_PORT)
export GLAUTH_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-harness-ldap-1"'
export GLAUTH_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-harness-ldap-1"'
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapGlauthOutageLive"
```
### 5c. HistorianGateway LiveIntegration soak (`Category=LiveIntegration`) — needs VPN
Exercises the gateway read/write/alarm/EnsureTags paths against a **real AVEVA historian**. Requires
the corporate **VPN up** and a running gateway (see §6). Full deploy detail: `reference_histgw_local_deploy`.
```bash
export HISTGW_GATEWAY_ENDPOINT=http://localhost:5221 # local gw, h2c
export HISTGW_GATEWAY_ALLOW_UNTRUSTED=true # required for h2c
export HISTGW_GATEWAY_APIKEY=histgw_placeholder_placeholder # gw ApiKeys:Mode=Disabled
export HISTGW_TEST_TAG=SysTimeSec
export HISTGW_WRITE_SANDBOX_TAG=HistGW.LiveTest.Sandbox
export HISTGW_BOOL_SANDBOX_TAG=HistGW.LiveTest.BoolSandbox
export HISTGW_ALARM_SOURCE=OtOpcUaLiveValidation
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests --filter "Category=LiveIntegration"
```
## 6. HistorianGateway — deploy for live testing
The gateway sidecar runs locally in Docker (`~/Desktop/HistorianGateway/docker/`) pointed at the
**real AVEVA historian** over VPN. It's the backend for OtOpcUa's historian read/alarm/continuous paths.
- **Hosts (VPN up):** historian gRPC CAP `wonder-sql-vd03.zmr.zimmer.com:32565` (TLS); Runtime SQL `:1433`
(`wonderapp`/`Wonderapp1`, SQL login); Galaxy GR SQL `wonder-app-vd03.zmr.zimmer.com:1433`.
- **Deploy:** publish FDD on host, then `docker compose up -d --build`. gRPC h2c `localhost:5221`,
dashboard/health `localhost:5220`, `ApiKeys:Mode=Disabled`. Corporate hostnames go in a gitignored
`docker/.env` / secrets, never in committed compose.
- **Known deploy fixes** (committed to the gateway repo): FQDN via `.env`; `libgssapi-krb5-2 gss-ntlmssp`
in the Dockerfile (historian v6 + SQL paths auth `NAM\dohertj2` via Negotiate/GSSAPI); `RuntimeDb` on.
- Full recipe + gotchas: `reference_histgw_local_deploy` · historian VM: `reference_ww_historian_vm`.
## 7. docker-dev local rig — `docker-dev/docker-compose.yml`
A 2-node OtOpcUa cluster built from source for AdminUI / OPC UA `/run` verification.
| Surface | Endpoint | Notes |
|---|---|---|
| AdminUI | `http://localhost:9200` (via Traefik) | round-robins central-1/2 — **rebuild both** to verify a UI change |
| OPC UA | `opc.tcp://localhost:4840` (central-1) · `:4841` (central-2) | data plane |
| Login | disabled (`Security__Auth__DisableLogin=true`) | auto-authenticated full-access admin — no sign-in needed |
| LDAP (data plane) | shared GLAuth `10.100.0.35:3893` (`DevStubMode=false`) | gates Client.CLI read/write **role** ops only |
```bash
docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2 traefik
# headless deploy: POST http://localhost:9200/api/deployments (X-Api-Key)
docker compose -f docker-dev/docker-compose.yml down -v # reset
```
## 8. Shared GLAuth (data-plane auth) — `10.100.0.35:3893`
The project-wide LDAP for docker-dev + live OPC UA data-plane auth (distinct from the harness
GLAuth on :3894). Container `zb-shared-glauth`, baseDN `dc=zb,dc=local`, bind
`cn=serviceaccount,dc=zb,dc=local`/`serviceaccount123`, test users password `password`. **Source of
truth + deploy runbook:** [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/).
## 9. Deployment (production rig)
The deploy host runs the OtOpcUa server + `mxaccessgw` + the HistorianGateway sidecar as
NSSM-wrapped services from `C:\publish\`. Refresh after landing changes with
`scripts/install/Refresh-Services.ps1` (stops reverse-dependency order → backs up → publishes →
restarts → smoke-verifies ports 4840/4841). Production creds use Integrated Security / gMSA (SQL)
and a real directory (LDAP) — never the dev values in this doc.
> The legacy Wonderware TCP/ArchestrA sidecar was retired; the historian backend is now the
> `ZB.MOM.WW.HistorianGateway` sidecar (see §6 and `docs/Historian.md`).
@@ -13,6 +13,12 @@
<ItemGroup>
<PackageReference Include="Microsoft.Data.Sqlite"/>
<!--
Direct pin of the native SQLite bundle to 2.1.12 (first version outside the
CVE-2025-6965 / GHSA-2m69-gcr7-jv3q range) — overrides the 2.1.11 that
Microsoft.Data.Sqlite pulls transitively. See Directory.Packages.props.
-->
<PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3"/>
<PackageReference Include="Serilog"/>
</ItemGroup>
@@ -50,6 +50,8 @@ public sealed class SubscribeCommand : S7CommandBase
{
await driver.InitializeAsync("{}", ct);
// Route every data-change event to the CliFx console (not System.Console — the
// analyzer flags it + IConsole is the testable abstraction).
driver.OnDataChange += (_, e) =>
{
var line = $"[{DateTime.UtcNow:HH:mm:ss.fff}] " +
@@ -75,10 +75,10 @@ public static class AbCipDriverFactoryExtensions
{
Enabled = dto.Probe?.Enabled ?? true,
Interval = TimeSpan.FromMilliseconds(dto.Probe?.IntervalMs ?? 5_000),
Timeout = TimeSpan.FromMilliseconds(dto.Probe?.TimeoutMs ?? 2_000),
Timeout = PositiveTimeoutOrDefault(dto.Probe?.TimeoutMs, 2_000),
ProbeTagPath = dto.Probe?.ProbeTagPath,
},
Timeout = TimeSpan.FromMilliseconds(dto.TimeoutMs ?? 2_000),
Timeout = PositiveTimeoutOrDefault(dto.TimeoutMs, 2_000),
EnableControllerBrowse = dto.EnableControllerBrowse ?? false,
EnableAlarmProjection = dto.EnableAlarmProjection ?? false,
AlarmPollInterval = TimeSpan.FromMilliseconds(dto.AlarmPollIntervalMs ?? 1_000),
@@ -86,6 +86,20 @@ public static class AbCipDriverFactoryExtensions
};
}
/// <summary>
/// Maps an operator-supplied timeout (ms) to a positive <see cref="TimeSpan"/>, substituting
/// <paramref name="defaultMs"/> for a null / zero / negative value. libplctag's managed
/// <c>Tag.Timeout</c> setter throws <see cref="ArgumentOutOfRangeException"/> for a
/// non-positive value, which would otherwise fault tag creation on every read/write; clamping
/// here keeps a misconfigured <c>TimeoutMs: 0</c> running on the default bound instead. (Driver
/// timeout hardening, sibling of the S7 R2-01 read-leg fix.)
/// </summary>
/// <param name="ms">The operator-supplied timeout in milliseconds, or null.</param>
/// <param name="defaultMs">The default applied when <paramref name="ms"/> is null or non-positive.</param>
/// <returns>A strictly positive timeout.</returns>
private static TimeSpan PositiveTimeoutOrDefault(int? ms, int defaultMs) =>
TimeSpan.FromMilliseconds(ms is int v and > 0 ? v : defaultMs);
private static AbCipTagDefinition BuildTag(AbCipTagDto t, string driverInstanceId) =>
new(
Name: t.Name ?? throw new InvalidOperationException(
@@ -264,6 +264,8 @@ public sealed class AbLegacyDriver : IDriver, IReadable, IWritable, ITagDiscover
if (status != 0)
{
// Evict the stale handle so the next call re-creates it (mirrors AbCip).
EvictRuntime(device, def.Name);
results[i] = new DataValueSnapshot(null,
AbLegacyStatusMapper.MapLibplctagStatus(status), null, now);
_health = new DriverHealth(DriverState.Degraded, _health.LastSuccessfulRead,
@@ -287,6 +289,8 @@ public sealed class AbLegacyDriver : IDriver, IReadable, IWritable, ITagDiscover
catch (OperationCanceledException) { throw; }
catch (Exception ex)
{
// Transport exception — evict so the next read creates a fresh handle (mirrors AbCip).
EvictRuntime(device, def.Name);
results[i] = new DataValueSnapshot(null,
AbLegacyStatusMapper.BadCommunicationError, null, now);
_health = new DriverHealth(DriverState.Degraded, _health.LastSuccessfulRead, ex.Message);
@@ -360,9 +364,16 @@ public sealed class AbLegacyDriver : IDriver, IReadable, IWritable, ITagDiscover
opLock.Release();
}
results[i] = new WriteResult(status == 0
? AbLegacyStatusMapper.Good
: AbLegacyStatusMapper.MapLibplctagStatus(status));
if (status != 0)
{
// Evict the stale handle so the next call re-creates it (mirrors AbCip).
EvictRuntime(device, def.Name);
results[i] = new WriteResult(AbLegacyStatusMapper.MapLibplctagStatus(status));
}
else
{
results[i] = new WriteResult(AbLegacyStatusMapper.Good);
}
}
catch (OperationCanceledException) { throw; }
catch (NotSupportedException nse)
@@ -380,6 +391,8 @@ public sealed class AbLegacyDriver : IDriver, IReadable, IWritable, ITagDiscover
}
catch (Exception ex)
{
// Transport exception — evict so the next write/read creates a fresh handle (mirrors AbCip).
EvictRuntime(device, def.Name);
results[i] = new WriteResult(AbLegacyStatusMapper.BadCommunicationError);
_health = new DriverHealth(DriverState.Degraded, _health.LastSuccessfulRead, ex.Message);
}
@@ -685,6 +698,26 @@ public sealed class AbLegacyDriver : IDriver, IReadable, IWritable, ITagDiscover
}
}
/// <summary>
/// Evict the runtime for <paramref name="tagName"/> from the device's cache and dispose it so
/// the next read/write call re-creates and re-initializes a fresh libplctag handle. Called
/// after a non-zero libplctag status or a transport exception on a read/write — a non-zero
/// status can mean the controller dropped the connection or the handle became permanently
/// invalid (e.g. after a PLC download), and reusing it would return the same failure forever.
/// Mirrors <c>AbCipDriver.EvictRuntime</c> and the probe loop's recreate-on-failure behaviour
/// (previously AbLegacy leaned only on the probe loop + libplctag's internal recovery, so a
/// data-path fault recovered more slowly than in AbCip).
/// </summary>
/// <param name="device">The device whose runtime cache holds the stale handle.</param>
/// <param name="tagName">The tag name key of the runtime to evict.</param>
private static void EvictRuntime(DeviceState device, string tagName)
{
if (device.Runtimes.TryRemove(tagName, out var stale))
{
try { stale.Dispose(); } catch { /* best-effort */ }
}
}
/// <summary>
/// Synchronous teardown. Mirrors the body of
/// <see cref="ShutdownAsync"/> but never wraps the async path in
@@ -83,10 +83,10 @@ public static class AbLegacyDriverFactoryExtensions
{
Enabled = dto.Probe?.Enabled ?? true,
Interval = TimeSpan.FromMilliseconds(dto.Probe?.IntervalMs ?? 5_000),
Timeout = TimeSpan.FromMilliseconds(dto.Probe?.TimeoutMs ?? 2_000),
Timeout = PositiveTimeoutOrDefault(dto.Probe?.TimeoutMs, 2_000),
ProbeAddress = dto.Probe?.ProbeAddress ?? "S:0",
},
Timeout = TimeSpan.FromMilliseconds(dto.TimeoutMs ?? 2_000),
Timeout = PositiveTimeoutOrDefault(dto.TimeoutMs, 2_000),
};
return new AbLegacyDriver(
@@ -95,6 +95,20 @@ public static class AbLegacyDriverFactoryExtensions
logger: loggerFactory?.CreateLogger<AbLegacyDriver>());
}
/// <summary>
/// Maps an operator-supplied timeout (ms) to a positive <see cref="TimeSpan"/>, substituting
/// <paramref name="defaultMs"/> for a null / zero / negative value. libplctag's managed
/// <c>Tag.Timeout</c> setter throws <see cref="ArgumentOutOfRangeException"/> for a
/// non-positive value, which would otherwise fault tag creation on every read/write; clamping
/// here keeps a misconfigured <c>TimeoutMs: 0</c> running on the default bound instead. (Driver
/// timeout hardening, sibling of the S7 R2-01 read-leg fix.)
/// </summary>
/// <param name="ms">The operator-supplied timeout in milliseconds, or null.</param>
/// <param name="defaultMs">The default applied when <paramref name="ms"/> is null or non-positive.</param>
/// <returns>A strictly positive timeout.</returns>
private static TimeSpan PositiveTimeoutOrDefault(int? ms, int defaultMs) =>
TimeSpan.FromMilliseconds(ms is int v and > 0 ? v : defaultMs);
private static T ParseEnum<T>(string? raw, string driverInstanceId, string field,
string? tagName = null, T? fallback = null) where T : struct, Enum
{
@@ -84,9 +84,9 @@ public static class FocasDriverFactoryExtensions
{
Enabled = dto.Probe?.Enabled ?? true,
Interval = TimeSpan.FromMilliseconds(dto.Probe?.IntervalMs ?? 5_000),
Timeout = TimeSpan.FromMilliseconds(dto.Probe?.TimeoutMs ?? 2_000),
Timeout = PositiveTimeoutOrDefault(dto.Probe?.TimeoutMs, 2_000),
},
Timeout = TimeSpan.FromMilliseconds(dto.TimeoutMs ?? 2_000),
Timeout = PositiveTimeoutOrDefault(dto.TimeoutMs, 2_000),
FixedTree = BuildFixedTree(dto.FixedTree),
AlarmProjection = BuildAlarmProjection(dto.AlarmProjection),
HandleRecycle = BuildHandleRecycle(dto.HandleRecycle),
@@ -96,6 +96,20 @@ public static class FocasDriverFactoryExtensions
return new FocasDriver(options, driverInstanceId, clientFactory);
}
/// <summary>
/// Maps an operator-supplied timeout (ms) to a positive <see cref="TimeSpan"/>, substituting
/// <paramref name="defaultMs"/> for a null / zero / negative value. A non-positive
/// <c>Timeout</c> would make <see cref="SynchronizedFocasClient"/> DISABLE its per-call
/// wall-clock ceiling (falling back to the caller's long-lived poll token), reintroducing the
/// exact unbounded-read-on-a-frozen-peer wedge the S7 R2-01 read-leg fix eliminated. Clamping
/// here at the config boundary guarantees the deadline can never be authored away.
/// </summary>
/// <param name="ms">The operator-supplied timeout in milliseconds, or null.</param>
/// <param name="defaultMs">The default applied when <paramref name="ms"/> is null or non-positive.</param>
/// <returns>A strictly positive timeout.</returns>
private static TimeSpan PositiveTimeoutOrDefault(int? ms, int defaultMs) =>
TimeSpan.FromMilliseconds(ms is int v and > 0 ? v : defaultMs);
/// <summary>
/// Builds the appropriate <see cref="IFocasClientFactory"/> based on the config DTO's backend selection.
/// </summary>
@@ -9,11 +9,14 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Mapping;
/// </summary>
/// <remarks>
/// Only the nine numeric types are historizable on the gateway's analog write path. Two of them
/// fall back to a wider historian type because the narrower one's write path is deferred upstream:
/// <see cref="DriverDataType.UInt16"/> maps to <see cref="HistorianDataType.Uint4"/> (the historian's
/// <c>UInt2</c> write path is not proven). String / DateTime / Reference are not historized in v1
/// and throw <see cref="NotSupportedException"/>; callers that want to skip them without catching an
/// exception should consult <see cref="IsHistorizable(DriverDataType)"/> first.
/// fall back to a wider/adjacent historian type because the exact one's write path is not usable
/// upstream: <see cref="DriverDataType.UInt16"/> maps to <see cref="HistorianDataType.Uint4"/> (the
/// historian's <c>UInt2</c> write path is not proven), and <see cref="DriverDataType.Boolean"/> maps
/// to <see cref="HistorianDataType.Int2"/> — the historian's <c>Int1</c> analog-tag creation path is
/// server-degenerate (issue #441 / HistorianGateway #11: <c>EnsureTags(Int1)</c> stores an unusable
/// stub), so Boolean historizes as a small 0/1 integer instead. String / DateTime / Reference are not
/// historized in v1 and throw <see cref="NotSupportedException"/>; callers that want to skip them
/// without catching an exception should consult <see cref="IsHistorizable(DriverDataType)"/> first.
/// </remarks>
internal static class HistorianTypeMapper
{
@@ -25,7 +28,8 @@ internal static class HistorianTypeMapper
/// </exception>
public static HistorianDataType ToHistorianDataType(DriverDataType dataType) => dataType switch
{
DriverDataType.Boolean => HistorianDataType.Int1,
DriverDataType.Boolean => HistorianDataType.Int2, // Int1 analog-tag creation is server-degenerate (issue #441) → store 0/1
DriverDataType.Int16 => HistorianDataType.Int2,
DriverDataType.Int32 => HistorianDataType.Int4,
DriverDataType.Int64 => HistorianDataType.Int8,
@@ -87,6 +87,7 @@ public interface IS7PlcFactory
internal sealed class S7PlcAdapter : IS7Plc
{
private readonly Plc _plc;
private readonly TimeSpan _operationTimeout;
/// <summary>Initializes a new instance of the <see cref="S7PlcAdapter"/> class.</summary>
/// <param name="cpuType">The CPU family.</param>
@@ -98,9 +99,14 @@ internal sealed class S7PlcAdapter : IS7Plc
public S7PlcAdapter(S7CpuType cpuType, string host, int port, short rack, short slot, TimeSpan timeout)
{
_plc = new Plc(S7CpuTypeMap.ToS7Net(cpuType), host, port, rack, slot);
_operationTimeout = timeout;
// S7netplus writes timeouts into the underlying TcpClient via ReadTimeout / WriteTimeout
// (milliseconds). Set before OpenAsync so the handshake itself honours the bound — matches
// the original inline construction in S7Driver.InitializeAsync.
// the original inline construction in S7Driver.InitializeAsync. NOTE: these map to the
// TcpClient's ReceiveTimeout / SendTimeout, which govern only SYNCHRONOUS socket calls; the
// async read/write paths S7.Net actually uses ignore them, so every data-plane op below is
// additionally bounded by S7OperationDeadline (a real wall-clock ceiling). OpenAsync is left
// to S7Driver.EnsureConnectedAsync's own CancelAfter, so it is not double-bounded here.
var ms = (int)timeout.TotalMilliseconds;
_plc.ReadTimeout = ms;
_plc.WriteTimeout = ms;
@@ -117,23 +123,30 @@ internal sealed class S7PlcAdapter : IS7Plc
/// <inheritdoc />
public Task<object?> ReadAsync(string address, CancellationToken cancellationToken) =>
_plc.ReadAsync(address, cancellationToken)!;
S7OperationDeadline.RunAsync(
t => _plc.ReadAsync(address, t)!, _operationTimeout, $"read of '{address}'", cancellationToken);
/// <inheritdoc />
public Task<byte[]> ReadBytesAsync(S7NetDataType area, int db, int startByteAdr, int count, CancellationToken cancellationToken) =>
_plc.ReadBytesAsync(area, db, startByteAdr, count, cancellationToken);
S7OperationDeadline.RunAsync(
t => _plc.ReadBytesAsync(area, db, startByteAdr, count, t),
_operationTimeout, $"block read of {area} db{db} @{startByteAdr}+{count}", cancellationToken);
/// <inheritdoc />
public Task WriteAsync(string address, object value, CancellationToken cancellationToken) =>
_plc.WriteAsync(address, value, cancellationToken);
S7OperationDeadline.RunAsync(
t => _plc.WriteAsync(address, value, t), _operationTimeout, $"write of '{address}'", cancellationToken);
/// <inheritdoc />
public Task WriteBytesAsync(S7NetDataType area, int db, int startByteAdr, byte[] value, CancellationToken cancellationToken) =>
_plc.WriteBytesAsync(area, db, startByteAdr, value, cancellationToken);
S7OperationDeadline.RunAsync(
t => _plc.WriteBytesAsync(area, db, startByteAdr, value, t),
_operationTimeout, $"block write of {area} db{db} @{startByteAdr}", cancellationToken);
/// <inheritdoc />
public async Task ReadStatusAsync(CancellationToken cancellationToken) =>
_ = await _plc.ReadStatusAsync(cancellationToken).ConfigureAwait(false);
public Task ReadStatusAsync(CancellationToken cancellationToken) =>
S7OperationDeadline.RunAsync(
t => _plc.ReadStatusAsync(t), _operationTimeout, "status read", cancellationToken);
/// <inheritdoc />
// Plc implements IDisposable explicitly, so cast to invoke it; disposal tears down the TcpClient.
@@ -1287,7 +1287,9 @@ public sealed class S7Driver
/// True when <paramref name="ex"/> (or any inner exception) is a socket-level / connection
/// loss OR an ISO-on-TCP framing/desync fault that a reopen can repair — a
/// <see cref="System.Net.Sockets.SocketException"/> / <see cref="System.IO.IOException"/> /
/// <see cref="ObjectDisposedException"/>; an S7.Net framing violation
/// <see cref="ObjectDisposedException"/>; a <see cref="TimeoutException"/> from a wire op that
/// blew its <see cref="S7OperationDeadline"/> wall-clock ceiling (an unresponsive but still
/// TCP-established peer — the READ-leg sibling of the connect-timeout fix, STAB-14); an S7.Net framing violation
/// (<see cref="TPKTInvalidException"/> / <see cref="TPDUInvalidException"/> /
/// <see cref="WrongNumberOfBytesException"/>); or a <see cref="PlcException"/> carrying
/// <see cref="ErrorCode.ConnectionError"/> or <see cref="ErrorCode.WrongNumberReceivedBytes"/>.
@@ -1314,6 +1316,13 @@ public sealed class S7Driver
{
if (e is System.Net.Sockets.SocketException or System.IO.IOException or ObjectDisposedException)
return true;
// A wire op that blew its wall-clock deadline (S7OperationDeadline — the async read/write
// ceiling S7.Net's ignored socket ReadTimeout/WriteTimeout can't provide) means the
// connection is unresponsive: discard + reopen. This is the READ-leg sibling of the
// connect-timeout fix (STAB-14) — without it a frozen-but-established peer wedges the
// poll loop forever instead of surfacing Bad + reconnecting.
if (e is TimeoutException)
return true;
// ISO-on-TCP framing/desync surface (STAB-15a): the stream position is untrustworthy —
// a half-read PDU left by a timeout/cancellation makes every later response mis-frame.
// NOTE: deliberately NOT System.IO.InvalidDataException — ReinterpretRawValue throws it
@@ -0,0 +1,80 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.S7;
/// <summary>
/// Bounds a single S7 wire operation with a wall-clock deadline, surfacing an overrun as a
/// <see cref="TimeoutException"/>.
/// </summary>
/// <remarks>
/// <para>
/// S7.Net's <c>ReadTimeout</c> / <c>WriteTimeout</c> set the underlying
/// <see cref="System.Net.Sockets.TcpClient"/>'s <c>ReceiveTimeout</c> / <c>SendTimeout</c>,
/// which govern only <b>synchronous</b> socket calls — the <c>async</c> read/write paths
/// S7.Net actually uses ignore them entirely. Without an explicit ceiling, a read on an
/// established-but-frozen connection (a frozen PLC, a firewall DROP, or a cable pulled
/// mid-flow — where the TCP session stays <i>open</i> but no reply ever arrives) blocks until
/// the OS TCP stack gives up, which can be minutes. That silently wedges the driver's poll
/// loop: no Bad tick, no reconnect — the STAB-14 "silently killed poll loop" failure, on the
/// READ leg that the connect-timeout fix (<c>S7Driver.EnsureConnectedAsync</c>'s
/// <c>CancelAfter</c>) never covered.
/// </para>
/// <para>
/// The timed-out operation surfaces as a <see cref="TimeoutException"/>, which
/// <c>S7Driver.IsS7ConnectionFatal</c> classifies connection-fatal → the handle is marked
/// dead → the next <c>EnsureConnectedAsync</c> reopens. The end-to-end proof is the live gate
/// <c>S7_1500ConnectTimeoutOutageTests</c> (a real snap7 sim frozen with <c>docker pause</c>).
/// </para>
/// </remarks>
internal static class S7OperationDeadline
{
/// <summary>Runs <paramref name="operation"/> under a <paramref name="timeout"/> wall-clock ceiling.</summary>
/// <typeparam name="T">The operation result type.</typeparam>
/// <param name="operation">The wire operation; receives a token cancelled at the deadline.</param>
/// <param name="timeout">The wall-clock ceiling for the operation.</param>
/// <param name="what">A short description of the operation for the timeout message (e.g. <c>read of 'DB1.DBW0'</c>).</param>
/// <param name="ct">The caller's cancellation token — its cancellation propagates as an <see cref="OperationCanceledException"/>, not a timeout.</param>
/// <returns>The operation result when it completes within the deadline.</returns>
/// <exception cref="TimeoutException">The operation did not complete within <paramref name="timeout"/>.</exception>
public static async Task<T> RunAsync<T>(
Func<CancellationToken, Task<T>> operation, TimeSpan timeout, string what, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(operation);
using var opCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
opCts.CancelAfter(timeout);
try
{
// WaitAsync guarantees control returns at the deadline even if the underlying op ignores
// its token; passing opCts.Token additionally lets an op that DOES honour cancellation
// unwind cleanly instead of leaking a wedged socket read behind an abandoned task.
return await operation(opCts.Token).WaitAsync(timeout, ct).ConfigureAwait(false);
}
catch (TimeoutException)
{
// WaitAsync's own deadline fired (the op ignored the token).
throw new TimeoutException(FormatMessage(what, timeout));
}
catch (OperationCanceledException) when (opCts.IsCancellationRequested && !ct.IsCancellationRequested)
{
// OUR deadline fired and the op honoured opCts.Token — not the caller's cancellation.
throw new TimeoutException(FormatMessage(what, timeout));
}
}
/// <summary>Runs a result-less <paramref name="operation"/> under a <paramref name="timeout"/> wall-clock ceiling.</summary>
/// <param name="operation">The wire operation; receives a token cancelled at the deadline.</param>
/// <param name="timeout">The wall-clock ceiling for the operation.</param>
/// <param name="what">A short description of the operation for the timeout message.</param>
/// <param name="ct">The caller's cancellation token.</param>
/// <returns>A task that completes when the operation completes within the deadline.</returns>
/// <exception cref="TimeoutException">The operation did not complete within <paramref name="timeout"/>.</exception>
public static Task RunAsync(
Func<CancellationToken, Task> operation, TimeSpan timeout, string what, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(operation);
return RunAsync(
async token => { await operation(token).ConfigureAwait(false); return true; },
timeout, what, ct);
}
private static string FormatMessage(string what, TimeSpan timeout) =>
$"S7 {what} did not complete within {(int)timeout.TotalMilliseconds} ms.";
}
@@ -27,7 +27,11 @@ public sealed record AbServerProfile(
public static class KnownProfiles
{
/// <summary>Gets all known server profiles.</summary>
public static IReadOnlyList<AbServerProfile> All { get; } =
/// <remarks>Expression-bodied (evaluated on access) so it observes the
/// <c>static readonly</c> profile fields below after they are initialised;
/// a <c>{ get; } =</c> initialiser here runs in textual order — before those
/// fields — and would capture an all-null list.</remarks>
public static IReadOnlyList<AbServerProfile> All =>
[ControlLogix, CompactLogix, Micro800, GuardLogix];
/// <summary>Gets the ControlLogix profile.</summary>
@@ -17,6 +17,33 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests;
[Trait("Category", "Unit")]
public sealed class AbCipFactoryArrayTagTests
{
/// <summary>
/// A non-positive <c>TimeoutMs</c> clamps to the 2 s default rather than reaching libplctag's
/// <c>Tag.Timeout</c> setter (which throws <see cref="ArgumentOutOfRangeException"/> for a
/// non-positive value, faulting every read/write). Driver timeout hardening.
/// </summary>
[Theory]
[InlineData(0)]
[InlineData(-1)]
public void ParseOptions_clamps_non_positive_TimeoutMs_to_the_default(int timeoutMs)
{
var json = $$$"""{"TimeoutMs":{{{timeoutMs}}},"Probe":{"TimeoutMs":{{{timeoutMs}}}}}""";
var opts = AbCipDriverFactoryExtensions.ParseOptions("drv-1", json);
opts.Timeout.ShouldBe(TimeSpan.FromMilliseconds(2_000));
opts.Timeout.ShouldBeGreaterThan(TimeSpan.Zero);
opts.Probe.Timeout.ShouldBe(TimeSpan.FromMilliseconds(2_000));
}
/// <summary>A positive <c>TimeoutMs</c> is honoured verbatim.</summary>
[Fact]
public void ParseOptions_honours_a_positive_TimeoutMs()
{
var opts = AbCipDriverFactoryExtensions.ParseOptions("drv-1", """{"TimeoutMs":500}""");
opts.Timeout.ShouldBe(TimeSpan.FromMilliseconds(500));
}
/// <summary>A driver-config tag declaring isArray/elementCount produces an array definition.</summary>
[Fact]
public void ParseOptions_threads_isArray_and_elementCount_into_tag_definition()
@@ -0,0 +1,109 @@
using libplctag;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.AbLegacy;
namespace ZB.MOM.WW.OtOpcUa.Driver.AbLegacy.Tests;
/// <summary>
/// Driver-timeout-hardening parity with AbCip: a read/write that fails with a non-zero libplctag
/// status or a transport exception must EVICT the cached runtime so the next call re-creates a
/// fresh handle — rather than returning the same failure forever until the probe loop happens to
/// recycle it. Previously AbLegacy evicted on neither path (unlike AbCip).
/// </summary>
[Trait("Category", "Unit")]
public sealed class AbLegacyEvictOnFailureTests
{
private const string Device = "ab://10.0.0.5/1,0";
// Probe disabled — the probe loop also calls _tagFactory.Create, which would perturb the
// creation count these tests assert on.
private static (AbLegacyDriver drv, FakeAbLegacyTagFactory factory) NewDriver(params AbLegacyTagDefinition[] tags)
{
var factory = new FakeAbLegacyTagFactory();
var drv = new AbLegacyDriver(new AbLegacyDriverOptions
{
Devices = [new AbLegacyDeviceOptions(Device)],
Tags = tags,
Probe = new AbLegacyProbeOptions { Enabled = false },
}, "drv-1", factory);
return (drv, factory);
}
/// <summary>A non-zero libplctag status on read evicts the runtime; the next read creates a fresh handle and recovers.</summary>
[Fact]
public async Task Read_nonzero_status_evicts_runtime_so_next_read_creates_fresh_handle()
{
var (drv, factory) = NewDriver(
new AbLegacyTagDefinition("Counter", Device, "N7:0", AbLegacyDataType.Int));
var callCount = 0;
factory.Customise = p =>
{
callCount++;
return callCount == 1
? new FakeAbLegacyTag(p) { Status = (int)Status.ErrorBadConnection }
: new FakeAbLegacyTag(p) { Status = 0, Value = 42 };
};
await drv.InitializeAsync("{}", CancellationToken.None);
// First read — bad status → runtime evicted, no reuse.
var first = await drv.ReadAsync(["Counter"], CancellationToken.None);
first.Single().StatusCode.ShouldNotBe(AbLegacyStatusMapper.Good);
// Second read — fresh handle, succeeds.
var second = await drv.ReadAsync(["Counter"], CancellationToken.None);
second.Single().StatusCode.ShouldBe(AbLegacyStatusMapper.Good);
second.Single().Value.ShouldBe(42);
callCount.ShouldBe(2); // one failed handle + one fresh handle
}
/// <summary>A transport exception on read evicts the runtime; the next read creates a fresh handle and recovers.</summary>
[Fact]
public async Task Read_transport_exception_evicts_runtime_so_next_read_creates_fresh_handle()
{
var (drv, factory) = NewDriver(
new AbLegacyTagDefinition("Counter", Device, "N7:0", AbLegacyDataType.Int));
var callCount = 0;
factory.Customise = p =>
{
callCount++;
return callCount == 1
? new FakeAbLegacyTag(p) { ThrowOnRead = true, Exception = new System.Net.Sockets.SocketException() }
: new FakeAbLegacyTag(p) { Status = 0, Value = 7 };
};
await drv.InitializeAsync("{}", CancellationToken.None);
var first = await drv.ReadAsync(["Counter"], CancellationToken.None);
first.Single().StatusCode.ShouldBe(AbLegacyStatusMapper.BadCommunicationError);
var second = await drv.ReadAsync(["Counter"], CancellationToken.None);
second.Single().StatusCode.ShouldBe(AbLegacyStatusMapper.Good);
second.Single().Value.ShouldBe(7);
callCount.ShouldBe(2);
}
/// <summary>A non-zero libplctag status on write evicts the runtime; the next write creates a fresh handle and recovers.</summary>
[Fact]
public async Task Write_nonzero_status_evicts_runtime_so_next_write_creates_fresh_handle()
{
var (drv, factory) = NewDriver(
new AbLegacyTagDefinition("Counter", Device, "N7:0", AbLegacyDataType.Int));
var callCount = 0;
factory.Customise = p =>
{
callCount++;
return callCount == 1
? new FakeAbLegacyTag(p) { Status = (int)Status.ErrorBadConnection }
: new FakeAbLegacyTag(p) { Status = 0 };
};
await drv.InitializeAsync("{}", CancellationToken.None);
var first = await drv.WriteAsync([new WriteRequest("Counter", (short)5)], CancellationToken.None);
first.Single().StatusCode.ShouldNotBe(AbLegacyStatusMapper.Good);
var second = await drv.WriteAsync([new WriteRequest("Counter", (short)5)], CancellationToken.None);
second.Single().StatusCode.ShouldBe(AbLegacyStatusMapper.Good);
callCount.ShouldBe(2);
}
}
@@ -415,7 +415,12 @@ class FocasMockServer:
timer_type = self._block_u32(request_block, 8)
key_map = {0: "power_on", 1: "operating", 2: "cutting", 3: "cycle"}
seconds = int(self.store.snapshot()["timers"].get(key_map.get(timer_type, "power_on"), 0))
return self._u32(seconds // 60) + self._u32((seconds % 60) * 1000)
# cnc_rdtimer's minute+msec fields are LITTLE-endian on the wire — unlike the big-endian
# block envelope and every other payload. Hardware-validated against a live FANUC 31i-B
# (2026-06-25); the driver's FocasWireClient.ParseTimer decodes them little-endian. Encoding
# them big-endian here (the old bug) made the driver read minute=60 as 0x3C000000. See
# docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md.
return self._u32_le(seconds // 60) + self._u32_le((seconds % 60) * 1000)
def _wire_alarms(self, request_block: bytes) -> bytes:
requested = max(self._block_u32(request_block, 12), 1)
@@ -481,6 +486,11 @@ class FocasMockServer:
def _u32(self, value: int) -> bytes:
return int(value).to_bytes(4, "big", signed=True)
def _u32_le(self, value: int) -> bytes:
# Little-endian u32 — only cnc_rdtimer's minute/msec fields use this on real hardware
# (see _wire_timer). Every other wire field is big-endian via _u32.
return int(value).to_bytes(4, "little", signed=True)
def _ascii_fixed(self, value: str, length: int) -> bytes:
return value.encode("ascii", errors="replace")[:length].ljust(length, b" ")
@@ -29,6 +29,11 @@ public sealed class FocasSimFixture : IAsyncDisposable
/// <summary>Gets the TCP port of the focas-mock simulator.</summary>
public int Port { get; }
/// <summary>Gets the <c>focas://host:port</c> device URI the driver connects to — derived from
/// <see cref="Host"/>/<see cref="Port"/> so tests honour <c>OTOPCUA_FOCAS_SIM_ENDPOINT</c> (a remote
/// fixture on the Docker host) instead of assuming a localhost-colocated mock.</summary>
public string DeviceUri => $"focas://{Host}:{Port}";
/// <summary>focas-mock profile stem the fixture should load (e.g. <c>fwlib30i64</c>,
/// <c>ThirtyOne_i</c> — both resolve via the mock's alias table). Null when unset.</summary>
public string? ExpectedProfile { get; }
@@ -23,7 +23,9 @@ public sealed class WireBackendCoverageTests
/// <param name="fx">The FOCAS simulation fixture.</param>
public WireBackendCoverageTests(FocasSimFixture fx) => _fx = fx;
private const string DeviceHost = "focas://127.0.0.1:8193";
// Derived from the fixture endpoint (OTOPCUA_FOCAS_SIM_ENDPOINT) so the driver dials the SAME
// mock the fixture probed — a remote fixture on the Docker host, not a hardcoded 127.0.0.1.
private string DeviceHost => _fx.DeviceUri;
/// <summary>Verifies that user tag reads route via the wire backend.</summary>
[Fact]
@@ -26,7 +26,9 @@ public sealed class WireBackendTests
/// <param name="fx">The FOCAS simulator fixture.</param>
public WireBackendTests(FocasSimFixture fx) => _fx = fx;
private const string DeviceHost = "focas://127.0.0.1:8193";
// Derived from the fixture endpoint (OTOPCUA_FOCAS_SIM_ENDPOINT) so the driver dials the SAME
// mock the fixture probed — a remote fixture on the Docker host, not a hardcoded 127.0.0.1.
private string DeviceHost => _fx.DeviceUri;
/// <summary>Verifies that identity axes and dynamic data populate via the wire backend.</summary>
[Fact]
@@ -57,6 +57,40 @@ public sealed class FocasFactoryConfigTests
drv.Options.Devices[0].Series.ShouldBe(FocasCncSeries.Thirty_i);
}
// ---- Driver timeout hardening: a non-positive TimeoutMs must not disable the per-call ceiling ----
/// <summary>
/// A misconfigured <c>TimeoutMs: 0</c> (or negative) must clamp to the 2 s default, NOT map to
/// <see cref="TimeSpan.Zero"/> — a zero/negative <c>Timeout</c> makes <c>SynchronizedFocasClient</c>
/// disable its per-call wall-clock ceiling, reintroducing the unbounded-read-on-a-frozen-peer
/// wedge the S7 R2-01 read-leg fix eliminated. Guards the factory clamp.
/// </summary>
[Theory]
[InlineData(0)]
[InlineData(-1)]
[InlineData(-5000)]
public void CreateInstance_clamps_non_positive_TimeoutMs_to_the_default(int timeoutMs)
{
var json = $$$"""{"Backend":"unimplemented","TimeoutMs":{{{timeoutMs}}},"Probe":{"TimeoutMs":{{{timeoutMs}}}}}""";
var drv = FocasDriverFactoryExtensions.CreateInstance("drv-1", json);
drv.Options.Timeout.ShouldBe(TimeSpan.FromMilliseconds(2_000));
drv.Options.Timeout.ShouldBeGreaterThan(TimeSpan.Zero);
drv.Options.Probe.Timeout.ShouldBe(TimeSpan.FromMilliseconds(2_000));
}
/// <summary>A positive <c>TimeoutMs</c> is honoured verbatim (the clamp only rewrites non-positive values).</summary>
[Fact]
public void CreateInstance_honours_a_positive_TimeoutMs()
{
const string json = """{"Backend":"unimplemented","TimeoutMs":750}""";
var drv = FocasDriverFactoryExtensions.CreateInstance("drv-1", json);
drv.Options.Timeout.ShouldBe(TimeSpan.FromMilliseconds(750));
}
/// <summary>Verifies that the AlarmProjection configuration section is mapped to driver options.</summary>
[Fact]
public void CreateInstance_maps_AlarmProjection_section_onto_options()
@@ -26,24 +26,23 @@ public sealed class GatewayTagProvisionerTests
Assert.NotNull(fake.LastEnsureDefinitions);
Assert.Equal(2, fake.LastEnsureDefinitions!.Count);
Assert.Equal(HistorianDataType.Float, fake.LastEnsureDefinitions[0].DataType);
Assert.Equal(HistorianDataType.Int1, fake.LastEnsureDefinitions[1].DataType);
Assert.Equal(HistorianDataType.Int2, fake.LastEnsureDefinitions[1].DataType);
Assert.Equal(2, result.Requested);
Assert.Equal(0, result.Skipped);
}
/// <summary>
/// archreview 06/U-7 — pins the 0.2.0 Boolean→Int1 provisioning semantics at TWO levels. The
/// 0.1.0→0.2.0 client bump made <c>HistorianTagDefinition.DataType</c> proto3-optional (explicit
/// presence): under 0.1.0 the provisioner's <c>Int1</c> (wire 0) was indistinguishable from unset,
/// so the gateway defaulted Boolean tags to Float; under 0.2.0 the same call carries explicit
/// presence and provisions Int1 (the intended behavior). Asserting BOTH the mapped type AND
/// <c>HasDataType</c> compile-pins the package floor (the <c>HasDataType</c>/<c>ClearDataType</c>
/// members exist only on the 0.2.0 proto3-optional contract — a downgrade breaks the build) and
/// assert-pins that the provisioner always sends explicit presence (a future contract change that
/// stopped sending it would break the assert). This is a pin, not a fix — it passes on current code.
/// Pins the Boolean→Int2 provisioning semantics at TWO levels. Boolean maps to <c>Int2</c> because
/// the historian's <c>Int1</c> analog-tag creation path is server-degenerate — <c>EnsureTags(Int1)</c>
/// stores an unusable stub (issue #441 / HistorianGateway #11) — so Boolean historizes as a small 0/1
/// integer. Asserting BOTH the mapped type AND <c>HasDataType</c> compile-pins the package floor (the
/// <c>HasDataType</c>/<c>ClearDataType</c> members exist only on the 0.2.0+ proto3-optional contract —
/// a downgrade breaks the build) and assert-pins that the provisioner always sends explicit presence
/// (a future contract change that stopped sending it would default the tag to the SDK's Float and
/// break this assert). This is a pin, not a fix — it passes on current code.
/// </summary>
[Fact]
public async Task Boolean_definition_carries_explicit_Int1_presence()
public async Task Boolean_definition_carries_explicit_Int2_presence()
{
var fake = new FakeHistorianGatewayClient { EnsureTagsResult = new TagOperationResults() };
var p = Provisioner(fake);
@@ -54,9 +53,9 @@ public sealed class GatewayTagProvisionerTests
var defs = fake.LastEnsureDefinitions!;
Assert.Single(defs);
Assert.Equal(HistorianDataType.Int1, defs[0].DataType);
// 0.2.0 proto3-optional presence witness — explicit presence is what flips the gateway from its
// SDK-default Float to the requested Int1. Without this, a Boolean tag would provision as Float.
Assert.Equal(HistorianDataType.Int2, defs[0].DataType);
// proto3-optional presence witness — explicit presence is what flips the gateway from its
// SDK-default Float to the requested Int2. Without this, a Boolean tag would provision as Float.
Assert.True(defs[0].HasDataType);
}
@@ -255,25 +255,26 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
}
/// <summary>
/// archreview 06/U-7 — Boolean provisioning leg on the 0.2.0 contract. Through the <b>real</b>
/// issue #441 — Boolean historization leg. Through the <b>real</b>
/// <see cref="GatewayTagProvisioner"/>, <c>EnsureTags</c> a <see cref="DriverDataType.Boolean"/>
/// request for the dedicated Boolean sandbox tag (mapped to <c>Int1</c> with explicit proto3
/// presence), assert it is ensured (not failed), then <c>WriteLiveValues</c> a <c>1.0</c> and read
/// it back over a wide window. This retires "the live EnsureTags leg has never run on 0.2.0" for
/// the Boolean path. The ±12h read window + timezone caveat mirror <see cref="Write_then_read_on_sandbox_tag"/>.
/// request for the dedicated Boolean sandbox tag (mapped to <c>Int2</c> — the historian's
/// <c>Int1</c> analog-tag creation path is server-degenerate, see issue #441 / HistorianGateway
/// #11, so Boolean historizes as a small 0/1 integer), assert it is ensured (not failed), then
/// <c>WriteLiveValues</c> a <c>1.0</c> and read it back over a wide window. The ±12h read window
/// + timezone caveat mirror <see cref="Write_then_read_on_sandbox_tag"/>.
/// </summary>
[Fact]
[Trait("Category", "LiveIntegration")]
public async Task EnsureTags_boolean_sandbox_provisions_as_Int1()
public async Task EnsureTags_boolean_sandbox_provisions_as_Int2()
{
if (_fx.NotConfigured) Assert.Skip(_fx.SkipReason!);
if (_fx.BoolSandboxTag is null)
Assert.Skip("Skipped: set HISTGW_BOOL_SANDBOX_TAG to a writable Boolean sandbox tag (e.g. HistGW.LiveTest.BoolSandbox) to run the 0.2.0 Boolean EnsureTags leg.");
Assert.Skip("Skipped: set HISTGW_BOOL_SANDBOX_TAG to a writable Boolean sandbox tag (e.g. HistGW.LiveTest.BoolSandbox) to run the Boolean historization leg.");
var ct = TestContext.Current.CancellationToken;
var tag = _fx.BoolSandboxTag;
// EnsureTags (Boolean → Int1) through the REAL provisioner seam (the same code path deploys use).
// EnsureTags (Boolean → Int2) through the REAL provisioner seam (the same code path deploys use).
await using var writeClient = _fx.CreateClient();
var provisioner = new GatewayTagProvisioner(writeClient, NullLogger<GatewayTagProvisioner>.Instance);
var provision = await provisioner.EnsureTagsAsync(
@@ -281,7 +282,7 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
ct);
provision.Ensured.ShouldBe(1,
"the Boolean sandbox tag must be ensured (needs the gateway on 0.2.0 contracts + an API key carrying historian:tags:write).");
"the Boolean sandbox tag must be ensured as Int2 (needs the gateway on 0.2.0+ contracts + an API key carrying historian:tags:write).");
provision.Failed.ShouldBe(0);
// WriteLiveValues a Boolean-as-1.0 and read it back (SQL live path can lag a flush cadence → poll).
@@ -308,14 +309,16 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
hit.ShouldNotBeNull($"the written Boolean sample ({written}) should read back from '{tag}'.");
TestContext.Current.SendDiagnosticMessage(
$"Boolean round-trip: EnsureTags(Int1) + WriteLiveValues '{tag}'={written} → read back at {hit!.SourceTimestampUtc:O}.");
$"Boolean round-trip: EnsureTags(Int2) + WriteLiveValues '{tag}'={written} → read back at {hit!.SourceTimestampUtc:O}.");
}
/// <summary>
/// archreview 06/U-7 retype probe — answers the assessment note's open question: what does the
/// deployed AVEVA Historian do when <c>EnsureTags</c> (an <b>upsert</b>) is asked to change an
/// existing tag's type? Against the <b>dedicated Boolean sandbox tag only</b> (never the shared
/// Float write sandbox), EnsureTags first as <c>Float</c> then as <c>Int1</c> and assert the second
/// Float write sandbox), EnsureTags first as <c>Float</c> then as <c>Int2</c> (both supported
/// analog types — <c>Int2</c> is the type Boolean now historizes as; the previously-probed
/// <c>Int1</c> is a documented server-degenerate exclusion, issue #441) and assert the second
/// call's per-tag outcome is <em>deterministic and observable</em> — either <c>Success=true</c>
/// (retype / tag-versioning accepted) or <c>Success=false</c> with a non-empty <c>Error</c> (mapped
/// to the provisioner's <c>failed</c> tally). The assertion accepts both because the contract
@@ -336,7 +339,7 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
await using var client = _fx.CreateClient();
// First establish the tag as Float, then ask EnsureTags to retype it to Int1 (Boolean).
// First establish the tag as Float, then ask EnsureTags to retype it to Int2.
await client.EnsureTagsAsync(
new[]
{
@@ -357,9 +360,9 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
new HistorianTagDefinition
{
TagName = tag,
DataType = HistorianDataType.Int1,
DataType = HistorianDataType.Int2,
EngineeringUnit = string.Empty,
Description = "OtOpcUa live validation retype probe (stage 2: Int1)",
Description = "OtOpcUa live validation retype probe (stage 2: Int2)",
StorageRateMs = 1000, // 0 makes the gateway's EnsureTags throw (storageRateMs must be > 0).
},
},
@@ -372,7 +375,7 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
"or Success=false with a non-empty Error (mapped to the provisioner's failed tally).");
TestContext.Current.SendDiagnosticMessage(
$"retype probe '{tag}' Float→Int1: Success={outcome.Success}, Error='{outcome.Error}'. " +
$"retype probe '{tag}' Float→Int2: Success={outcome.Success}, Error='{outcome.Error}'. " +
"(records the deployed historian's in-place analog-retype policy — feed this back into docs/Historian.md).");
}
}
@@ -8,7 +8,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.Mapping;
public sealed class HistorianTypeMapperTests
{
[Theory]
[InlineData(DriverDataType.Boolean, HistorianDataType.Int1)]
[InlineData(DriverDataType.Boolean, HistorianDataType.Int2)] // Int1 analog-tag creation is server-degenerate (issue #441) → store 0/1
[InlineData(DriverDataType.Int16, HistorianDataType.Int2)]
[InlineData(DriverDataType.Int32, HistorianDataType.Int4)]
[InlineData(DriverDataType.Int64, HistorianDataType.Int8)]
@@ -48,6 +48,31 @@ service + starting another. The integration tests discriminate by a
separate `MODBUS_SIM_PROFILE` env var so they skip correctly when the
wrong profile is live.
> **⚠️ Profile-cycling gotcha (bit us in the 2026-07 sweep — see
> `archreview/plans/INTEGRATION-SWEEP-STATUS.md`).** Each profile is a
> *separate compose service* selected by `--profile`, so a plain
> `docker compose down` (no `--profile`) does **not** stop the
> currently-running profile container — it keeps holding `:5020`. Start
> another profile and the new container silently fails to bind the port
> (compose still reports "Started"); every test then hits the **stale**
> profile's data and fails with wrong values / Illegal-Data-Address. The
> symptom is deceptive: reads succeed but return the previous profile's
> seed (e.g. Mitsubishi's `D0` reads DL205's `0xCAFE`). Always tear the
> old container down **by name / with its profile** before bringing up the
> next, and verify the new container actually published `:5020`:
>
> ```bash
> # reliable cycle (force-remove ALL modbus containers, then bring one up)
> docker rm -f $(docker ps -aq --filter name=otopcua-modbus --filter name=otopcua-pymodbus)
> docker compose --profile <profile> up -d --force-recreate
> docker ps --filter name=otopcua --format '{{.Names}} {{.Ports}}' # MUST show 0.0.0.0:5020->5020
> ```
>
> The `exception_injection` service uses a different container name
> (`otopcua-modbus-exception-injector`, not `otopcua-pymodbus-*`), so a
> name-filtered `docker rm` that only matches `otopcua-pymodbus` will miss
> it — the `--force-recreate` above covers that case.
### Profile coverage matrix
The two general-purpose profiles cover disjoint test sets. A full pass
@@ -63,6 +63,40 @@ public sealed class S7DriverReconnectTests
factory.Created[1].Disposed.ShouldBeFalse();
}
/// <summary>
/// R2-01 read-leg gate. A <see cref="TimeoutException"/> — what <c>S7OperationDeadline</c>
/// raises when a wire read blows its wall-clock ceiling on an established-but-frozen peer
/// (the async read S7.Net's ignored socket <c>ReadTimeout</c> can't bound) — is classified
/// connection-fatal: the handle is marked dead and the NEXT read reopens a fresh connection,
/// exactly like a socket drop. Before the fix a read-timeout was NOT fatal, so the wedged
/// handle was reused and the poll loop never recovered (<c>Created.Count</c> pinned at 1).
/// </summary>
[Fact]
public async Task Read_reopens_after_a_wire_operation_timeout()
{
var factory = new FakeS7PlcFactory();
using var drv = new S7Driver(Options(), "s7-read-timeout-fatal", factory);
await drv.InitializeAsync("{}", TestContext.Current.CancellationToken);
factory.Created.Count.ShouldBe(1);
// The next read blows its deadline — surfaces as a TimeoutException, the shape
// S7OperationDeadline raises for a frozen-but-established peer.
factory.Created[0].NextReadThrows = new TimeoutException("S7 read of 'DB1.DBW0' did not complete within 250 ms.");
// Read #1 — timeout → Degraded, handle marked dead, no reopen yet.
var bad = await drv.ReadAsync(["W0"], TestContext.Current.CancellationToken);
bad[0].StatusCode.ShouldNotBe(0u);
drv.GetHealth().State.ShouldBe(DriverState.Degraded);
factory.Created.Count.ShouldBe(1);
// Read #2 — disposes the dead handle, opens a SECOND connection, reads Good, recovers.
var recovered = await drv.ReadAsync(["W0"], TestContext.Current.CancellationToken);
recovered[0].StatusCode.ShouldBe(0u);
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
factory.Created.Count.ShouldBe(2);
factory.Created[0].Disposed.ShouldBeTrue();
}
/// <summary>
/// A protocol / data-address error (S7.Net <see cref="ErrorCode.ReadData"/>) is NOT a
/// connection loss — the driver keeps the same connection and does not churn a reopen.
@@ -0,0 +1,95 @@
using System.Diagnostics;
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Driver.S7.Tests;
/// <summary>
/// Unit tests for <see cref="S7OperationDeadline"/> — the wall-clock ceiling that bounds every
/// S7 async wire op (R2-01 read-leg gate). Async S7.Net reads/writes ignore the socket
/// ReadTimeout/WriteTimeout, so a frozen-but-established peer would otherwise block for minutes;
/// these pin the deadline behaviour without a live PLC.
/// </summary>
[Trait("Category", "Unit")]
public sealed class S7OperationDeadlineTests
{
private static readonly TimeSpan ShortTimeout = TimeSpan.FromMilliseconds(150);
/// <summary>An operation that completes within the deadline returns its value untouched.</summary>
[Fact]
public async Task Completes_within_deadline_returns_value()
{
var result = await S7OperationDeadline.RunAsync(
_ => Task.FromResult(42), ShortTimeout, "read", TestContext.Current.CancellationToken);
result.ShouldBe(42);
}
/// <summary>
/// An operation that ignores its token and never completes is cut at the deadline and
/// surfaces a <see cref="TimeoutException"/> — the frozen-established-peer shape. Uses a bare
/// <see cref="TaskCompletionSource"/> that never completes, so the token is genuinely ignored.
/// </summary>
[Fact]
public async Task Token_ignoring_hang_times_out()
{
var neverCompletes = new TaskCompletionSource<int>().Task;
var sw = Stopwatch.StartNew();
var ex = await Should.ThrowAsync<TimeoutException>(async () =>
await S7OperationDeadline.RunAsync(
_ => neverCompletes, ShortTimeout, "read of 'DB1.DBW0'", TestContext.Current.CancellationToken));
sw.Elapsed.ShouldBeLessThan(TimeSpan.FromSeconds(5));
ex.Message.ShouldContain("did not complete within 150 ms");
}
/// <summary>
/// An operation that honours its token (the well-behaved async-read case) is cancelled at the
/// deadline; the resulting cancellation is normalised to a <see cref="TimeoutException"/>,
/// NOT surfaced as an <see cref="OperationCanceledException"/>.
/// </summary>
[Fact]
public async Task Token_honouring_hang_times_out_as_timeout_not_cancel()
{
await Should.ThrowAsync<TimeoutException>(async () =>
await S7OperationDeadline.RunAsync(
async token =>
{
await Task.Delay(Timeout.Infinite, token);
return 0;
},
ShortTimeout, "read", TestContext.Current.CancellationToken));
}
/// <summary>
/// Caller cancellation (as opposed to the deadline firing) propagates as an
/// <see cref="OperationCanceledException"/> — the deadline wrapper must not swallow genuine
/// cancellation into a timeout.
/// </summary>
[Fact]
public async Task Caller_cancellation_propagates_as_operation_cancelled()
{
using var cts = new CancellationTokenSource();
cts.CancelAfter(50);
// A long deadline so the CALLER token fires first, not the internal deadline.
await Should.ThrowAsync<OperationCanceledException>(async () =>
await S7OperationDeadline.RunAsync(
async token =>
{
await Task.Delay(Timeout.Infinite, token);
return 0;
},
TimeSpan.FromSeconds(30), "read", cts.Token));
}
/// <summary>The result-less overload also bounds a hang and surfaces a <see cref="TimeoutException"/>.</summary>
[Fact]
public async Task Resultless_overload_times_out()
{
var neverCompletes = new TaskCompletionSource().Task;
await Should.ThrowAsync<TimeoutException>(async () =>
await S7OperationDeadline.RunAsync(
_ => neverCompletes, ShortTimeout, "write of 'DB1.DBW0'", TestContext.Current.CancellationToken));
}
}
@@ -138,6 +138,59 @@ public sealed class ConfigComposerTests : ControlPlaneActorTestBase
node.DeviceHost.ShouldBe("10.9.9.9:8193");
}
/// <summary>
/// Verifies that a <see cref="DriverInstance"/>'s per-instance <c>ResilienceConfig</c> override
/// survives the real <see cref="ConfigComposer.SnapshotAndFlattenAsync"/> →
/// <see cref="DeploymentArtifact.ParseDriverInstances(ReadOnlySpan{byte})"/> seam (arch-review
/// follow-up #10 sub-gap). The composer serialises the whole entity, so the override rides the
/// artifact into <c>DriverInstanceSpec.ResilienceConfig</c> — which <c>DriverHostActor</c> layers
/// onto the driver's Polly pipeline. Guards the one leg the hand-rolled
/// <c>DeploymentArtifactTests.ParseDriverInstances_carries_ResilienceConfig_onto_the_spec</c>
/// cannot: if the composer ever stopped emitting the column (a projection, a <c>[JsonIgnore]</c>),
/// authored resilience policy would silently revert to tier defaults in production while the
/// hand-built artifact tests stayed green.
/// </summary>
[Fact]
public async Task ResilienceConfig_survives_ConfigComposer_to_ParseDriverInstances_round_trip()
{
const string resilienceJson =
"{\"capabilityPolicies\":{\"Read\":{\"timeoutSeconds\":5,\"retryCount\":7,\"breakerFailureThreshold\":3}}}";
var f = NewInMemoryDbFactory();
await using (var db = f.CreateDbContext())
{
db.ServerClusters.Add(NewCluster("c1"));
db.Namespaces.Add(new Namespace
{
NamespaceId = "ns-eq", ClusterId = "c1",
Kind = NamespaceKind.Equipment, NamespaceUri = "urn:eq",
});
db.DriverInstances.Add(new DriverInstance
{
DriverInstanceId = "drv-1", ClusterId = "c1", NamespaceId = "ns-eq",
Name = "Modbus", DriverType = "Modbus", DriverConfig = "{}",
ResilienceConfig = resilienceJson,
});
// A second instance with no override proves the null case rides the same real composer path.
db.DriverInstances.Add(new DriverInstance
{
DriverInstanceId = "drv-2", ClusterId = "c1", NamespaceId = "ns-eq",
Name = "Modbus2", DriverType = "Modbus", DriverConfig = "{}",
});
await db.SaveChangesAsync();
}
await using var readDb = f.CreateDbContext();
var artifact = await ConfigComposer.SnapshotAndFlattenAsync(readDb);
var specs = DeploymentArtifact.ParseDriverInstances(artifact.Blob);
var withOverride = specs.Single(s => s.DriverInstanceId == "drv-1");
withOverride.ResilienceConfig.ShouldBe(resilienceJson, "the authored override must ride the artifact byte-for-byte");
var withoutOverride = specs.Single(s => s.DriverInstanceId == "drv-2");
withoutOverride.ResilienceConfig.ShouldBeNull("an unset override stays null ⇒ tier defaults at spawn");
}
private static readonly DateTime FixedTimestamp = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
private static ServerCluster NewCluster(string id) => new()
@@ -23,6 +23,7 @@ public sealed class DeployHappyPathTests
public async Task StartDeployment_seals_after_both_nodes_apply()
{
await using var harness = await TwoNodeClusterHarness.StartAsync();
await harness.SeedDefaultClusterAsync();
await using var scope = harness.NodeA.Services.CreateAsyncScope();
var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
@@ -59,6 +60,7 @@ public sealed class DeployHappyPathTests
public async Task Replaying_dispatch_to_same_revision_is_idempotent_no_op()
{
await using var harness = await TwoNodeClusterHarness.StartAsync();
await harness.SeedDefaultClusterAsync();
await using var scope = harness.NodeA.Services.CreateAsyncScope();
var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
@@ -218,8 +218,10 @@ public sealed class DriverReconnectE2eTests
Name = "Reconnect E2E Cluster",
Enterprise = "zb",
Site = "central",
NodeCount = 1,
RedundancyMode = RedundancyMode.None,
// Both harness nodes are Enabled driver members, so NodeCount must be 2 (Warm) to
// satisfy the ClusterEnabledNodeCountMismatch validator + the SQL CHECK constraint.
NodeCount = 2,
RedundancyMode = RedundancyMode.Warm,
CreatedBy = "test",
});
@@ -240,6 +242,18 @@ public sealed class DriverReconnectE2eTests
CreatedBy = "test",
});
// The harness runs BOTH nodes as driver members, so the deployment records a
// NodeDeploymentState for node B too — seed its ClusterNode row so the real-SQL FK
// (FK_NodeDeploymentState_ClusterNode_NodeId) resolves. (In-memory ignores FKs.)
db.ClusterNodes.Add(new ClusterNode
{
NodeId = harness.NodeBNodeId,
ClusterId = ClusterId,
Host = TwoNodeClusterHarness.LoopbackHost,
ApplicationUri = "urn:zb:reconnect-e2e:node-b",
CreatedBy = "test",
});
db.DriverInstances.Add(new DriverInstance
{
DriverInstanceId = DriverId,
@@ -7,6 +7,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using ZB.MOM.WW.OtOpcUa.Configuration.Validation;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
@@ -28,19 +29,28 @@ public sealed class EquipmentNamespaceMaterializationTests
{
private static CancellationToken Ct => TestContext.Current.CancellationToken;
// Equipment.EquipmentId must equal DraftValidator.DeriveEquipmentId(EquipmentUuid) — the
// 'EquipmentIdNotDerived' rule (EquipmentId = 'EQ-' + first 12 hex of the UUID). A fixed UUID
// keeps the derived id stable for the assertions below.
private static readonly Guid EquipmentUuid = Guid.Parse("11111111-1111-1111-1111-111111111111");
private static readonly string EquipmentId = DraftValidator.DeriveEquipmentId(EquipmentUuid); // EQ-111111111111
/// <summary>Verifies a deployed Equipment namespace carries its signal into the composed artifact.</summary>
[Fact]
public async Task Deploying_an_equipment_namespace_carries_the_signal_into_the_artifact()
{
await using var harness = await TwoNodeClusterHarness.StartAsync();
// Seed the parent ServerCluster + ClusterNode rows the SQL FKs require (the c1 config below
// FKs to ServerCluster; on the in-memory provider this is unnecessary — FKs aren't enforced).
await harness.SeedDefaultClusterAsync("c1");
await SeedEquipmentNamespaceAsync(harness);
await using var scope = harness.NodeA.Services.CreateAsyncScope();
var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
var result = await client.StartDeploymentAsync(createdBy: "alice@test", Ct);
result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted);
result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted, $"Deploy not accepted: {result.Message}");
var deploymentId = result.DeploymentId!.Value.Value;
// The artifact is composed + persisted at dispatch time; assert on it without waiting for
@@ -65,13 +75,13 @@ public sealed class EquipmentNamespaceMaterializationTests
// ParseComposition extracted it as an EquipmentTag with FullName pulled from TagConfig.
var tag = composition.EquipmentTags.ShouldHaveSingleItem();
tag.TagId.ShouldBe("tag-speed");
tag.EquipmentId.ShouldBe("eq-1");
tag.EquipmentId.ShouldBe(EquipmentId);
tag.Name.ShouldBe("Speed");
tag.DataType.ShouldBe("Float");
tag.FullName.ShouldBe("40001");
// The equipment folder browses by its friendly UNS Name, not the MachineCode.
composition.EquipmentNodes.ShouldContain(e => e.EquipmentId == "eq-1" && e.DisplayName == "station-1");
composition.EquipmentNodes.ShouldContain(e => e.EquipmentId == EquipmentId && e.DisplayName == "station-1");
composition.UnsAreas.ShouldContain(a => a.UnsAreaId == "nw-area-filling" && a.DisplayName == "filling");
composition.UnsLines.ShouldContain(l => l.UnsLineId == "nw-line-1" && l.DisplayName == "line-1");
}
@@ -95,12 +105,13 @@ public sealed class EquipmentNamespaceMaterializationTests
db.UnsLines.Add(new UnsLine { UnsLineId = "nw-line-1", UnsAreaId = "nw-area-filling", Name = "line-1" });
db.Equipment.Add(new Equipment
{
EquipmentId = "eq-1", DriverInstanceId = "drv-modbus", UnsLineId = "nw-line-1",
EquipmentId = EquipmentId, EquipmentUuid = EquipmentUuid,
DriverInstanceId = "drv-modbus", UnsLineId = "nw-line-1",
Name = "station-1", MachineCode = "STATION_001",
});
db.Tags.Add(new Tag
{
TagId = "tag-speed", DriverInstanceId = "drv-modbus", EquipmentId = "eq-1",
TagId = "tag-speed", DriverInstanceId = "drv-modbus", EquipmentId = EquipmentId,
Name = "Speed", DataType = "Float", AccessLevel = TagAccessLevel.Read,
TagConfig = "{\"FullName\":\"40001\"}",
});
@@ -58,6 +58,7 @@ public sealed class FailoverDuringDeployTests
// dispatch time — when only node A is Up, only one ApplyAck is expected and the
// deployment seals without B ever participating.
await using var harness = await TwoNodeClusterHarness.StartAsync();
await harness.SeedDefaultClusterAsync();
await harness.StopNodeBAsync();
await harness.WaitForClusterSizeAsync(1, TimeSpan.FromSeconds(20));
@@ -45,6 +45,7 @@ public sealed class FleetDiagnosticsRoundTripTests
public async Task GetDiagnostics_after_deploy_reports_current_revision()
{
await using var harness = await TwoNodeClusterHarness.StartAsync();
await harness.SeedDefaultClusterAsync();
await using var scope = harness.NodeA.Services.CreateAsyncScope();
var adminOps = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
@@ -0,0 +1,190 @@
using System.Diagnostics;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
using ZB.MOM.WW.Auth.Abstractions.Roles;
using ZB.MOM.WW.OtOpcUa.Host.OpcUa;
using ZB.MOM.WW.OtOpcUa.Security.Ldap;
using LdapTransport = ZB.MOM.WW.Auth.Abstractions.Ldap.LdapTransport;
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
/// <summary>
/// R2-08 03/S2 LIVE gate — the directory-outage circuit exercised against a REAL GLAuth server
/// (the image + config PR #451 swapped in for the retired bitnami/openldap), driven through the
/// production auth path (<see cref="OtOpcUaLdapAuthService"/> + <see cref="LdapOpcUaUserAuthenticator"/>).
/// The offline <see cref="LdapAuthResilienceTests"/> prove the circuit logic with fakes + an
/// unroutable-host blackhole; this proves the same behaviour against an actual directory that is
/// paused mid-run (a real outage) and unpaused (recovery), and doubles as the first end-to-end
/// verification that the GLAuth swap binds correctly.
/// </summary>
/// <remarks>
/// Triple-gated so it never runs by accident and always restores the shared GLAuth:
/// <list type="number">
/// <item><c>GLAUTH_LIVE_HOST</c> — the GLAuth host (e.g. <c>10.100.0.35</c>); absent ⇒ skip;</item>
/// <item><c>GLAUTH_OUTAGE_START_CMD</c> — pauses the GLAuth container (a real outage); and</item>
/// <item><c>GLAUTH_OUTAGE_STOP_CMD</c> — unpauses it.</item>
/// </list>
/// A <c>try/finally</c> ALWAYS runs the stop command once the pause succeeded. Operator recipe:
/// <code>
/// ssh dohertj2@10.100.0.35 "cd ~/otopcua-harness &amp;&amp; docker compose up -d ldap"
/// export GLAUTH_LIVE_HOST=10.100.0.35
/// export GLAUTH_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-harness-ldap-1"'
/// export GLAUTH_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-harness-ldap-1"'
/// dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapGlauthOutageLive"
/// </code>
/// </remarks>
[Trait("Category", "Integration")]
[Trait("Category", "LiveLdap")]
public sealed class LdapGlauthOutageLiveTests
{
private const string HostEnvVar = "GLAUTH_LIVE_HOST";
private const string PortEnvVar = "GLAUTH_LIVE_PORT";
private const string StartCmdEnvVar = "GLAUTH_OUTAGE_START_CMD";
private const string StopCmdEnvVar = "GLAUTH_OUTAGE_STOP_CMD";
private const int OutageThreshold = 3;
private const int CooldownSeconds = 2;
private static readonly TimeSpan AuthTimeout = TimeSpan.FromMilliseconds(2_500);
/// <summary>
/// Healthy real binds succeed; pausing GLAuth makes consecutive binds fail bounded (never a
/// multi-second SDK stall) and opens the circuit into a sub-second fast-deny; unpausing +
/// cooldown lets a half-open probe close the circuit and binds succeed again.
/// </summary>
[Fact]
public async Task Real_glauth_outage_opens_circuit_then_recovers_on_unpause()
{
var host = Environment.GetEnvironmentVariable(HostEnvVar);
if (string.IsNullOrWhiteSpace(host))
Assert.Skip($"Set {HostEnvVar} (+ {StartCmdEnvVar}/{StopCmdEnvVar}) to run the live GLAuth outage cycle.");
var port = int.TryParse(Environment.GetEnvironmentVariable(PortEnvVar), out var p) ? p : 3894;
var startCmd = Environment.GetEnvironmentVariable(StartCmdEnvVar);
var stopCmd = Environment.GetEnvironmentVariable(StopCmdEnvVar);
if (string.IsNullOrWhiteSpace(startCmd) || string.IsNullOrWhiteSpace(stopCmd))
Assert.Skip($"Set both {StartCmdEnvVar} and {StopCmdEnvVar} (docker pause/unpause) to run this destructive live test.");
var ct = TestContext.Current.CancellationToken;
var options = new LdapOptions
{
Enabled = true,
DevStubMode = false,
Server = host!,
Port = port,
Transport = LdapTransport.None,
AllowInsecure = true,
SearchBase = "dc=zb,dc=local",
ServiceAccountDn = "cn=serviceaccount,dc=zb,dc=local",
ServiceAccountPassword = "serviceaccount123",
ConnectionTimeoutMs = 2_000,
AuthTimeoutMs = (int)AuthTimeout.TotalMilliseconds,
OutageFailureThreshold = OutageThreshold,
OutageCooldownSeconds = CooldownSeconds,
};
var authenticator = BuildAuthenticator(options);
// ── 1. Healthy directory: real binds succeed, wrong creds deny (user-side, not an outage). ──
var alice = await authenticator.AuthenticateUserNameAsync("alice", "alice123", ct);
alice.Success.ShouldBeTrue("alice should bind against the live GLAuth");
alice.Roles.ShouldNotBeEmpty("a healthy bind returns alice's mapped group roles");
var bob = await authenticator.AuthenticateUserNameAsync("bob", "bob123", ct);
bob.Success.ShouldBeTrue("bob should bind against the live GLAuth");
var wrong = await authenticator.AuthenticateUserNameAsync("alice", "wrong-password", ct);
wrong.Success.ShouldBeFalse("a wrong password is a user-side deny");
// ── 2. Outage: pause GLAuth. Consecutive binds fail BOUNDED, then the circuit fast-denies. ──
var paused = false;
try
{
await RunShellCommandAsync(startCmd!, ct);
paused = true;
for (var i = 0; i < OutageThreshold; i++)
{
var sw = Stopwatch.StartNew();
var outage = await authenticator.AuthenticateUserNameAsync("alice", "alice123", ct);
sw.Stop();
outage.Success.ShouldBeFalse($"bind #{i + 1} during the outage must fail closed");
// The whole flow is wall-clock bounded — never the SDK's default ~10-20 s stall.
sw.Elapsed.ShouldBeLessThan(AuthTimeout + TimeSpan.FromSeconds(2),
"the authenticator must bound a hung directory call, not stall on it");
}
// Circuit is now open — the next call fast-denies WITHOUT a directory round-trip (sub-second).
var blockedSw = Stopwatch.StartNew();
var blocked = await authenticator.AuthenticateUserNameAsync("alice", "alice123", ct);
blockedSw.Stop();
blocked.Success.ShouldBeFalse();
blocked.Error.ShouldNotBeNull();
blocked.Error!.ShouldContain("unavailable", Case.Insensitive,
"an open outage circuit fast-denies 'backend unavailable'");
blockedSw.Elapsed.ShouldBeLessThan(TimeSpan.FromSeconds(1),
"an open circuit fast-denies without a directory call");
}
finally
{
if (paused)
await RunShellCommandAsync(stopCmd!, CancellationToken.None); // ALWAYS restore GLAuth
}
// ── 3. Recovery: after the cooldown the half-open probe reaches the (restored) directory. ──
await Task.Delay(TimeSpan.FromSeconds(CooldownSeconds) + TimeSpan.FromSeconds(1), ct);
var recovered = false;
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
while (DateTime.UtcNow < deadline)
{
var probe = await authenticator.AuthenticateUserNameAsync("alice", "alice123", ct);
if (probe.Success) { recovered = true; break; }
await Task.Delay(TimeSpan.FromMilliseconds(500), ct);
}
recovered.ShouldBeTrue("the circuit must half-open and close once GLAuth is unpaused");
}
private static LdapOpcUaUserAuthenticator BuildAuthenticator(LdapOptions options)
{
var ldap = new OtOpcUaLdapAuthService(Options.Create(options), NullLogger<OtOpcUaLdapAuthService>.Instance);
return new LdapOpcUaUserAuthenticator(
ldap, ScopeFactoryWith(new PassthroughMapper()), Options.Create(options),
NullLogger<LdapOpcUaUserAuthenticator>.Instance);
}
/// <summary>Builds an <see cref="IServiceScopeFactory"/> whose scopes resolve the supplied mapper.</summary>
private static IServiceScopeFactory ScopeFactoryWith(IGroupRoleMapper<string> mapper)
{
var services = new ServiceCollection();
services.AddScoped(_ => mapper);
return services.BuildServiceProvider().GetRequiredService<IServiceScopeFactory>();
}
/// <summary>Runs an SSH/docker one-liner through the login shell (verbatim, as in the S7 outage gate).</summary>
private static async Task RunShellCommandAsync(string command, CancellationToken ct)
{
using var proc = new Process
{
StartInfo = new ProcessStartInfo("/bin/sh", $"-c \"{command.Replace("\"", "\\\"")}\"")
{
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
},
};
proc.Start();
await proc.WaitForExitAsync(ct);
proc.ExitCode.ShouldBe(0, $"outage command failed: {await proc.StandardError.ReadToEndAsync(ct)}");
await Task.Delay(TimeSpan.FromSeconds(1), ct); // let the pause/unpause take effect
}
/// <summary>Passes LDAP groups straight through as roles — the live gate asserts on the bind, not the mapping.</summary>
private sealed class PassthroughMapper : IGroupRoleMapper<string>
{
public Task<GroupRoleMapping<string>> MapAsync(IReadOnlyList<string> groups, CancellationToken ct)
=> Task.FromResult(new GroupRoleMapping<string>(groups, Scope: null));
}
}
@@ -14,6 +14,8 @@ using ZB.MOM.WW.OtOpcUa.AdminUI.Clients;
using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using ZB.MOM.WW.OtOpcUa.ControlPlane;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Host.Health;
@@ -36,9 +38,11 @@ namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
/// <item><c>OTOPCUA_HARNESS_USE_SQL=1</c> → swap the in-memory DB for SQL Server on
/// <c>localhost:14331</c> (see <c>docker-compose.yml</c>). Each harness gets a unique
/// database name (<c>OtOpcUa_Harness_{guid}</c>) created via <c>EnsureCreated</c>
/// and dropped via <c>EnsureDeleted</c> on dispose.</item>
/// and dropped via <c>EnsureDeleted</c> on dispose. Override the SQL / LDAP host with
/// <c>OTOPCUA_HARNESS_SQL_HOST</c> / <c>OTOPCUA_HARNESS_LDAP_HOST</c> to point at native
/// fixtures on the shared Docker host (avoids the arm64-Mac emulated-mssql slowdown).</item>
/// <item><c>OTOPCUA_HARNESS_USE_LDAP=1</c> → drop the stub and point <c>LdapAuthService</c>
/// at OpenLDAP on <c>localhost:3894</c>.</item>
/// at the GLAuth container on <c>localhost:3894</c> (see Docker/glauth/config.toml).</item>
/// </list>
///
/// <para>Why not <c>WebApplicationFactory&lt;Program&gt;</c>? Program.cs reads <c>OTOPCUA_ROLES</c>
@@ -114,6 +118,53 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
public Task<OtOpcUaConfigDbContext> CreateConfigDbContextAsync()
=> NodeA.Services.GetRequiredService<IDbContextFactory<OtOpcUaConfigDbContext>>().CreateDbContextAsync();
/// <summary>
/// Seeds a default <see cref="ServerCluster"/> plus a <see cref="ClusterNode"/> row for BOTH
/// harness nodes (<see cref="NodeANodeId"/> / <see cref="NodeBNodeId"/>) so the real-SQL FK
/// constraint <c>FK_NodeDeploymentState_ClusterNode_NodeId</c> is satisfied when a deployment
/// records per-node state. The EF in-memory provider ignores FK constraints, so deploy E2E
/// tests pass without this; against SQL Server each node's <c>NodeDeploymentState</c> INSERT
/// fails without its parent <see cref="ClusterNode"/> row and the deployment never seals.
/// No-op unless <c>OTOPCUA_HARNESS_USE_SQL=1</c> (in-memory needs no seeding). Call once, before
/// <c>StartDeploymentAsync</c>, in tests that don't already seed their own cluster + both nodes.
/// </summary>
/// <param name="clusterId">Cluster id for the seeded rows. Defaults to <c>MAIN</c>.</param>
public async Task SeedDefaultClusterAsync(string clusterId = "MAIN")
{
if (!Mode.UseSqlServer) return;
await using var db = await CreateConfigDbContextAsync();
db.ServerClusters.Add(new ServerCluster
{
ClusterId = clusterId,
Name = "Harness Default Cluster",
Enterprise = "zb",
Site = "central",
// 2-node cluster → Warm/Hot per CK_ServerCluster_RedundancyMode_NodeCount
// (NodeCount=1↔None, NodeCount=2↔Warm|Hot). The harness runs two nodes.
NodeCount = 2,
RedundancyMode = RedundancyMode.Warm,
CreatedBy = "harness",
});
db.ClusterNodes.AddRange(
new ClusterNode
{
NodeId = NodeANodeId,
ClusterId = clusterId,
Host = LoopbackHost,
ApplicationUri = "urn:zb:harness:node-a",
CreatedBy = "harness",
},
new ClusterNode
{
NodeId = NodeBNodeId,
ClusterId = clusterId,
Host = LoopbackHost,
ApplicationUri = "urn:zb:harness:node-b",
CreatedBy = "harness",
});
await db.SaveChangesAsync();
}
/// <summary>Boots both nodes and waits up to <paramref name="formationTimeout"/> for cluster convergence.</summary>
/// <param name="formationTimeout">Maximum time to wait for cluster formation; defaults to 20 seconds if not provided.</param>
/// <param name="driverFactory">Optional opt-in <see cref="IDriverFactory"/> both nodes register so deployed
@@ -131,7 +182,12 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
if (harness.Mode.UseSqlServer)
{
harness._sqlDbName = $"OtOpcUa_Harness_{Guid.NewGuid():N}";
harness._sqlConnString = $"Server=localhost,14331;Database={harness._sqlDbName};User Id=sa;Password=OtOpcUa!Harness123;TrustServerCertificate=True;";
// SQL host is overridable so the heavy 2-node E2E tests can target NATIVE-amd64 SQL on the
// shared Docker host instead of the arm64 Mac's emulated mssql:2022 (no arm64 image → ~5-10×
// slower → the deploy/failover E2E tests time out). Set OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331
// with the harness sql container brought up there. Defaults to localhost,14331.
var sqlHost = Environment.GetEnvironmentVariable("OTOPCUA_HARNESS_SQL_HOST") ?? "localhost,14331";
harness._sqlConnString = $"Server={sqlHost};Database={harness._sqlDbName};User Id=sa;Password=OtOpcUa!Harness123;TrustServerCertificate=True;";
await EnsureSqlSchemaCreatedAsync(harness._sqlConnString);
}
@@ -295,13 +351,16 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
// Bound section is Security:Ldap (see LdapOptions.SectionName); Transport replaces the
// old UseTls bool and AllowInsecure replaces AllowInsecureLdap (Task 1.4).
configOverrides["Security:Ldap:Enabled"] = "true";
configOverrides["Security:Ldap:Server"] = "localhost";
// Overridable alongside the SQL host so the GLAuth container can live on the Docker host too.
configOverrides["Security:Ldap:Server"] = Environment.GetEnvironmentVariable("OTOPCUA_HARNESS_LDAP_HOST") ?? "localhost";
configOverrides["Security:Ldap:Port"] = "3894";
configOverrides["Security:Ldap:Transport"] = "None";
configOverrides["Security:Ldap:AllowInsecure"] = "true";
configOverrides["Security:Ldap:SearchBase"] = "dc=zb,dc=local";
configOverrides["Security:Ldap:ServiceAccountDn"] = "cn=admin,dc=zb,dc=local";
configOverrides["Security:Ldap:ServiceAccountPassword"] = "ldapadmin";
// GLAuth's search-capable bind account (see Docker/glauth/config.toml) — matches the
// shared GLAuth the data-plane binds against. Replaced the retired OpenLDAP cn=admin.
configOverrides["Security:Ldap:ServiceAccountDn"] = "cn=serviceaccount,dc=zb,dc=local";
configOverrides["Security:Ldap:ServiceAccountPassword"] = "serviceaccount123";
}
builder.Configuration.AddInMemoryCollection(configOverrides);
@@ -6,8 +6,8 @@
#
# 1. EF behaviors that diverge between provider implementations — index uniqueness,
# RowVersion concurrency, JSON column round-trips, EF migration application.
# 2. Real LDAP binds against an OpenLDAP server with the dev users from
# C:\publish\glauth\auth.md.
# 2. Real LDAP binds against a GLAuth server (./glauth/config.toml) with dev users
# alice (full access) / bob (read-only) and the cn=serviceaccount bind account.
#
# Activate by setting these env vars before running the suite:
#
@@ -43,15 +43,13 @@ services:
retries: 20
ldap:
# OpenLDAP image — same one docker-dev/ uses, just on a different port. Dev users
# alice/bob match the GLAuth fixtures so AuthEndpoints contract tests share creds.
image: bitnami/openldap:2.6
environment:
LDAP_ROOT: "dc=zb,dc=local"
LDAP_ADMIN_USERNAME: "admin"
LDAP_ADMIN_PASSWORD: "ldapadmin"
LDAP_USERS: "alice,bob"
LDAP_PASSWORDS: "alice123,bob123"
LDAP_USER_DC: "ou=FleetAdmin"
# GLAuth — the same LDAP server the rest of the project uses (docker-dev + the live
# OPC UA data-plane auth, both against the shared GLAuth on :3893). Replaced the retired
# bitnami/openldap:2.6 (Bitnami deprecated their free catalog in 2025). Seeded by
# ./glauth/config.toml with a search-capable serviceaccount + dev users alice/bob.
# Listens on :3893 in-container; mapped to host :3894 to run beside docker-dev's :3893.
image: glauth/glauth:latest
volumes:
- ./glauth/config.toml:/app/config/config.cfg:ro
ports:
- "3894:1389"
- "3894:3893"
@@ -0,0 +1,78 @@
# GLAuth config for the Host.IntegrationTests real-LDAP mode (OTOPCUA_HARNESS_USE_LDAP=1).
#
# Replaces the retired bitnami/openldap:2.6 image (Bitnami deprecated their free catalog
# in 2025) with GLAuth — the same LDAP server the rest of the project uses (docker-dev +
# the live OPC UA data-plane auth, both against the shared GLAuth on 10.100.0.35:3893).
# Unifying on GLAuth removes the lone OpenLDAP outlier and the gone-image breakage.
#
# Mirrors scadaproj/infra/glauth/config.toml (source of truth) but trimmed to the harness's
# needs: baseDN dc=zb,dc=local, a search-capable service account, and two dev users
# (alice = full-access, bob = read-only). Bind DN form is GLAuth's cn=<name>,dc=zb,dc=local.
#
# Listens on :3893 inside the container; the compose maps host :3894 -> :3893 so this runs
# side-by-side with docker-dev's shared GLAuth (:3893) without a port clash.
[ldap]
enabled = true
listen = "0.0.0.0:3893"
[ldaps]
enabled = false
[backend]
datastore = "config"
baseDN = "dc=zb,dc=local"
[behaviors]
# Tests may bind bad passwords on purpose (outage / wrong-cred paths) — don't lock out.
LimitFailedBinds = false
# ── Groups (gidnumbers match scadaproj/infra/glauth so GroupToRole maps identically) ──
[[groups]]
name = "ReadOnly"
gidnumber = 5601
[[groups]]
name = "WriteOperate"
gidnumber = 5602
[[groups]]
name = "WriteTune"
gidnumber = 5603
[[groups]]
name = "WriteConfigure"
gidnumber = 5604
[[groups]]
name = "AlarmAck"
gidnumber = 5605
# ── Users ─────────────────────────────────────────────────────────────
# Service account the harness binds first to search for the user entry.
# DN: cn=serviceaccount,dc=zb,dc=local password: serviceaccount123
[[users]]
name = "serviceaccount"
uidnumber = 5999
primarygroup = 5601
passsha256 = "af29d0e5c9801ae98a999ed3915e1cf428a64b4b62b3cf221b6336cce0398419"
[[users.capabilities]]
action = "search"
object = "*"
# alice — full access (member of every write/ack role group). password: alice123
[[users]]
name = "alice"
givenname = "Alice"
sn = "Admin"
mail = "alice@zb.local"
uidnumber = 5010
primarygroup = 5604
othergroups = [5601, 5602, 5603, 5605]
passsha256 = "4e40e8ffe0ee32fa53e139147ed559229a5930f89c2204706fc174beb36210b3"
# bob — read-only. password: bob123
[[users]]
name = "bob"
givenname = "Bob"
sn = "Reader"
mail = "bob@zb.local"
uidnumber = 5011
primarygroup = 5601
passsha256 = "8d059c3640b97180dd2ee453e20d34ab0cb0f2eccbe87d01915a8e578a202b11"
@@ -72,40 +72,44 @@ public sealed class DualEndpointTests
// SDK 1.5.374 sync-style session-open path — mirrors src/Client/.../DefaultSessionFactory.cs
// and DefaultApplicationConfigurationFactory.cs. The 1.5.378 telemetry/async overloads are
// not available at this pinned version.
var appConfig = new ApplicationConfiguration
var clientPki = TestClientSecurity.AllocatePkiRoot();
try
{
ApplicationName = "OtOpcUa.DualEndpointClient",
ApplicationUri = $"urn:OtOpcUa.DualEndpointClient.{Guid.NewGuid():N}",
ApplicationType = ApplicationType.Client,
SecurityConfiguration = new SecurityConfiguration
var appConfig = new ApplicationConfiguration
{
ApplicationCertificate = new CertificateIdentifier(),
AutoAcceptUntrustedCertificates = true,
},
ClientConfiguration = new ClientConfiguration { DefaultSessionTimeout = 60_000 },
};
await appConfig.ValidateAsync(ApplicationType.Client, default);
appConfig.CertificateValidator.CertificateValidation += (_, e) => e.Accept = true;
ApplicationName = "OtOpcUa.DualEndpointClient",
ApplicationUri = $"urn:OtOpcUa.DualEndpointClient.{Guid.NewGuid():N}",
ApplicationType = ApplicationType.Client,
SecurityConfiguration = TestClientSecurity.Build(clientPki),
ClientConfiguration = new ClientConfiguration { DefaultSessionTimeout = 60_000 },
};
await appConfig.ValidateAsync(ApplicationType.Client, default);
appConfig.CertificateValidator.CertificateValidation += (_, e) => e.Accept = true;
// 1.5.378: the discovery/session/read client surface moved to async.
var endpoint = await CoreClientUtils.SelectEndpointAsync(
appConfig, endpointUrl, false, DefaultTelemetry.Create(_ => { }), default);
var endpointConfiguration = EndpointConfiguration.Create(appConfig);
var configuredEndpoint = new ConfiguredEndpoint(null, endpoint, endpointConfiguration);
// 1.5.378: the discovery/session/read client surface moved to async.
var endpoint = await CoreClientUtils.SelectEndpointAsync(
appConfig, endpointUrl, false, DefaultTelemetry.Create(_ => { }), default);
var endpointConfiguration = EndpointConfiguration.Create(appConfig);
var configuredEndpoint = new ConfiguredEndpoint(null, endpoint, endpointConfiguration);
using var session = await new DefaultSessionFactory(DefaultTelemetry.Create(_ => { })).CreateAsync(
appConfig,
configuredEndpoint,
updateBeforeConnect: false,
checkDomain: false,
sessionName: "DualEndpointTests",
sessionTimeout: 60_000,
identity: new UserIdentity(new AnonymousIdentityToken()),
preferredLocales: null,
ct: default);
using var session = await new DefaultSessionFactory(DefaultTelemetry.Create(_ => { })).CreateAsync(
appConfig,
configuredEndpoint,
updateBeforeConnect: false,
checkDomain: false,
sessionName: "DualEndpointTests",
sessionTimeout: 60_000,
identity: new UserIdentity(new AnonymousIdentityToken()),
preferredLocales: null,
ct: default);
var value = await session.ReadValueAsync(VariableIds.Server_ServerArray, default);
return (string[])value.Value;
var value = await session.ReadValueAsync(VariableIds.Server_ServerArray, default);
return (string[])value.Value;
}
finally
{
if (Directory.Exists(clientPki)) Directory.Delete(clientPki, recursive: true);
}
}
private static int AllocateFreePort()
@@ -215,9 +215,11 @@ public sealed class SubscriptionSurvivalTests
sink.WriteValue(nodeIdAString, 11, Commons.OpcUa.OpcUaQuality.Good, DateTime.UtcNow);
await WaitUntilAsync(() => { lock (gate) return receivedA.Any(v => Equals(v.Value, 11)); }, TimeSpan.FromSeconds(5));
// B is gone: a re-read returns BadNodeIdUnknown.
var bRead = await session.ReadValueAsync(nodeIdB, ct);
bRead.StatusCode.ShouldBe((StatusCode)StatusCodes.BadNodeIdUnknown);
// B is gone: a re-read surfaces BadNodeIdUnknown. The 1.5.378 node-cache read path
// throws ServiceResultException for an unknown node rather than returning a bad DataValue.
var bReadEx = await Should.ThrowAsync<ServiceResultException>(
async () => await session.ReadValueAsync(nodeIdB, ct));
bReadEx.StatusCode.ShouldBe(StatusCodes.BadNodeIdUnknown);
}
finally
{
@@ -296,9 +298,11 @@ public sealed class SubscriptionSurvivalTests
sink.WriteValue(nodeIdAString, 11, Commons.OpcUa.OpcUaQuality.Good, DateTime.UtcNow);
await WaitUntilAsync(() => { lock (gate) return receivedA.Any(v => Equals(v.Value, 11)); }, TimeSpan.FromSeconds(5));
// B is gone.
// B is gone. (1.5.378 node-cache read throws for an unknown node — see above.)
var nodeIdB = new NodeId(Commons.OpcUa.EquipmentNodeIds.Variable("eq-1", "", "B"), ns);
(await session.ReadValueAsync(nodeIdB, ct)).StatusCode.ShouldBe((StatusCode)StatusCodes.BadNodeIdUnknown);
var bReadEx = await Should.ThrowAsync<ServiceResultException>(
async () => await session.ReadValueAsync(nodeIdB, ct));
bReadEx.StatusCode.ShouldBe(StatusCodes.BadNodeIdUnknown);
// C is browsable + readable.
var nodeIdC = new NodeId(Commons.OpcUa.EquipmentNodeIds.Variable("eq-1", "", "C"), ns);
@@ -331,11 +335,10 @@ public sealed class SubscriptionSurvivalTests
ApplicationName = "OtOpcUa.SubscriptionSurvivalClient",
ApplicationUri = $"urn:OtOpcUa.SubscriptionSurvivalClient.{Guid.NewGuid():N}",
ApplicationType = ApplicationType.Client,
SecurityConfiguration = new SecurityConfiguration
{
ApplicationCertificate = new CertificateIdentifier(),
AutoAcceptUntrustedCertificates = true,
},
// Directory cert stores rooted at a throwaway temp PKI dir — ValidateAsync rejects a
// bare SecurityConfiguration ("TrustedIssuerCertificates StorePath must be specified").
// The returned session outlives this method, so the dir is left for the OS temp sweep.
SecurityConfiguration = TestClientSecurity.Build(TestClientSecurity.AllocatePkiRoot()),
ClientConfiguration = new ClientConfiguration { DefaultSessionTimeout = 60_000 },
};
await appConfig.ValidateAsync(ApplicationType.Client, ct);
@@ -0,0 +1,47 @@
using Opc.Ua;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests;
/// <summary>
/// Builds the client-side <see cref="SecurityConfiguration"/> the integration tests use to
/// open a real OPC UA session. The SDK's <c>ApplicationConfiguration.ValidateAsync</c> requires
/// the trusted-issuer / trusted-peer / rejected stores to carry an explicit <c>StorePath</c>
/// (it throws <c>TrustedIssuerCertificates StorePath must be specified</c> otherwise), so — unlike
/// a bare <c>new SecurityConfiguration()</c> — each store is rooted at a throwaway PKI directory,
/// mirroring <c>src/Client/.../DefaultApplicationConfigurationFactory.cs</c>.
/// </summary>
internal static class TestClientSecurity
{
/// <summary>Allocates a fresh throwaway client PKI root under the temp directory.</summary>
/// <returns>An absolute path that does not yet exist; the SDK creates the stores under it.</returns>
public static string AllocatePkiRoot() =>
Path.Combine(Path.GetTempPath(), $"otopcua-client-pki-{Guid.NewGuid():N}");
/// <summary>Builds an auto-accepting client security config with Directory stores under <paramref name="pkiRoot"/>.</summary>
/// <param name="pkiRoot">Root directory for the own / issuer / trusted / rejected stores.</param>
/// <returns>A validated-ready <see cref="SecurityConfiguration"/>.</returns>
public static SecurityConfiguration Build(string pkiRoot) => new()
{
ApplicationCertificate = new CertificateIdentifier
{
StoreType = CertificateStoreType.Directory,
StorePath = Path.Combine(pkiRoot, "own"),
},
TrustedIssuerCertificates = new CertificateTrustList
{
StoreType = CertificateStoreType.Directory,
StorePath = Path.Combine(pkiRoot, "issuer"),
},
TrustedPeerCertificates = new CertificateTrustList
{
StoreType = CertificateStoreType.Directory,
StorePath = Path.Combine(pkiRoot, "trusted"),
},
RejectedCertificateStore = new CertificateTrustList
{
StoreType = CertificateStoreType.Directory,
StorePath = Path.Combine(pkiRoot, "rejected"),
},
AutoAcceptUntrustedCertificates = true,
};
}