Compare commits

...

24 Commits

Author SHA1 Message Date
dohertj2 152a5645ad fix(harness): deploy E2E tests pass against real SQL (#452)
v2-ci / unit-tests (push) Blocked by required conditions
v2-ci / build (push) Has started running
2026-07-15 06:43:37 -04:00
Joseph Doherty 859d63178a fix(harness): make deploy E2E tests pass against real SQL (seed cluster FKs)
v2-ci / unit-tests (pull_request) Blocked by required conditions
v2-ci / build (pull_request) Has started running
Integration-sweep follow-up #6 (SQL leg). The sweep flagged 7 heavy 2-node
deploy/failover/reconnect E2E tests as 'time out under amd64-emulated SQL' and
suggested raising deadlines. That diagnosis was WRONG: run against real
(native-amd64) SQL on the Docker host, they still hang — even at 4x deadline.
Root cause is that these tests had never actually run against real SQL, only the
EF in-memory provider, which does NOT enforce foreign keys.

Two distinct defects, both hidden by in-memory:

1. Missing FK-parent seed (5 tests). The deploy records a NodeDeploymentState row
   per node, FK'd to ClusterNode (FK_NodeDeploymentState_ClusterNode_NodeId). The
   tests never seeded ServerCluster + ClusterNode, so on SQL each node-state INSERT
   throws and the deploy never seals (waits the full deadline regardless of size).
   Added TwoNodeClusterHarness.SeedDefaultClusterAsync (seeds a ServerCluster +
   a ClusterNode per node; Warm/NodeCount=2 to satisfy the SQL CHECK constraint
   CK_ServerCluster_RedundancyMode_NodeCount AND the ClusterEnabledNodeCountMismatch
   validator). Called it in DeployHappyPath x2 / Failover / FleetDiagnostics /
   EquipmentNamespace; fixed DriverReconnect to seed node B + NodeCount=2.

2. Stale EquipmentId (EquipmentNamespace test; pre-existing on BOTH providers).
   Seeded EquipmentId='eq-1', but the later-added EquipmentIdNotDerived validator
   requires EquipmentId == DeriveEquipmentId(EquipmentUuid) ('EQ-'+first 12 hex).
   Fixed the seed to a canonical id + matching UUID. Only surfaced once the FK fix
   let the deploy reach equipment validation.

Also added OTOPCUA_HARNESS_SQL_HOST / OTOPCUA_HARNESS_LDAP_HOST overrides so the
harness fixtures can run on the native-amd64 Docker host (avoids arm64 mssql
emulation entirely).

Verified: the 5 SQL-backed classes go 11/11 green vs native SQL; in-memory default
unregressed (12/12). RoslynVirtualTagEvaluatorTests racing test is a pre-existing
flaky race (not SQL-backed; passes in isolation) — left as-is.
2026-07-15 06:43:15 -04:00
dohertj2 779afbb66f fix(harness): swap retired bitnami/openldap for GLAuth (#451)
v2-ci / build (push) Successful in 4m0s
v2-ci / unit-tests (push) Failing after 9m29s
2026-07-15 06:04:17 -04:00
Joseph Doherty 3a48eb0c70 fix(harness): swap retired bitnami/openldap for GLAuth in the Host real-LDAP mode
v2-ci / build (pull_request) Successful in 4m13s
v2-ci / unit-tests (pull_request) Failing after 10m18s
The Host.IntegrationTests opt-in real-LDAP mode (OTOPCUA_HARNESS_USE_LDAP=1)
used bitnami/openldap:2.6, gone since Bitnami deprecated their free image
catalog in 2025. Replaced it with GLAuth — the same LDAP server the rest of
the project already uses (docker-dev + the live OPC UA data-plane auth, both
against the shared GLAuth on :3893) — removing the lone OpenLDAP outlier and
the gone-image breakage.

- Added tests/.../Host.IntegrationTests/glauth/config.toml: baseDN dc=zb,dc=local,
  a search-capable serviceaccount, dev users alice (full access) / bob (read-only),
  and role groups with the same gidnumbers as scadaproj/infra/glauth so GroupToRole
  maps identically.
- Compose ldap service -> glauth/glauth:latest, mounting the config, host :3894 -> :3893.
- Repointed the harness real-LDAP override from the OpenLDAP cn=admin/ldapadmin to
  GLAuth's cn=serviceaccount/serviceaccount123.

Live-verified against a deployed container on :3894: the serviceaccount bind
searches and returns alice + her 5 memberOf groups; alice/bob bind with the
correct password; a wrong password yields Invalid credentials (49) — exactly the
OtOpcUaLdapAuthService flow (proven identical to the data-plane's shared-GLAuth path).

NB: real-LDAP mode is opt-in; the default Host run uses StubLdapAuthService, so
this never blocked the default suite. Integration-sweep follow-up #6 (LDAP leg);
the amd64-emulated-SQL deadline leg remains open.
2026-07-15 06:03:50 -04:00
dohertj2 18481f3d11 fix(focas-tests): honour fixture endpoint + fix mock timer endianness (#450)
v2-ci / build (push) Successful in 4m34s
v2-ci / unit-tests (push) Failing after 10m37s
2026-07-15 05:52:03 -04:00
Joseph Doherty 7982043673 fix(focas-tests): honour fixture endpoint + fix mock timer endianness (9F/1P -> 10/10)
v2-ci / build (pull_request) Successful in 4m31s
v2-ci / unit-tests (pull_request) Failing after 10m26s
Two test/fixture bugs; NO OtOpcUa driver change.

1. Topology: FocasSimFixture reads an env-overridable endpoint
   (OTOPCUA_FOCAS_SIM_ENDPOINT, default localhost:8193) and exposes Host/Port,
   but WireBackendTests + WireBackendCoverageTests hardcoded
   focas://127.0.0.1:8193. Against a remote fixture the skip-gate passed
   (remote reachable) but the driver dialed localhost -> KeyNotFound. Added a
   DeviceUri property to the fixture; each test now derives DeviceHost from it.
   Recovered 8 of 9.

2. Mock timer endianness (surfaced once #1 let the tests reach the mock): the
   last failure read 60397977600 for a 3600 s power-on timer (= 0x3C000000 x 60).
   The focas-mock's _wire_timer encoded the minute/msec fields big-endian (_u32),
   but cnc_rdtimer's timer fields are LITTLE-endian on real hardware — a
   documented, live-31i-B-validated quirk that FocasWireClient.ParseTimer decodes
   LE (docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md). The driver was
   correct; the mock was wrong. _wire_timer now emits little-endian (_u32_le).

Suite 9F/1P -> 10/10 against the remote fixture at ~/otopcua-focas.
Integration-sweep follow-up #3.
2026-07-15 05:51:47 -04:00
dohertj2 00d8422cef docs(sweep): AbLegacy verified green — fixture-deployment gap, no code (#449)
v2-ci / build (push) Successful in 4m28s
v2-ci / unit-tests (push) Failing after 11m5s
2026-07-15 05:38:37 -04:00
Joseph Doherty f381b471f9 docs(sweep): AbLegacy verified green — fixture-deployment gap, not a code defect (#4)
v2-ci / build (pull_request) Successful in 4m26s
v2-ci / unit-tests (pull_request) Has started running
The sweep's AbLegacy 4F was a fixture-deployment gap, not a driver/test defect.
The suite needs ab_server in PCCC mode (--plc=SLC500|Micrologix|PLC/5), which
reuses the AbCip image (already on the host) and binds :44818. No PCCC container
was deployed, and the 4F-not-skip means :44818 was answering CIP (a leftover
AbCip container) so the PCCC reads returned Bad — same shared-port contamination
class as the Modbus false alarm.

Deployed the PCCC compose to ~/otopcua-ablegacy and ran each family with
AB_LEGACY_COMPOSE_PROFILE set to match the running container (required — without
it the [Theory] parametrises over all 3 families and 2 fail against a single-
family container): slc500 2/2, micrologix 1/1 (+1 skip), plc5 1/1 (+1 skip).
Skips are family-inapplicable file types. No code change.

Corrected the sweep record + added the AbLegacy bring-up + run recipe.
Integration-sweep follow-up #4.
2026-07-15 05:38:18 -04:00
dohertj2 ab1644daa4 docs(modbus-tests): sweep #2 was a fixture-cycling false alarm (#448)
v2-ci / build (push) Successful in 4m19s
v2-ci / unit-tests (push) Failing after 10m47s
2026-07-15 05:13:12 -04:00
Joseph Doherty 00f39c849e docs(modbus-tests): document profile-cycling gotcha; sweep #2 was a false alarm
v2-ci / build (pull_request) Successful in 4m13s
v2-ci / unit-tests (pull_request) Failing after 9m41s
The 2026-07 integration sweep flagged the Modbus non-standard profiles as
8F/5F/3F/8F and guessed 'pymodbus 4.x drift'. Re-investigation shows that was
wrong on both counts:

- The image runs the PINNED pymodbus 3.13.0 (verified via import pymodbus;
  __version__). The '...removed in v4' log is 3.13.0's forward-deprecation
  warning, misread as running 4.x.
- The failures were a fixture-cycling artifact: each profile is a separate
  profile-gated compose service sharing :5020, and a plain 'docker compose down'
  (no --profile) does NOT stop the running profile container. It kept holding
  :5020; the next '--profile up' container silently failed to bind (compose
  still says 'Started'), so every later profile run hit the STALE sim. Proven by
  a raw Modbus probe: the 'mitsubishi' sim served DL205's 0xCAFE at HR0.

After force-removing all modbus containers and bringing up each profile cleanly
(verifying it actually published :5020), all four pass: dl205 16/16,
mitsubishi 13/13, s7_1500 10/10, exception_injection 17/17. No OtOpcUa or test
code change needed.

Added a profile-cycling gotcha + reliable-cycle recipe to Docker/README.md and
corrected the sweep record. Integration-sweep follow-up #2.
2026-07-15 05:12:52 -04:00
dohertj2 271562e00e fix(opcuaserver-tests): fix cert-store gap + SDK read-throws drift (#447)
v2-ci / build (push) Successful in 7m17s
v2-ci / unit-tests (push) Failing after 12m55s
2026-07-15 03:34:35 -04:00
Joseph Doherty de9d9697fd fix(opcuaserver-tests): supply client cert-store paths + adapt to SDK read-throws
v2-ci / build (pull_request) Successful in 7m23s
v2-ci / unit-tests (pull_request) Failing after 15m33s
OpcUaServer.IntegrationTests was 4F. Two distinct, previously-masked issues:

1. Cert-store gap: the in-test client SecurityConfiguration used a bare
   new SecurityConfiguration()/CertificateIdentifier(), so ValidateAsync threw
   'TrustedIssuerCertificates StorePath must be specified' before any connect.
   Added TestClientSecurity helper building Directory own/issuer/trusted/rejected
   stores under a throwaway temp PKI dir (mirrors DefaultApplicationConfigurationFactory);
   wired into DualEndpointTests + SubscriptionSurvivalTests.

2. Fixing #1 unmasked an SDK-version drift: the 1.5.378 node-cache read path
   throws ServiceResultException(BadNodeIdUnknown) for a removed node instead of
   returning a bad DataValue. The two subscription-survival assertions now expect
   the throw via Should.ThrowAsync. Behavior under test (removed node -> unknown)
   is unchanged; only the delivery mechanism differs.

Suite 4F -> 4/4 green. Integration-sweep follow-up #5.
2026-07-15 03:34:14 -04:00
dohertj2 c1482802d4 fix(abcip-tests): fix static-init NRE blocking AbCip.IntegrationTests (#446)
v2-ci / build (push) Successful in 4m15s
v2-ci / unit-tests (push) Failing after 18m20s
2026-07-15 03:29:03 -04:00
Joseph Doherty 2ba867b0f8 docs(archreview): mark integration-sweep follow-up #1 (AbCip static-init) done
v2-ci / build (pull_request) Successful in 4m19s
v2-ci / unit-tests (pull_request) Failing after 17m55s
2026-07-15 03:28:24 -04:00
Joseph Doherty 326b22a7ba fix(abcip-tests): expression-body KnownProfiles.All to fix static-init NRE
KnownProfiles.All used a { get; } = [...] initialiser declared textually
before the static readonly profile fields it references, so C# static-init
order captured an all-null list. ForFamily() then NRE'd on p.Family, faulting
AbServerFixture..ctor and blocking the whole AbCip.IntegrationTests suite.

Expression-body it so it evaluates on access, after the fields are set.
Live-verified against the controllogix fixture: suite goes 6F/1P/3skip ->
10 pass / 0 fail / 3 skip (skips = emulator-mode tests).

Integration-sweep follow-up #1 (archreview/plans/INTEGRATION-SWEEP-STATUS.md).
2026-07-15 03:28:04 -04:00
dohertj2 b94ea3d412 Merge pull request 'docs(archreview): integration-suite sweep results + follow-ups' (#445)
v2-ci / build (push) Successful in 4m50s
v2-ci / unit-tests (push) Failing after 9m2s
2026-07-15 01:26:14 -04:00
Joseph Doherty 1ee73d76d6 docs(archreview): record integration-suite sweep results + follow-ups (2026-07-14)
v2-ci / build (pull_request) Successful in 4m46s
v2-ci / unit-tests (pull_request) Failing after 10m56s
Ran the deferred *.IntegrationTests sweep (Mac + docker-host fixtures,
serial). No OtOpcUa regression: OpcUaClient(+Browser)/S7/Modbus-standard
green, Host 105/117, TwinCAT clean-skip. The rest fail on pre-existing
test/fixture bit-rot of distinct kinds (AbCip static-init bug, Modbus
pymodbus-4.x sim drift, FOCAS 127.0.0.1-mock topology, AbLegacy no
fixture, OpcUaServer cert-store config, Host openldap:2.6 image gone).

New INTEGRATION-SWEEP-STATUS.md has the full table, per-suite evidence,
7 follow-ups, and re-run fixture recipes; STATUS.md links it.
2026-07-15 01:25:55 -04:00
dohertj2 306728885d Merge pull request 'docs(archreview): R2-04 T13/T15 live-passed' (#444)
v2-ci / build (push) Successful in 4m13s
v2-ci / unit-tests (push) Failing after 10m54s
2026-07-14 15:14:05 -04:00
Joseph Doherty eccb15bdab docs(archreview): R2-04 T13/T15 live-passed 2026-07-14
v2-ci / build (pull_request) Successful in 4m8s
v2-ci / unit-tests (pull_request) Failing after 11m0s
Both R2-04 verification gates for the S4 primary-gate default-deny run:
- T15 (docker-dev 2-node rig, rebuilt from master 365a90e6 + Modbus sim):
  ServiceLevel 250/240; boot-window write rejected 'not primary (role
  unknown)' (meter reason=role-unknown,site=write); steady-state
  secondary rejected 'not primary' + node reverted (read-back 1234 not
  9999, meter reason=secondary); primary write reaches device; clean
  apply (no apply.failed). The behavior-affecting S4 slice is trusted.
- T13 (in-process PrimaryGateFailoverTests): 1/1, ran without SQL; the
  real DPS-delivered snapshot drives the gate; negative control covered.

STATUS.md + R2-04 tasks.json updated.
2026-07-14 15:13:43 -04:00
dohertj2 365a90e63d Merge pull request 'fix(s7-cli): restore OnDataChange CliFx-console rationale comment' (#443)
v2-ci / build (push) Successful in 3m17s
v2-ci / unit-tests (push) Failing after 8m2s
2026-07-14 14:49:26 -04:00
Joseph Doherty d327243fd3 fix(s7-cli): restore OnDataChange CliFx-console rationale comment
v2-ci / build (pull_request) Failing after 15s
v2-ci / unit-tests (pull_request) Has been skipped
The 9cad9ed0 'strip tracking-ID comments' XML-doc sweep collaterally
removed the comment explaining why the S7 subscribe command routes
OnDataChange through the CliFx IConsole (not System.Console), leaving
SubscribeCommandConsoleHandlerCommentTests red. The Modbus copy still
carries the comment verbatim and the test guards S7-mirrors-Modbus
parity, so restore it (matching the Modbus wording) rather than weaken
the guard.

S7.Cli tests 49/49.
2026-07-14 14:49:10 -04:00
dohertj2 38e21df296 Merge pull request 'fix(historian): map Boolean historization to Int2, not degenerate Int1 (#441)' (#442)
v2-ci / build (push) Successful in 3m24s
v2-ci / unit-tests (push) Failing after 8m26s
2026-07-14 12:38:04 -04:00
Joseph Doherty 9f62310b49 fix(historian): map Boolean historization to Int2, not degenerate Int1 (#441)
v2-ci / build (pull_request) Successful in 3m24s
v2-ci / unit-tests (pull_request) Failing after 8m19s
The historian's Int1 analog-tag creation path is server-degenerate:
EnsureTags(Int1) stores an unusable stub (live-probed on the deployed
2023 R2 historian; root-caused + documented gateway-side in
HistorianGateway #11 / PR #16). HistorianTypeMapper mapped
DriverDataType.Boolean -> HistorianDataType.Int1, so Boolean historized
tags failed auto-provisioning.

Map Boolean -> Int2 instead (store 0/1). Int2 is a proven-creatable
analog type, and the value-write path already sends every value as a
double on WriteLiveValues regardless of tag type, so Boolean 0.0/1.0
round-trips cleanly. Both Float and Int2 are supported, so a
pre-existing Float Boolean tag retypes cleanly on its next deploy.

- HistorianTypeMapper: Boolean => Int2 (+ remarks)
- Unit pins: HistorianTypeMapperTests, GatewayTagProvisionerTests
  (Boolean_definition_carries_explicit_Int2_presence)
- Live gate: EnsureTags_boolean_sandbox_provisions_as_Int2 (was _Int1);
  retype probe reframed Float->Int2 (drops the dead Int1 leg)
- Docs: docs/Historian.md 'Boolean historization maps to Int2', CLAUDE.md,
  archreview STATUS.md (R2-06 live gate -> 6/6)

Resolves #441.
2026-07-14 11:57:27 -04:00
dohertj2 7f79cd5941 Merge pull request 'deps: bump HistorianGateway Client+Contracts to 0.3.0' (#440) from bump-historiangateway-0.3.0 into master
v2-ci / build (push) Successful in 4m17s
v2-ci / unit-tests (push) Failing after 12m24s
2026-07-14 08:18:07 -04:00
27 changed files with 839 additions and 184 deletions
+8 -7
View File
@@ -180,7 +180,7 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
The server uses LDAP-based user authentication via the `Security:Ldap` section in `appsettings.json`. When enabled, credentials are validated by LDAP bind against a GLAuth server, and LDAP group membership maps to OPC UA permissions: `ReadOnly` (browse/read), `WriteOperate` (write FreeAccess/Operate attributes), `WriteTune` (write Tune attributes), `WriteConfigure` (write Configure attributes), `AlarmAck` (alarm acknowledgment). `LdapOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`) implements `IOpcUaUserAuthenticator`, delegating the LDAP bind + group lookup to `OtOpcUaLdapAuthService` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`, an `ILdapAuthService`). See `docs/security.md` for the full guide.
Dev/test LDAP is the **shared GLAuth** running on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`, plaintext/`Transport=None`). It is managed via `scadaproj/infra/glauth/` (source of truth + deploy runbook). Single bind account `cn=serviceaccount,dc=zb,dc=local` / `serviceaccount123`; all test users password `password`. The docker-dev compose binds this shared instance directly — `DevStubMode` is no longer used. The per-VM NSSM GLAuth at `C:\publish\glauth\` and the old base DNs `dc=lmxopcua,dc=local` / `dc=otopcua,dc=local` are obsolete. (The integration-test harness under `tests/.../Host.IntegrationTests/` uses a separate ephemeral bitnami/openldap on port 3894 for automated tests — that is distinct from the shared dev GLAuth.)
Dev/test LDAP is the **shared GLAuth** running on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`, plaintext/`Transport=None`). It is managed via `scadaproj/infra/glauth/` (source of truth + deploy runbook). Single bind account `cn=serviceaccount,dc=zb,dc=local` / `serviceaccount123`; all test users password `password`. The docker-dev compose binds this shared instance directly — `DevStubMode` is no longer used. The per-VM NSSM GLAuth at `C:\publish\glauth\` and the old base DNs `dc=lmxopcua,dc=local` / `dc=otopcua,dc=local` are obsolete. (The integration-test harness under `tests/.../Host.IntegrationTests/` runs its own ephemeral **GLAuth** container on port 3894 — seeded by `tests/.../Host.IntegrationTests/glauth/config.toml` with dev users alice/bob + the `cn=serviceaccount` bind account — for the opt-in `OTOPCUA_HARNESS_USE_LDAP=1` real-bind mode; distinct from the shared dev GLAuth on 3893. It replaced the retired `bitnami/openldap:2.6` in PR #451, 2026-07-15.)
## Library Preferences
@@ -319,11 +319,12 @@ When the section is disabled the applier binds the no-op `NullHistorianProvision
run. Every dispatch logs a tally (`dispatched/requested/ensured/skipped/failed`); `dispatched=N` with
`requested=0` flags the dormant no-op. The API key must carry `historian:tags:write` for `EnsureTags`. (PR
#423 shipped `GatewayTagProvisioner` but left this wiring out, so provisioning was dormant; restored 2026-06-27.)
**0.2.0 DataType delta (archreview 06/U-7):** `EnsureTags` is an **upsert** (create-or-update, not
skip-if-exists), and the 0.2.0 `HistorianTagDefinition.DataType` proto3-optional change means Boolean tags now
provision as **Int1** (0.1.0 silently defaulted them to Float). A pre-bump Float Boolean is asked to retype
on its next deploy; the AVEVA-side retype policy is pinned by the live retype probe — see the "0.2.0 DataType
provisioning delta" section in `docs/Historian.md`.
**DataType mapping (issue #441):** `EnsureTags` is an **upsert** (create-or-update, not skip-if-exists).
`HistorianTypeMapper` maps **`Boolean → Int2`** (Boolean historizes as a small 0/1 integer) because the
historian's `Int1` analog-tag creation path is **server-degenerate**`EnsureTags(Int1)` stores an unusable
stub (live-probed; gateway-side resolution in HistorianGateway #11 / PR #16). Both Float and Int2 are
supported analog types, so a pre-existing Float Boolean tag retypes cleanly to Int2 on its next deploy — see
the "Boolean historization maps to Int2" section in `docs/Historian.md`.
### Gateway-side prerequisites
@@ -358,7 +359,7 @@ export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222 # absolute gateway
export HISTGW_GATEWAY_APIKEY=histgw_<id>_<secret> # must carry historian:read + historian:write (+ tags:write) scopes
export HISTGW_TEST_TAG=<existing-tag> # read round-trip
export HISTGW_WRITE_SANDBOX_TAG=<writable-float-tag> # e.g. HistGW.LiveTest.Sandbox — write round-trip (EnsureTags + write)
export HISTGW_BOOL_SANDBOX_TAG=<writable-bool-tag> # e.g. HistGW.LiveTest.BoolSandbox — 0.2.0 Boolean EnsureTags leg + retype probe (06/U-7)
export HISTGW_BOOL_SANDBOX_TAG=<writable-bool-tag> # e.g. HistGW.LiveTest.BoolSandbox — Boolean→Int2 EnsureTags leg + retype probe (issue #441)
export HISTGW_ALARM_SOURCE=<source-name> # alarm SendEvent → ReadEvents round-trip
dotnet test --filter "Category=LiveIntegration"
```
@@ -0,0 +1,226 @@
# Integration-test sweep — status + follow-ups (2026-07-14)
**Context:** ran the deferred `*.IntegrationTests` sweep (the round-2 STATUS.md "Full integration-suite
sweep" item that discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24).
**How:** tests run **on the Mac** (repo + .NET 10 SDK), pointing at **driver fixtures brought up on the
docker host `10.100.0.35`**, one driver at a time (serial — the suites have a ~16 GB/run memory leak).
Master at run time: `30672888` (branch `master`, clean).
**Docker host `10.100.0.35` (`DOCKER`) note:** has `dotnet 10.0.300`, 22 cores, but only ~7 GB free RAM
(the rest is shared SQL/GLAuth/fixtures) and **no repo checkout** — so the tests can't run *on* it (OOM risk
to shared infra). Run on the Mac against its fixtures. Fixture dirs live at `/opt/otopcua-<driver>/` (owned,
need sudo to create new ones — I deployed FOCAS to `~/otopcua-focas` on the host instead).
---
## Results
| Suite | Result | Root cause of failures |
|---|---|---|
| **OpcUaClient** | ✅ 4/4 | — (opc-plc `:50000`) |
| **OpcUaClient.Browser** | ✅ 1/1 | — |
| **S7** | ✅ 3/3 (2 skip) | skips = outage tests needing `S7_TIMEOUT_OUTAGE_*` env |
| **Modbus** (profile=standard) | ✅ 6/6 | — |
| **Host** | ⚠️ **105/117** (7F, 5 skip) | 7 heavy E2E time out under amd64-emulated SQL (OPEN); LDAP image swapped OpenLDAP→GLAuth 2026-07-15 (PR #451). The 5 skips are driver-fixture-gated E2E, not LDAP |
| **TwinCAT** | ⏭️ 13 skip | no TC3 XAR fixture exists (expected clean-skip) |
| **Modbus** (dl205 / mitsubishi / s7_1500 / exception_injection) | ✅ 16/13/10/17 (was 8F/5F/3F/8F) | **NOT drift** — fixture-cycling artifact, re-verified GREEN 2026-07-15 — see below |
| **AbCip** | ✅ 10P/3skip (was 6F/1P/3skip) | static-init-order bug FIXED 2026-07-15 (`326b22a7`) — see below |
| **FOCAS** | ✅ 10/10 (was 9F/1P) | topology + mock-timer-endianness FIXED 2026-07-15 — see below |
| **AbLegacy** | ✅ slc500 2/2, micrologix 1/1, plc5 1/1 (was 4F) | fixture-deployment gap FIXED 2026-07-15 — see below |
| **OpcUaServer** | ✅ 4/4 (was 4F) | cert-store gap + SDK read-throws drift FIXED 2026-07-15 — see below |
### Headline conclusion
**No OtOpcUa production regression.** Every failure is **pre-existing test/fixture infrastructure rot** in
long-deferred suites — or, in the Modbus case, a **fixture-cycling false alarm** (see below). The drivers are
fine: 4 suites fully green, **all Modbus profiles green** (standard + dl205/mitsubishi/s7_1500/exception once
correctly bound — the sweep's non-standard "failures" were the stale-container artifact), Host 105/117 green,
and all driver **unit** suites + the live `/run` gates pass. These integration suites were DEFERRED in round 2
and never run, so they'd bit-rotted in several distinct ways.
**Follow-up progress (2026-07-15): ALL 7 CLOSED.** #1 AbCip (PR #446), #5 OpcUaServer (PR #447), #2 Modbus
(PR #448 — false alarm), #4 AbLegacy (PR #449 — fixture gap), #3 FOCAS (PR #450 — topology + mock-timer), #7
(fixture deploys), #6 LDAP-image (PR #451 — OpenLDAP→GLAuth), and #6 SQL-deploy-E2E (PR #452 — misdiagnosed
"emulation timing"; really missing FK-parent seeds + a stale EquipmentId, hidden by the in-memory provider).
**Every driver + server integration suite is now verified GREEN. Zero OtOpcUa production regressions across the
entire sweep** — every failure was fixture/harness/test rot.
---
## Per-suite failure detail (evidence)
- **Modbus non-standard profiles** — ✅ **RE-VERIFIED GREEN 2026-07-15 (PR #448). The sweep's 8F/5F/3F/8F was a
false alarm — a fixture-cycling artifact, NOT pymodbus drift and NOT an OtOpcUa regression.** Correcting the
sweep's original diagnosis:
- The image genuinely runs **pymodbus 3.13.0** (`docker run --entrypoint python ... -c 'import pymodbus;
print(pymodbus.__version__)'` → `3.13.0`; the Dockerfile already pins `pymodbus[simulator]==3.13.0`). The
*"…removed in v4"* log line is 3.13.0's **forward-deprecation warning**, not evidence it runs 4.x — the
original sweep note misread it.
- Root cause: each Modbus profile is a **separate profile-gated compose service** sharing `:5020`. A plain
`docker compose down` (no `--profile`) does **not** stop the running profile container, so it keeps holding
`:5020`; the next `--profile … up` container silently fails to bind (compose still says "Started"). Every
later profile run then hit the **stale** container. Proven by a raw Modbus probe: the "mitsubishi" sim
served DL205's `0xCAFE` at HR0 (DL205's V0 marker) and Illegal-Data-Address for mitsubishi-only addresses,
while `docker ps` showed the mitsubishi container up with **no** port mapping and DL205 still holding `:5020`.
- After force-removing all modbus containers and bringing up each profile cleanly (verifying it actually
published `:5020`), all four pass: **dl205 16/16, mitsubishi 13/13, s7_1500 10/10, exception_injection
17/17** (the injector needs `--force-recreate` — it uses a different container name). **No code change** —
added a profile-cycling gotcha + reliable-cycle recipe to `Docker/README.md`.
- **AbCip** — **static-init-order bug in the test harness** (always-fail, blocks the whole suite).
`tests/.../AbCip.IntegrationTests/AbServerProfile.cs`: `KnownProfiles.All { get; } = [ControlLogix, …]` is
declared at **line 30**, *before* the `static readonly` profile fields (line 34+). C# runs static
initializers in textual order → `All` captures `[null,null,null,null]` → `ForFamily` NREs at `p.Family`
(line 61) and `AbServerFixture..ctor` gets null. **Fix (trivial, safe):** make `All` an expression-bodied
property: `public static IReadOnlyList<AbServerProfile> All => [ControlLogix, CompactLogix, Micro800,
GuardLogix];` (evaluated on access, after the fields are set). Unblocks the suite; then re-run to see if the
actual sim tests pass.
- **FOCAS** — ✅ **FIXED 2026-07-15 (PR #450). Two test/fixture bugs, no OtOpcUa driver change.**
1. *Topology:* `FocasSimFixture` already reads an env-overridable endpoint (`OTOPCUA_FOCAS_SIM_ENDPOINT`,
default `localhost:8193`) and exposes `Host`/`Port`, but `WireBackendTests` + `WireBackendCoverageTests`
**hardcoded** `focas://127.0.0.1:8193`. Against a remote fixture the skip-gate passed (remote reachable) but
the driver dialed localhost → KeyNotFound. Fix: added a `DeviceUri` property to the fixture and derived each
test's `DeviceHost` from it — recovered **8 of 9**.
2. *Mock timer endianness (newly surfaced):* the last failure (`Timers_populate`) read `60397977600` for a
`3600 s` power-on timer (= `0x3C000000 × 60`). Root cause: the focas-mock's `_wire_timer` encoded the
`minute`/`msec` fields **big-endian** (`_u32`), but `cnc_rdtimer`'s timer fields are **little-endian** on
real hardware — a documented, live-31i-B-validated quirk (`FocasWireClient.ParseTimer` decodes them LE, per
`docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md`). **The driver was correct; the mock was wrong.** Fix:
`_wire_timer` now emits little-endian (`_u32_le`) → suite **10/10**. Fixture deployed at `~/otopcua-focas`.
- **AbLegacy** — ✅ **FIXED 2026-07-15 (PR #449). Fixture-deployment gap, not a code defect.** The AbLegacy suite
needs `ab_server` in **PCCC mode** (`--plc=SLC500|Micrologix|PLC/5`), which reuses the **AbCip image**
(`otopcua-ab-server:libplctag-release`, already on the host) and binds `:44818`. Two issues: (a) no PCCC
container was deployed (`~/otopcua-ablegacy` didn't exist); (b) the sweep's *4F-not-skip* means `:44818` was
reachable but answering **CIP** (a leftover AbCip container) — the PCCC reads returned Bad. Same shared-port
contamination class as the Modbus false alarm. Deployed the compose to `~/otopcua-ablegacy` on the host (image
already present, no build needed) and ran each family with `AB_LEGACY_COMPOSE_PROFILE` set to match the running
container (**required** — without it the `[Theory]` parametrises over all 3 families and 2 fail against a
single-family container): **slc500 2/2, micrologix 1/1 (+1 skip), plc5 1/1 (+1 skip)** — the skips are
family-inapplicable file types (e.g. no `F8` float file seeded on micrologix/plc5). **No code change.**
- **OpcUaServer** — ✅ **FIXED 2026-07-15** (PR #447). Was 4F on `Opc.Ua.ServiceResultException :
TrustedIssuerCertificates StorePath must be specified` — the in-test *client* `SecurityConfiguration`
omitted the trusted-issuer/peer/rejected store paths that `ValidateAsync` requires. Fix: `TestClientSecurity`
helper rooting Directory stores at a throwaway PKI dir (mirrors `DefaultApplicationConfigurationFactory`).
Fixing it unmasked a second drift — the 1.5.378 node-cache read path throws
`ServiceResultException(BadNodeIdUnknown)` for a removed node rather than returning a bad `DataValue`;
the two subscription-survival assertions now use `Should.ThrowAsync`. Suite **4F → 4/4 green**.
- **Host** — largely healthy (105/117). Two independent issues:
- **LDAP image — ✅ FIXED 2026-07-15 (PR #451).** The opt-in real-LDAP mode (`OTOPCUA_HARNESS_USE_LDAP=1`)
used `bitnami/openldap:2.6`, gone since Bitnami deprecated their free catalog in 2025. **Replaced it with
GLAuth** — the same LDAP server the rest of the project uses (docker-dev + the live OPC UA data-plane auth,
both against the shared GLAuth on `:3893`), removing the lone OpenLDAP outlier. Added a harness
`glauth/config.toml` (base DN `dc=zb,dc=local`, search-capable `serviceaccount`, dev users alice=full /
bob=read-only, role groups with the same gidnumbers as `scadaproj/infra/glauth`) + repointed the harness's
service-account override to `cn=serviceaccount`. Live-verified all three bind legs against a deployed
container on `:3894` (serviceaccount search returns alice + 5 memberOf groups; alice/bob bind OK; wrong
password → `Invalid credentials (49)`). NB: the real-LDAP mode is **opt-in** — the default Host run uses
`StubLdapAuthService`, so this never blocked the default suite (the sweep's "5 LDAP skips" were actually the
driver-fixture-gated E2E tests, not LDAP).
- **SQL-backed deploy E2E — ✅ FIXED 2026-07-15 (PR #452). The sweep's "amd64-emulated-SQL timing" diagnosis
was WRONG.** Running the 7 heavy E2E against real (native-amd64) SQL on the Docker host proved they don't
time out for speed — they had **never actually run against real SQL**, only the EF in-memory provider (which
ignores FK constraints). Two distinct root causes, both hidden by in-memory:
- *Missing FK-parent seed (5 tests):* the deploy writes a `NodeDeploymentState` row per node, which FKs to
`ClusterNode` (`FK_NodeDeploymentState_ClusterNode_NodeId`). The tests never seeded `ServerCluster` +
`ClusterNode` rows, so on SQL each node-state INSERT throws → the deploy never seals (waits the full
deadline regardless of how high — proven at 4× it still hung). Added `TwoNodeClusterHarness.SeedDefaultClusterAsync`
(seeds a `ServerCluster` + a `ClusterNode` per node; `Warm`/NodeCount=2 to satisfy both the SQL CHECK
constraint `CK_ServerCluster_RedundancyMode_NodeCount` and the `ClusterEnabledNodeCountMismatch` validator)
and called it in `DeployHappyPathTests`×2 / `FailoverDuringDeployTests` / `FleetDiagnosticsRoundTripTests` /
`EquipmentNamespaceMaterializationTests`; fixed `DriverReconnectE2eTests` to seed node B + declare NodeCount=2.
- *Stale `EquipmentId` (1 test, pre-existing on BOTH providers):* `EquipmentNamespaceMaterializationTests`
seeded `EquipmentId="eq-1"`, but the `EquipmentIdNotDerived` validator (added later) requires
`EquipmentId == DeriveEquipmentId(EquipmentUuid)` = `EQ-` + first 12 hex of the UUID. Fixed the seed to a
canonical id + matching UUID. (Was failing on in-memory too; only surfaced once the FK fix let the deploy
reach equipment validation.)
- Added `OTOPCUA_HARNESS_SQL_HOST` / `OTOPCUA_HARNESS_LDAP_HOST` env overrides so the fixtures can live on the
native-amd64 Docker host (`docker compose up -d sql` there; run with `OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331`).
- Result: the 5 SQL-backed classes go **11/11 green vs native SQL**, and the in-memory default is unregressed
(12/12). `RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts…` is a **pre-existing flaky
race test** (not SQL-backed; passes in isolation, flaked once under heavy parallel load) — left as-is.
- **TwinCAT** — 13 skip cleanly (no TC3 XAR fixture exists; unit-only per the driver's design). Expected.
---
## Follow-up backlog (distinct, not one fix)
1. ~~**AbCip static-init bug** — trivial always-fail test bug. Expression-body `KnownProfiles.All`.~~
✅ **DONE 2026-07-15** (commit `326b22a7`, PR #446). Live-verified against the controllogix fixture:
suite went **6F/1P/3skip → 10 pass / 0 fail / 3 skip** (skips = emulator-mode tests needing a different
fixture mode).
2. ~~**Modbus non-standard profiles** — pin pymodbus in the fixture Dockerfile.~~
✅ **RESOLVED 2026-07-15 (PR #448) — false alarm, no code change.** Image already runs the pinned
pymodbus 3.13.0; the 8F/5F/3F/8F was a fixture-cycling artifact (stale profile container holding `:5020`).
Re-verified all four profiles GREEN (16/13/10/17). Added a profile-cycling gotcha + reliable-cycle recipe
to `Docker/README.md`.
3. ~~**FOCAS** — reconcile the 127.0.0.1-mock topology with the remote-fixture model.~~
✅ **DONE 2026-07-15 (PR #450).** Tests now derive `DeviceHost` from the fixture endpoint (recovered 8/9);
also fixed a newly-surfaced mock bug — `_wire_timer` encoded the timer big-endian but real hardware +
driver use little-endian. Suite **10/10**. No OtOpcUa driver change.
4. ~~**AbLegacy** — deploy an AbLegacy sim fixture (none on host).~~
✅ **DONE 2026-07-15 (PR #449) — fixture-deployment gap, no code defect.** Deployed the PCCC compose to
`~/otopcua-ablegacy` (reuses the AbCip image, already on host). All 3 families GREEN (slc500 2/2, micrologix
1/1, plc5 1/1). Recipe added below; `AB_LEGACY_COMPOSE_PROFILE` MUST match the running container.
5. ~~**OpcUaServer** — supply cert-store StorePath in the harness config.~~
✅ **DONE 2026-07-15** (PR #447). Added `TestClientSecurity` helper building a validated client
`SecurityConfiguration` with Directory own/issuer/trusted/rejected stores under a throwaway PKI dir
(mirrors `DefaultApplicationConfigurationFactory`); wired into both `DualEndpointTests` +
`SubscriptionSurvivalTests`. That unmasked a **second** drift: the 1.5.378 node-cache read path
*throws* `ServiceResultException(BadNodeIdUnknown)` for a removed node instead of returning a bad
`DataValue` — updated the two subscription-survival assertions to `Should.ThrowAsync`. Suite **4F → 4/4**.
6. **Host** — ⬐ split into two:
- ~~replace `bitnami/openldap:2.6` (image gone)~~ ✅ **DONE 2026-07-15 (PR #451)** — swapped the harness LDAP
to **GLAuth** (unifies on the LDAP the rest of the project uses); live-verified the binds on `:3894`.
- ~~**amd64-emulated-SQL timing**~~ ✅ **DONE 2026-07-15 (PR #452) — misdiagnosed.** Not a timing/emulation
issue: the deploy E2E tests had never run against real SQL (in-memory ignores FKs) and lacked the
`ServerCluster`/`ClusterNode` seed the `NodeDeploymentState` FK requires → deploy never seals. Added a
harness seeding helper + `OTOPCUA_HARNESS_SQL_HOST` override; also fixed a stale `EquipmentId` in one test.
11/11 green vs native SQL, in-memory unregressed. (Roslyn racing test = pre-existing flaky, left as-is.)
7. ~~**Fixture deployment gap** — FOCAS + AbLegacy fixtures aren't on the host.~~
✅ **RESOLVED 2026-07-15.** Both now deployed under the home dir: `~/otopcua-ablegacy` (PR #449) +
`~/otopcua-focas` (PR #450). (`/opt` still needs sudo; the home-dir deploys work fine.)
---
## Fixture setup recipes (to re-run the sweep)
Bring up on the docker host (distinct ports, can coexist except Modbus profiles share `:5020`):
```bash
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-modbus && docker compose --profile standard up -d' # :5020
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-abcip && docker compose --profile controllogix up -d' # :44818
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d' # :1102
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-opcuaclient && docker compose up -d' # :50000
ssh dohertj2@10.100.0.35 'cd ~/otopcua-ablegacy && docker compose --profile slc500 up -d' # :44818 (PCCC; reuses AbCip image)
ssh dohertj2@10.100.0.35 'cd ~/otopcua-focas && docker compose up -d --build' # :8193
```
Run per suite (env → project):
| Suite | Env | Project |
|---|---|---|
| OpcUaClient(+Browser) | `OPCUA_SIM_ENDPOINT=opc.tcp://10.100.0.35:50000` | `tests/Drivers/…OpcUaClient(.Browser).IntegrationTests` |
| S7 | `S7_SIM_ENDPOINT=10.100.0.35:1102` | `tests/Drivers/…S7.IntegrationTests` |
| Modbus | `MODBUS_SIM_ENDPOINT=10.100.0.35:5020` `MODBUS_SIM_PROFILE=<standard\|dl205\|mitsubishi\|s7_1500\|exception_injection>` (swap the host sim to match) | `tests/Drivers/…Modbus.IntegrationTests` |
| AbCip | `AB_SERVER_ENDPOINT=10.100.0.35:44818` `AB_SERVER_PROFILE=controllogix` | `tests/Drivers/…AbCip.IntegrationTests` |
| AbLegacy | `AB_LEGACY_ENDPOINT=10.100.0.35:44818` `AB_LEGACY_COMPOSE_PROFILE=<slc500\|micrologix\|plc5>` (**must** match the running PCCC container — else the `[Theory]` runs all 3 families and 2 fail) | `tests/Drivers/…AbLegacy.IntegrationTests` |
| FOCAS | `OTOPCUA_FOCAS_SIM_ENDPOINT=10.100.0.35:8193` (tests now honour this — no longer hardcoded to 127.0.0.1) | `tests/Drivers/…FOCAS.IntegrationTests` |
| Host | `OTOPCUA_HARNESS_USE_SQL=1` (+ SQL `:14331` via the harness `docker-compose.yml up sql`) — set `OTOPCUA_HARNESS_USE_LDAP=1` to exercise real binds against the **GLAuth** service (`up ldap`, host `:3894`; users alice/bob, svc `cn=serviceaccount`/`serviceaccount123`) | `tests/Server/…Host.IntegrationTests` |
**Modbus profile cycling:** the sim services share `:5020`, so to run a profile:
`ssh … 'cd /opt/otopcua-modbus && docker compose down && docker compose --profile <p> up -d'`, then run with
`MODBUS_SIM_PROFILE=<p>` (each class skips unless its profile is active). Wait for `:5020` before running —
TCP opens before the register map fully loads.
---
## Current state (post-sweep)
- **All fixtures torn down** — modbus/abcip/s7/opcuaclient/focas on the host removed (ports 5020/44818/1102/
50000/8193 all closed); the local Host SQL fixture `down -v`'d. Shared host is clean.
- **Repo:** `master @ 30672888`, clean. No sweep changes committed (this file is the only artifact).
- **Nothing merged for the sweep** — it's a read-only test run; the follow-ups above are unstarted.
@@ -2,20 +2,122 @@
"planPath": "archreview/plans/R2-04-failure-visibility-plan.md",
"lastUpdated": "2026-07-12",
"tasks": [
{ "id": "T1", "subject": "Extend AddressSpaceApplyOutcome with RebuildFailed/FailedNodes + SafeRebuild returns bool (01/S-1)", "status": "completed", "blockedBy": [] },
{ "id": "T2", "subject": "Safe* helpers return bool; Apply tallies removal-pass failures into FailedNodes", "status": "completed", "blockedBy": ["T1"] },
{ "id": "T3", "subject": "Materialise* passes return swallowed-failure counts (void -> int, five methods)", "status": "completed", "blockedBy": ["T2"] },
{ "id": "T4", "subject": "OpcUaApplyFailed counter + OpcUaPublishActor Error/meter surfacing of degraded applies", "status": "completed", "blockedBy": ["T3"] },
{ "id": "T5", "subject": "Galaxy internal IMxWriteOps seam under GatewayGalaxyDataWriter (sealed SDK session untestable) (06/S-1)", "status": "completed", "blockedBy": [] },
{ "id": "T6", "subject": "Galaxy fail-closed write on AdviseSupervisory failure -> BadCommunicationError, no raw write", "status": "completed", "blockedBy": ["T5"] },
{ "id": "T7", "subject": "Galaxy counters galaxy.writes.advise_failed + galaxy.writes.unconfirmed (empty-statuses Good metered)", "status": "completed", "blockedBy": ["T6"] },
{ "id": "T8", "subject": "Galaxy skip-gated live smoke: normal advised write still returns Good post-change", "status": "completed", "blockedBy": ["T6"] },
{ "id": "T9", "subject": "PrimaryGatePolicy pure class + truth-table tests + otopcua.redundancy.primary_gate_denied counter (03/S4)", "status": "completed", "blockedBy": [] },
{ "id": "T10", "subject": "Wire DriverHostActor gates (:977/:1039/:1095) through PrimaryGatePolicy with member-count provider + S5 log-once", "status": "completed", "blockedBy": ["T9"] },
{ "id": "T11", "subject": "DriverHostActorPrimaryGateTests: deny-on-unknown multi-node, allow single-node, ack/emit variants, meter tags", "status": "completed", "blockedBy": ["T10"] },
{ "id": "T12", "subject": "ScriptedAlarmHostActor alerts-emit gate adopts PrimaryGatePolicy (consistency extension)", "status": "completed", "blockedBy": ["T9"] },
{ "id": "T13", "subject": "In-process 2-node delivery-path test (TwoNodeClusterHarness): DPS-delivered snapshot drives the gate", "status": "deferred-live", "note": "rig/live; Host.IntegrationTests (~16GB/run leak + needs SQL/2-node rig) — implemented + build-verified, controller runs it in the serialized heavy pass", "blockedBy": ["T11"] },
{ "id": "T14", "subject": "docs/Redundancy.md: gate policy table, boot-window client experience, new meters, Galaxy fail-closed semantics", "status": "completed", "blockedBy": [] },
{ "id": "T15", "subject": "LIVE GATE: 2-node docker-dev rig verify (ServiceLevel 240/100, boot-window write rejected+reverted, single /alerts row)", "status": "deferred-live", "note": "rig/live; 2-node docker-dev rig session (rebuild both centrals) — merge-blocking gate the controller runs in the serialized heavy pass", "blockedBy": ["T10", "T11", "T12", "T13"] }
{
"id": "T1",
"subject": "Extend AddressSpaceApplyOutcome with RebuildFailed/FailedNodes + SafeRebuild returns bool (01/S-1)",
"status": "completed",
"blockedBy": []
},
{
"id": "T2",
"subject": "Safe* helpers return bool; Apply tallies removal-pass failures into FailedNodes",
"status": "completed",
"blockedBy": [
"T1"
]
},
{
"id": "T3",
"subject": "Materialise* passes return swallowed-failure counts (void -> int, five methods)",
"status": "completed",
"blockedBy": [
"T2"
]
},
{
"id": "T4",
"subject": "OpcUaApplyFailed counter + OpcUaPublishActor Error/meter surfacing of degraded applies",
"status": "completed",
"blockedBy": [
"T3"
]
},
{
"id": "T5",
"subject": "Galaxy internal IMxWriteOps seam under GatewayGalaxyDataWriter (sealed SDK session untestable) (06/S-1)",
"status": "completed",
"blockedBy": []
},
{
"id": "T6",
"subject": "Galaxy fail-closed write on AdviseSupervisory failure -> BadCommunicationError, no raw write",
"status": "completed",
"blockedBy": [
"T5"
]
},
{
"id": "T7",
"subject": "Galaxy counters galaxy.writes.advise_failed + galaxy.writes.unconfirmed (empty-statuses Good metered)",
"status": "completed",
"blockedBy": [
"T6"
]
},
{
"id": "T8",
"subject": "Galaxy skip-gated live smoke: normal advised write still returns Good post-change",
"status": "completed",
"blockedBy": [
"T6"
]
},
{
"id": "T9",
"subject": "PrimaryGatePolicy pure class + truth-table tests + otopcua.redundancy.primary_gate_denied counter (03/S4)",
"status": "completed",
"blockedBy": []
},
{
"id": "T10",
"subject": "Wire DriverHostActor gates (:977/:1039/:1095) through PrimaryGatePolicy with member-count provider + S5 log-once",
"status": "completed",
"blockedBy": [
"T9"
]
},
{
"id": "T11",
"subject": "DriverHostActorPrimaryGateTests: deny-on-unknown multi-node, allow single-node, ack/emit variants, meter tags",
"status": "completed",
"blockedBy": [
"T10"
]
},
{
"id": "T12",
"subject": "ScriptedAlarmHostActor alerts-emit gate adopts PrimaryGatePolicy (consistency extension)",
"status": "completed",
"blockedBy": [
"T9"
]
},
{
"id": "T13",
"subject": "In-process 2-node delivery-path test (TwoNodeClusterHarness): DPS-delivered snapshot drives the gate",
"status": "completed",
"note": "DONE 2026-07-14: PrimaryGateFailoverTests PASS 1/1 (3s), ran without SQL (gate test needs only the in-process 2-node cluster + DPS snapshot). Delivered snapshot promotes exactly one Primary; negative-control (broken subscribe) times out. See STATUS.md.",
"blockedBy": [
"T11"
]
},
{
"id": "T14",
"subject": "docs/Redundancy.md: gate policy table, boot-window client experience, new meters, Galaxy fail-closed semantics",
"status": "completed",
"blockedBy": []
},
{
"id": "T15",
"subject": "LIVE GATE: 2-node docker-dev rig verify (ServiceLevel 240/100, boot-window write rejected+reverted, single /alerts row)",
"status": "completed",
"note": "DONE 2026-07-14: docker-dev 2-node rig rebuilt from master 365a90e6 + Modbus sim up. ServiceLevel 250/240; boot-window write rejected \"not primary (role unknown)\" (meter reason=role-unknown,site=write); steady-state secondary rejected \"not primary\" + node reverted (read-back 1234 not 9999, meter reason=secondary); primary write reaches device; clean apply (no apply.failed). Full record in STATUS.md.",
"blockedBy": [
"T10",
"T11",
"T12",
"T13"
]
}
]
}
}
+72 -16
View File
@@ -221,9 +221,9 @@ Client.UI 126, Core.AlarmHistorian 29, Core.ScriptedAlarms 71, and all drivers (
Every plan's integration-suite and live-`/run` gates were deferred (not skipped) so nothing OOMs the box and the
shared docker-dev rig / VPN infra isn't contended. To run one-at-a-time:
- **Full integration-suite sweep** (10 `*.IntegrationTests` projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24.
- **Full integration-suite sweep** (10 `*.IntegrationTests` projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24. **RAN 2026-07-14 — see [`INTEGRATION-SWEEP-STATUS.md`](INTEGRATION-SWEEP-STATUS.md).** Outcome: OpcUaClient(+Browser)/S7/Modbus-standard green, Host 105/117, TwinCAT clean-skip; the rest fail on **pre-existing test/fixture bit-rot** (AbCip static-init bug, Modbus pymodbus-4.x sim drift, FOCAS 127.0.0.1-mock topology, AbLegacy no fixture, OpcUaServer cert-store config, Host openldap:2.6 image gone). **No OtOpcUa regression.** 7 follow-ups in that file.
- **docker-dev `:9200` live-`/run`** (rebuild BOTH centrals): R2-02 T15 (ResilienceConfig warning banner / input lockout / unknown-key survival), R2-03 T12 (VT fail→Bad→recover + P7 no-recompile), R2-10 T10 (breaker-open on both centrals + recovery + FOLLOWUP-10 log line), R2-11 T22 (typed-editor Writable checkbox), **R2-07 T12 (surgical add/remove/mixed — F10b inertness risk, MUST run)**, R2-05 T15 (over-gating regression only — deny unobservable under `DisableLogin`).
- **2-node redundancy rig:** **R2-04 T13/T15 — behavior-affecting S4 primary-gate default-deny; MUST run before trusting the failover-role change.**
- **2-node redundancy rig:** ~~R2-04 T13/T15~~ **BOTH ✅ PASSED 2026-07-14** — T15 live docker-dev gate + T13 in-process delivery-path test (see below).
- **Genuinely external / needs-user:** R2-06 T12 (VPN to a 0.2.0 HistorianGateway for the live EnsureTags retype probe), R2-08 live legs (real GLAuth outage/recovery cycle; `Category=LiveIntegration` gateway soak), R2-01 T11 (S7 SYN-blackhole outage on the `10.100.0.35` s7_1500 fixture).
### Deferred behavior-change follow-up (deliberately not shipped)
@@ -280,6 +280,55 @@ gates (R2-06 VPN, R2-08 live LDAP, R2-01 S7 blackhole) still need their infra.
---
## R2-04 T15 — 2-node primary-gate LIVE GATE ✅ PASSED (2026-07-14)
Ran the S4 primary-gate live recipe on the docker-dev 2-node central cluster **rebuilt from master `365a90e6`**
(`up -d --build central-1 central-2`). Brought up the **Modbus sim** (`otopcua-pymodbus-standard` on
`10.100.0.35:5020`) so `MAIN`'s Modbus equipment tags read **Good** (a data-less rig materialises every node
`Bad_WaitingForInitialData`, and the CLI write reads-first — so a live driver is required to have a writable-Good
node to drive the gate). Made `EQ-55297329838d/ModbusHR200` writable (`TagConfig.writable=true` via SQL +
`POST :9200/api/deployments`). Data-plane writes bound `multi-role`/`password` (carries the `WriteOperate` LDAP role).
**All core assertions PASS:**
- **Step 1 — ServiceLevel split:** `Server_ServiceLevel` (`i=2267`) = **250 on `:4840` (primary) / 240 on `:4841`
(secondary)** — differentiated, no split-brain. (Plan said 240/100; the deployment's healthy values are 250/240 —
the invariant "primary > secondary" holds.)
- **Step 2 — boot-window default-DENY (the NEW S4 behavior):** restarting **both** centrals (to widen the
role-null-with-≥2-members window — a single-node restart's window is sub-second because cluster-join and the first
redundancy snapshot land near-simultaneously) caught, ~5 s into boot:
`WRN Operator write to EQ-55297329838d/ModbusHR200 rejected: not primary (role unknown)`. Meter
`otopcua_redundancy_primary_gate_denied_total{reason="role-unknown",site="write"} = 1`.
- **Step 3 — steady-state secondary DENY + #5 revert:** write `9999` to `:4841` → CLI reports optimistic
`Write successful` (synchronous, by design) but read-back = **1234 (the primary's value, reverted — NOT 9999)**;
central-2 logs `rejected: not primary`; meter `reason="secondary",site="write" = 1`.
- **Step 3b — primary write reaches the device:** write `1234` to `:4840` → `Write successful` and read-back on
`:4841` = `1234` (the Modbus driver wrote through).
- **T4 sanity — clean apply:** both centrals logged `AddressSpaceApplier: applied plan (kind=PureAdd … failed=0)` /
`materialised (… failed=0)`; **no `[ERR]` / `apply.failed` / degraded-apply line**; the `apply_failed` meter is
absent (never incremented).
- **Step 4 — `/alerts`:** endpoint 200, **0 rows** — no active alarm on the data-less rig, so trivially no
dual-primary double-emission (the double-row regression is only observable with a firing alarm, which needs live
driver data this rig lacks; the alerts-emit gate shares the same `PrimaryGatePolicy` and is unit-covered).
**Net:** the R2-04 `PrimaryGatePolicy` membership-aware **default-DENY-on-unknown** is confirmed live end-to-end —
both denial reasons (`role-unknown`, `secondary`) fire the log + the new meter, the #5 optimistic-write revert
corrects the node, and the primary path is unaffected. The behavior-affecting S4 slice is trusted.
**R2-04 T13 — in-process delivery-path test ✅ PASSED (2026-07-14).**
`dotnet test …Host.IntegrationTests --filter FullyQualifiedName~PrimaryGateFailoverTests` → **1/1, 3 s**. Ran
**without SQL** (`OTOPCUA_HARNESS_USE_SQL` unset) — the gate test needs only the in-process 2-node Akka cluster +
the DPS-delivered redundancy snapshot, not a ConfigDb deploy. The `TwoNodeClusterHarness` formed the cluster and
the **real DistributedPubSub-delivered** snapshot promoted exactly one node to Primary (Secondary's
`DriverHostActor.RouteNodeWrite` → `not primary`; Primary passed the gate, rejecting only with a mapping reason).
Its built-in negative control (a broken redundancy-topic subscribe leaves both stuck `role unknown` → 45 s timeout)
means the PASS proves genuine snapshot delivery drives the gate — the half the pure-TestKit unit guards can't cover.
**Rig mutations (this pass):** `ModbusHR200.TagConfig.writable=true`; Modbus sim brought up on the shared docker host
(`10.100.0.35`). Reset the central rig with `docker compose -f docker-dev/docker-compose.yml down -v`; the sim with
`ssh dohertj2@10.100.0.35 'cd /opt/otopcua-modbus && docker compose --profile standard down'`.
---
## Post-round-2 loose-end closure (2026-07-13)
- **Pre-existing red unit test closed** (theme-#4 "CI to all suites" hygiene; predates round 2, not an R2 plan).
@@ -327,24 +376,31 @@ convention) + a regression unit test + `StorageRateMs` on the live-suite inline
- ✅ `Write_then_read_on_sandbox_tag` — `EnsureTags(Float)` + `WriteLiveValues` (SQL) + read-back.
- ✅ `Alarm_SendEvent_is_acked` + ✅ `Alarm_SendEvent_then_ReadEvents` — alarm `SendEvent` + SQL `ReadEvents`.
- ✅ `EnsureTags_type_change_outcome_is_observable` — now observes a **real** retype outcome (not a storage-rate error).
- `EnsureTags_boolean_sandbox_provisions_as_Int1` — **Int1 (Boolean) tag creation is not supported by the
gateway's histsdk on this historian**: `EnsureTags(Int1)` returns `ProtocolEvidenceMissingException` (the vendored
client's "no capture evidence for this type" guard → gRPC `Unimplemented`), both fresh and as a retype target.
A **type sweep** confirmed it is Int1-specific: **Int2/UInt2/Int4/UInt4/Float/Double all provision `success=True`;
only Int1 fails.** Root cause: `histsdk` `HistorianTagWriteProtocol.GetAnalogDataTypeCode` has no captured
CTagMetadata wire code for Int1. This is a **gateway/histsdk type-support gap**, not an OtOpcUa defect — **filed as
[historiangw#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11)**.
- ⚠️ `EnsureTags_boolean_sandbox_provisions_as_Int1` (as originally written) — **Int1 (Boolean) tag creation is not
supported by the gateway's histsdk on this historian**: `EnsureTags(Int1)` returns `ProtocolEvidenceMissingException`
(the vendored client's "no capture evidence for this type" guard → gRPC `Unimplemented`), both fresh and as a retype
target. A **type sweep** confirmed it is Int1-specific: **Int2/UInt2/Int4/UInt4/Float/Double all provision
`success=True`; only Int1 fails.** Root cause: `histsdk` `HistorianTagWriteProtocol.GetAnalogDataTypeCode` has no
captured CTagMetadata wire code for Int1 a **gateway/histsdk type-support gap**, not an OtOpcUa defect
(**filed as [historiangw#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11)**, which routed the
decision back as **[issue #441](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/441)**).
**✅ RESOLVED (issue #441, 2026-07-14):** `HistorianTypeMapper` now maps **`Boolean → Int2`** (0/1 integer) instead
of the degenerate `Int1`. Int2 is a proven-creatable analog type on this historian, and the value-write path already
sends every value as a `double` on `WriteLiveValues` regardless of tag type, so Boolean `0.0/1.0` round-trips
cleanly. The live gate test is renamed `EnsureTags_boolean_sandbox_provisions_as_Int2` and now passes → **6/6**. Unit
pins updated (`HistorianTypeMapperTests`, `GatewayTagProvisionerTests.Boolean_definition_carries_explicit_Int2_presence`);
gateway driver unit `103/103`.
**✅ U-7 retype policy — ANSWERED (via `grpcurl` against the real historian):**
- `EnsureTags` is a **real create-or-update upsert** — re-`EnsureTags` of an existing tag at the same type → `success`.
- An **in-place retype between SUPPORTED types is ACCEPTED** — `Float→Double` on an existing tag returns `success`
(of the three 06/U-7 hypotheses: **accepted-retyped**, not rejected).
- **But the specific 0.2.0 Boolean case (Float→Int1) can't retype** — Int1 isn't a creatable type here, so a
pre-0.2.0 **Float Boolean tag asked to become Int1 fails the provision** with `ProtocolEvidenceMissingException`
— it is **neither silently retyped nor data-lost**; the provision just fails (and is metered `failed=N`). That is
the practically-relevant U-7 answer for Boolean tags on this deployment.
- **The specific `Int1` case can't retype** — Int1 isn't a creatable type here. Rather than accept a broken Boolean
path, OtOpcUa **remapped `Boolean → Int2`** (issue #441, above), which retypes cleanly from a pre-existing Float
Boolean tag (both are supported analog types) and never hits the Int1 gap.
The R2-06 **code** deliverables (S-11 fail-fast + `ServerHistorianOptionsValidator`; U-7 Boolean→Int1 mapping +
The R2-06 **code** deliverables (S-11 fail-fast + `ServerHistorianOptionsValidator`; Boolean provisioning mapping +
upsert; C-7 doc) were already unit-covered + merged; this live gate additionally **found + fixed the StorageRateMs
provisioning bug** and **answered the U-7 retype policy**. The one remaining red test is a documented gateway
histsdk Int1-support gap.
provisioning bug**, **answered the U-7 retype policy**, and surfaced the Int1 histsdk gap that drove the
**Boolean→Int2 remap (issue #441)** — closing the last red test to reach **6/6**.
+36 -40
View File
@@ -141,48 +141,44 @@ for this node), which is the signal to check that `ServerHistorian:Enabled=true`
> so a HistoryRead on it runs to the full `CallTimeout` (~30 s) before erroring. Auto-provisioning on deploy
> keeps freshly-historized tags from hitting that slow path. See the `CallTimeout` row above.
### 0.2.0 DataType provisioning delta (archreview 06/U-7)
### Boolean historization maps to Int2 (issue #441)
`EnsureTags` is contract-documented (HistorianGateway 0.2.0) as an **upsert** — *"creates or updates
`EnsureTags` is contract-documented (HistorianGateway 0.2.0+) as an **upsert** — *"creates or updates
historian tags according to the given definitions"* — that maps per-tag onto AVEVA's
`HistorianClient.EnsureTagAsync(definition) -> bool`. It is **not idempotent** (not routed through the
idempotent retry pipeline) and **not skip-if-exists**.
The 0.1.0→0.2.0 client bump made `HistorianTagDefinition.DataType` **proto3-optional** (explicit presence).
Under 0.1.0, the provisioner's `Boolean → Int1` mapping sent wire value `0`, indistinguishable from unset, so
the gateway defaulted such tags to the SDK-default **Float**. Under 0.2.0 the same call carries explicit
presence, so Boolean tags now provision as **Int1** (the pre-existing intent of `HistorianTypeMapper` — no
mapper code change is needed or wanted). Consequences:
`HistorianTypeMapper` maps **`DriverDataType.Boolean → HistorianDataType.Int2`** and Boolean values
historize as a small **0/1 integer**. Int2 is used rather than the semantically-exact `Int1` because the
historian's **`Int1` analog-tag creation path is server-degenerate** — `EnsureTags(Int1)` stores an
unusable stub, so a Boolean tag mapped to `Int1` never becomes historizable. This is an evidence-based
decision from a live probe against the deployed 2023 R2 historian; the root cause and gateway-side
resolution are tracked in [issue #441](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/441) and
HistorianGateway [#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11) / PR #16 (which
documents `Int1` as an explicit, evidence-based exclusion). The value-write path is unaffected: the
recorder sends every value as a `double` on `WriteLiveValues` regardless of tag type, so a Boolean
`0.0/1.0` round-trips cleanly into the Int2 tag.
1. **New Boolean tags provision correctly as Int1.** This is the fix arriving as a silent, zero-diff
semantic change from the bump.
2. **Pre-bump Boolean tags exist historian-side as Float.** On the next deploy that dispatches provisioning
for such a tag, the gateway is asked (via the upsert) to update it to Int1.
**Observed live (2026-07-13, real 0.2.0 gateway → real AVEVA historian on `wonder-sql-vd03`, probed via
`grpcurl` — archreview R2-06 T12):**
- `EnsureTags` is a **real create-or-update upsert** — re-`EnsureTags` of an existing tag at the same type
returns `Success=true`.
- An **in-place retype between SUPPORTED types is ACCEPTED**`Float→Double` on an existing tag returns
`Success=true`.
- A type sweep confirmed **`Int2/UInt2/Int4/UInt4/Float/Double` all provision `Success=true`; only `Int1`
fails** (`Success=false, Error="ProtocolEvidenceMissingException"` — the vendored client's "no capture
evidence for this type" guard → gRPC `Unimplemented`), both for a fresh tag and as a retype target. Mapping
Boolean to `Int2` sidesteps that gap entirely.
**Observed live (2026-07-13, real 0.2.0 gateway → real AVEVA historian on `wonder-sql-vd03`, probed via
`grpcurl` — archreview R2-06 T12):**
- `EnsureTags` is a **real create-or-update upsert** — re-`EnsureTags` of an existing tag at the same type
returns `Success=true`.
- An **in-place retype between SUPPORTED types is ACCEPTED**`Float→Double` on an existing tag returns
`Success=true` (hypothesis **(ii) accepted as a plain retype**, not rejected).
- **BUT `Int1` — the 0.2.0 Boolean target — is *not a creatable type* on this historian's histsdk:**
`EnsureTags(Int1)` returns `Success=false, Error="ProtocolEvidenceMissingException"` (the vendored client's
"no capture evidence for this type" guard → gRPC `Unimplemented`), both for a fresh tag and as a retype
target. A type sweep confirmed it is **Int1-specific**: `Int2/UInt2/Int4/UInt4/Float/Double` all provision
`Success=true`; only `Int1` fails.
- **Net for a pre-bump Float Boolean tag asked to become Int1:** the provision **fails** (per-tag
`Success=false``failed=N` tally). It is **neither silently retyped nor data-lost** — the tag stays Float
and remains readable; the Int1 provision simply doesn't land until the gateway's histsdk gains `Int1`
create-evidence — filed as
[historiangw#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11). (Prerequisite for reaching this
outcome at all: the provisioner must stamp a **positive
`StorageRateMs`** — a `0` made `EnsureTags` throw `ArgumentOutOfRangeException` and fail *every* provision;
fixed in `GatewayTagProvisioner`, PR #439.)
3. **The failure mode is bounded and observable either way.** Provisioning is fire-and-forget and never
blocks/fails a deploy; a per-tag rejection surfaces only as the tally's `failed` count. Values written to
a still-Float tag by the recorder remain readable (Float supersets Int1's 0/1 range); nothing is lost
pending a reconcile. A bulk retype sweep is **not** done in code — assess via the probe first, then decide
whether reconciliation is a one-time operator action.
> **Prerequisite for provisioning to land at all:** the provisioner must stamp a **positive `StorageRateMs`**
> — a `0` made `EnsureTags` throw `ArgumentOutOfRangeException` and fail *every* provision (fixed in
> `GatewayTagProvisioner`, PR #439).
**Migration note for pre-existing Boolean tags.** A Boolean tag historian-side as Float (e.g. from the
0.1.0-era default) is asked, on its next deploy, to upsert to Int2. Both are supported analog types, so the
retype is accepted. Provisioning is fire-and-forget and never blocks/fails a deploy; a per-tag rejection (if
any) surfaces only as the tally's `failed` count, and values written to a still-Float tag remain readable
(Float supersets the 0/1 range) pending reconcile. A bulk retype sweep is **not** done in code.
---
@@ -407,11 +403,11 @@ The gateway-backed driver also ships an env-gated live suite (`Category=LiveInte
`HISTGW_GATEWAY_ENDPOINT` / `HISTGW_GATEWAY_APIKEY` / `HISTGW_TEST_TAG` / `HISTGW_WRITE_SANDBOX_TAG` /
`HISTGW_BOOL_SANDBOX_TAG` / `HISTGW_ALARM_SOURCE` env vars (it skips cleanly when they are absent).
`HISTGW_BOOL_SANDBOX_TAG` (e.g. `HistGW.LiveTest.BoolSandbox`, a **dedicated** writable Boolean tag —
never the shared Float write sandbox) drives the 0.2.0 Boolean `EnsureTags` leg
(`EnsureTags_boolean_sandbox_provisions_as_Int1`) and the retype probe
(`EnsureTags_type_change_outcome_is_observable`) that resolve the
[0.2.0 DataType provisioning delta](#020-datatype-provisioning-delta-archreview-06u-7) open question. Running
the full suite against a **0.2.0** gateway (all six tests, none skipped) also discharges 06/U-2's
never the shared Float write sandbox) drives the Boolean historization leg
(`EnsureTags_boolean_sandbox_provisions_as_Int2`) and the retype probe
(`EnsureTags_type_change_outcome_is_observable`) that back the
[Boolean historization maps to Int2](#boolean-historization-maps-to-int2-issue-441) decision. Running
the full suite against the gateway (all six tests, none skipped) also discharges 06/U-2's
"full documented run" for the historian suite.
See [AlarmHistorian.md](AlarmHistorian.md) for the alarm write path and
@@ -50,6 +50,8 @@ public sealed class SubscribeCommand : S7CommandBase
{
await driver.InitializeAsync("{}", ct);
// Route every data-change event to the CliFx console (not System.Console — the
// analyzer flags it + IConsole is the testable abstraction).
driver.OnDataChange += (_, e) =>
{
var line = $"[{DateTime.UtcNow:HH:mm:ss.fff}] " +
@@ -9,11 +9,14 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Mapping;
/// </summary>
/// <remarks>
/// Only the nine numeric types are historizable on the gateway's analog write path. Two of them
/// fall back to a wider historian type because the narrower one's write path is deferred upstream:
/// <see cref="DriverDataType.UInt16"/> maps to <see cref="HistorianDataType.Uint4"/> (the historian's
/// <c>UInt2</c> write path is not proven). String / DateTime / Reference are not historized in v1
/// and throw <see cref="NotSupportedException"/>; callers that want to skip them without catching an
/// exception should consult <see cref="IsHistorizable(DriverDataType)"/> first.
/// fall back to a wider/adjacent historian type because the exact one's write path is not usable
/// upstream: <see cref="DriverDataType.UInt16"/> maps to <see cref="HistorianDataType.Uint4"/> (the
/// historian's <c>UInt2</c> write path is not proven), and <see cref="DriverDataType.Boolean"/> maps
/// to <see cref="HistorianDataType.Int2"/> — the historian's <c>Int1</c> analog-tag creation path is
/// server-degenerate (issue #441 / HistorianGateway #11: <c>EnsureTags(Int1)</c> stores an unusable
/// stub), so Boolean historizes as a small 0/1 integer instead. String / DateTime / Reference are not
/// historized in v1 and throw <see cref="NotSupportedException"/>; callers that want to skip them
/// without catching an exception should consult <see cref="IsHistorizable(DriverDataType)"/> first.
/// </remarks>
internal static class HistorianTypeMapper
{
@@ -25,7 +28,8 @@ internal static class HistorianTypeMapper
/// </exception>
public static HistorianDataType ToHistorianDataType(DriverDataType dataType) => dataType switch
{
DriverDataType.Boolean => HistorianDataType.Int1,
DriverDataType.Boolean => HistorianDataType.Int2, // Int1 analog-tag creation is server-degenerate (issue #441) → store 0/1
DriverDataType.Int16 => HistorianDataType.Int2,
DriverDataType.Int32 => HistorianDataType.Int4,
DriverDataType.Int64 => HistorianDataType.Int8,
@@ -27,7 +27,11 @@ public sealed record AbServerProfile(
public static class KnownProfiles
{
/// <summary>Gets all known server profiles.</summary>
public static IReadOnlyList<AbServerProfile> All { get; } =
/// <remarks>Expression-bodied (evaluated on access) so it observes the
/// <c>static readonly</c> profile fields below after they are initialised;
/// a <c>{ get; } =</c> initialiser here runs in textual order — before those
/// fields — and would capture an all-null list.</remarks>
public static IReadOnlyList<AbServerProfile> All =>
[ControlLogix, CompactLogix, Micro800, GuardLogix];
/// <summary>Gets the ControlLogix profile.</summary>
@@ -415,7 +415,12 @@ class FocasMockServer:
timer_type = self._block_u32(request_block, 8)
key_map = {0: "power_on", 1: "operating", 2: "cutting", 3: "cycle"}
seconds = int(self.store.snapshot()["timers"].get(key_map.get(timer_type, "power_on"), 0))
return self._u32(seconds // 60) + self._u32((seconds % 60) * 1000)
# cnc_rdtimer's minute+msec fields are LITTLE-endian on the wire — unlike the big-endian
# block envelope and every other payload. Hardware-validated against a live FANUC 31i-B
# (2026-06-25); the driver's FocasWireClient.ParseTimer decodes them little-endian. Encoding
# them big-endian here (the old bug) made the driver read minute=60 as 0x3C000000. See
# docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md.
return self._u32_le(seconds // 60) + self._u32_le((seconds % 60) * 1000)
def _wire_alarms(self, request_block: bytes) -> bytes:
requested = max(self._block_u32(request_block, 12), 1)
@@ -481,6 +486,11 @@ class FocasMockServer:
def _u32(self, value: int) -> bytes:
return int(value).to_bytes(4, "big", signed=True)
def _u32_le(self, value: int) -> bytes:
# Little-endian u32 — only cnc_rdtimer's minute/msec fields use this on real hardware
# (see _wire_timer). Every other wire field is big-endian via _u32.
return int(value).to_bytes(4, "little", signed=True)
def _ascii_fixed(self, value: str, length: int) -> bytes:
return value.encode("ascii", errors="replace")[:length].ljust(length, b" ")
@@ -29,6 +29,11 @@ public sealed class FocasSimFixture : IAsyncDisposable
/// <summary>Gets the TCP port of the focas-mock simulator.</summary>
public int Port { get; }
/// <summary>Gets the <c>focas://host:port</c> device URI the driver connects to — derived from
/// <see cref="Host"/>/<see cref="Port"/> so tests honour <c>OTOPCUA_FOCAS_SIM_ENDPOINT</c> (a remote
/// fixture on the Docker host) instead of assuming a localhost-colocated mock.</summary>
public string DeviceUri => $"focas://{Host}:{Port}";
/// <summary>focas-mock profile stem the fixture should load (e.g. <c>fwlib30i64</c>,
/// <c>ThirtyOne_i</c> — both resolve via the mock's alias table). Null when unset.</summary>
public string? ExpectedProfile { get; }
@@ -23,7 +23,9 @@ public sealed class WireBackendCoverageTests
/// <param name="fx">The FOCAS simulation fixture.</param>
public WireBackendCoverageTests(FocasSimFixture fx) => _fx = fx;
private const string DeviceHost = "focas://127.0.0.1:8193";
// Derived from the fixture endpoint (OTOPCUA_FOCAS_SIM_ENDPOINT) so the driver dials the SAME
// mock the fixture probed — a remote fixture on the Docker host, not a hardcoded 127.0.0.1.
private string DeviceHost => _fx.DeviceUri;
/// <summary>Verifies that user tag reads route via the wire backend.</summary>
[Fact]
@@ -26,7 +26,9 @@ public sealed class WireBackendTests
/// <param name="fx">The FOCAS simulator fixture.</param>
public WireBackendTests(FocasSimFixture fx) => _fx = fx;
private const string DeviceHost = "focas://127.0.0.1:8193";
// Derived from the fixture endpoint (OTOPCUA_FOCAS_SIM_ENDPOINT) so the driver dials the SAME
// mock the fixture probed — a remote fixture on the Docker host, not a hardcoded 127.0.0.1.
private string DeviceHost => _fx.DeviceUri;
/// <summary>Verifies that identity axes and dynamic data populate via the wire backend.</summary>
[Fact]
@@ -26,24 +26,23 @@ public sealed class GatewayTagProvisionerTests
Assert.NotNull(fake.LastEnsureDefinitions);
Assert.Equal(2, fake.LastEnsureDefinitions!.Count);
Assert.Equal(HistorianDataType.Float, fake.LastEnsureDefinitions[0].DataType);
Assert.Equal(HistorianDataType.Int1, fake.LastEnsureDefinitions[1].DataType);
Assert.Equal(HistorianDataType.Int2, fake.LastEnsureDefinitions[1].DataType);
Assert.Equal(2, result.Requested);
Assert.Equal(0, result.Skipped);
}
/// <summary>
/// archreview 06/U-7 — pins the 0.2.0 Boolean→Int1 provisioning semantics at TWO levels. The
/// 0.1.0→0.2.0 client bump made <c>HistorianTagDefinition.DataType</c> proto3-optional (explicit
/// presence): under 0.1.0 the provisioner's <c>Int1</c> (wire 0) was indistinguishable from unset,
/// so the gateway defaulted Boolean tags to Float; under 0.2.0 the same call carries explicit
/// presence and provisions Int1 (the intended behavior). Asserting BOTH the mapped type AND
/// <c>HasDataType</c> compile-pins the package floor (the <c>HasDataType</c>/<c>ClearDataType</c>
/// members exist only on the 0.2.0 proto3-optional contract — a downgrade breaks the build) and
/// assert-pins that the provisioner always sends explicit presence (a future contract change that
/// stopped sending it would break the assert). This is a pin, not a fix — it passes on current code.
/// Pins the Boolean→Int2 provisioning semantics at TWO levels. Boolean maps to <c>Int2</c> because
/// the historian's <c>Int1</c> analog-tag creation path is server-degenerate — <c>EnsureTags(Int1)</c>
/// stores an unusable stub (issue #441 / HistorianGateway #11) — so Boolean historizes as a small 0/1
/// integer. Asserting BOTH the mapped type AND <c>HasDataType</c> compile-pins the package floor (the
/// <c>HasDataType</c>/<c>ClearDataType</c> members exist only on the 0.2.0+ proto3-optional contract —
/// a downgrade breaks the build) and assert-pins that the provisioner always sends explicit presence
/// (a future contract change that stopped sending it would default the tag to the SDK's Float and
/// break this assert). This is a pin, not a fix — it passes on current code.
/// </summary>
[Fact]
public async Task Boolean_definition_carries_explicit_Int1_presence()
public async Task Boolean_definition_carries_explicit_Int2_presence()
{
var fake = new FakeHistorianGatewayClient { EnsureTagsResult = new TagOperationResults() };
var p = Provisioner(fake);
@@ -54,9 +53,9 @@ public sealed class GatewayTagProvisionerTests
var defs = fake.LastEnsureDefinitions!;
Assert.Single(defs);
Assert.Equal(HistorianDataType.Int1, defs[0].DataType);
// 0.2.0 proto3-optional presence witness — explicit presence is what flips the gateway from its
// SDK-default Float to the requested Int1. Without this, a Boolean tag would provision as Float.
Assert.Equal(HistorianDataType.Int2, defs[0].DataType);
// proto3-optional presence witness — explicit presence is what flips the gateway from its
// SDK-default Float to the requested Int2. Without this, a Boolean tag would provision as Float.
Assert.True(defs[0].HasDataType);
}
@@ -255,25 +255,26 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
}
/// <summary>
/// archreview 06/U-7 — Boolean provisioning leg on the 0.2.0 contract. Through the <b>real</b>
/// issue #441 — Boolean historization leg. Through the <b>real</b>
/// <see cref="GatewayTagProvisioner"/>, <c>EnsureTags</c> a <see cref="DriverDataType.Boolean"/>
/// request for the dedicated Boolean sandbox tag (mapped to <c>Int1</c> with explicit proto3
/// presence), assert it is ensured (not failed), then <c>WriteLiveValues</c> a <c>1.0</c> and read
/// it back over a wide window. This retires "the live EnsureTags leg has never run on 0.2.0" for
/// the Boolean path. The ±12h read window + timezone caveat mirror <see cref="Write_then_read_on_sandbox_tag"/>.
/// request for the dedicated Boolean sandbox tag (mapped to <c>Int2</c> — the historian's
/// <c>Int1</c> analog-tag creation path is server-degenerate, see issue #441 / HistorianGateway
/// #11, so Boolean historizes as a small 0/1 integer), assert it is ensured (not failed), then
/// <c>WriteLiveValues</c> a <c>1.0</c> and read it back over a wide window. The ±12h read window
/// + timezone caveat mirror <see cref="Write_then_read_on_sandbox_tag"/>.
/// </summary>
[Fact]
[Trait("Category", "LiveIntegration")]
public async Task EnsureTags_boolean_sandbox_provisions_as_Int1()
public async Task EnsureTags_boolean_sandbox_provisions_as_Int2()
{
if (_fx.NotConfigured) Assert.Skip(_fx.SkipReason!);
if (_fx.BoolSandboxTag is null)
Assert.Skip("Skipped: set HISTGW_BOOL_SANDBOX_TAG to a writable Boolean sandbox tag (e.g. HistGW.LiveTest.BoolSandbox) to run the 0.2.0 Boolean EnsureTags leg.");
Assert.Skip("Skipped: set HISTGW_BOOL_SANDBOX_TAG to a writable Boolean sandbox tag (e.g. HistGW.LiveTest.BoolSandbox) to run the Boolean historization leg.");
var ct = TestContext.Current.CancellationToken;
var tag = _fx.BoolSandboxTag;
// EnsureTags (Boolean → Int1) through the REAL provisioner seam (the same code path deploys use).
// EnsureTags (Boolean → Int2) through the REAL provisioner seam (the same code path deploys use).
await using var writeClient = _fx.CreateClient();
var provisioner = new GatewayTagProvisioner(writeClient, NullLogger<GatewayTagProvisioner>.Instance);
var provision = await provisioner.EnsureTagsAsync(
@@ -281,7 +282,7 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
ct);
provision.Ensured.ShouldBe(1,
"the Boolean sandbox tag must be ensured (needs the gateway on 0.2.0 contracts + an API key carrying historian:tags:write).");
"the Boolean sandbox tag must be ensured as Int2 (needs the gateway on 0.2.0+ contracts + an API key carrying historian:tags:write).");
provision.Failed.ShouldBe(0);
// WriteLiveValues a Boolean-as-1.0 and read it back (SQL live path can lag a flush cadence → poll).
@@ -308,14 +309,16 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
hit.ShouldNotBeNull($"the written Boolean sample ({written}) should read back from '{tag}'.");
TestContext.Current.SendDiagnosticMessage(
$"Boolean round-trip: EnsureTags(Int1) + WriteLiveValues '{tag}'={written} → read back at {hit!.SourceTimestampUtc:O}.");
$"Boolean round-trip: EnsureTags(Int2) + WriteLiveValues '{tag}'={written} → read back at {hit!.SourceTimestampUtc:O}.");
}
/// <summary>
/// archreview 06/U-7 retype probe — answers the assessment note's open question: what does the
/// deployed AVEVA Historian do when <c>EnsureTags</c> (an <b>upsert</b>) is asked to change an
/// existing tag's type? Against the <b>dedicated Boolean sandbox tag only</b> (never the shared
/// Float write sandbox), EnsureTags first as <c>Float</c> then as <c>Int1</c> and assert the second
/// Float write sandbox), EnsureTags first as <c>Float</c> then as <c>Int2</c> (both supported
/// analog types — <c>Int2</c> is the type Boolean now historizes as; the previously-probed
/// <c>Int1</c> is a documented server-degenerate exclusion, issue #441) and assert the second
/// call's per-tag outcome is <em>deterministic and observable</em> — either <c>Success=true</c>
/// (retype / tag-versioning accepted) or <c>Success=false</c> with a non-empty <c>Error</c> (mapped
/// to the provisioner's <c>failed</c> tally). The assertion accepts both because the contract
@@ -336,7 +339,7 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
await using var client = _fx.CreateClient();
// First establish the tag as Float, then ask EnsureTags to retype it to Int1 (Boolean).
// First establish the tag as Float, then ask EnsureTags to retype it to Int2.
await client.EnsureTagsAsync(
new[]
{
@@ -357,9 +360,9 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
new HistorianTagDefinition
{
TagName = tag,
DataType = HistorianDataType.Int1,
DataType = HistorianDataType.Int2,
EngineeringUnit = string.Empty,
Description = "OtOpcUa live validation retype probe (stage 2: Int1)",
Description = "OtOpcUa live validation retype probe (stage 2: Int2)",
StorageRateMs = 1000, // 0 makes the gateway's EnsureTags throw (storageRateMs must be > 0).
},
},
@@ -372,7 +375,7 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
"or Success=false with a non-empty Error (mapped to the provisioner's failed tally).");
TestContext.Current.SendDiagnosticMessage(
$"retype probe '{tag}' Float→Int1: Success={outcome.Success}, Error='{outcome.Error}'. " +
$"retype probe '{tag}' Float→Int2: Success={outcome.Success}, Error='{outcome.Error}'. " +
"(records the deployed historian's in-place analog-retype policy — feed this back into docs/Historian.md).");
}
}
@@ -8,7 +8,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.Mapping;
public sealed class HistorianTypeMapperTests
{
[Theory]
[InlineData(DriverDataType.Boolean, HistorianDataType.Int1)]
[InlineData(DriverDataType.Boolean, HistorianDataType.Int2)] // Int1 analog-tag creation is server-degenerate (issue #441) → store 0/1
[InlineData(DriverDataType.Int16, HistorianDataType.Int2)]
[InlineData(DriverDataType.Int32, HistorianDataType.Int4)]
[InlineData(DriverDataType.Int64, HistorianDataType.Int8)]
@@ -48,6 +48,31 @@ service + starting another. The integration tests discriminate by a
separate `MODBUS_SIM_PROFILE` env var so they skip correctly when the
wrong profile is live.
> **⚠️ Profile-cycling gotcha (bit us in the 2026-07 sweep — see
> `archreview/plans/INTEGRATION-SWEEP-STATUS.md`).** Each profile is a
> *separate compose service* selected by `--profile`, so a plain
> `docker compose down` (no `--profile`) does **not** stop the
> currently-running profile container — it keeps holding `:5020`. Start
> another profile and the new container silently fails to bind the port
> (compose still reports "Started"); every test then hits the **stale**
> profile's data and fails with wrong values / Illegal-Data-Address. The
> symptom is deceptive: reads succeed but return the previous profile's
> seed (e.g. Mitsubishi's `D0` reads DL205's `0xCAFE`). Always tear the
> old container down **by name / with its profile** before bringing up the
> next, and verify the new container actually published `:5020`:
>
> ```bash
> # reliable cycle (force-remove ALL modbus containers, then bring one up)
> docker rm -f $(docker ps -aq --filter name=otopcua-modbus --filter name=otopcua-pymodbus)
> docker compose --profile <profile> up -d --force-recreate
> docker ps --filter name=otopcua --format '{{.Names}} {{.Ports}}' # MUST show 0.0.0.0:5020->5020
> ```
>
> The `exception_injection` service uses a different container name
> (`otopcua-modbus-exception-injector`, not `otopcua-pymodbus-*`), so a
> name-filtered `docker rm` that only matches `otopcua-pymodbus` will miss
> it — the `--force-recreate` above covers that case.
### Profile coverage matrix
The two general-purpose profiles cover disjoint test sets. A full pass
@@ -23,6 +23,7 @@ public sealed class DeployHappyPathTests
public async Task StartDeployment_seals_after_both_nodes_apply()
{
await using var harness = await TwoNodeClusterHarness.StartAsync();
await harness.SeedDefaultClusterAsync();
await using var scope = harness.NodeA.Services.CreateAsyncScope();
var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
@@ -59,6 +60,7 @@ public sealed class DeployHappyPathTests
public async Task Replaying_dispatch_to_same_revision_is_idempotent_no_op()
{
await using var harness = await TwoNodeClusterHarness.StartAsync();
await harness.SeedDefaultClusterAsync();
await using var scope = harness.NodeA.Services.CreateAsyncScope();
var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
@@ -218,8 +218,10 @@ public sealed class DriverReconnectE2eTests
Name = "Reconnect E2E Cluster",
Enterprise = "zb",
Site = "central",
NodeCount = 1,
RedundancyMode = RedundancyMode.None,
// Both harness nodes are Enabled driver members, so NodeCount must be 2 (Warm) to
// satisfy the ClusterEnabledNodeCountMismatch validator + the SQL CHECK constraint.
NodeCount = 2,
RedundancyMode = RedundancyMode.Warm,
CreatedBy = "test",
});
@@ -240,6 +242,18 @@ public sealed class DriverReconnectE2eTests
CreatedBy = "test",
});
// The harness runs BOTH nodes as driver members, so the deployment records a
// NodeDeploymentState for node B too — seed its ClusterNode row so the real-SQL FK
// (FK_NodeDeploymentState_ClusterNode_NodeId) resolves. (In-memory ignores FKs.)
db.ClusterNodes.Add(new ClusterNode
{
NodeId = harness.NodeBNodeId,
ClusterId = ClusterId,
Host = TwoNodeClusterHarness.LoopbackHost,
ApplicationUri = "urn:zb:reconnect-e2e:node-b",
CreatedBy = "test",
});
db.DriverInstances.Add(new DriverInstance
{
DriverInstanceId = DriverId,
@@ -7,6 +7,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using ZB.MOM.WW.OtOpcUa.Configuration.Validation;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
@@ -28,19 +29,28 @@ public sealed class EquipmentNamespaceMaterializationTests
{
private static CancellationToken Ct => TestContext.Current.CancellationToken;
// Equipment.EquipmentId must equal DraftValidator.DeriveEquipmentId(EquipmentUuid) — the
// 'EquipmentIdNotDerived' rule (EquipmentId = 'EQ-' + first 12 hex of the UUID). A fixed UUID
// keeps the derived id stable for the assertions below.
private static readonly Guid EquipmentUuid = Guid.Parse("11111111-1111-1111-1111-111111111111");
private static readonly string EquipmentId = DraftValidator.DeriveEquipmentId(EquipmentUuid); // EQ-111111111111
/// <summary>Verifies a deployed Equipment namespace carries its signal into the composed artifact.</summary>
[Fact]
public async Task Deploying_an_equipment_namespace_carries_the_signal_into_the_artifact()
{
await using var harness = await TwoNodeClusterHarness.StartAsync();
// Seed the parent ServerCluster + ClusterNode rows the SQL FKs require (the c1 config below
// FKs to ServerCluster; on the in-memory provider this is unnecessary — FKs aren't enforced).
await harness.SeedDefaultClusterAsync("c1");
await SeedEquipmentNamespaceAsync(harness);
await using var scope = harness.NodeA.Services.CreateAsyncScope();
var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
var result = await client.StartDeploymentAsync(createdBy: "alice@test", Ct);
result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted);
result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted, $"Deploy not accepted: {result.Message}");
var deploymentId = result.DeploymentId!.Value.Value;
// The artifact is composed + persisted at dispatch time; assert on it without waiting for
@@ -65,13 +75,13 @@ public sealed class EquipmentNamespaceMaterializationTests
// ParseComposition extracted it as an EquipmentTag with FullName pulled from TagConfig.
var tag = composition.EquipmentTags.ShouldHaveSingleItem();
tag.TagId.ShouldBe("tag-speed");
tag.EquipmentId.ShouldBe("eq-1");
tag.EquipmentId.ShouldBe(EquipmentId);
tag.Name.ShouldBe("Speed");
tag.DataType.ShouldBe("Float");
tag.FullName.ShouldBe("40001");
// The equipment folder browses by its friendly UNS Name, not the MachineCode.
composition.EquipmentNodes.ShouldContain(e => e.EquipmentId == "eq-1" && e.DisplayName == "station-1");
composition.EquipmentNodes.ShouldContain(e => e.EquipmentId == EquipmentId && e.DisplayName == "station-1");
composition.UnsAreas.ShouldContain(a => a.UnsAreaId == "nw-area-filling" && a.DisplayName == "filling");
composition.UnsLines.ShouldContain(l => l.UnsLineId == "nw-line-1" && l.DisplayName == "line-1");
}
@@ -95,12 +105,13 @@ public sealed class EquipmentNamespaceMaterializationTests
db.UnsLines.Add(new UnsLine { UnsLineId = "nw-line-1", UnsAreaId = "nw-area-filling", Name = "line-1" });
db.Equipment.Add(new Equipment
{
EquipmentId = "eq-1", DriverInstanceId = "drv-modbus", UnsLineId = "nw-line-1",
EquipmentId = EquipmentId, EquipmentUuid = EquipmentUuid,
DriverInstanceId = "drv-modbus", UnsLineId = "nw-line-1",
Name = "station-1", MachineCode = "STATION_001",
});
db.Tags.Add(new Tag
{
TagId = "tag-speed", DriverInstanceId = "drv-modbus", EquipmentId = "eq-1",
TagId = "tag-speed", DriverInstanceId = "drv-modbus", EquipmentId = EquipmentId,
Name = "Speed", DataType = "Float", AccessLevel = TagAccessLevel.Read,
TagConfig = "{\"FullName\":\"40001\"}",
});
@@ -58,6 +58,7 @@ public sealed class FailoverDuringDeployTests
// dispatch time — when only node A is Up, only one ApplyAck is expected and the
// deployment seals without B ever participating.
await using var harness = await TwoNodeClusterHarness.StartAsync();
await harness.SeedDefaultClusterAsync();
await harness.StopNodeBAsync();
await harness.WaitForClusterSizeAsync(1, TimeSpan.FromSeconds(20));
@@ -45,6 +45,7 @@ public sealed class FleetDiagnosticsRoundTripTests
public async Task GetDiagnostics_after_deploy_reports_current_revision()
{
await using var harness = await TwoNodeClusterHarness.StartAsync();
await harness.SeedDefaultClusterAsync();
await using var scope = harness.NodeA.Services.CreateAsyncScope();
var adminOps = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
@@ -14,6 +14,8 @@ using ZB.MOM.WW.OtOpcUa.AdminUI.Clients;
using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using ZB.MOM.WW.OtOpcUa.ControlPlane;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Host.Health;
@@ -36,9 +38,11 @@ namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
/// <item><c>OTOPCUA_HARNESS_USE_SQL=1</c> → swap the in-memory DB for SQL Server on
/// <c>localhost:14331</c> (see <c>docker-compose.yml</c>). Each harness gets a unique
/// database name (<c>OtOpcUa_Harness_{guid}</c>) created via <c>EnsureCreated</c>
/// and dropped via <c>EnsureDeleted</c> on dispose.</item>
/// and dropped via <c>EnsureDeleted</c> on dispose. Override the SQL / LDAP host with
/// <c>OTOPCUA_HARNESS_SQL_HOST</c> / <c>OTOPCUA_HARNESS_LDAP_HOST</c> to point at native
/// fixtures on the shared Docker host (avoids the arm64-Mac emulated-mssql slowdown).</item>
/// <item><c>OTOPCUA_HARNESS_USE_LDAP=1</c> → drop the stub and point <c>LdapAuthService</c>
/// at OpenLDAP on <c>localhost:3894</c>.</item>
/// at the GLAuth container on <c>localhost:3894</c> (see Docker/glauth/config.toml).</item>
/// </list>
///
/// <para>Why not <c>WebApplicationFactory&lt;Program&gt;</c>? Program.cs reads <c>OTOPCUA_ROLES</c>
@@ -114,6 +118,53 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
public Task<OtOpcUaConfigDbContext> CreateConfigDbContextAsync()
=> NodeA.Services.GetRequiredService<IDbContextFactory<OtOpcUaConfigDbContext>>().CreateDbContextAsync();
/// <summary>
/// Seeds a default <see cref="ServerCluster"/> plus a <see cref="ClusterNode"/> row for BOTH
/// harness nodes (<see cref="NodeANodeId"/> / <see cref="NodeBNodeId"/>) so the real-SQL FK
/// constraint <c>FK_NodeDeploymentState_ClusterNode_NodeId</c> is satisfied when a deployment
/// records per-node state. The EF in-memory provider ignores FK constraints, so deploy E2E
/// tests pass without this; against SQL Server each node's <c>NodeDeploymentState</c> INSERT
/// fails without its parent <see cref="ClusterNode"/> row and the deployment never seals.
/// No-op unless <c>OTOPCUA_HARNESS_USE_SQL=1</c> (in-memory needs no seeding). Call once, before
/// <c>StartDeploymentAsync</c>, in tests that don't already seed their own cluster + both nodes.
/// </summary>
/// <param name="clusterId">Cluster id for the seeded rows. Defaults to <c>MAIN</c>.</param>
public async Task SeedDefaultClusterAsync(string clusterId = "MAIN")
{
if (!Mode.UseSqlServer) return;
await using var db = await CreateConfigDbContextAsync();
db.ServerClusters.Add(new ServerCluster
{
ClusterId = clusterId,
Name = "Harness Default Cluster",
Enterprise = "zb",
Site = "central",
// 2-node cluster → Warm/Hot per CK_ServerCluster_RedundancyMode_NodeCount
// (NodeCount=1↔None, NodeCount=2↔Warm|Hot). The harness runs two nodes.
NodeCount = 2,
RedundancyMode = RedundancyMode.Warm,
CreatedBy = "harness",
});
db.ClusterNodes.AddRange(
new ClusterNode
{
NodeId = NodeANodeId,
ClusterId = clusterId,
Host = LoopbackHost,
ApplicationUri = "urn:zb:harness:node-a",
CreatedBy = "harness",
},
new ClusterNode
{
NodeId = NodeBNodeId,
ClusterId = clusterId,
Host = LoopbackHost,
ApplicationUri = "urn:zb:harness:node-b",
CreatedBy = "harness",
});
await db.SaveChangesAsync();
}
/// <summary>Boots both nodes and waits up to <paramref name="formationTimeout"/> for cluster convergence.</summary>
/// <param name="formationTimeout">Maximum time to wait for cluster formation; defaults to 20 seconds if not provided.</param>
/// <param name="driverFactory">Optional opt-in <see cref="IDriverFactory"/> both nodes register so deployed
@@ -131,7 +182,12 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
if (harness.Mode.UseSqlServer)
{
harness._sqlDbName = $"OtOpcUa_Harness_{Guid.NewGuid():N}";
harness._sqlConnString = $"Server=localhost,14331;Database={harness._sqlDbName};User Id=sa;Password=OtOpcUa!Harness123;TrustServerCertificate=True;";
// SQL host is overridable so the heavy 2-node E2E tests can target NATIVE-amd64 SQL on the
// shared Docker host instead of the arm64 Mac's emulated mssql:2022 (no arm64 image → ~5-10×
// slower → the deploy/failover E2E tests time out). Set OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331
// with the harness sql container brought up there. Defaults to localhost,14331.
var sqlHost = Environment.GetEnvironmentVariable("OTOPCUA_HARNESS_SQL_HOST") ?? "localhost,14331";
harness._sqlConnString = $"Server={sqlHost};Database={harness._sqlDbName};User Id=sa;Password=OtOpcUa!Harness123;TrustServerCertificate=True;";
await EnsureSqlSchemaCreatedAsync(harness._sqlConnString);
}
@@ -295,13 +351,16 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
// Bound section is Security:Ldap (see LdapOptions.SectionName); Transport replaces the
// old UseTls bool and AllowInsecure replaces AllowInsecureLdap (Task 1.4).
configOverrides["Security:Ldap:Enabled"] = "true";
configOverrides["Security:Ldap:Server"] = "localhost";
// Overridable alongside the SQL host so the GLAuth container can live on the Docker host too.
configOverrides["Security:Ldap:Server"] = Environment.GetEnvironmentVariable("OTOPCUA_HARNESS_LDAP_HOST") ?? "localhost";
configOverrides["Security:Ldap:Port"] = "3894";
configOverrides["Security:Ldap:Transport"] = "None";
configOverrides["Security:Ldap:AllowInsecure"] = "true";
configOverrides["Security:Ldap:SearchBase"] = "dc=zb,dc=local";
configOverrides["Security:Ldap:ServiceAccountDn"] = "cn=admin,dc=zb,dc=local";
configOverrides["Security:Ldap:ServiceAccountPassword"] = "ldapadmin";
// GLAuth's search-capable bind account (see Docker/glauth/config.toml) — matches the
// shared GLAuth the data-plane binds against. Replaced the retired OpenLDAP cn=admin.
configOverrides["Security:Ldap:ServiceAccountDn"] = "cn=serviceaccount,dc=zb,dc=local";
configOverrides["Security:Ldap:ServiceAccountPassword"] = "serviceaccount123";
}
builder.Configuration.AddInMemoryCollection(configOverrides);
@@ -6,8 +6,8 @@
#
# 1. EF behaviors that diverge between provider implementations — index uniqueness,
# RowVersion concurrency, JSON column round-trips, EF migration application.
# 2. Real LDAP binds against an OpenLDAP server with the dev users from
# C:\publish\glauth\auth.md.
# 2. Real LDAP binds against a GLAuth server (./glauth/config.toml) with dev users
# alice (full access) / bob (read-only) and the cn=serviceaccount bind account.
#
# Activate by setting these env vars before running the suite:
#
@@ -43,15 +43,13 @@ services:
retries: 20
ldap:
# OpenLDAP image — same one docker-dev/ uses, just on a different port. Dev users
# alice/bob match the GLAuth fixtures so AuthEndpoints contract tests share creds.
image: bitnami/openldap:2.6
environment:
LDAP_ROOT: "dc=zb,dc=local"
LDAP_ADMIN_USERNAME: "admin"
LDAP_ADMIN_PASSWORD: "ldapadmin"
LDAP_USERS: "alice,bob"
LDAP_PASSWORDS: "alice123,bob123"
LDAP_USER_DC: "ou=FleetAdmin"
# GLAuth — the same LDAP server the rest of the project uses (docker-dev + the live
# OPC UA data-plane auth, both against the shared GLAuth on :3893). Replaced the retired
# bitnami/openldap:2.6 (Bitnami deprecated their free catalog in 2025). Seeded by
# ./glauth/config.toml with a search-capable serviceaccount + dev users alice/bob.
# Listens on :3893 in-container; mapped to host :3894 to run beside docker-dev's :3893.
image: glauth/glauth:latest
volumes:
- ./glauth/config.toml:/app/config/config.cfg:ro
ports:
- "3894:1389"
- "3894:3893"
@@ -0,0 +1,78 @@
# GLAuth config for the Host.IntegrationTests real-LDAP mode (OTOPCUA_HARNESS_USE_LDAP=1).
#
# Replaces the retired bitnami/openldap:2.6 image (Bitnami deprecated their free catalog
# in 2025) with GLAuth — the same LDAP server the rest of the project uses (docker-dev +
# the live OPC UA data-plane auth, both against the shared GLAuth on 10.100.0.35:3893).
# Unifying on GLAuth removes the lone OpenLDAP outlier and the gone-image breakage.
#
# Mirrors scadaproj/infra/glauth/config.toml (source of truth) but trimmed to the harness's
# needs: baseDN dc=zb,dc=local, a search-capable service account, and two dev users
# (alice = full-access, bob = read-only). Bind DN form is GLAuth's cn=<name>,dc=zb,dc=local.
#
# Listens on :3893 inside the container; the compose maps host :3894 -> :3893 so this runs
# side-by-side with docker-dev's shared GLAuth (:3893) without a port clash.
[ldap]
enabled = true
listen = "0.0.0.0:3893"
[ldaps]
enabled = false
[backend]
datastore = "config"
baseDN = "dc=zb,dc=local"
[behaviors]
# Tests may bind bad passwords on purpose (outage / wrong-cred paths) — don't lock out.
LimitFailedBinds = false
# ── Groups (gidnumbers match scadaproj/infra/glauth so GroupToRole maps identically) ──
[[groups]]
name = "ReadOnly"
gidnumber = 5601
[[groups]]
name = "WriteOperate"
gidnumber = 5602
[[groups]]
name = "WriteTune"
gidnumber = 5603
[[groups]]
name = "WriteConfigure"
gidnumber = 5604
[[groups]]
name = "AlarmAck"
gidnumber = 5605
# ── Users ─────────────────────────────────────────────────────────────
# Service account the harness binds first to search for the user entry.
# DN: cn=serviceaccount,dc=zb,dc=local password: serviceaccount123
[[users]]
name = "serviceaccount"
uidnumber = 5999
primarygroup = 5601
passsha256 = "af29d0e5c9801ae98a999ed3915e1cf428a64b4b62b3cf221b6336cce0398419"
[[users.capabilities]]
action = "search"
object = "*"
# alice — full access (member of every write/ack role group). password: alice123
[[users]]
name = "alice"
givenname = "Alice"
sn = "Admin"
mail = "alice@zb.local"
uidnumber = 5010
primarygroup = 5604
othergroups = [5601, 5602, 5603, 5605]
passsha256 = "4e40e8ffe0ee32fa53e139147ed559229a5930f89c2204706fc174beb36210b3"
# bob — read-only. password: bob123
[[users]]
name = "bob"
givenname = "Bob"
sn = "Reader"
mail = "bob@zb.local"
uidnumber = 5011
primarygroup = 5601
passsha256 = "8d059c3640b97180dd2ee453e20d34ab0cb0f2eccbe87d01915a8e578a202b11"
@@ -72,40 +72,44 @@ public sealed class DualEndpointTests
// SDK 1.5.374 sync-style session-open path — mirrors src/Client/.../DefaultSessionFactory.cs
// and DefaultApplicationConfigurationFactory.cs. The 1.5.378 telemetry/async overloads are
// not available at this pinned version.
var appConfig = new ApplicationConfiguration
var clientPki = TestClientSecurity.AllocatePkiRoot();
try
{
ApplicationName = "OtOpcUa.DualEndpointClient",
ApplicationUri = $"urn:OtOpcUa.DualEndpointClient.{Guid.NewGuid():N}",
ApplicationType = ApplicationType.Client,
SecurityConfiguration = new SecurityConfiguration
var appConfig = new ApplicationConfiguration
{
ApplicationCertificate = new CertificateIdentifier(),
AutoAcceptUntrustedCertificates = true,
},
ClientConfiguration = new ClientConfiguration { DefaultSessionTimeout = 60_000 },
};
await appConfig.ValidateAsync(ApplicationType.Client, default);
appConfig.CertificateValidator.CertificateValidation += (_, e) => e.Accept = true;
ApplicationName = "OtOpcUa.DualEndpointClient",
ApplicationUri = $"urn:OtOpcUa.DualEndpointClient.{Guid.NewGuid():N}",
ApplicationType = ApplicationType.Client,
SecurityConfiguration = TestClientSecurity.Build(clientPki),
ClientConfiguration = new ClientConfiguration { DefaultSessionTimeout = 60_000 },
};
await appConfig.ValidateAsync(ApplicationType.Client, default);
appConfig.CertificateValidator.CertificateValidation += (_, e) => e.Accept = true;
// 1.5.378: the discovery/session/read client surface moved to async.
var endpoint = await CoreClientUtils.SelectEndpointAsync(
appConfig, endpointUrl, false, DefaultTelemetry.Create(_ => { }), default);
var endpointConfiguration = EndpointConfiguration.Create(appConfig);
var configuredEndpoint = new ConfiguredEndpoint(null, endpoint, endpointConfiguration);
// 1.5.378: the discovery/session/read client surface moved to async.
var endpoint = await CoreClientUtils.SelectEndpointAsync(
appConfig, endpointUrl, false, DefaultTelemetry.Create(_ => { }), default);
var endpointConfiguration = EndpointConfiguration.Create(appConfig);
var configuredEndpoint = new ConfiguredEndpoint(null, endpoint, endpointConfiguration);
using var session = await new DefaultSessionFactory(DefaultTelemetry.Create(_ => { })).CreateAsync(
appConfig,
configuredEndpoint,
updateBeforeConnect: false,
checkDomain: false,
sessionName: "DualEndpointTests",
sessionTimeout: 60_000,
identity: new UserIdentity(new AnonymousIdentityToken()),
preferredLocales: null,
ct: default);
using var session = await new DefaultSessionFactory(DefaultTelemetry.Create(_ => { })).CreateAsync(
appConfig,
configuredEndpoint,
updateBeforeConnect: false,
checkDomain: false,
sessionName: "DualEndpointTests",
sessionTimeout: 60_000,
identity: new UserIdentity(new AnonymousIdentityToken()),
preferredLocales: null,
ct: default);
var value = await session.ReadValueAsync(VariableIds.Server_ServerArray, default);
return (string[])value.Value;
var value = await session.ReadValueAsync(VariableIds.Server_ServerArray, default);
return (string[])value.Value;
}
finally
{
if (Directory.Exists(clientPki)) Directory.Delete(clientPki, recursive: true);
}
}
private static int AllocateFreePort()
@@ -215,9 +215,11 @@ public sealed class SubscriptionSurvivalTests
sink.WriteValue(nodeIdAString, 11, Commons.OpcUa.OpcUaQuality.Good, DateTime.UtcNow);
await WaitUntilAsync(() => { lock (gate) return receivedA.Any(v => Equals(v.Value, 11)); }, TimeSpan.FromSeconds(5));
// B is gone: a re-read returns BadNodeIdUnknown.
var bRead = await session.ReadValueAsync(nodeIdB, ct);
bRead.StatusCode.ShouldBe((StatusCode)StatusCodes.BadNodeIdUnknown);
// B is gone: a re-read surfaces BadNodeIdUnknown. The 1.5.378 node-cache read path
// throws ServiceResultException for an unknown node rather than returning a bad DataValue.
var bReadEx = await Should.ThrowAsync<ServiceResultException>(
async () => await session.ReadValueAsync(nodeIdB, ct));
bReadEx.StatusCode.ShouldBe(StatusCodes.BadNodeIdUnknown);
}
finally
{
@@ -296,9 +298,11 @@ public sealed class SubscriptionSurvivalTests
sink.WriteValue(nodeIdAString, 11, Commons.OpcUa.OpcUaQuality.Good, DateTime.UtcNow);
await WaitUntilAsync(() => { lock (gate) return receivedA.Any(v => Equals(v.Value, 11)); }, TimeSpan.FromSeconds(5));
// B is gone.
// B is gone. (1.5.378 node-cache read throws for an unknown node — see above.)
var nodeIdB = new NodeId(Commons.OpcUa.EquipmentNodeIds.Variable("eq-1", "", "B"), ns);
(await session.ReadValueAsync(nodeIdB, ct)).StatusCode.ShouldBe((StatusCode)StatusCodes.BadNodeIdUnknown);
var bReadEx = await Should.ThrowAsync<ServiceResultException>(
async () => await session.ReadValueAsync(nodeIdB, ct));
bReadEx.StatusCode.ShouldBe(StatusCodes.BadNodeIdUnknown);
// C is browsable + readable.
var nodeIdC = new NodeId(Commons.OpcUa.EquipmentNodeIds.Variable("eq-1", "", "C"), ns);
@@ -331,11 +335,10 @@ public sealed class SubscriptionSurvivalTests
ApplicationName = "OtOpcUa.SubscriptionSurvivalClient",
ApplicationUri = $"urn:OtOpcUa.SubscriptionSurvivalClient.{Guid.NewGuid():N}",
ApplicationType = ApplicationType.Client,
SecurityConfiguration = new SecurityConfiguration
{
ApplicationCertificate = new CertificateIdentifier(),
AutoAcceptUntrustedCertificates = true,
},
// Directory cert stores rooted at a throwaway temp PKI dir — ValidateAsync rejects a
// bare SecurityConfiguration ("TrustedIssuerCertificates StorePath must be specified").
// The returned session outlives this method, so the dir is left for the OS temp sweep.
SecurityConfiguration = TestClientSecurity.Build(TestClientSecurity.AllocatePkiRoot()),
ClientConfiguration = new ClientConfiguration { DefaultSessionTimeout = 60_000 },
};
await appConfig.ValidateAsync(ApplicationType.Client, ct);
@@ -0,0 +1,47 @@
using Opc.Ua;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests;
/// <summary>
/// Builds the client-side <see cref="SecurityConfiguration"/> the integration tests use to
/// open a real OPC UA session. The SDK's <c>ApplicationConfiguration.ValidateAsync</c> requires
/// the trusted-issuer / trusted-peer / rejected stores to carry an explicit <c>StorePath</c>
/// (it throws <c>TrustedIssuerCertificates StorePath must be specified</c> otherwise), so — unlike
/// a bare <c>new SecurityConfiguration()</c> — each store is rooted at a throwaway PKI directory,
/// mirroring <c>src/Client/.../DefaultApplicationConfigurationFactory.cs</c>.
/// </summary>
internal static class TestClientSecurity
{
/// <summary>Allocates a fresh throwaway client PKI root under the temp directory.</summary>
/// <returns>An absolute path that does not yet exist; the SDK creates the stores under it.</returns>
public static string AllocatePkiRoot() =>
Path.Combine(Path.GetTempPath(), $"otopcua-client-pki-{Guid.NewGuid():N}");
/// <summary>Builds an auto-accepting client security config with Directory stores under <paramref name="pkiRoot"/>.</summary>
/// <param name="pkiRoot">Root directory for the own / issuer / trusted / rejected stores.</param>
/// <returns>A validated-ready <see cref="SecurityConfiguration"/>.</returns>
public static SecurityConfiguration Build(string pkiRoot) => new()
{
ApplicationCertificate = new CertificateIdentifier
{
StoreType = CertificateStoreType.Directory,
StorePath = Path.Combine(pkiRoot, "own"),
},
TrustedIssuerCertificates = new CertificateTrustList
{
StoreType = CertificateStoreType.Directory,
StorePath = Path.Combine(pkiRoot, "issuer"),
},
TrustedPeerCertificates = new CertificateTrustList
{
StoreType = CertificateStoreType.Directory,
StorePath = Path.Combine(pkiRoot, "trusted"),
},
RejectedCertificateStore = new CertificateTrustList
{
StoreType = CertificateStoreType.Directory,
StorePath = Path.Combine(pkiRoot, "rejected"),
},
AutoAcceptUntrustedCertificates = true,
};
}