Compare commits
45 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 57003a23a5 | |||
| b0eb653bad | |||
| b980eeb148 | |||
| d7470c7960 | |||
| b364a499e5 | |||
| 0c005c9a72 | |||
| 809e7886d8 | |||
| 5908339d67 | |||
| b461b2bcf8 | |||
| e0ff908d26 | |||
| 09066eb88f | |||
| 1d3b6d1264 | |||
| c9cf5015e5 | |||
| 7bca0eeca0 | |||
| 5f12636cbc | |||
| 4d2dcff5df | |||
| a19c6b987c | |||
| a5a0f96d49 | |||
| c21e21423f | |||
| 4dc937fcb8 | |||
| 88e0977ae6 | |||
| 152a5645ad | |||
| 859d63178a | |||
| 779afbb66f | |||
| 3a48eb0c70 | |||
| 18481f3d11 | |||
| 7982043673 | |||
| 00d8422cef | |||
| f381b471f9 | |||
| ab1644daa4 | |||
| 00f39c849e | |||
| 271562e00e | |||
| de9d9697fd | |||
| c1482802d4 | |||
| 2ba867b0f8 | |||
| 326b22a7ba | |||
| b94ea3d412 | |||
| 1ee73d76d6 | |||
| 306728885d | |||
| eccb15bdab | |||
| 365a90e63d | |||
| d327243fd3 | |||
| 38e21df296 | |||
| 9f62310b49 | |||
| 7f79cd5941 |
@@ -111,10 +111,12 @@ dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests # a single tes
|
||||
dotnet test --filter "FullyQualifiedName~MyTestClass.MyMethod" # a single test
|
||||
```
|
||||
|
||||
`Directory.Build.props` carries a temporary `NuGetAuditSuppress` for
|
||||
CVE-2025-6965 / GHSA-2m69-gcr7-jv3q (transitive native `SQLitePCLRaw.lib.e_sqlite3`,
|
||||
no upstream fix yet) — so restore/build not flagging it is expected; remove the
|
||||
suppression once a patched bundle ships.
|
||||
CVE-2025-6965 / GHSA-2m69-gcr7-jv3q (transitive native `SQLitePCLRaw.lib.e_sqlite3`)
|
||||
is **resolved** — the temporary `NuGetAuditSuppress` was removed once the patched bundle
|
||||
shipped. `Core.AlarmHistorian` now carries a surgical direct pin of
|
||||
`SQLitePCLRaw.bundle_e_sqlite3` to `2.1.12` (the first fixed native bundle, embedding the
|
||||
SQLite 3.50.2+ patch), overriding the `2.1.11` that `Microsoft.Data.Sqlite` pulls
|
||||
transitively. `dotnet restore`/`build` is clean with the audit active.
|
||||
|
||||
Test projects live under `tests/<module>/` (Core, Server, Drivers,
|
||||
Drivers/Cli, Client, Tooling) — there is no single unit-test project.
|
||||
@@ -125,6 +127,11 @@ central SQL Server.
|
||||
|
||||
## Docker Workflow (driver fixtures + central SQL Server)
|
||||
|
||||
> **Infra index:** [`infra/README.md`](infra/README.md) is the canonical index of all testing +
|
||||
> deployment infrastructure — the Docker host layout, driver fixtures, the env-gated **live-gate**
|
||||
> recipes (S7 blackhole, GLAuth outage, HistorianGateway LiveIntegration), the integration-test
|
||||
> harness, the docker-dev rig, and the deploy setup. Start there; the sections below + `docs/v2/dev-environment.md` carry the detail.
|
||||
|
||||
> **Migrated 2026-04-28**: Docker config + host moved off this dev VM (DESKTOP-6JL3KKO) onto the shared Linux Docker host (`DOCKER`, 10.100.0.35) so the dev VM could shed WSL2/Hyper-V and have its GPU re-attached via ESXi passthrough. Docker Desktop is no longer installed here. All checked-in `appsettings.json` defaults, fixture-class default endpoints, and `e2e-config.sample.json` were rewritten to target `10.100.0.35`. The driver fixture compose files under `tests/.../Docker/docker-compose.yml` now carry a `project: lmxopcua` label on every service. See `docs/v2/dev-environment.md` for the full rewrite (header dated 2026-04-28).
|
||||
|
||||
Docker workloads run on a shared Linux host at **`10.100.0.35`** — not on this VM. Stacks live at `/opt/otopcua-<driver>/` on the host and carry the `project=lmxopcua` label so they're discoverable via `docker ps --filter label=project=lmxopcua`.
|
||||
@@ -180,7 +187,7 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
|
||||
|
||||
The server uses LDAP-based user authentication via the `Security:Ldap` section in `appsettings.json`. When enabled, credentials are validated by LDAP bind against a GLAuth server, and LDAP group membership maps to OPC UA permissions: `ReadOnly` (browse/read), `WriteOperate` (write FreeAccess/Operate attributes), `WriteTune` (write Tune attributes), `WriteConfigure` (write Configure attributes), `AlarmAck` (alarm acknowledgment). `LdapOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`) implements `IOpcUaUserAuthenticator`, delegating the LDAP bind + group lookup to `OtOpcUaLdapAuthService` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`, an `ILdapAuthService`). See `docs/security.md` for the full guide.
|
||||
|
||||
Dev/test LDAP is the **shared GLAuth** running on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`, plaintext/`Transport=None`). It is managed via `scadaproj/infra/glauth/` (source of truth + deploy runbook). Single bind account `cn=serviceaccount,dc=zb,dc=local` / `serviceaccount123`; all test users password `password`. The docker-dev compose binds this shared instance directly — `DevStubMode` is no longer used. The per-VM NSSM GLAuth at `C:\publish\glauth\` and the old base DNs `dc=lmxopcua,dc=local` / `dc=otopcua,dc=local` are obsolete. (The integration-test harness under `tests/.../Host.IntegrationTests/` uses a separate ephemeral bitnami/openldap on port 3894 for automated tests — that is distinct from the shared dev GLAuth.)
|
||||
Dev/test LDAP is the **shared GLAuth** running on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`, plaintext/`Transport=None`). It is managed via `scadaproj/infra/glauth/` (source of truth + deploy runbook). Single bind account `cn=serviceaccount,dc=zb,dc=local` / `serviceaccount123`; all test users password `password`. The docker-dev compose binds this shared instance directly — `DevStubMode` is no longer used. The per-VM NSSM GLAuth at `C:\publish\glauth\` and the old base DNs `dc=lmxopcua,dc=local` / `dc=otopcua,dc=local` are obsolete. (The integration-test harness under `tests/.../Host.IntegrationTests/` runs its own ephemeral **GLAuth** container on port 3894 — seeded by `tests/.../Host.IntegrationTests/glauth/config.toml` with dev users alice/bob + the `cn=serviceaccount` bind account — for the opt-in `OTOPCUA_HARNESS_USE_LDAP=1` real-bind mode; distinct from the shared dev GLAuth on 3893. It replaced the retired `bitnami/openldap:2.6` in PR #451, 2026-07-15.)
|
||||
|
||||
## Library Preferences
|
||||
|
||||
@@ -319,11 +326,12 @@ When the section is disabled the applier binds the no-op `NullHistorianProvision
|
||||
run. Every dispatch logs a tally (`dispatched/requested/ensured/skipped/failed`); `dispatched=N` with
|
||||
`requested=0` flags the dormant no-op. The API key must carry `historian:tags:write` for `EnsureTags`. (PR
|
||||
#423 shipped `GatewayTagProvisioner` but left this wiring out, so provisioning was dormant; restored 2026-06-27.)
|
||||
**0.2.0 DataType delta (archreview 06/U-7):** `EnsureTags` is an **upsert** (create-or-update, not
|
||||
skip-if-exists), and the 0.2.0 `HistorianTagDefinition.DataType` proto3-optional change means Boolean tags now
|
||||
provision as **Int1** (0.1.0 silently defaulted them to Float). A pre-bump Float Boolean is asked to retype
|
||||
on its next deploy; the AVEVA-side retype policy is pinned by the live retype probe — see the "0.2.0 DataType
|
||||
provisioning delta" section in `docs/Historian.md`.
|
||||
**DataType mapping (issue #441):** `EnsureTags` is an **upsert** (create-or-update, not skip-if-exists).
|
||||
`HistorianTypeMapper` maps **`Boolean → Int2`** (Boolean historizes as a small 0/1 integer) because the
|
||||
historian's `Int1` analog-tag creation path is **server-degenerate** — `EnsureTags(Int1)` stores an unusable
|
||||
stub (live-probed; gateway-side resolution in HistorianGateway #11 / PR #16). Both Float and Int2 are
|
||||
supported analog types, so a pre-existing Float Boolean tag retypes cleanly to Int2 on its next deploy — see
|
||||
the "Boolean historization maps to Int2" section in `docs/Historian.md`.
|
||||
|
||||
### Gateway-side prerequisites
|
||||
|
||||
@@ -358,7 +366,7 @@ export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222 # absolute gateway
|
||||
export HISTGW_GATEWAY_APIKEY=histgw_<id>_<secret> # must carry historian:read + historian:write (+ tags:write) scopes
|
||||
export HISTGW_TEST_TAG=<existing-tag> # read round-trip
|
||||
export HISTGW_WRITE_SANDBOX_TAG=<writable-float-tag> # e.g. HistGW.LiveTest.Sandbox — write round-trip (EnsureTags + write)
|
||||
export HISTGW_BOOL_SANDBOX_TAG=<writable-bool-tag> # e.g. HistGW.LiveTest.BoolSandbox — 0.2.0 Boolean EnsureTags leg + retype probe (06/U-7)
|
||||
export HISTGW_BOOL_SANDBOX_TAG=<writable-bool-tag> # e.g. HistGW.LiveTest.BoolSandbox — Boolean→Int2 EnsureTags leg + retype probe (issue #441)
|
||||
export HISTGW_ALARM_SOURCE=<source-name> # alarm SendEvent → ReadEvents round-trip
|
||||
dotnet test --filter "Category=LiveIntegration"
|
||||
```
|
||||
|
||||
@@ -16,21 +16,6 @@
|
||||
<LangVersion>latest</LangVersion>
|
||||
</PropertyGroup>
|
||||
|
||||
<!--
|
||||
NuGet audit suppression: CVE-2025-6965 / GHSA-2m69-gcr7-jv3q affects the transitive native
|
||||
bundle SQLitePCLRaw.lib.e_sqlite3 (advisory range <= 2.1.11), pulled in via
|
||||
Microsoft.Data.Sqlite 9.0.0 -> SQLitePCLRaw.bundle_e_sqlite3 by Core.AlarmHistorian's
|
||||
store-and-forward sink. As of 2026-06-20 the maintainer has NOT shipped a patched bundle
|
||||
(SQLite fixed the underlying flaw upstream in 3.50.2 but no SQLitePCLRaw release embeds it
|
||||
yet), so there is no package version to bump to. The flaw requires attacker-controlled
|
||||
pathological aggregate SQL; the sink only issues internal parameterised statements, so the
|
||||
exposure here is negligible. REMOVE this suppression once SQLitePCLRaw ships a patched native
|
||||
bundle and bump Microsoft.Data.Sqlite to a release that references it.
|
||||
-->
|
||||
<ItemGroup>
|
||||
<NuGetAuditSuppress Include="https://github.com/advisories/GHSA-2m69-gcr7-jv3q" />
|
||||
</ItemGroup>
|
||||
|
||||
<!--
|
||||
Wire the custom OTOPCUA0001 analyzer (UnwrappedCapabilityCallAnalyzer) into every project so
|
||||
the CapabilityInvoker-wrapping rule is enforced tree-wide, not just against the analyzer's own
|
||||
|
||||
@@ -51,7 +51,7 @@
|
||||
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp.Scripting" Version="4.12.0" />
|
||||
<PackageVersion Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="5.0.0" />
|
||||
<PackageVersion Include="Microsoft.Data.SqlClient" Version="6.1.1" />
|
||||
<PackageVersion Include="Microsoft.Data.Sqlite" Version="9.0.0" />
|
||||
<PackageVersion Include="Microsoft.Data.Sqlite" Version="10.0.7" />
|
||||
<PackageVersion Include="Microsoft.EntityFrameworkCore" Version="10.0.7" />
|
||||
<PackageVersion Include="Microsoft.EntityFrameworkCore.Design" Version="10.0.7" />
|
||||
<PackageVersion Include="Microsoft.EntityFrameworkCore.InMemory" Version="10.0.7" />
|
||||
@@ -95,6 +95,14 @@
|
||||
<PackageVersion Include="Serilog.Sinks.Console" Version="6.0.0" />
|
||||
<PackageVersion Include="Serilog.Sinks.File" Version="7.0.0" />
|
||||
<PackageVersion Include="Shouldly" Version="4.3.0" />
|
||||
<!--
|
||||
Surgical transitive pin: Microsoft.Data.Sqlite pulls SQLitePCLRaw.bundle_e_sqlite3 2.1.11
|
||||
(native lib.e_sqlite3 2.1.11 — the top of the CVE-2025-6965 / GHSA-2m69-gcr7-jv3q vulnerable
|
||||
range, <= 2.1.11). 2.1.12 is the first fixed native bundle (embeds the SQLite 3.50.2+ patch).
|
||||
A direct PackageReference in Core.AlarmHistorian promotes this to the resolved version,
|
||||
clearing the advisory without CentralPackageTransitivePinningEnabled (which breaks the Roslyn split).
|
||||
-->
|
||||
<PackageVersion Include="SQLitePCLRaw.bundle_e_sqlite3" Version="2.1.12" />
|
||||
<PackageVersion Include="System.CommandLine" Version="2.0.5" />
|
||||
<PackageVersion Include="System.Data.SqlClient" Version="4.9.0" />
|
||||
<PackageVersion Include="System.IdentityModel.Tokens.Jwt" Version="8.11.0" />
|
||||
|
||||
@@ -172,7 +172,13 @@ are the new STAB-14..17 below.
|
||||
> `when (token.IsCancellationRequested)` filters on the read/write ensure-wrappers (`:488`/`:1000`) and all
|
||||
> three poll-loop OCE catches — so a non-teardown OCE degrades/backs-off instead of killing the loop. Unit
|
||||
> guards T1–T4 (`S7DriverReconnectTests`, incl. the poll-loop-survival regression). Live gate
|
||||
> `S7_1500ConnectTimeoutOutageTests` (env-gated SYN-blackhole, `deferred-live`).
|
||||
> `S7_1500ConnectTimeoutOutageTests` **CLOSED 2026-07-15** — running it (blackhole via `docker pause`)
|
||||
> exposed that the fix above covered only the *connect* leg: S7.Net's socket `ReadTimeout`/`WriteTimeout`
|
||||
> are ignored by its async read/write paths, so a read on an established-but-frozen peer blocked for
|
||||
> minutes and wedged the poll loop. Fixed (PR #453, master `4dc937fc`) by a new `S7OperationDeadline`
|
||||
> wall-clock ceiling on every data-plane wire op (`S7PlcAdapter` → `TimeoutException`) + `IsS7ConnectionFatal`
|
||||
> now treating `TimeoutException` as fatal → mark-dead → reopen (the READ-leg sibling of STAB-14). Live gate
|
||||
> now GREEN; Driver.S7.Tests 246→260.
|
||||
|
||||
**STAB-14 — High — NEW — S7's reconnect wrapper rethrows connect-timeout
|
||||
cancellation, and the poll loop's bare OCE catch then permanently kills the
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
# Integration-test sweep — status + follow-ups (2026-07-14)
|
||||
|
||||
**Context:** ran the deferred `*.IntegrationTests` sweep (the round-2 STATUS.md "Full integration-suite
|
||||
sweep" item that discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24).
|
||||
**How:** tests run **on the Mac** (repo + .NET 10 SDK), pointing at **driver fixtures brought up on the
|
||||
docker host `10.100.0.35`**, one driver at a time (serial — the suites have a ~16 GB/run memory leak).
|
||||
Master at run time: `30672888` (branch `master`, clean).
|
||||
|
||||
**Docker host `10.100.0.35` (`DOCKER`) note:** has `dotnet 10.0.300`, 22 cores, but only ~7 GB free RAM
|
||||
(the rest is shared SQL/GLAuth/fixtures) and **no repo checkout** — so the tests can't run *on* it (OOM risk
|
||||
to shared infra). Run on the Mac against its fixtures. Fixture dirs live at `/opt/otopcua-<driver>/` (owned,
|
||||
need sudo to create new ones — I deployed FOCAS to `~/otopcua-focas` on the host instead).
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
| Suite | Result | Root cause of failures |
|
||||
|---|---|---|
|
||||
| **OpcUaClient** | ✅ 4/4 | — (opc-plc `:50000`) |
|
||||
| **OpcUaClient.Browser** | ✅ 1/1 | — |
|
||||
| **S7** | ✅ 3/3 (2 skip) | skips = outage tests needing `S7_TIMEOUT_OUTAGE_*` env |
|
||||
| **Modbus** (profile=standard) | ✅ 6/6 | — |
|
||||
| **Host** | ⚠️ **105/117** (7F, 5 skip) | 7 heavy E2E time out under amd64-emulated SQL (OPEN); LDAP image swapped OpenLDAP→GLAuth 2026-07-15 (PR #451). The 5 skips are driver-fixture-gated E2E, not LDAP |
|
||||
| **TwinCAT** | ⏭️ 13 skip | no TC3 XAR fixture exists (expected clean-skip) |
|
||||
| **Modbus** (dl205 / mitsubishi / s7_1500 / exception_injection) | ✅ 16/13/10/17 (was 8F/5F/3F/8F) | **NOT drift** — fixture-cycling artifact, re-verified GREEN 2026-07-15 — see below |
|
||||
| **AbCip** | ✅ 10P/3skip (was 6F/1P/3skip) | static-init-order bug FIXED 2026-07-15 (`326b22a7`) — see below |
|
||||
| **FOCAS** | ✅ 10/10 (was 9F/1P) | topology + mock-timer-endianness FIXED 2026-07-15 — see below |
|
||||
| **AbLegacy** | ✅ slc500 2/2, micrologix 1/1, plc5 1/1 (was 4F) | fixture-deployment gap FIXED 2026-07-15 — see below |
|
||||
| **OpcUaServer** | ✅ 4/4 (was 4F) | cert-store gap + SDK read-throws drift FIXED 2026-07-15 — see below |
|
||||
|
||||
### Headline conclusion
|
||||
**No OtOpcUa production regression.** Every failure is **pre-existing test/fixture infrastructure rot** in
|
||||
long-deferred suites — or, in the Modbus case, a **fixture-cycling false alarm** (see below). The drivers are
|
||||
fine: 4 suites fully green, **all Modbus profiles green** (standard + dl205/mitsubishi/s7_1500/exception once
|
||||
correctly bound — the sweep's non-standard "failures" were the stale-container artifact), Host 105/117 green,
|
||||
and all driver **unit** suites + the live `/run` gates pass. These integration suites were DEFERRED in round 2
|
||||
and never run, so they'd bit-rotted in several distinct ways.
|
||||
|
||||
**Follow-up progress (2026-07-15): ALL 7 CLOSED.** #1 AbCip (PR #446), #5 OpcUaServer (PR #447), #2 Modbus
|
||||
(PR #448 — false alarm), #4 AbLegacy (PR #449 — fixture gap), #3 FOCAS (PR #450 — topology + mock-timer), #7
|
||||
(fixture deploys), #6 LDAP-image (PR #451 — OpenLDAP→GLAuth), and #6 SQL-deploy-E2E (PR #452 — misdiagnosed
|
||||
"emulation timing"; really missing FK-parent seeds + a stale EquipmentId, hidden by the in-memory provider).
|
||||
**Every driver + server integration suite is now verified GREEN. Zero OtOpcUa production regressions across the
|
||||
entire sweep** — every failure was fixture/harness/test rot.
|
||||
|
||||
---
|
||||
|
||||
## Per-suite failure detail (evidence)
|
||||
|
||||
- **Modbus non-standard profiles** — ✅ **RE-VERIFIED GREEN 2026-07-15 (PR #448). The sweep's 8F/5F/3F/8F was a
|
||||
false alarm — a fixture-cycling artifact, NOT pymodbus drift and NOT an OtOpcUa regression.** Correcting the
|
||||
sweep's original diagnosis:
|
||||
- The image genuinely runs **pymodbus 3.13.0** (`docker run --entrypoint python ... -c 'import pymodbus;
|
||||
print(pymodbus.__version__)'` → `3.13.0`; the Dockerfile already pins `pymodbus[simulator]==3.13.0`). The
|
||||
*"…removed in v4"* log line is 3.13.0's **forward-deprecation warning**, not evidence it runs 4.x — the
|
||||
original sweep note misread it.
|
||||
- Root cause: each Modbus profile is a **separate profile-gated compose service** sharing `:5020`. A plain
|
||||
`docker compose down` (no `--profile`) does **not** stop the running profile container, so it keeps holding
|
||||
`:5020`; the next `--profile … up` container silently fails to bind (compose still says "Started"). Every
|
||||
later profile run then hit the **stale** container. Proven by a raw Modbus probe: the "mitsubishi" sim
|
||||
served DL205's `0xCAFE` at HR0 (DL205's V0 marker) and Illegal-Data-Address for mitsubishi-only addresses,
|
||||
while `docker ps` showed the mitsubishi container up with **no** port mapping and DL205 still holding `:5020`.
|
||||
- After force-removing all modbus containers and bringing up each profile cleanly (verifying it actually
|
||||
published `:5020`), all four pass: **dl205 16/16, mitsubishi 13/13, s7_1500 10/10, exception_injection
|
||||
17/17** (the injector needs `--force-recreate` — it uses a different container name). **No code change** —
|
||||
added a profile-cycling gotcha + reliable-cycle recipe to `Docker/README.md`.
|
||||
|
||||
- **AbCip** — **static-init-order bug in the test harness** (always-fail, blocks the whole suite).
|
||||
`tests/.../AbCip.IntegrationTests/AbServerProfile.cs`: `KnownProfiles.All { get; } = [ControlLogix, …]` is
|
||||
declared at **line 30**, *before* the `static readonly` profile fields (line 34+). C# runs static
|
||||
initializers in textual order → `All` captures `[null,null,null,null]` → `ForFamily` NREs at `p.Family`
|
||||
(line 61) and `AbServerFixture..ctor` gets null. **Fix (trivial, safe):** make `All` an expression-bodied
|
||||
property: `public static IReadOnlyList<AbServerProfile> All => [ControlLogix, CompactLogix, Micro800,
|
||||
GuardLogix];` (evaluated on access, after the fields are set). Unblocks the suite; then re-run to see if the
|
||||
actual sim tests pass.
|
||||
|
||||
- **FOCAS** — ✅ **FIXED 2026-07-15 (PR #450). Two test/fixture bugs, no OtOpcUa driver change.**
|
||||
1. *Topology:* `FocasSimFixture` already reads an env-overridable endpoint (`OTOPCUA_FOCAS_SIM_ENDPOINT`,
|
||||
default `localhost:8193`) and exposes `Host`/`Port`, but `WireBackendTests` + `WireBackendCoverageTests`
|
||||
**hardcoded** `focas://127.0.0.1:8193`. Against a remote fixture the skip-gate passed (remote reachable) but
|
||||
the driver dialed localhost → KeyNotFound. Fix: added a `DeviceUri` property to the fixture and derived each
|
||||
test's `DeviceHost` from it — recovered **8 of 9**.
|
||||
2. *Mock timer endianness (newly surfaced):* the last failure (`Timers_populate`) read `60397977600` for a
|
||||
`3600 s` power-on timer (= `0x3C000000 × 60`). Root cause: the focas-mock's `_wire_timer` encoded the
|
||||
`minute`/`msec` fields **big-endian** (`_u32`), but `cnc_rdtimer`'s timer fields are **little-endian** on
|
||||
real hardware — a documented, live-31i-B-validated quirk (`FocasWireClient.ParseTimer` decodes them LE, per
|
||||
`docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md`). **The driver was correct; the mock was wrong.** Fix:
|
||||
`_wire_timer` now emits little-endian (`_u32_le`) → suite **10/10**. Fixture deployed at `~/otopcua-focas`.
|
||||
|
||||
- **AbLegacy** — ✅ **FIXED 2026-07-15 (PR #449). Fixture-deployment gap, not a code defect.** The AbLegacy suite
|
||||
needs `ab_server` in **PCCC mode** (`--plc=SLC500|Micrologix|PLC/5`), which reuses the **AbCip image**
|
||||
(`otopcua-ab-server:libplctag-release`, already on the host) and binds `:44818`. Two issues: (a) no PCCC
|
||||
container was deployed (`~/otopcua-ablegacy` didn't exist); (b) the sweep's *4F-not-skip* means `:44818` was
|
||||
reachable but answering **CIP** (a leftover AbCip container) — the PCCC reads returned Bad. Same shared-port
|
||||
contamination class as the Modbus false alarm. Deployed the compose to `~/otopcua-ablegacy` on the host (image
|
||||
already present, no build needed) and ran each family with `AB_LEGACY_COMPOSE_PROFILE` set to match the running
|
||||
container (**required** — without it the `[Theory]` parametrises over all 3 families and 2 fail against a
|
||||
single-family container): **slc500 2/2, micrologix 1/1 (+1 skip), plc5 1/1 (+1 skip)** — the skips are
|
||||
family-inapplicable file types (e.g. no `F8` float file seeded on micrologix/plc5). **No code change.**
|
||||
|
||||
- **OpcUaServer** — ✅ **FIXED 2026-07-15** (PR #447). Was 4F on `Opc.Ua.ServiceResultException :
|
||||
TrustedIssuerCertificates StorePath must be specified` — the in-test *client* `SecurityConfiguration`
|
||||
omitted the trusted-issuer/peer/rejected store paths that `ValidateAsync` requires. Fix: `TestClientSecurity`
|
||||
helper rooting Directory stores at a throwaway PKI dir (mirrors `DefaultApplicationConfigurationFactory`).
|
||||
Fixing it unmasked a second drift — the 1.5.378 node-cache read path throws
|
||||
`ServiceResultException(BadNodeIdUnknown)` for a removed node rather than returning a bad `DataValue`;
|
||||
the two subscription-survival assertions now use `Should.ThrowAsync`. Suite **4F → 4/4 green**.
|
||||
|
||||
- **Host** — largely healthy (105/117). Two independent issues:
|
||||
- **LDAP image — ✅ FIXED 2026-07-15 (PR #451).** The opt-in real-LDAP mode (`OTOPCUA_HARNESS_USE_LDAP=1`)
|
||||
used `bitnami/openldap:2.6`, gone since Bitnami deprecated their free catalog in 2025. **Replaced it with
|
||||
GLAuth** — the same LDAP server the rest of the project uses (docker-dev + the live OPC UA data-plane auth,
|
||||
both against the shared GLAuth on `:3893`), removing the lone OpenLDAP outlier. Added a harness
|
||||
`glauth/config.toml` (base DN `dc=zb,dc=local`, search-capable `serviceaccount`, dev users alice=full /
|
||||
bob=read-only, role groups with the same gidnumbers as `scadaproj/infra/glauth`) + repointed the harness's
|
||||
service-account override to `cn=serviceaccount`. Live-verified all three bind legs against a deployed
|
||||
container on `:3894` (serviceaccount search returns alice + 5 memberOf groups; alice/bob bind OK; wrong
|
||||
password → `Invalid credentials (49)`). NB: the real-LDAP mode is **opt-in** — the default Host run uses
|
||||
`StubLdapAuthService`, so this never blocked the default suite (the sweep's "5 LDAP skips" were actually the
|
||||
driver-fixture-gated E2E tests, not LDAP).
|
||||
- **SQL-backed deploy E2E — ✅ FIXED 2026-07-15 (PR #452). The sweep's "amd64-emulated-SQL timing" diagnosis
|
||||
was WRONG.** Running the 7 heavy E2E against real (native-amd64) SQL on the Docker host proved they don't
|
||||
time out for speed — they had **never actually run against real SQL**, only the EF in-memory provider (which
|
||||
ignores FK constraints). Two distinct root causes, both hidden by in-memory:
|
||||
- *Missing FK-parent seed (5 tests):* the deploy writes a `NodeDeploymentState` row per node, which FKs to
|
||||
`ClusterNode` (`FK_NodeDeploymentState_ClusterNode_NodeId`). The tests never seeded `ServerCluster` +
|
||||
`ClusterNode` rows, so on SQL each node-state INSERT throws → the deploy never seals (waits the full
|
||||
deadline regardless of how high — proven at 4× it still hung). Added `TwoNodeClusterHarness.SeedDefaultClusterAsync`
|
||||
(seeds a `ServerCluster` + a `ClusterNode` per node; `Warm`/NodeCount=2 to satisfy both the SQL CHECK
|
||||
constraint `CK_ServerCluster_RedundancyMode_NodeCount` and the `ClusterEnabledNodeCountMismatch` validator)
|
||||
and called it in `DeployHappyPathTests`×2 / `FailoverDuringDeployTests` / `FleetDiagnosticsRoundTripTests` /
|
||||
`EquipmentNamespaceMaterializationTests`; fixed `DriverReconnectE2eTests` to seed node B + declare NodeCount=2.
|
||||
- *Stale `EquipmentId` (1 test, pre-existing on BOTH providers):* `EquipmentNamespaceMaterializationTests`
|
||||
seeded `EquipmentId="eq-1"`, but the `EquipmentIdNotDerived` validator (added later) requires
|
||||
`EquipmentId == DeriveEquipmentId(EquipmentUuid)` = `EQ-` + first 12 hex of the UUID. Fixed the seed to a
|
||||
canonical id + matching UUID. (Was failing on in-memory too; only surfaced once the FK fix let the deploy
|
||||
reach equipment validation.)
|
||||
- Added `OTOPCUA_HARNESS_SQL_HOST` / `OTOPCUA_HARNESS_LDAP_HOST` env overrides so the fixtures can live on the
|
||||
native-amd64 Docker host (`docker compose up -d sql` there; run with `OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331`).
|
||||
- Result: the 5 SQL-backed classes go **11/11 green vs native SQL**, and the in-memory default is unregressed
|
||||
(12/12). `RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts…` is a **pre-existing flaky
|
||||
race test** (not SQL-backed; passes in isolation, flaked once under heavy parallel load) — left as-is.
|
||||
|
||||
- **TwinCAT** — 13 skip cleanly (no TC3 XAR fixture exists; unit-only per the driver's design). Expected.
|
||||
|
||||
---
|
||||
|
||||
## Follow-up backlog (distinct, not one fix)
|
||||
|
||||
1. ~~**AbCip static-init bug** — trivial always-fail test bug. Expression-body `KnownProfiles.All`.~~
|
||||
✅ **DONE 2026-07-15** (commit `326b22a7`, PR #446). Live-verified against the controllogix fixture:
|
||||
suite went **6F/1P/3skip → 10 pass / 0 fail / 3 skip** (skips = emulator-mode tests needing a different
|
||||
fixture mode).
|
||||
2. ~~**Modbus non-standard profiles** — pin pymodbus in the fixture Dockerfile.~~
|
||||
✅ **RESOLVED 2026-07-15 (PR #448) — false alarm, no code change.** Image already runs the pinned
|
||||
pymodbus 3.13.0; the 8F/5F/3F/8F was a fixture-cycling artifact (stale profile container holding `:5020`).
|
||||
Re-verified all four profiles GREEN (16/13/10/17). Added a profile-cycling gotcha + reliable-cycle recipe
|
||||
to `Docker/README.md`.
|
||||
3. ~~**FOCAS** — reconcile the 127.0.0.1-mock topology with the remote-fixture model.~~
|
||||
✅ **DONE 2026-07-15 (PR #450).** Tests now derive `DeviceHost` from the fixture endpoint (recovered 8/9);
|
||||
also fixed a newly-surfaced mock bug — `_wire_timer` encoded the timer big-endian but real hardware +
|
||||
driver use little-endian. Suite **10/10**. No OtOpcUa driver change.
|
||||
4. ~~**AbLegacy** — deploy an AbLegacy sim fixture (none on host).~~
|
||||
✅ **DONE 2026-07-15 (PR #449) — fixture-deployment gap, no code defect.** Deployed the PCCC compose to
|
||||
`~/otopcua-ablegacy` (reuses the AbCip image, already on host). All 3 families GREEN (slc500 2/2, micrologix
|
||||
1/1, plc5 1/1). Recipe added below; `AB_LEGACY_COMPOSE_PROFILE` MUST match the running container.
|
||||
5. ~~**OpcUaServer** — supply cert-store StorePath in the harness config.~~
|
||||
✅ **DONE 2026-07-15** (PR #447). Added `TestClientSecurity` helper building a validated client
|
||||
`SecurityConfiguration` with Directory own/issuer/trusted/rejected stores under a throwaway PKI dir
|
||||
(mirrors `DefaultApplicationConfigurationFactory`); wired into both `DualEndpointTests` +
|
||||
`SubscriptionSurvivalTests`. That unmasked a **second** drift: the 1.5.378 node-cache read path
|
||||
*throws* `ServiceResultException(BadNodeIdUnknown)` for a removed node instead of returning a bad
|
||||
`DataValue` — updated the two subscription-survival assertions to `Should.ThrowAsync`. Suite **4F → 4/4**.
|
||||
6. **Host** — ⬐ split into two:
|
||||
- ~~replace `bitnami/openldap:2.6` (image gone)~~ ✅ **DONE 2026-07-15 (PR #451)** — swapped the harness LDAP
|
||||
to **GLAuth** (unifies on the LDAP the rest of the project uses); live-verified the binds on `:3894`.
|
||||
- ~~**amd64-emulated-SQL timing**~~ ✅ **DONE 2026-07-15 (PR #452) — misdiagnosed.** Not a timing/emulation
|
||||
issue: the deploy E2E tests had never run against real SQL (in-memory ignores FKs) and lacked the
|
||||
`ServerCluster`/`ClusterNode` seed the `NodeDeploymentState` FK requires → deploy never seals. Added a
|
||||
harness seeding helper + `OTOPCUA_HARNESS_SQL_HOST` override; also fixed a stale `EquipmentId` in one test.
|
||||
11/11 green vs native SQL, in-memory unregressed. (Roslyn racing test = pre-existing flaky, left as-is.)
|
||||
7. ~~**Fixture deployment gap** — FOCAS + AbLegacy fixtures aren't on the host.~~
|
||||
✅ **RESOLVED 2026-07-15.** Both now deployed under the home dir: `~/otopcua-ablegacy` (PR #449) +
|
||||
`~/otopcua-focas` (PR #450). (`/opt` still needs sudo; the home-dir deploys work fine.)
|
||||
|
||||
---
|
||||
|
||||
## Fixture setup recipes (to re-run the sweep)
|
||||
|
||||
Bring up on the docker host (distinct ports, can coexist except Modbus profiles share `:5020`):
|
||||
|
||||
```bash
|
||||
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-modbus && docker compose --profile standard up -d' # :5020
|
||||
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-abcip && docker compose --profile controllogix up -d' # :44818
|
||||
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d' # :1102
|
||||
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-opcuaclient && docker compose up -d' # :50000
|
||||
ssh dohertj2@10.100.0.35 'cd ~/otopcua-ablegacy && docker compose --profile slc500 up -d' # :44818 (PCCC; reuses AbCip image)
|
||||
ssh dohertj2@10.100.0.35 'cd ~/otopcua-focas && docker compose up -d --build' # :8193
|
||||
```
|
||||
|
||||
Run per suite (env → project):
|
||||
|
||||
| Suite | Env | Project |
|
||||
|---|---|---|
|
||||
| OpcUaClient(+Browser) | `OPCUA_SIM_ENDPOINT=opc.tcp://10.100.0.35:50000` | `tests/Drivers/…OpcUaClient(.Browser).IntegrationTests` |
|
||||
| S7 | `S7_SIM_ENDPOINT=10.100.0.35:1102` | `tests/Drivers/…S7.IntegrationTests` |
|
||||
| Modbus | `MODBUS_SIM_ENDPOINT=10.100.0.35:5020` `MODBUS_SIM_PROFILE=<standard\|dl205\|mitsubishi\|s7_1500\|exception_injection>` (swap the host sim to match) | `tests/Drivers/…Modbus.IntegrationTests` |
|
||||
| AbCip | `AB_SERVER_ENDPOINT=10.100.0.35:44818` `AB_SERVER_PROFILE=controllogix` | `tests/Drivers/…AbCip.IntegrationTests` |
|
||||
| AbLegacy | `AB_LEGACY_ENDPOINT=10.100.0.35:44818` `AB_LEGACY_COMPOSE_PROFILE=<slc500\|micrologix\|plc5>` (**must** match the running PCCC container — else the `[Theory]` runs all 3 families and 2 fail) | `tests/Drivers/…AbLegacy.IntegrationTests` |
|
||||
| FOCAS | `OTOPCUA_FOCAS_SIM_ENDPOINT=10.100.0.35:8193` (tests now honour this — no longer hardcoded to 127.0.0.1) | `tests/Drivers/…FOCAS.IntegrationTests` |
|
||||
| Host | `OTOPCUA_HARNESS_USE_SQL=1` (+ SQL `:14331` via the harness `docker-compose.yml up sql`) — set `OTOPCUA_HARNESS_USE_LDAP=1` to exercise real binds against the **GLAuth** service (`up ldap`, host `:3894`; users alice/bob, svc `cn=serviceaccount`/`serviceaccount123`) | `tests/Server/…Host.IntegrationTests` |
|
||||
|
||||
**Modbus profile cycling:** the sim services share `:5020`, so to run a profile:
|
||||
`ssh … 'cd /opt/otopcua-modbus && docker compose down && docker compose --profile <p> up -d'`, then run with
|
||||
`MODBUS_SIM_PROFILE=<p>` (each class skips unless its profile is active). Wait for `:5020` before running —
|
||||
TCP opens before the register map fully loads.
|
||||
|
||||
---
|
||||
|
||||
## Current state (post-sweep)
|
||||
|
||||
- **All fixtures torn down** — modbus/abcip/s7/opcuaclient/focas on the host removed (ports 5020/44818/1102/
|
||||
50000/8193 all closed); the local Host SQL fixture `down -v`'d. Shared host is clean.
|
||||
- **Repo:** `master @ 30672888`, clean. No sweep changes committed (this file is the only artifact).
|
||||
- **Nothing merged for the sweep** — it's a read-only test run; the follow-ups above are unstarted.
|
||||
@@ -2,20 +2,122 @@
|
||||
"planPath": "archreview/plans/R2-04-failure-visibility-plan.md",
|
||||
"lastUpdated": "2026-07-12",
|
||||
"tasks": [
|
||||
{ "id": "T1", "subject": "Extend AddressSpaceApplyOutcome with RebuildFailed/FailedNodes + SafeRebuild returns bool (01/S-1)", "status": "completed", "blockedBy": [] },
|
||||
{ "id": "T2", "subject": "Safe* helpers return bool; Apply tallies removal-pass failures into FailedNodes", "status": "completed", "blockedBy": ["T1"] },
|
||||
{ "id": "T3", "subject": "Materialise* passes return swallowed-failure counts (void -> int, five methods)", "status": "completed", "blockedBy": ["T2"] },
|
||||
{ "id": "T4", "subject": "OpcUaApplyFailed counter + OpcUaPublishActor Error/meter surfacing of degraded applies", "status": "completed", "blockedBy": ["T3"] },
|
||||
{ "id": "T5", "subject": "Galaxy internal IMxWriteOps seam under GatewayGalaxyDataWriter (sealed SDK session untestable) (06/S-1)", "status": "completed", "blockedBy": [] },
|
||||
{ "id": "T6", "subject": "Galaxy fail-closed write on AdviseSupervisory failure -> BadCommunicationError, no raw write", "status": "completed", "blockedBy": ["T5"] },
|
||||
{ "id": "T7", "subject": "Galaxy counters galaxy.writes.advise_failed + galaxy.writes.unconfirmed (empty-statuses Good metered)", "status": "completed", "blockedBy": ["T6"] },
|
||||
{ "id": "T8", "subject": "Galaxy skip-gated live smoke: normal advised write still returns Good post-change", "status": "completed", "blockedBy": ["T6"] },
|
||||
{ "id": "T9", "subject": "PrimaryGatePolicy pure class + truth-table tests + otopcua.redundancy.primary_gate_denied counter (03/S4)", "status": "completed", "blockedBy": [] },
|
||||
{ "id": "T10", "subject": "Wire DriverHostActor gates (:977/:1039/:1095) through PrimaryGatePolicy with member-count provider + S5 log-once", "status": "completed", "blockedBy": ["T9"] },
|
||||
{ "id": "T11", "subject": "DriverHostActorPrimaryGateTests: deny-on-unknown multi-node, allow single-node, ack/emit variants, meter tags", "status": "completed", "blockedBy": ["T10"] },
|
||||
{ "id": "T12", "subject": "ScriptedAlarmHostActor alerts-emit gate adopts PrimaryGatePolicy (consistency extension)", "status": "completed", "blockedBy": ["T9"] },
|
||||
{ "id": "T13", "subject": "In-process 2-node delivery-path test (TwoNodeClusterHarness): DPS-delivered snapshot drives the gate", "status": "deferred-live", "note": "rig/live; Host.IntegrationTests (~16GB/run leak + needs SQL/2-node rig) — implemented + build-verified, controller runs it in the serialized heavy pass", "blockedBy": ["T11"] },
|
||||
{ "id": "T14", "subject": "docs/Redundancy.md: gate policy table, boot-window client experience, new meters, Galaxy fail-closed semantics", "status": "completed", "blockedBy": [] },
|
||||
{ "id": "T15", "subject": "LIVE GATE: 2-node docker-dev rig verify (ServiceLevel 240/100, boot-window write rejected+reverted, single /alerts row)", "status": "deferred-live", "note": "rig/live; 2-node docker-dev rig session (rebuild both centrals) — merge-blocking gate the controller runs in the serialized heavy pass", "blockedBy": ["T10", "T11", "T12", "T13"] }
|
||||
{
|
||||
"id": "T1",
|
||||
"subject": "Extend AddressSpaceApplyOutcome with RebuildFailed/FailedNodes + SafeRebuild returns bool (01/S-1)",
|
||||
"status": "completed",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "T2",
|
||||
"subject": "Safe* helpers return bool; Apply tallies removal-pass failures into FailedNodes",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"T1"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "T3",
|
||||
"subject": "Materialise* passes return swallowed-failure counts (void -> int, five methods)",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"T2"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "T4",
|
||||
"subject": "OpcUaApplyFailed counter + OpcUaPublishActor Error/meter surfacing of degraded applies",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"T3"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "T5",
|
||||
"subject": "Galaxy internal IMxWriteOps seam under GatewayGalaxyDataWriter (sealed SDK session untestable) (06/S-1)",
|
||||
"status": "completed",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "T6",
|
||||
"subject": "Galaxy fail-closed write on AdviseSupervisory failure -> BadCommunicationError, no raw write",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"T5"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "T7",
|
||||
"subject": "Galaxy counters galaxy.writes.advise_failed + galaxy.writes.unconfirmed (empty-statuses Good metered)",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"T6"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "T8",
|
||||
"subject": "Galaxy skip-gated live smoke: normal advised write still returns Good post-change",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"T6"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "T9",
|
||||
"subject": "PrimaryGatePolicy pure class + truth-table tests + otopcua.redundancy.primary_gate_denied counter (03/S4)",
|
||||
"status": "completed",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "T10",
|
||||
"subject": "Wire DriverHostActor gates (:977/:1039/:1095) through PrimaryGatePolicy with member-count provider + S5 log-once",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"T9"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "T11",
|
||||
"subject": "DriverHostActorPrimaryGateTests: deny-on-unknown multi-node, allow single-node, ack/emit variants, meter tags",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"T10"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "T12",
|
||||
"subject": "ScriptedAlarmHostActor alerts-emit gate adopts PrimaryGatePolicy (consistency extension)",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"T9"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "T13",
|
||||
"subject": "In-process 2-node delivery-path test (TwoNodeClusterHarness): DPS-delivered snapshot drives the gate",
|
||||
"status": "completed",
|
||||
"note": "DONE 2026-07-14: PrimaryGateFailoverTests PASS 1/1 (3s), ran without SQL (gate test needs only the in-process 2-node cluster + DPS snapshot). Delivered snapshot promotes exactly one Primary; negative-control (broken subscribe) times out. See STATUS.md.",
|
||||
"blockedBy": [
|
||||
"T11"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "T14",
|
||||
"subject": "docs/Redundancy.md: gate policy table, boot-window client experience, new meters, Galaxy fail-closed semantics",
|
||||
"status": "completed",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "T15",
|
||||
"subject": "LIVE GATE: 2-node docker-dev rig verify (ServiceLevel 240/100, boot-window write rejected+reverted, single /alerts row)",
|
||||
"status": "completed",
|
||||
"note": "DONE 2026-07-14: docker-dev 2-node rig rebuilt from master 365a90e6 + Modbus sim up. ServiceLevel 250/240; boot-window write rejected \"not primary (role unknown)\" (meter reason=role-unknown,site=write); steady-state secondary rejected \"not primary\" + node reverted (read-back 1234 not 9999, meter reason=secondary); primary write reaches device; clean apply (no apply.failed). Full record in STATUS.md.",
|
||||
"blockedBy": [
|
||||
"T10",
|
||||
"T11",
|
||||
"T12",
|
||||
"T13"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
+93
-24
@@ -171,12 +171,12 @@ what the unit leg cannot; (c) migrate the 3 xunit v2 holdouts once Akka ships an
|
||||
|---|---|---|---|---|
|
||||
| **R2-06** | 06/S-11 (rank 7), 06/U-7 (rank 8), 06/C-7 | `r2/06-serverhistorian-failfast` (off `1676c8f4`) | `3a049a13`…`4640ecb1` (10 task commits) | **Code-complete, offline-verified.** S-11: fail-fast `ServerHistorianOptionsValidator` (consumer-gated on `ServerHistorian:Enabled` OR `AlarmHistorian:Enabled`, fails empty/non-http(s) Endpoint at startup) + `AddValidatedOptions` wiring + adapter defense-in-depth (`TryCreate`→named `InvalidOperationException`, no more raw `UriFormatException` crash-loop) + `Validate()` warning parity + alarm-only-mode startup warning. U-7: pinned Boolean→Int1 explicit-presence (`HasDataType`) + `HISTGW_BOOL_SANDBOX_TAG` live Boolean EnsureTags leg + retype probe (skip-clean offline) + upsert/retype migration note (`docs/Historian.md`). C-7: keep-populating decision + dead-`Quality` doc comments (0.2.0 C-002). **Live leg operator-gated (T12) — pending a VPN run on a 0.2.0 gateway** (records the observed in-place retype policy back into `docs/Historian.md`; also discharges 06/U-2's full documented run). |
|
||||
| **R2-05** | 04/C-1 (High) | `r2/05-adminui-authz` (off `1676c8f4`) | `183b72b7`…`6c1f05cb` (5 commits) | **Code-complete, offline-verified.** All 38 routable AdminUI pages carry an explicit named-policy `@attribute` via new `AdminUiPolicies` constants (`Security/Auth/`). New `ConfigEditor` (Administrator+Designer) gates the 20-page config-authoring surface incl. the 8 driver pages that author live `ResilienceConfig` + `/api/script-analysis/*`; new `AuthenticatedRead` on the 16 read pages; `FleetAdmin`/`DriverOperator` unchanged (literals→constants). Anti-drift guard `PageAuthorizationGuardTests` (bans bare `[Authorize]`/`Roles=`/unknown-policy/unclassified) + `AdminUiPoliciesTests` semantics matrix. Corrections folded in: Reservations + ClusterRedundancy read-only (→`AuthenticatedRead`); Home had no attribute (now `AuthenticatedRead`). AdminUI.Tests 517/517, Security.Tests 80/80. **Live `/run` positive pass deferred (T15)** — docker-dev `DisableLogin=true` auto-admin can't observe deny; negative proof is CI-side (both guard tests). |
|
||||
| **R2-01** | 05/STAB-14 (High regression, rank 1), 05/STAB-15 (rank 3), 05/STAB-8 seam | `r2/01-s7-fault-hardening` (off `1676c8f4`) | `fcdcf0e2`…`bdb7ed41` (13 task commits) | **Code-complete, offline-verified.** All prod changes in `S7Driver.cs`. STAB-14: connect-timeout surfaces as `TimeoutException` (not an escaping OCE) at the `EnsureConnectedAsync`/`InitializeAsync` source + `when(token.IsCancellationRequested)` filters on the read/write ensure-wrappers and all three poll-loop OCE catches — subscription poll loops survive an unreachable-host outage and recover (T2 regression RED→GREEN). STAB-15: `IsS7ConnectionFatal` broadened to the ISO-on-TCP framing surface (TPKT/TPDU/WrongNumberOfBytes/`PlcException{WrongNumberReceivedBytes}`, NOT `InvalidDataException`); mid-PDU cancel marks handle dead; probe classifies+marks-dead under the gate. STAB-8 (S7 part): `EnsureConnectedAsync` documented as the R2-09 connect-throttle seam (note-only). Driver.S7.Tests 246/246. **Live outage gate deferred (T11)** — `S7_1500ConnectTimeoutOutageTests` (env `S7_TIMEOUT_OUTAGE_START_CMD`/`STOP_CMD`, SYN-blackhole) skips clean offline; the connect-*timeout* class the `docker restart` bounce can't produce. (Pre-existing unrelated CLI docs-scan test fails at base — out of scope.) |
|
||||
| **R2-01** | 05/STAB-14 (High regression, rank 1), 05/STAB-15 (rank 3), 05/STAB-8 seam | `r2/01-s7-fault-hardening` (off `1676c8f4`) | `fcdcf0e2`…`bdb7ed41` (13 task commits) | **Code-complete, offline-verified.** All prod changes in `S7Driver.cs`. STAB-14: connect-timeout surfaces as `TimeoutException` (not an escaping OCE) at the `EnsureConnectedAsync`/`InitializeAsync` source + `when(token.IsCancellationRequested)` filters on the read/write ensure-wrappers and all three poll-loop OCE catches — subscription poll loops survive an unreachable-host outage and recover (T2 regression RED→GREEN). STAB-15: `IsS7ConnectionFatal` broadened to the ISO-on-TCP framing surface (TPKT/TPDU/WrongNumberOfBytes/`PlcException{WrongNumberReceivedBytes}`, NOT `InvalidDataException`); mid-PDU cancel marks handle dead; probe classifies+marks-dead under the gate. STAB-8 (S7 part): `EnsureConnectedAsync` documented as the R2-09 connect-throttle seam (note-only). Driver.S7.Tests 246/246. **Live outage gate CLOSED 2026-07-15 (T11) — and it uncovered + fixed a real prod gap (PR #453, master `4dc937fc`).** Running `S7_1500ConnectTimeoutOutageTests` against the live s7_1500 sim (blackholed via `docker pause`, the ACK-less-established shape) exposed that STAB-14 fixed only the *connect* leg: S7.Net's `ReadTimeout`/`WriteTimeout` map to the TcpClient's *synchronous* receive/send timeout, which async reads ignore, so a read on an established-but-frozen peer blocked for minutes — the poll loop wedged (no Bad tick, no reconnect), which offline fakes (open-hang only) couldn't surface. Fix = new `S7OperationDeadline` wall-clock ceiling on every data-plane wire op in `S7PlcAdapter` (→ `TimeoutException`) + `IsS7ConnectionFatal` now classifies `TimeoutException` fatal → mark-dead → reopen (the read-leg sibling of the connect-timeout fix). Driver.S7.Tests 246→260 (6 new: 5 deadline + 1 driver-reaction); **live gate now GREEN** (8s: baseline Good → pause blackhole → Bad tick → unpause → recovered Good). (Pre-existing unrelated CLI docs-scan test fails at base — out of scope.) |
|
||||
| **R2-03** | 02/S13 (rank 5), 02/S12, 02/P7 | `r2/03-vt-failure-quality` (off `1676c8f4`) | `b9775485`…`6bb91e8d` (12 task commits) | **Code-complete, offline-verified.** S13: VT script failure/timeout degrades the node to `OpcUaQuality.Bad` (carrying last-known value) instead of freezing on last-Good forever — additive `EvaluationResult.Quality`, bridge historizes `0x80020000u` BadInternalError, published once per Good→Bad transition gated on all deps arrived (inputs-ready, no cold-start flash), recovery force-publishes next Good. S12: apply-boundary `ClearCompiledScripts` racing an in-flight eval into ObjectDisposedException retried once in `RoslynVirtualTagEvaluator` (re-fetch→recompile into current generation); stress guard reproduces pre-fix, deterministically green post-fix. P7: apply-boundary clear gated on a changed deployed expression set (`HashSet.SetEquals`, ordinal) — identical redeploy / rename-with-same-expressions no longer forces full VT recompile. Runtime.Tests 372/372, RoslynVirtualTagEvaluatorTests 20/20. **Deferred (T11/T12):** full cross-suite sweep + docker-dev live-`/run` (fail→Bad→recover + P7 no-recompile spot-check). |
|
||||
| **R2-02** | 01/S-6 (High, rank 2), 01/U-6, 01/S-8=03/S12 (rank 4), 04/C-7, 01/S-7=03/S13 | `r2/02-resilience-config-hardening` (off `1676c8f4`) | `4be13429`…`dd82e740` (17 task commits) | **Code-complete, offline-verified.** **S-6:** parser clamps timeout/retry/breaker overrides (timeout≤0→tier default, breaker==1→2, caps) with diagnostics + belt-and-braces `Build` clamp so pipeline construction can never throw Polly `ValidationException` (new `ResiliencePolicyRanges` in Core.Abstractions); brick-repro + `Build_NeverThrows_ForHostileDirectOptions` + factory-seam wiring proof; AdminUI `Validate()` range warnings. **S-8:** parser forces Write/AlarmAck retry→0 (spec invariant) **and** dispatch flips to `isIdempotent:false` (RED-first on the Runtime wiring test); non-idempotent arm caches the no-retry snapshot once/invoker (01/P-4) + tracker in-flight accounting; `WriteIdempotentAttribute` false claim corrected; retry cells disabled. **U-6:** bulkhead knob deleted end-to-end (migration-free; stored keys ignored) + `Options_properties_are_exactly_the_pipeline_wired_set` inertness guard. **C-7:** `ResilienceFormModel` JsonObject-bag round-trip preserves unknown keys; malformed JSON → `ParseFailed` lock + verbatim `ToJson` + discard button. **S-7:** per-`Create` `Interlocked` options-generation in the pipeline-cache key kills the respawn stale-pipeline race. Core resilience 121/121, AdminUI form 10/10, Analyzers 32/32, Runtime 364/364; 0 production OTOPCUA0001. **Deferred (T15/T18):** docker-dev `:9200` `/run` (warning banner, input lockout, unknown-key survival) + whole-solution sweep. |
|
||||
| **R2-10** | 03/U10=04/U-5 (Medium/Low, action #10) | `r2/10-resilience-observability` (off `46fedda3`) | `205d3980`…`<docs>` (11 task commits) | **Code-complete, offline-verified.** The resilience status tracker now has a production reader, mirroring the driver-health flow. Core truthfulness: `RecordBreakerClose` + derived `IsBreakerOpen` (breaker-open state derivable), `OnClosed` records the close, and `RecordSuccess` wired onto the invoker success paths (`ConsecutiveFailures` is a true consecutive counter, not stuck-forever). Transport: `DriverResilienceStatusChanged` Commons contract → periodic 5 s full-snapshot publisher `DriverResilienceStatusPublisherService` (driver nodes, registered in `AddOtOpcUaDriverFactories` with wiring guard `ResilienceStatusReaderRegistrationTests` — closes the built-but-never-wired class) → `driver-resilience-status` DPS topic → per-admin-node `DriverResilienceStatusBridge` → `IDriverResilienceStatusStore` → resilience chips (breaker-OPEN / consecutive-failures / in-flight / staleness) on `DriverStatusPanel.razor`, read in-process (self-HubConnection ban honored). Core.Tests resilience 31/31 (incl. deterministic breaker-close via a controllable TimeProvider), Host.IntegrationTests publisher+guard 6/6, AdminUI.Tests store 5/5; full-solution build clean. **Deferred (T10):** docker-dev live-`/run` — S7 dead-endpoint breaker-open observed on BOTH centrals + recovery-clear + staleness-dim (also the first live sighting of the #10 `circuit-breaker OPENED` log line, closing FOLLOWUP-10's behavioral gate). |
|
||||
|
||||
| **R2-08** | 03/S2 (High, action #8), 03/S3 (High) | `r2/08-ldap-historyread-async` (off `1a698cbb`) | `ce6d9e21`…`<docs>` (14 task commits) | **Code-complete, offline-verified.** **S2 (LDAP async/timeout):** the SDK raises the impersonation callback synchronously under a `SessionManager`-wide `m_eventLock` (verified vs. 1.5.378 — a hung bind serializes ALL activations), so the fix is entirely at the `LdapOpcUaUserAuthenticator` boundary: five new `Security:Ldap` keys (`ConnectionTimeoutMs`=10000 projected into the shared lib, `AuthTimeoutMs`=15000 hard whole-flow `WaitAsync` bound → fail-closed "timed out", `MaxConcurrentAuthentications`=8 `SemaphoreSlim` with release-on-core-completion so orphaned binds still count, `OutageFailureThreshold`=3/`OutageCooldownSeconds`=15 directory-outage circuit → fast-deny "unavailable", half-open probe) + `LdapAuthResult.IsSystemFailure` seam (mapped from `ServiceAccountBindFailed`) distinguishing outage from credential-deny + `LdapOptionsValidator` rules + `HandleImpersonation` contract doc + `docs/security.md`. Stall-repro RED→GREEN; offline outage-sim through the real `HandleImpersonation` against an unroutable host. **S3 (HistoryRead fan-out):** all four HistoryRead arms async-ified (`ServeNodeAsync`/`ServeRawPagedAsync`/`ServeEventsAsync`, OCE→`BadTimeout`) then bounded per-node fan-out (`RunBounded` semaphore + single arm-boundary bridge) + per-request deadline CTS (`HistoryReadDeadline`=60s) + process-wide lazy batch limiter (`MaxConcurrentHistoryReadBatches`=16 → flood degrades to `BadTooManyOperations`); 16×4=64 in-flight cap; three new `ServerHistorian` keys + node-manager props + hosted-service wiring + `docs/Historian.md`/CLAUDE.md tables. Sequential-serving RED→GREEN. Host.IntegrationTests (LDAP resilience) 13/13, Security.Tests 11/11, Runtime.Tests options 11/11, OpcUaServer.Tests HistoryRead 30/30. **Deferred (live):** S2 docker-dev unreachable-LDAP + GLAuth outage/recovery cycle; S3 `Category=LiveIntegration` gateway soak. |
|
||||
| **R2-08** | 03/S2 (High, action #8), 03/S3 (High) | `r2/08-ldap-historyread-async` (off `1a698cbb`) | `ce6d9e21`…`<docs>` (14 task commits) | **Code-complete, offline-verified.** **S2 (LDAP async/timeout):** the SDK raises the impersonation callback synchronously under a `SessionManager`-wide `m_eventLock` (verified vs. 1.5.378 — a hung bind serializes ALL activations), so the fix is entirely at the `LdapOpcUaUserAuthenticator` boundary: five new `Security:Ldap` keys (`ConnectionTimeoutMs`=10000 projected into the shared lib, `AuthTimeoutMs`=15000 hard whole-flow `WaitAsync` bound → fail-closed "timed out", `MaxConcurrentAuthentications`=8 `SemaphoreSlim` with release-on-core-completion so orphaned binds still count, `OutageFailureThreshold`=3/`OutageCooldownSeconds`=15 directory-outage circuit → fast-deny "unavailable", half-open probe) + `LdapAuthResult.IsSystemFailure` seam (mapped from `ServiceAccountBindFailed`) distinguishing outage from credential-deny + `LdapOptionsValidator` rules + `HandleImpersonation` contract doc + `docs/security.md`. Stall-repro RED→GREEN; offline outage-sim through the real `HandleImpersonation` against an unroutable host. **S3 (HistoryRead fan-out):** all four HistoryRead arms async-ified (`ServeNodeAsync`/`ServeRawPagedAsync`/`ServeEventsAsync`, OCE→`BadTimeout`) then bounded per-node fan-out (`RunBounded` semaphore + single arm-boundary bridge) + per-request deadline CTS (`HistoryReadDeadline`=60s) + process-wide lazy batch limiter (`MaxConcurrentHistoryReadBatches`=16 → flood degrades to `BadTooManyOperations`); 16×4=64 in-flight cap; three new `ServerHistorian` keys + node-manager props + hosted-service wiring + `docs/Historian.md`/CLAUDE.md tables. Sequential-serving RED→GREEN. Host.IntegrationTests (LDAP resilience) 13/13, Security.Tests 11/11, Runtime.Tests options 11/11, OpcUaServer.Tests HistoryRead 30/30. **S2 GLAuth outage/recovery LIVE gate CLOSED 2026-07-15 (PR #455, master `5f12636c`)** — new env-gated `LdapGlauthOutageLiveTests` drives the circuit through the production path (`OtOpcUaLdapAuthService`+`LdapOpcUaUserAuthenticator`) against a REAL GLAuth on `10.100.0.35:3894` (paused/unpaused via `docker pause`); GREEN (12s): healthy alice/bob binds w/ mapped roles → pause → 3 bounded failures → circuit opens → sub-second "unavailable" → unpause → half-open close → recover. Also the FIRST end-to-end proof the PR #451 GLAuth swap binds correctly. Skips clean offline (1 skipped, 8ms). **S3 `Category=LiveIntegration` gateway soak — GREEN 2026-07-15 (6/6)** against the running local 0.2.0 gateway (`http://localhost:5221` h2c) + the real AVEVA historian over VPN; the Galaxy read round-trip exercises the `GatewayHistorianDataSource` read path S3 hardened. **R2-08 fully live-verified.** |
|
||||
| **R2-11** | 01/C-1, 01/P-1 + 05/CONV-2, UNDER-1, UNDER-6 | `r2/11-tagconfig-consolidation` (off `46fedda3`) | first `2e14fe1f` … (24 task commits) | **Code-complete, offline-verified.** **Part 1 (C-1/P-1):** single `TagConfigIntent.Parse` + `DeviceConfigIntent` in Commons replace the four byte-parity TagConfig parse copies (composer/artifact/walker/DraftValidator) — composer now parses once per tag (P-1); a permanent `Runtime.Tests/TagConfigCorpusParityTests` (30-blob corpus) guards compose≡decode; `grep "MUST parse identically" src/` = 0. **Part 2 (CONV-2):** shared strict-capable `TagConfigJson` readers in Core.Abstractions replace the six copy-pasted `ReadEnum`s; all six driver parsers gain `Inspect()` warnings + honour the `writable` key (FOCAS **forced read-only**, 05/UNDER-1); FOCAS equipment tags run the capability-matrix pre-flight (05/UNDER-6); Modbus probe parses the factory DTO shape (`timeoutMs` no longer dropped). New `EquipmentTagConfigInspector` in ControlPlane + `AdminOperationsActor` deploy-gate wiring with `Deployment:TagConfigValidationMode` (**Warn default | Error opt-in**), proven through the actor (F10b consuming test). AdminUI: `writable` checkbox in the six typed editors. **Phase-C follow-up (deferred, recorded here):** flip the runtime parsers to strict (typo'd enum ⇒ `BadNodeIdUnknown` instead of wrong-width `Good`) once fleets have run `Error` mode clean — behavior change deliberately not shipped in this plan. **Deferred (T22/T24):** docker-dev `/run` of one typed editor's Writable checkbox + whole-solution serial sweep. |
|
||||
|
||||
| **R2-07 (Phases 1+2+3)** | 03/P1 (action #7, surgical pure-adds/removes/mixed) | `r2/07-surgical-pure-adds` (off `1a698cbb`) | `bb226f45`…`e2b47638` (T1–T14) | **Phases 1+2 code-complete, offline-verified.** New pure `AddressSpaceChangeClassifier` (Empty/AttributeOnly/PureAdd/PureRemove/AddRemoveMix/Rebuild; first-match-wins, default-closed Rebuild; the F10b tag/vtag surgical-eligibility predicates moved here as `internal static`). `AddressSpaceApplier.Apply` routes by kind. **Phase 1 (PureAdd/AttributeOnly):** skip `RebuildAddressSpace` (the idempotent Materialise passes create exactly the added nodes; existing subscriptions survive); adds announced via `ComputeAddAnnouncements`+`AnnounceAddedNodes` (dedup affected-parent → existing guard-covered `RaiseNodesAddedModelChange`), fired by `OpcUaPublishActor` after the 4 passes gated on `!outcome.RebuildCalled`; `MaterialiseAlarmCondition` made idempotent (skip-if-present-and-same-kind keeps the condition NodeState alive). **NO new sink members in Phase 1.** **Phase 2 (PureRemove):** 3 new `ISurgicalAddressSpaceSink` members (`RemoveVariableNode`/`RemoveAlarmConditionNode`/`RemoveEquipmentSubtree`) — guard-first with the **observed negative control** (`DeferredSinkForwardingReflectionTests` fails member-named until `DeferredAddressSpaceSink` forwards) + `SdkAddressSpaceSink` forwards + `OtOpcUaNodeManager` in-place teardown (scoped map cleanup, notifier demotion via `RemoveRootNotifier`, `BuildNodesRemovedModelChange` verb=NodeDeleted reported outside `Lock`). Applier `PureRemove` arm: terminal Bad / `RemovedConditionState` writes (closing the removed-alarm-tag today-gap) → subsumption partition (removed-equipment children folded into the subtree remove) → in-place removes → any false/throw ⇒ full-rebuild ratchet; non-surgical sink ⇒ rebuild. **Phase 3 (AddRemoveMix):** removes-then-adds within one apply — the remove pass runs in `Apply` (no rebuild), then the publish actor's Materialise passes create the adds + `AnnounceAddedNodes` announces them; an id reused across the remove+add sets is remove-then-recreated as a fresh NodeState (correct — a different tag). **Guard stays green (Commons.Tests 17/17), OpcUaServer.Tests 336/336, Runtime.Tests 398/398.** **03/P1 marked REMEDIATED in `archreview/03-server-runtime.md`.** **Deferred-live (T5, T6, T12, T14):** `SubscriptionSurvivalTests` add + remove + mixed cases (authored + compile-green; NOT run — `*.IntegrationTests` memory constraint) and the docker-dev live-`/run` add/remove/mixed gates (`kind=PureAdd/PureRemove/AddRemoveMix, rebuild=False`; survivor subscription uninterrupted; removed node → `BadNodeIdUnknown`) — MUST run in the serial heavy pass (F10b inertness risk). |
|
||||
@@ -221,10 +221,10 @@ Client.UI 126, Core.AlarmHistorian 29, Core.ScriptedAlarms 71, and all drivers (
|
||||
Every plan's integration-suite and live-`/run` gates were deferred (not skipped) so nothing OOMs the box and the
|
||||
shared docker-dev rig / VPN infra isn't contended. To run one-at-a-time:
|
||||
|
||||
- **Full integration-suite sweep** (10 `*.IntegrationTests` projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24.
|
||||
- **Full integration-suite sweep** (10 `*.IntegrationTests` projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24. **RAN 2026-07-14 — see [`INTEGRATION-SWEEP-STATUS.md`](INTEGRATION-SWEEP-STATUS.md).** Outcome: OpcUaClient(+Browser)/S7/Modbus-standard green, Host 105/117, TwinCAT clean-skip; the rest fail on **pre-existing test/fixture bit-rot** (AbCip static-init bug, Modbus pymodbus-4.x sim drift, FOCAS 127.0.0.1-mock topology, AbLegacy no fixture, OpcUaServer cert-store config, Host openldap:2.6 image gone). **No OtOpcUa regression.** 7 follow-ups in that file.
|
||||
- **docker-dev `:9200` live-`/run`** (rebuild BOTH centrals): R2-02 T15 (ResilienceConfig warning banner / input lockout / unknown-key survival), R2-03 T12 (VT fail→Bad→recover + P7 no-recompile), R2-10 T10 (breaker-open on both centrals + recovery + FOLLOWUP-10 log line), R2-11 T22 (typed-editor Writable checkbox), **R2-07 T12 (surgical add/remove/mixed — F10b inertness risk, MUST run)**, R2-05 T15 (over-gating regression only — deny unobservable under `DisableLogin`).
|
||||
- **2-node redundancy rig:** **R2-04 T13/T15 — behavior-affecting S4 primary-gate default-deny; MUST run before trusting the failover-role change.**
|
||||
- **Genuinely external / needs-user:** R2-06 T12 (VPN to a 0.2.0 HistorianGateway for the live EnsureTags retype probe), R2-08 live legs (real GLAuth outage/recovery cycle; `Category=LiveIntegration` gateway soak), R2-01 T11 (S7 SYN-blackhole outage on the `10.100.0.35` s7_1500 fixture).
|
||||
- **2-node redundancy rig:** ~~R2-04 T13/T15~~ **BOTH ✅ PASSED 2026-07-14** — T15 live docker-dev gate + T13 in-process delivery-path test (see below).
|
||||
- **Genuinely external / needs-user:** ~~ALL CLOSED as of 2026-07-15.~~ ~~R2-06 T12~~ (6/6 on the VPN, 2026-07-13). ~~R2-08 S3 `Category=LiveIntegration` gateway soak~~ **CLOSED 2026-07-15** (6/6 vs. the running local 0.2.0 gateway + real historian over VPN). ~~R2-01 T11~~ **CLOSED 2026-07-15** (S7 blackhole via `docker pause`; found + fixed the read-leg wall-clock gap, PR #453). ~~R2-08 GLAuth outage cycle~~ **CLOSED 2026-07-15** (`LdapGlauthOutageLiveTests` vs. the harness GLAuth paused via `docker pause`, PR #455). **Every Round-2 live gate is now GREEN.**
|
||||
|
||||
### Deferred behavior-change follow-up (deliberately not shipped)
|
||||
**R2-11 Phase C** — flip the runtime driver parsers to strict (typo'd enum ⇒ `BadNodeIdUnknown` instead of a
|
||||
@@ -234,9 +234,22 @@ wrong-width `Good`). Land only after fleets have run `Deployment:TagConfigValida
|
||||
|
||||
**Bottom line:** R2-02, R2-11, R2-07 (both surgical phases), R2-04 (achievable failover), and R2-10 (resilience pipeline in
|
||||
dispatch) verified live on the docker-dev central cluster. R2-05 partial-by-design; R2-03 unit-verified (rig has no driver
|
||||
data). Everything still open needs infra this box lacks: the `*.IntegrationTests` sweep needs the `10.100.0.35` driver
|
||||
fixtures + ephemeral LDAP up; R2-06 needs VPN to a 0.2.0 gateway; R2-08 a live GLAuth outage cycle; R2-01 the S7 blackhole
|
||||
fixture. Those are the only Round-2 items not yet exercised.
|
||||
data). R2-01's S7 blackhole gate is **CLOSED (2026-07-15)** — run via `docker pause` on the `10.100.0.35` s7_1500 fixture;
|
||||
it found + fixed the read-leg wall-clock gap (PR #453). R2-08's GLAuth outage-cycle leg is **CLOSED (2026-07-15, PR #455)** —
|
||||
`LdapGlauthOutageLiveTests` vs. the harness GLAuth on `10.100.0.35`. R2-08 S3 `Category=LiveIntegration` gateway soak is
|
||||
**CLOSED (2026-07-15, 6/6)** and R2-06 (6/6, 2026-07-13) — both vs. the local 0.2.0 gateway + real historian over VPN.
|
||||
**Every Round-2 live gate is now GREEN — no Round-2 items remain unexercised.**
|
||||
|
||||
### Residual follow-ups — tracked in Gitea (2026-07-15)
|
||||
|
||||
The remaining non-blocking items are filed as issues (none gate the remediation, which is complete):
|
||||
|
||||
| Issue | Item | Nature |
|
||||
|---|---|---|
|
||||
| **#456** | #10 — observe retry/breaker firing in live dispatch; thread `ResilienceConfig` into the deploy artifact (tier-defaults only today) | niche / sub-gap |
|
||||
| **#457** | R2-11 Phase C — flip runtime parsers to strict, after fleets run `TagConfigValidationMode=Error` clean | deliberately deferred (operational rollout) |
|
||||
| **#458** | Remove the `NuGetAuditSuppress` for CVE-2025-6965 once a patched SQLitePCLRaw bundle ships | upstream-gated |
|
||||
| **#459** | SBR topology — crashing the OLDEST Akka node does not fail over | pre-registered user-deferred design gap |
|
||||
|
||||
Brought up the 2-node central cluster (`central-1`+`central-2`, SQL, traefik) from source at merged master `6aeb53a9`
|
||||
(single-tenant rig; drove the `:9200` AdminUI via browser + Client.CLI on `:4840`/`:4841`).
|
||||
@@ -275,8 +288,57 @@ Brought up the 2-node central cluster (`central-1`+`central-2`, SQL, traefik) fr
|
||||
|
||||
**Rig note:** the pass mutated the persisted ConfigDb (added a `drv-modbus-line3-01` Modbus driver on MAIN; removed
|
||||
`capper-03`'s `production-count` virtual tag for the PureRemove test). Reset with
|
||||
`docker compose -f docker-dev/docker-compose.yml down -v`. The remaining integration-suite sweep + genuinely-external
|
||||
gates (R2-06 VPN, R2-08 live LDAP, R2-01 S7 blackhole) still need their infra.
|
||||
`docker compose -f docker-dev/docker-compose.yml down -v`. R2-01's S7 blackhole gate is now CLOSED (2026-07-15, PR #453);
|
||||
**all Round-2 live gates are now GREEN** (R2-06 + R2-08 S3 verified on the VPN 2026-07-15; R2-01 + R2-08 GLAuth self-serviced on `10.100.0.35`).
|
||||
|
||||
---
|
||||
|
||||
## R2-04 T15 — 2-node primary-gate LIVE GATE ✅ PASSED (2026-07-14)
|
||||
|
||||
Ran the S4 primary-gate live recipe on the docker-dev 2-node central cluster **rebuilt from master `365a90e6`**
|
||||
(`up -d --build central-1 central-2`). Brought up the **Modbus sim** (`otopcua-pymodbus-standard` on
|
||||
`10.100.0.35:5020`) so `MAIN`'s Modbus equipment tags read **Good** (a data-less rig materialises every node
|
||||
`Bad_WaitingForInitialData`, and the CLI write reads-first — so a live driver is required to have a writable-Good
|
||||
node to drive the gate). Made `EQ-55297329838d/ModbusHR200` writable (`TagConfig.writable=true` via SQL +
|
||||
`POST :9200/api/deployments`). Data-plane writes bound `multi-role`/`password` (carries the `WriteOperate` LDAP role).
|
||||
|
||||
**All core assertions PASS:**
|
||||
- **Step 1 — ServiceLevel split:** `Server_ServiceLevel` (`i=2267`) = **250 on `:4840` (primary) / 240 on `:4841`
|
||||
(secondary)** — differentiated, no split-brain. (Plan said 240/100; the deployment's healthy values are 250/240 —
|
||||
the invariant "primary > secondary" holds.)
|
||||
- **Step 2 — boot-window default-DENY (the NEW S4 behavior):** restarting **both** centrals (to widen the
|
||||
role-null-with-≥2-members window — a single-node restart's window is sub-second because cluster-join and the first
|
||||
redundancy snapshot land near-simultaneously) caught, ~5 s into boot:
|
||||
`WRN Operator write to EQ-55297329838d/ModbusHR200 rejected: not primary (role unknown)`. Meter
|
||||
`otopcua_redundancy_primary_gate_denied_total{reason="role-unknown",site="write"} = 1`.
|
||||
- **Step 3 — steady-state secondary DENY + #5 revert:** write `9999` to `:4841` → CLI reports optimistic
|
||||
`Write successful` (synchronous, by design) but read-back = **1234 (the primary's value, reverted — NOT 9999)**;
|
||||
central-2 logs `rejected: not primary`; meter `reason="secondary",site="write" = 1`.
|
||||
- **Step 3b — primary write reaches the device:** write `1234` to `:4840` → `Write successful` and read-back on
|
||||
`:4841` = `1234` (the Modbus driver wrote through).
|
||||
- **T4 sanity — clean apply:** both centrals logged `AddressSpaceApplier: applied plan (kind=PureAdd … failed=0)` /
|
||||
`materialised (… failed=0)`; **no `[ERR]` / `apply.failed` / degraded-apply line**; the `apply_failed` meter is
|
||||
absent (never incremented).
|
||||
- **Step 4 — `/alerts`:** endpoint 200, **0 rows** — no active alarm on the data-less rig, so trivially no
|
||||
dual-primary double-emission (the double-row regression is only observable with a firing alarm, which needs live
|
||||
driver data this rig lacks; the alerts-emit gate shares the same `PrimaryGatePolicy` and is unit-covered).
|
||||
|
||||
**Net:** the R2-04 `PrimaryGatePolicy` membership-aware **default-DENY-on-unknown** is confirmed live end-to-end —
|
||||
both denial reasons (`role-unknown`, `secondary`) fire the log + the new meter, the #5 optimistic-write revert
|
||||
corrects the node, and the primary path is unaffected. The behavior-affecting S4 slice is trusted.
|
||||
|
||||
**R2-04 T13 — in-process delivery-path test ✅ PASSED (2026-07-14).**
|
||||
`dotnet test …Host.IntegrationTests --filter FullyQualifiedName~PrimaryGateFailoverTests` → **1/1, 3 s**. Ran
|
||||
**without SQL** (`OTOPCUA_HARNESS_USE_SQL` unset) — the gate test needs only the in-process 2-node Akka cluster +
|
||||
the DPS-delivered redundancy snapshot, not a ConfigDb deploy. The `TwoNodeClusterHarness` formed the cluster and
|
||||
the **real DistributedPubSub-delivered** snapshot promoted exactly one node to Primary (Secondary's
|
||||
`DriverHostActor.RouteNodeWrite` → `not primary`; Primary passed the gate, rejecting only with a mapping reason).
|
||||
Its built-in negative control (a broken redundancy-topic subscribe leaves both stuck `role unknown` → 45 s timeout)
|
||||
means the PASS proves genuine snapshot delivery drives the gate — the half the pure-TestKit unit guards can't cover.
|
||||
|
||||
**Rig mutations (this pass):** `ModbusHR200.TagConfig.writable=true`; Modbus sim brought up on the shared docker host
|
||||
(`10.100.0.35`). Reset the central rig with `docker compose -f docker-dev/docker-compose.yml down -v`; the sim with
|
||||
`ssh dohertj2@10.100.0.35 'cd /opt/otopcua-modbus && docker compose --profile standard down'`.
|
||||
|
||||
---
|
||||
|
||||
@@ -327,24 +389,31 @@ convention) + a regression unit test + `StorageRateMs` on the live-suite inline
|
||||
- ✅ `Write_then_read_on_sandbox_tag` — `EnsureTags(Float)` + `WriteLiveValues` (SQL) + read-back.
|
||||
- ✅ `Alarm_SendEvent_is_acked` + ✅ `Alarm_SendEvent_then_ReadEvents` — alarm `SendEvent` + SQL `ReadEvents`.
|
||||
- ✅ `EnsureTags_type_change_outcome_is_observable` — now observes a **real** retype outcome (not a storage-rate error).
|
||||
- ❌ `EnsureTags_boolean_sandbox_provisions_as_Int1` — **Int1 (Boolean) tag creation is not supported by the
|
||||
gateway's histsdk on this historian**: `EnsureTags(Int1)` returns `ProtocolEvidenceMissingException` (the vendored
|
||||
client's "no capture evidence for this type" guard → gRPC `Unimplemented`), both fresh and as a retype target.
|
||||
A **type sweep** confirmed it is Int1-specific: **Int2/UInt2/Int4/UInt4/Float/Double all provision `success=True`;
|
||||
only Int1 fails.** Root cause: `histsdk` `HistorianTagWriteProtocol.GetAnalogDataTypeCode` has no captured
|
||||
CTagMetadata wire code for Int1. This is a **gateway/histsdk type-support gap**, not an OtOpcUa defect — **filed as
|
||||
[historiangw#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11)**.
|
||||
- ⚠️ `EnsureTags_boolean_sandbox_provisions_as_Int1` (as originally written) — **Int1 (Boolean) tag creation is not
|
||||
supported by the gateway's histsdk on this historian**: `EnsureTags(Int1)` returns `ProtocolEvidenceMissingException`
|
||||
(the vendored client's "no capture evidence for this type" guard → gRPC `Unimplemented`), both fresh and as a retype
|
||||
target. A **type sweep** confirmed it is Int1-specific: **Int2/UInt2/Int4/UInt4/Float/Double all provision
|
||||
`success=True`; only Int1 fails.** Root cause: `histsdk` `HistorianTagWriteProtocol.GetAnalogDataTypeCode` has no
|
||||
captured CTagMetadata wire code for Int1 — a **gateway/histsdk type-support gap**, not an OtOpcUa defect
|
||||
(**filed as [historiangw#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11)**, which routed the
|
||||
decision back as **[issue #441](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/441)**).
|
||||
|
||||
**✅ RESOLVED (issue #441, 2026-07-14):** `HistorianTypeMapper` now maps **`Boolean → Int2`** (0/1 integer) instead
|
||||
of the degenerate `Int1`. Int2 is a proven-creatable analog type on this historian, and the value-write path already
|
||||
sends every value as a `double` on `WriteLiveValues` regardless of tag type, so Boolean `0.0/1.0` round-trips
|
||||
cleanly. The live gate test is renamed `EnsureTags_boolean_sandbox_provisions_as_Int2` and now passes → **6/6**. Unit
|
||||
pins updated (`HistorianTypeMapperTests`, `GatewayTagProvisionerTests.Boolean_definition_carries_explicit_Int2_presence`);
|
||||
gateway driver unit `103/103`.
|
||||
|
||||
**✅ U-7 retype policy — ANSWERED (via `grpcurl` against the real historian):**
|
||||
- `EnsureTags` is a **real create-or-update upsert** — re-`EnsureTags` of an existing tag at the same type → `success`.
|
||||
- An **in-place retype between SUPPORTED types is ACCEPTED** — `Float→Double` on an existing tag returns `success`
|
||||
(of the three 06/U-7 hypotheses: **accepted-retyped**, not rejected).
|
||||
- **But the specific 0.2.0 Boolean case (Float→Int1) can't retype** — Int1 isn't a creatable type here, so a
|
||||
pre-0.2.0 **Float Boolean tag asked to become Int1 fails the provision** with `ProtocolEvidenceMissingException`
|
||||
— it is **neither silently retyped nor data-lost**; the provision just fails (and is metered `failed=N`). That is
|
||||
the practically-relevant U-7 answer for Boolean tags on this deployment.
|
||||
- **The specific `Int1` case can't retype** — Int1 isn't a creatable type here. Rather than accept a broken Boolean
|
||||
path, OtOpcUa **remapped `Boolean → Int2`** (issue #441, above), which retypes cleanly from a pre-existing Float
|
||||
Boolean tag (both are supported analog types) and never hits the Int1 gap.
|
||||
|
||||
The R2-06 **code** deliverables (S-11 fail-fast + `ServerHistorianOptionsValidator`; U-7 Boolean→Int1 mapping +
|
||||
The R2-06 **code** deliverables (S-11 fail-fast + `ServerHistorianOptionsValidator`; Boolean provisioning mapping +
|
||||
upsert; C-7 doc) were already unit-covered + merged; this live gate additionally **found + fixed the StorageRateMs
|
||||
provisioning bug** and **answered the U-7 retype policy**. The one remaining red test is a documented gateway
|
||||
histsdk Int1-support gap.
|
||||
provisioning bug**, **answered the U-7 retype policy**, and surfaced the Int1 histsdk gap that drove the
|
||||
**Boolean→Int2 remap (issue #441)** — closing the last red test to reach **6/6**.
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
# Follow-up #10 / issue #456 — live proof that authored ResilienceConfig reaches the runtime pipeline
|
||||
|
||||
**Date:** 2026-07-15 · **Rig:** docker-dev (central-1 + central-2, MAIN cluster, Warm/2-node) built from source.
|
||||
|
||||
## What this proves
|
||||
|
||||
The #456 "sub-gap" worried that per-instance `DriverInstance.ResilienceConfig` never reaches the
|
||||
runtime resilience pipeline (so it runs tier-defaults only). This is **disproven** — both by a new
|
||||
regression test and by this live capture.
|
||||
|
||||
## Setup
|
||||
|
||||
Seeded on MAIN (SQL): an Equipment namespace `ns-res` + an `OpcUaClient` driver `drv-res` carrying a
|
||||
**non-default** override:
|
||||
|
||||
```json
|
||||
"ResilienceConfig": {"capabilityPolicies":{"Subscribe":{"retryCount":999}}}
|
||||
```
|
||||
|
||||
`retryCount:999` is deliberately over the cap (100) so the parser emits an unmistakable clamp
|
||||
diagnostic that echoes the authored value. Deployed headless: `POST /api/deployments` → Accepted
|
||||
(deploymentId 80b4be3c…).
|
||||
|
||||
## Evidence
|
||||
|
||||
**1. The persisted deployment artifact bytes carry it** (`dbo.Deployment.ArtifactBlob`, 4055 bytes):
|
||||
`CHARINDEX('ResilienceConfig') > 0` = YES, `CHARINDEX('999') > 0` = YES; the `drv-res` object slice
|
||||
shows `…"DriverType":"OpcUaClient"…"DriverConfig":"…"…"Resil…"`.
|
||||
|
||||
**2. The runtime parsed + applied it — on BOTH redundancy nodes:**
|
||||
|
||||
```
|
||||
central-1 [WRN] Driver resilience config for instance=drv-res type=OpcUaClient: Subscribe.retryCount 999 exceeds the cap (100); clamped.
|
||||
central-1 [INF] DriverHost central-1:4053: spawned OpcUaClient driver drv-res (stub=False)
|
||||
central-2 [WRN] Driver resilience config for instance=drv-res type=OpcUaClient: Subscribe.retryCount 999 exceeds the cap (100); clamped.
|
||||
central-2 [INF] DriverHost central-2:4053: spawned OpcUaClient driver drv-res (stub=False)
|
||||
```
|
||||
|
||||
The authored value `999` appears verbatim in the runtime log. If the artifact did not carry
|
||||
`ResilienceConfig` (the feared sub-gap), `spec.ResilienceConfig` would be null, the parser would take
|
||||
pure tier-defaults, and **no clamp diagnostic would fire**. The diagnostic firing = the authored
|
||||
policy rode DB → ConfigComposer → artifact → DriverHostActor → DriverCapabilityInvokerFactory.Create →
|
||||
DriverResilienceOptionsParser. This is arch-review follow-up #10 acceptance item 2 ("re-verify the
|
||||
behavioral gate with a non-default policy"), verified live on both nodes.
|
||||
|
||||
## Regression guard
|
||||
|
||||
`ConfigComposerTests.ResilienceConfig_survives_ConfigComposer_to_ParseDriverInstances_round_trip`
|
||||
(master 809e7886) proves the same composer→artifact→parse leg deterministically, guarding against a
|
||||
future projection / `[JsonIgnore]` silently reverting authored policy to tier defaults.
|
||||
|
||||
## Still open (acceptance item 1)
|
||||
|
||||
The raw retry/breaker log line **on an actual driver fault** was NOT captured. It requires a
|
||||
connect-then-fault-mid-flight trigger: the wrapped capability calls (`Subscribe`/`Discover`/`Write`)
|
||||
only fire after a driver reaches **Connected** (`DriverInstanceActor` `ResubscribeDesired` runs on the
|
||||
`Become(Connected)` transition), so a driver at a dead endpoint stays in Reconnecting and never
|
||||
exercises the wrapped path. The reliable trigger is an **idempotent Write to a Connected-then-killed
|
||||
driver** (LDAP-gated Client.CLI write) or an S7 dead-socket reconnect. The retry/breaker **behaviour**
|
||||
is already proven deterministically (`CapabilityInvokerTests` retry, the pipeline-builder logging test,
|
||||
`DriverCapabilityInvokerFactoryTests` override-applies-to-behaviour). This remaining line is
|
||||
operator-confidence only.
|
||||
@@ -0,0 +1,44 @@
|
||||
# #459 — LIVE finding: keep-oldest downs the survivor when the oldest crashes (2-container rig)
|
||||
|
||||
**Date:** 2026-07-15 · **Rig:** docker-dev, central-1 (seed/oldest, :4840) + central-2 (:4841), MAIN Warm/2-node.
|
||||
|
||||
## Headline
|
||||
On a **real 2-container rig**, hard-killing the **oldest** node makes the **younger survivor down ITSELF**
|
||||
→ both processes exit → **no failover, total outage**. This CONFIRMS #459 (the premise is real, not stale)
|
||||
and **disproves Option A** (keep `keep-oldest` is insufficient). Reproduced twice, deterministically.
|
||||
|
||||
## Baseline (both nodes healthy, converged)
|
||||
- `Server.ServiceLevel` (NodeId `i=2267`): central-1 (oldest) = **250** (healthy leader), central-2 = **240** (healthy follower). Matches the documented tiering.
|
||||
|
||||
## The kill
|
||||
`docker kill otopcua-dev-central-1-1` (true process death, not a graceful Leave). central-2'"'"'s SBR:
|
||||
|
||||
```
|
||||
[WRN] SBR took decision Akka.Cluster.SBR.DownReachable and is downing
|
||||
[akka.tcp://otopcua@central-2:4053] including myself, [1] unreachable of [2] members
|
||||
[INF] Cluster Node - Marking node [central-2:4053] as [Down]
|
||||
[INF] Self downed, stopping ClusterSingletonManager
|
||||
[INF] CoordinatedShutdown invoked due to [ClusterDowningReason]. Exiting with [ExitCode:3].
|
||||
```
|
||||
|
||||
Result: central-2 Exited (255); `opc.tcp://localhost:4841` → Connection refused. No survivor.
|
||||
|
||||
## Why (this is textbook keep-oldest, not a misconfig)
|
||||
`keep-oldest` keeps the partition CONTAINING the oldest and downs the other side. When the oldest CRASHES,
|
||||
the survivor (younger) is "the side without the oldest" → it downs itself. `down-if-alone = on` does NOT
|
||||
rescue it: that knob only makes the OLDEST down itself when the oldest is isolated-but-alive; a crashed
|
||||
oldest can'"'"'t execute it, and the younger applies strict keep-oldest → `DownReachable` (self-down). This is
|
||||
the documented 2-node keep-oldest limitation; Akka recommends 3+ members / a real quorum.
|
||||
|
||||
## Why the in-process test gave false confidence
|
||||
`HardKillFailoverTests.Hard_kill_of_oldest_node_fails_over_to_survivor` PASSES, but `HardKillNodeAAsync`
|
||||
only calls `provider.Transport.Shutdown()` — node A'"'"'s **ActorSystem stays alive** (a transport partition
|
||||
with the oldest still running), NOT a process crash. So the in-process test exercises a different scenario
|
||||
than a real `docker kill`. It should be treated as non-representative for the oldest-CRASH case
|
||||
(test-quality follow-up: make it kill the process, or reframe what it proves).
|
||||
|
||||
## Consequence
|
||||
Option A (keep `keep-oldest`) does not provide oldest-crash failover on a real 2-node deployment. The fix
|
||||
requires **Option B** (3rd witness/lighthouse member + `keep-majority` — survives any single crash) or
|
||||
**Option C** (DB-lease arbiter decoupling Primary from Akka oldest). A 2-node cluster fundamentally cannot
|
||||
auto-survive an arbitrary single-node crash under any SBR strategy without a tiebreaker.
|
||||
@@ -133,6 +133,14 @@ services:
|
||||
# host raise the VM memory or run fewer host services.
|
||||
mem_limit: 2g
|
||||
mem_reservation: 1g
|
||||
# Restart supervision. A 2-node keep-oldest SBR downs the LONE survivor when the
|
||||
# OLDEST node crashes (the survivor is "the side without the oldest" → DownReachable
|
||||
# → run-coordinated-shutdown-when-down terminates it). Recovery is by exit-and-rejoin:
|
||||
# the supervisor restarts the exited node and it re-forms/rejoins as a fresh
|
||||
# incarnation (the ScadaBridge pattern). Without a restart policy the exited node
|
||||
# stays down and the cluster looks like a total outage. Inherited by every host node
|
||||
# via the anchor merge.
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
sql: { condition: service_healthy }
|
||||
migrator: { condition: service_completed_successfully }
|
||||
@@ -143,7 +151,11 @@ services:
|
||||
Cluster__Hostname: "0.0.0.0"
|
||||
Cluster__Port: "4053"
|
||||
Cluster__PublicHostname: "central-1"
|
||||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
|
||||
# re-join the mesh via EITHER peer, not only central-1. With restart supervision this is
|
||||
# the 2-node keep-oldest exit-and-rejoin recovery path.
|
||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
|
||||
Cluster__Roles__0: "admin"
|
||||
Cluster__Roles__1: "driver"
|
||||
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
||||
@@ -204,7 +216,9 @@ services:
|
||||
Cluster__Hostname: "0.0.0.0"
|
||||
Cluster__Port: "4053"
|
||||
Cluster__PublicHostname: "central-2"
|
||||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1.
|
||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
|
||||
Cluster__Roles__0: "admin"
|
||||
Cluster__Roles__1: "driver"
|
||||
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
||||
|
||||
+32
-36
@@ -141,48 +141,44 @@ for this node), which is the signal to check that `ServerHistorian:Enabled=true`
|
||||
> so a HistoryRead on it runs to the full `CallTimeout` (~30 s) before erroring. Auto-provisioning on deploy
|
||||
> keeps freshly-historized tags from hitting that slow path. See the `CallTimeout` row above.
|
||||
|
||||
### 0.2.0 DataType provisioning delta (archreview 06/U-7)
|
||||
### Boolean historization maps to Int2 (issue #441)
|
||||
|
||||
`EnsureTags` is contract-documented (HistorianGateway 0.2.0) as an **upsert** — *"creates or updates
|
||||
`EnsureTags` is contract-documented (HistorianGateway 0.2.0+) as an **upsert** — *"creates or updates
|
||||
historian tags according to the given definitions"* — that maps per-tag onto AVEVA's
|
||||
`HistorianClient.EnsureTagAsync(definition) -> bool`. It is **not idempotent** (not routed through the
|
||||
idempotent retry pipeline) and **not skip-if-exists**.
|
||||
|
||||
The 0.1.0→0.2.0 client bump made `HistorianTagDefinition.DataType` **proto3-optional** (explicit presence).
|
||||
Under 0.1.0, the provisioner's `Boolean → Int1` mapping sent wire value `0`, indistinguishable from unset, so
|
||||
the gateway defaulted such tags to the SDK-default **Float**. Under 0.2.0 the same call carries explicit
|
||||
presence, so Boolean tags now provision as **Int1** (the pre-existing intent of `HistorianTypeMapper` — no
|
||||
mapper code change is needed or wanted). Consequences:
|
||||
|
||||
1. **New Boolean tags provision correctly as Int1.** This is the fix arriving as a silent, zero-diff
|
||||
semantic change from the bump.
|
||||
2. **Pre-bump Boolean tags exist historian-side as Float.** On the next deploy that dispatches provisioning
|
||||
for such a tag, the gateway is asked (via the upsert) to update it to Int1.
|
||||
`HistorianTypeMapper` maps **`DriverDataType.Boolean → HistorianDataType.Int2`** and Boolean values
|
||||
historize as a small **0/1 integer**. Int2 is used rather than the semantically-exact `Int1` because the
|
||||
historian's **`Int1` analog-tag creation path is server-degenerate** — `EnsureTags(Int1)` stores an
|
||||
unusable stub, so a Boolean tag mapped to `Int1` never becomes historizable. This is an evidence-based
|
||||
decision from a live probe against the deployed 2023 R2 historian; the root cause and gateway-side
|
||||
resolution are tracked in [issue #441](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/441) and
|
||||
HistorianGateway [#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11) / PR #16 (which
|
||||
documents `Int1` as an explicit, evidence-based exclusion). The value-write path is unaffected: the
|
||||
recorder sends every value as a `double` on `WriteLiveValues` regardless of tag type, so a Boolean
|
||||
`0.0/1.0` round-trips cleanly into the Int2 tag.
|
||||
|
||||
**Observed live (2026-07-13, real 0.2.0 gateway → real AVEVA historian on `wonder-sql-vd03`, probed via
|
||||
`grpcurl` — archreview R2-06 T12):**
|
||||
- `EnsureTags` is a **real create-or-update upsert** — re-`EnsureTags` of an existing tag at the same type
|
||||
returns `Success=true`.
|
||||
- An **in-place retype between SUPPORTED types is ACCEPTED** — `Float→Double` on an existing tag returns
|
||||
`Success=true` (hypothesis **(ii) accepted as a plain retype**, not rejected).
|
||||
- **BUT `Int1` — the 0.2.0 Boolean target — is *not a creatable type* on this historian's histsdk:**
|
||||
`EnsureTags(Int1)` returns `Success=false, Error="ProtocolEvidenceMissingException"` (the vendored client's
|
||||
"no capture evidence for this type" guard → gRPC `Unimplemented`), both for a fresh tag and as a retype
|
||||
target. A type sweep confirmed it is **Int1-specific**: `Int2/UInt2/Int4/UInt4/Float/Double` all provision
|
||||
`Success=true`; only `Int1` fails.
|
||||
- **Net for a pre-bump Float Boolean tag asked to become Int1:** the provision **fails** (per-tag
|
||||
`Success=false` → `failed=N` tally). It is **neither silently retyped nor data-lost** — the tag stays Float
|
||||
and remains readable; the Int1 provision simply doesn't land until the gateway's histsdk gains `Int1`
|
||||
create-evidence — filed as
|
||||
[historiangw#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11). (Prerequisite for reaching this
|
||||
outcome at all: the provisioner must stamp a **positive
|
||||
`StorageRateMs`** — a `0` made `EnsureTags` throw `ArgumentOutOfRangeException` and fail *every* provision;
|
||||
fixed in `GatewayTagProvisioner`, PR #439.)
|
||||
3. **The failure mode is bounded and observable either way.** Provisioning is fire-and-forget and never
|
||||
blocks/fails a deploy; a per-tag rejection surfaces only as the tally's `failed` count. Values written to
|
||||
a still-Float tag by the recorder remain readable (Float supersets Int1's 0/1 range); nothing is lost
|
||||
pending a reconcile. A bulk retype sweep is **not** done in code — assess via the probe first, then decide
|
||||
whether reconciliation is a one-time operator action.
|
||||
`Success=true`.
|
||||
- A type sweep confirmed **`Int2/UInt2/Int4/UInt4/Float/Double` all provision `Success=true`; only `Int1`
|
||||
fails** (`Success=false, Error="ProtocolEvidenceMissingException"` — the vendored client's "no capture
|
||||
evidence for this type" guard → gRPC `Unimplemented`), both for a fresh tag and as a retype target. Mapping
|
||||
Boolean to `Int2` sidesteps that gap entirely.
|
||||
|
||||
> **Prerequisite for provisioning to land at all:** the provisioner must stamp a **positive `StorageRateMs`**
|
||||
> — a `0` made `EnsureTags` throw `ArgumentOutOfRangeException` and fail *every* provision (fixed in
|
||||
> `GatewayTagProvisioner`, PR #439).
|
||||
|
||||
**Migration note for pre-existing Boolean tags.** A Boolean tag historian-side as Float (e.g. from the
|
||||
0.1.0-era default) is asked, on its next deploy, to upsert to Int2. Both are supported analog types, so the
|
||||
retype is accepted. Provisioning is fire-and-forget and never blocks/fails a deploy; a per-tag rejection (if
|
||||
any) surfaces only as the tally's `failed` count, and values written to a still-Float tag remain readable
|
||||
(Float supersets the 0/1 range) pending reconcile. A bulk retype sweep is **not** done in code.
|
||||
|
||||
---
|
||||
|
||||
@@ -407,11 +403,11 @@ The gateway-backed driver also ships an env-gated live suite (`Category=LiveInte
|
||||
`HISTGW_GATEWAY_ENDPOINT` / `HISTGW_GATEWAY_APIKEY` / `HISTGW_TEST_TAG` / `HISTGW_WRITE_SANDBOX_TAG` /
|
||||
`HISTGW_BOOL_SANDBOX_TAG` / `HISTGW_ALARM_SOURCE` env vars (it skips cleanly when they are absent).
|
||||
`HISTGW_BOOL_SANDBOX_TAG` (e.g. `HistGW.LiveTest.BoolSandbox`, a **dedicated** writable Boolean tag —
|
||||
never the shared Float write sandbox) drives the 0.2.0 Boolean `EnsureTags` leg
|
||||
(`EnsureTags_boolean_sandbox_provisions_as_Int1`) and the retype probe
|
||||
(`EnsureTags_type_change_outcome_is_observable`) that resolve the
|
||||
[0.2.0 DataType provisioning delta](#020-datatype-provisioning-delta-archreview-06u-7) open question. Running
|
||||
the full suite against a **0.2.0** gateway (all six tests, none skipped) also discharges 06/U-2's
|
||||
never the shared Float write sandbox) drives the Boolean historization leg
|
||||
(`EnsureTags_boolean_sandbox_provisions_as_Int2`) and the retype probe
|
||||
(`EnsureTags_type_change_outcome_is_observable`) that back the
|
||||
[Boolean historization maps to Int2](#boolean-historization-maps-to-int2-issue-441) decision. Running
|
||||
the full suite against the gateway (all six tests, none skipped) also discharges 06/U-2's
|
||||
"full documented run" for the historian suite.
|
||||
|
||||
See [AlarmHistorian.md](AlarmHistorian.md) for the alarm write path and
|
||||
|
||||
+33
-10
@@ -161,23 +161,46 @@ provider reads the `split-brain-resolver` HOCON block in `akka.conf`. On top of
|
||||
`ServiceCollectionExtensions.BuildClusterOptions` sets the typed `ClusterOptions.SplitBrainResolver`
|
||||
(`KeepOldestOption { DownIfAlone = true }`) to make the strategy **explicit in code** rather than relying on
|
||||
the framework default — it is reinforcing, not the sole activator, and yields the same effective behavior. So
|
||||
the cluster is **not** running `NoDowning`; hard-crashed nodes are downed and both the cluster singletons and
|
||||
the `driver` role-leader fail over. (Only an *explicit* `NoDowning`, e.g.
|
||||
the cluster is **not** running `NoDowning`; hard-crashed nodes are downed and the cluster recovers (how it
|
||||
recovers depends on *which* node was lost — see the table below). (Only an *explicit* `NoDowning`, e.g.
|
||||
`akka.cluster.downing-provider-class = ""`, would leave both redundancy sides at ServiceLevel 240
|
||||
indefinitely.) The HOCON block carries the tuning: `active-strategy = keep-oldest`, `stable-after = 15s`,
|
||||
`keep-oldest.down-if-alone = on`, `failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone`
|
||||
must stay consistent with the typed option; `stable-after` lives only in HOCON because the typed option can't
|
||||
express it).
|
||||
|
||||
`keep-oldest` is the correct strategy for a 2-node warm-redundancy pair: under a clean partition the oldest
|
||||
member (typically the long-running primary) stays up and the smaller (or younger) side downs itself within
|
||||
~`stable-after` seconds; `down-if-alone` downs a node that loses its only peer. `keep-majority`/`static-quorum`
|
||||
are wrong for two nodes (no majority in a 1-1 split). The `RedundancyStateActor` on the surviving partition
|
||||
re-computes from the post-partition `Cluster.State`, and a hard-crashed (not gracefully stopped) node
|
||||
triggers the same failover — verified by `HardKillFailoverTests` (its negative control confirms failover
|
||||
survives removing the typed option, and only breaks under an explicit `NoDowning`).
|
||||
`keep-oldest` is the correct strategy for a 2-node warm-redundancy pair (`keep-majority`/`static-quorum`
|
||||
are wrong for two nodes — no majority in a 1-1 split), but its two loss cases recover **differently**, and
|
||||
this is inherent to any 2-node cluster:
|
||||
|
||||
There is no operator-driven role swap during a partition. Failover is what the cluster does automatically.
|
||||
| Node lost | What keep-oldest does | Recovery |
|
||||
|---|---|---|
|
||||
| **Younger** node (crash or partition) | The oldest is on the surviving side → it stays up and downs the younger side (`DownUnreachable`). | **In-place, fast.** The oldest keeps its singletons + `driver` role-leadership; `RedundancyStateActor` re-computes from the post-loss `Cluster.State`. |
|
||||
| **Oldest** node (crash or partition) | The lone survivor is "the side **without** the oldest" → keep-oldest downs the **survivor itself** (`DownReachable "including myself"`), and `run-coordinated-shutdown-when-down = on` terminates it. `down-if-alone` does **not** rescue a lone survivor (its rescue branch needs ≥ 2 surviving members, so it is a 3+-node feature). | **Exit-and-rejoin.** The self-downed survivor **and** the crashed oldest are both restarted by the service supervisor and re-form / rejoin. There is a brief restart-window outage; there is **no** in-place takeover. |
|
||||
|
||||
**Recovery therefore depends on three things being in place** (the [ScadaBridge](../../ScadaBridge/CLAUDE.md)
|
||||
sister project runs the same pattern):
|
||||
|
||||
1. **A service supervisor that restarts the process** on exit — production `Install-Services.ps1` sets
|
||||
`sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000`; the docker-dev rig sets
|
||||
`restart: unless-stopped`. Without it a downed node stays down and an oldest-crash looks like a total outage.
|
||||
2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) — it
|
||||
watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
|
||||
`IHostApplicationLifetime.StopApplication()` so the process exits (and the supervisor restarts it) instead of
|
||||
idling forever with a dead actor system.
|
||||
3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join the mesh via either peer.
|
||||
|
||||
> **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
|
||||
> sole `driver` role-leader, and it passes — but it simulates the crash with `provider.Transport.Shutdown()`,
|
||||
> which leaves node A's `ActorSystem` **alive** (a transport partition with the oldest still running), not a
|
||||
> real process death. It is therefore **not** representative of an oldest-process crash; a real 2-container
|
||||
> `docker kill` of the oldest downs the survivor (verified 2026-07-15, see
|
||||
> `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`). Treat the recovery guarantee for
|
||||
> the oldest as "exit-and-rejoin under supervision," not "in-place failover."
|
||||
|
||||
There is no operator-driven role swap during a partition. Failover / recovery is what the cluster + supervisor
|
||||
do automatically. **Instant in-place takeover on *any* single-node loss requires 3+ members** (an odd cluster
|
||||
or a lightweight witness) — a deliberate future option, not the current 2-node posture.
|
||||
|
||||
## Primary data-plane gate (writes, acks, alerts emit)
|
||||
|
||||
|
||||
+193
@@ -0,0 +1,193 @@
|
||||
# Infrastructure — testing & deployment
|
||||
|
||||
Canonical index of every piece of infrastructure OtOpcUa's tests and deployments
|
||||
depend on: the shared Docker host, the driver integration fixtures, the env-gated
|
||||
**live-gate** recipes, the integration-test harness, the local dev rig, and the
|
||||
gateway/deploy setups. This file is the single entry point; deeper history lives in
|
||||
the linked docs.
|
||||
|
||||
> **Credentials below are dev-only** (per decision #137) — never production values.
|
||||
> Production uses Integrated Security / gMSA (SQL) and a real directory (LDAP).
|
||||
|
||||
**Related deep docs**
|
||||
- [`docs/v2/dev-environment.md`](../docs/v2/dev-environment.md) — full historical inventory + Windows bootstrap.
|
||||
- [`docs/v2/Galaxy.ParityRig.md`](../docs/v2/Galaxy.ParityRig.md) — mxaccessgw / Galaxy parity-rig runbook.
|
||||
- `docs/drivers/*-Test-Fixture.md` — per-driver fixture coverage maps.
|
||||
- [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/) — the shared GLAuth source-of-truth + deploy runbook.
|
||||
- `reference_histgw_local_deploy`, `reference_ww_historian_vm` (memory) — HistorianGateway deploy detail.
|
||||
|
||||
---
|
||||
|
||||
## 1. Shared Linux Docker host — `10.100.0.35`
|
||||
|
||||
All Docker workloads (central SQL, driver fixtures, integration-harness deps) run on a
|
||||
shared Debian host, **not** on the dev machine. Discover everything with the
|
||||
`project=lmxopcua` label:
|
||||
|
||||
```bash
|
||||
ssh dohertj2@10.100.0.35 'docker ps --filter label=project=lmxopcua --format "{{.Names}}\t{{.Status}}"'
|
||||
```
|
||||
|
||||
- **From this Mac**, drive it over passwordless SSH (`ssh dohertj2@10.100.0.35 'docker …'`).
|
||||
`docker -H ssh://…` does **not** work from the Windows VM (stdio bridge hangs) — the
|
||||
Windows VM uses the `lmxopcua-fix.ps1` helper (`~/bin`) instead; the Mac uses SSH directly.
|
||||
- **Stacks** live at `/opt/otopcua-<driver>/` (root-owned) or `~/otopcua-<driver>/` (user-owned,
|
||||
where `/opt` needs sudo). Current layout:
|
||||
|
||||
| Stack dir | Purpose |
|
||||
|---|---|
|
||||
| `/opt/otopcua-mssql` | central SQL (always-on) |
|
||||
| `/opt/otopcua-modbus` · `/opt/otopcua-abcip` · `/opt/otopcua-s7` · `/opt/otopcua-opcuaclient` | driver fixtures |
|
||||
| `~/otopcua-ablegacy` · `~/otopcua-focas` | driver fixtures (user-owned) |
|
||||
| `~/otopcua-harness` | Host.IntegrationTests real-mode deps (SQL + GLAuth) — see §4 |
|
||||
|
||||
- The repo's `tests/.../Docker/` trees are the **source of truth**; the host dirs are mirrored
|
||||
deployments. Push changes with `rsync`/`lmxopcua-fix sync` before `docker compose up`.
|
||||
|
||||
## 2. Central SQL Server (always-on)
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Container | `otopcua-mssql` (`mcr.microsoft.com/mssql/server:2022-latest`), stack `/opt/otopcua-mssql` |
|
||||
| Endpoint | `10.100.0.35:14330` → `1433` |
|
||||
| Creds (dev) | `sa` / `OtOpcUaDev_2026!` |
|
||||
| Used by | `appsettings.json` `ConfigDb`; DB-backed test suites (`*.Configuration.Tests`, `*.Admin.Tests`, `*.Server.Tests`) |
|
||||
|
||||
Nuclear reset: `ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mssql && docker compose down -v && docker compose up -d'`.
|
||||
|
||||
## 3. Driver integration fixtures
|
||||
|
||||
Each driver's integration suite TCP-probes its fixture at collection init and **skips cleanly**
|
||||
when it isn't running. Bring one up, `dotnet test`, tear down. Repo compose lives under
|
||||
`tests/Drivers/….IntegrationTests/Docker/`.
|
||||
|
||||
| Driver | Fixture image | Endpoint | Bring up (on the host) |
|
||||
|---|---|---|---|
|
||||
| Modbus | `otopcua-pymodbus:3.13.0` | `10.100.0.35:5020` | `docker compose --profile <standard\|dl205\|mitsubishi\|s7_1500> up -d` |
|
||||
| AB CIP | `otopcua-ab-server:libplctag-release` | `10.100.0.35:44818` | `docker compose --profile <controllogix\|compactlogix\|micro800\|guardlogix> up -d` |
|
||||
| AB Legacy (PCCC) | `otopcua-ab-server` (PCCC mode) | `10.100.0.35:44818` | `AB_LEGACY_COMPOSE_PROFILE=<…> docker compose up -d` (stack `~/otopcua-ablegacy`) |
|
||||
| S7 | `otopcua-python-snap7` | `10.100.0.35:1102` | `docker compose --profile s7_1500 up -d` |
|
||||
| OpcUaClient | `mcr.microsoft.com/iotedge/opc-plc:2.14.10` | `opc.tcp://10.100.0.35:50000` | `docker compose up -d` |
|
||||
| FOCAS | `otopcua-focas-sim` (vendored focas-mock) | `10.100.0.35:8193` | `docker compose up -d` (stack `~/otopcua-focas`) |
|
||||
|
||||
**Endpoint overrides** point a suite at a real PLC instead of the sim:
|
||||
`MODBUS_SIM_ENDPOINT` · `AB_SERVER_ENDPOINT` · `S7_SIM_ENDPOINT` · `OPCUA_SIM_ENDPOINT`.
|
||||
|
||||
> **Fixture-cycling gotcha:** profile-gated services can share a port; a plain `docker compose down`
|
||||
> may leave a stale container bound. Force-remove before switching profiles:
|
||||
> `docker rm -f $(docker ps -aq --filter name=otopcua-<driver>)` then `up --force-recreate`.
|
||||
|
||||
## 4. Host.IntegrationTests real-mode harness
|
||||
|
||||
`tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests` runs on EF-InMemory + a stub LDAP by
|
||||
default (zero infra). Real-mode swaps in **SQL Server** + a real **GLAuth** bind. Stack:
|
||||
`~/otopcua-harness` (repo: `tests/Server/.../Host.IntegrationTests/docker-compose.yml` + `glauth/config.toml`).
|
||||
|
||||
| Dep | Container | Endpoint | Notes |
|
||||
|---|---|---|---|
|
||||
| SQL | `otopcua-harness-sql-1` | `10.100.0.35:14331` → 1433 | port 14331 avoids the always-on 14330 |
|
||||
| GLAuth | `otopcua-harness-ldap-1` | `10.100.0.35:3894` → 3893 | GLAuth (swapped in for retired bitnami/openldap, PR #451); users below |
|
||||
|
||||
Harness GLAuth dev users (`glauth/config.toml`, baseDN `dc=zb,dc=local`):
|
||||
`serviceaccount`/`serviceaccount123` (search), `alice`/`alice123` (full-access), `bob`/`bob123` (read-only).
|
||||
|
||||
```bash
|
||||
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d'
|
||||
export OTOPCUA_HARNESS_USE_SQL=1 OTOPCUA_HARNESS_USE_LDAP=1
|
||||
export OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331
|
||||
export OTOPCUA_HARNESS_LDAP_HOST=10.100.0.35 # harness uses port 3894
|
||||
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests
|
||||
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose down'
|
||||
```
|
||||
|
||||
## 5. Live-gate recipes (env-gated destructive tests)
|
||||
|
||||
These prove failure/recovery behavior against **real** infra and skip cleanly offline. Each is
|
||||
gated by env vars and always restores the shared resource in a `finally`.
|
||||
|
||||
### 5a. S7 connect-timeout blackhole (`S7_1500ConnectTimeoutOutageTests`)
|
||||
Proves an S7 subscription survives a SYN-blackhole outage (frozen PLC), which a `docker restart`
|
||||
can't reproduce. `docker pause` gives the ACK-less-established shape — no sudo/VPN.
|
||||
|
||||
```bash
|
||||
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d'
|
||||
export S7_TIMEOUT_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-python-snap7-s7_1500"'
|
||||
export S7_TIMEOUT_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-python-snap7-s7_1500"'
|
||||
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.IntegrationTests --filter "FullyQualifiedName~ConnectTimeoutOutage"
|
||||
```
|
||||
|
||||
### 5b. LDAP GLAuth outage/recovery (`LdapGlauthOutageLiveTests`)
|
||||
Drives the directory-outage circuit through the production auth path against the harness GLAuth
|
||||
(§4), paused/unpaused. Also verifies the GLAuth swap binds correctly.
|
||||
|
||||
```bash
|
||||
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d ldap'
|
||||
export GLAUTH_LIVE_HOST=10.100.0.35 # port defaults to 3894 (GLAUTH_LIVE_PORT)
|
||||
export GLAUTH_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-harness-ldap-1"'
|
||||
export GLAUTH_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-harness-ldap-1"'
|
||||
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapGlauthOutageLive"
|
||||
```
|
||||
|
||||
### 5c. HistorianGateway LiveIntegration soak (`Category=LiveIntegration`) — needs VPN
|
||||
Exercises the gateway read/write/alarm/EnsureTags paths against a **real AVEVA historian**. Requires
|
||||
the corporate **VPN up** and a running gateway (see §6). Full deploy detail: `reference_histgw_local_deploy`.
|
||||
|
||||
```bash
|
||||
export HISTGW_GATEWAY_ENDPOINT=http://localhost:5221 # local gw, h2c
|
||||
export HISTGW_GATEWAY_ALLOW_UNTRUSTED=true # required for h2c
|
||||
export HISTGW_GATEWAY_APIKEY=histgw_placeholder_placeholder # gw ApiKeys:Mode=Disabled
|
||||
export HISTGW_TEST_TAG=SysTimeSec
|
||||
export HISTGW_WRITE_SANDBOX_TAG=HistGW.LiveTest.Sandbox
|
||||
export HISTGW_BOOL_SANDBOX_TAG=HistGW.LiveTest.BoolSandbox
|
||||
export HISTGW_ALARM_SOURCE=OtOpcUaLiveValidation
|
||||
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests --filter "Category=LiveIntegration"
|
||||
```
|
||||
|
||||
## 6. HistorianGateway — deploy for live testing
|
||||
|
||||
The gateway sidecar runs locally in Docker (`~/Desktop/HistorianGateway/docker/`) pointed at the
|
||||
**real AVEVA historian** over VPN. It's the backend for OtOpcUa's historian read/alarm/continuous paths.
|
||||
|
||||
- **Hosts (VPN up):** historian gRPC CAP `wonder-sql-vd03.zmr.zimmer.com:32565` (TLS); Runtime SQL `:1433`
|
||||
(`wonderapp`/`Wonderapp1`, SQL login); Galaxy GR SQL `wonder-app-vd03.zmr.zimmer.com:1433`.
|
||||
- **Deploy:** publish FDD on host, then `docker compose up -d --build`. gRPC h2c `localhost:5221`,
|
||||
dashboard/health `localhost:5220`, `ApiKeys:Mode=Disabled`. Corporate hostnames go in a gitignored
|
||||
`docker/.env` / secrets, never in committed compose.
|
||||
- **Known deploy fixes** (committed to the gateway repo): FQDN via `.env`; `libgssapi-krb5-2 gss-ntlmssp`
|
||||
in the Dockerfile (historian v6 + SQL paths auth `NAM\dohertj2` via Negotiate/GSSAPI); `RuntimeDb` on.
|
||||
- Full recipe + gotchas: `reference_histgw_local_deploy` · historian VM: `reference_ww_historian_vm`.
|
||||
|
||||
## 7. docker-dev local rig — `docker-dev/docker-compose.yml`
|
||||
|
||||
A 2-node OtOpcUa cluster built from source for AdminUI / OPC UA `/run` verification.
|
||||
|
||||
| Surface | Endpoint | Notes |
|
||||
|---|---|---|
|
||||
| AdminUI | `http://localhost:9200` (via Traefik) | round-robins central-1/2 — **rebuild both** to verify a UI change |
|
||||
| OPC UA | `opc.tcp://localhost:4840` (central-1) · `:4841` (central-2) | data plane |
|
||||
| Login | disabled (`Security__Auth__DisableLogin=true`) | auto-authenticated full-access admin — no sign-in needed |
|
||||
| LDAP (data plane) | shared GLAuth `10.100.0.35:3893` (`DevStubMode=false`) | gates Client.CLI read/write **role** ops only |
|
||||
|
||||
```bash
|
||||
docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2 traefik
|
||||
# headless deploy: POST http://localhost:9200/api/deployments (X-Api-Key)
|
||||
docker compose -f docker-dev/docker-compose.yml down -v # reset
|
||||
```
|
||||
|
||||
## 8. Shared GLAuth (data-plane auth) — `10.100.0.35:3893`
|
||||
|
||||
The project-wide LDAP for docker-dev + live OPC UA data-plane auth (distinct from the harness
|
||||
GLAuth on :3894). Container `zb-shared-glauth`, baseDN `dc=zb,dc=local`, bind
|
||||
`cn=serviceaccount,dc=zb,dc=local`/`serviceaccount123`, test users password `password`. **Source of
|
||||
truth + deploy runbook:** [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/).
|
||||
|
||||
## 9. Deployment (production rig)
|
||||
|
||||
The deploy host runs the OtOpcUa server + `mxaccessgw` + the HistorianGateway sidecar as
|
||||
NSSM-wrapped services from `C:\publish\`. Refresh after landing changes with
|
||||
`scripts/install/Refresh-Services.ps1` (stops reverse-dependency order → backs up → publishes →
|
||||
restarts → smoke-verifies ports 4840/4841). Production creds use Integrated Security / gMSA (SQL)
|
||||
and a real directory (LDAP) — never the dev values in this doc.
|
||||
|
||||
> The legacy Wonderware TCP/ArchestrA sidecar was retired; the historian backend is now the
|
||||
> `ZB.MOM.WW.HistorianGateway` sidecar (see §6 and `docs/Historian.md`).
|
||||
+6
@@ -13,6 +13,12 @@
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Data.Sqlite"/>
|
||||
<!--
|
||||
Direct pin of the native SQLite bundle to 2.1.12 (first version outside the
|
||||
CVE-2025-6965 / GHSA-2m69-gcr7-jv3q range) — overrides the 2.1.11 that
|
||||
Microsoft.Data.Sqlite pulls transitively. See Directory.Packages.props.
|
||||
-->
|
||||
<PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3"/>
|
||||
<PackageReference Include="Serilog"/>
|
||||
</ItemGroup>
|
||||
|
||||
|
||||
@@ -50,6 +50,8 @@ public sealed class SubscribeCommand : S7CommandBase
|
||||
{
|
||||
await driver.InitializeAsync("{}", ct);
|
||||
|
||||
// Route every data-change event to the CliFx console (not System.Console — the
|
||||
// analyzer flags it + IConsole is the testable abstraction).
|
||||
driver.OnDataChange += (_, e) =>
|
||||
{
|
||||
var line = $"[{DateTime.UtcNow:HH:mm:ss.fff}] " +
|
||||
|
||||
@@ -75,10 +75,10 @@ public static class AbCipDriverFactoryExtensions
|
||||
{
|
||||
Enabled = dto.Probe?.Enabled ?? true,
|
||||
Interval = TimeSpan.FromMilliseconds(dto.Probe?.IntervalMs ?? 5_000),
|
||||
Timeout = TimeSpan.FromMilliseconds(dto.Probe?.TimeoutMs ?? 2_000),
|
||||
Timeout = PositiveTimeoutOrDefault(dto.Probe?.TimeoutMs, 2_000),
|
||||
ProbeTagPath = dto.Probe?.ProbeTagPath,
|
||||
},
|
||||
Timeout = TimeSpan.FromMilliseconds(dto.TimeoutMs ?? 2_000),
|
||||
Timeout = PositiveTimeoutOrDefault(dto.TimeoutMs, 2_000),
|
||||
EnableControllerBrowse = dto.EnableControllerBrowse ?? false,
|
||||
EnableAlarmProjection = dto.EnableAlarmProjection ?? false,
|
||||
AlarmPollInterval = TimeSpan.FromMilliseconds(dto.AlarmPollIntervalMs ?? 1_000),
|
||||
@@ -86,6 +86,20 @@ public static class AbCipDriverFactoryExtensions
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Maps an operator-supplied timeout (ms) to a positive <see cref="TimeSpan"/>, substituting
|
||||
/// <paramref name="defaultMs"/> for a null / zero / negative value. libplctag's managed
|
||||
/// <c>Tag.Timeout</c> setter throws <see cref="ArgumentOutOfRangeException"/> for a
|
||||
/// non-positive value, which would otherwise fault tag creation on every read/write; clamping
|
||||
/// here keeps a misconfigured <c>TimeoutMs: 0</c> running on the default bound instead. (Driver
|
||||
/// timeout hardening, sibling of the S7 R2-01 read-leg fix.)
|
||||
/// </summary>
|
||||
/// <param name="ms">The operator-supplied timeout in milliseconds, or null.</param>
|
||||
/// <param name="defaultMs">The default applied when <paramref name="ms"/> is null or non-positive.</param>
|
||||
/// <returns>A strictly positive timeout.</returns>
|
||||
private static TimeSpan PositiveTimeoutOrDefault(int? ms, int defaultMs) =>
|
||||
TimeSpan.FromMilliseconds(ms is int v and > 0 ? v : defaultMs);
|
||||
|
||||
private static AbCipTagDefinition BuildTag(AbCipTagDto t, string driverInstanceId) =>
|
||||
new(
|
||||
Name: t.Name ?? throw new InvalidOperationException(
|
||||
|
||||
@@ -264,6 +264,8 @@ public sealed class AbLegacyDriver : IDriver, IReadable, IWritable, ITagDiscover
|
||||
|
||||
if (status != 0)
|
||||
{
|
||||
// Evict the stale handle so the next call re-creates it (mirrors AbCip).
|
||||
EvictRuntime(device, def.Name);
|
||||
results[i] = new DataValueSnapshot(null,
|
||||
AbLegacyStatusMapper.MapLibplctagStatus(status), null, now);
|
||||
_health = new DriverHealth(DriverState.Degraded, _health.LastSuccessfulRead,
|
||||
@@ -287,6 +289,8 @@ public sealed class AbLegacyDriver : IDriver, IReadable, IWritable, ITagDiscover
|
||||
catch (OperationCanceledException) { throw; }
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Transport exception — evict so the next read creates a fresh handle (mirrors AbCip).
|
||||
EvictRuntime(device, def.Name);
|
||||
results[i] = new DataValueSnapshot(null,
|
||||
AbLegacyStatusMapper.BadCommunicationError, null, now);
|
||||
_health = new DriverHealth(DriverState.Degraded, _health.LastSuccessfulRead, ex.Message);
|
||||
@@ -360,9 +364,16 @@ public sealed class AbLegacyDriver : IDriver, IReadable, IWritable, ITagDiscover
|
||||
opLock.Release();
|
||||
}
|
||||
|
||||
results[i] = new WriteResult(status == 0
|
||||
? AbLegacyStatusMapper.Good
|
||||
: AbLegacyStatusMapper.MapLibplctagStatus(status));
|
||||
if (status != 0)
|
||||
{
|
||||
// Evict the stale handle so the next call re-creates it (mirrors AbCip).
|
||||
EvictRuntime(device, def.Name);
|
||||
results[i] = new WriteResult(AbLegacyStatusMapper.MapLibplctagStatus(status));
|
||||
}
|
||||
else
|
||||
{
|
||||
results[i] = new WriteResult(AbLegacyStatusMapper.Good);
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) { throw; }
|
||||
catch (NotSupportedException nse)
|
||||
@@ -380,6 +391,8 @@ public sealed class AbLegacyDriver : IDriver, IReadable, IWritable, ITagDiscover
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Transport exception — evict so the next write/read creates a fresh handle (mirrors AbCip).
|
||||
EvictRuntime(device, def.Name);
|
||||
results[i] = new WriteResult(AbLegacyStatusMapper.BadCommunicationError);
|
||||
_health = new DriverHealth(DriverState.Degraded, _health.LastSuccessfulRead, ex.Message);
|
||||
}
|
||||
@@ -685,6 +698,26 @@ public sealed class AbLegacyDriver : IDriver, IReadable, IWritable, ITagDiscover
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Evict the runtime for <paramref name="tagName"/> from the device's cache and dispose it so
|
||||
/// the next read/write call re-creates and re-initializes a fresh libplctag handle. Called
|
||||
/// after a non-zero libplctag status or a transport exception on a read/write — a non-zero
|
||||
/// status can mean the controller dropped the connection or the handle became permanently
|
||||
/// invalid (e.g. after a PLC download), and reusing it would return the same failure forever.
|
||||
/// Mirrors <c>AbCipDriver.EvictRuntime</c> and the probe loop's recreate-on-failure behaviour
|
||||
/// (previously AbLegacy leaned only on the probe loop + libplctag's internal recovery, so a
|
||||
/// data-path fault recovered more slowly than in AbCip).
|
||||
/// </summary>
|
||||
/// <param name="device">The device whose runtime cache holds the stale handle.</param>
|
||||
/// <param name="tagName">The tag name key of the runtime to evict.</param>
|
||||
private static void EvictRuntime(DeviceState device, string tagName)
|
||||
{
|
||||
if (device.Runtimes.TryRemove(tagName, out var stale))
|
||||
{
|
||||
try { stale.Dispose(); } catch { /* best-effort */ }
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Synchronous teardown. Mirrors the body of
|
||||
/// <see cref="ShutdownAsync"/> but never wraps the async path in
|
||||
|
||||
@@ -83,10 +83,10 @@ public static class AbLegacyDriverFactoryExtensions
|
||||
{
|
||||
Enabled = dto.Probe?.Enabled ?? true,
|
||||
Interval = TimeSpan.FromMilliseconds(dto.Probe?.IntervalMs ?? 5_000),
|
||||
Timeout = TimeSpan.FromMilliseconds(dto.Probe?.TimeoutMs ?? 2_000),
|
||||
Timeout = PositiveTimeoutOrDefault(dto.Probe?.TimeoutMs, 2_000),
|
||||
ProbeAddress = dto.Probe?.ProbeAddress ?? "S:0",
|
||||
},
|
||||
Timeout = TimeSpan.FromMilliseconds(dto.TimeoutMs ?? 2_000),
|
||||
Timeout = PositiveTimeoutOrDefault(dto.TimeoutMs, 2_000),
|
||||
};
|
||||
|
||||
return new AbLegacyDriver(
|
||||
@@ -95,6 +95,20 @@ public static class AbLegacyDriverFactoryExtensions
|
||||
logger: loggerFactory?.CreateLogger<AbLegacyDriver>());
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Maps an operator-supplied timeout (ms) to a positive <see cref="TimeSpan"/>, substituting
|
||||
/// <paramref name="defaultMs"/> for a null / zero / negative value. libplctag's managed
|
||||
/// <c>Tag.Timeout</c> setter throws <see cref="ArgumentOutOfRangeException"/> for a
|
||||
/// non-positive value, which would otherwise fault tag creation on every read/write; clamping
|
||||
/// here keeps a misconfigured <c>TimeoutMs: 0</c> running on the default bound instead. (Driver
|
||||
/// timeout hardening, sibling of the S7 R2-01 read-leg fix.)
|
||||
/// </summary>
|
||||
/// <param name="ms">The operator-supplied timeout in milliseconds, or null.</param>
|
||||
/// <param name="defaultMs">The default applied when <paramref name="ms"/> is null or non-positive.</param>
|
||||
/// <returns>A strictly positive timeout.</returns>
|
||||
private static TimeSpan PositiveTimeoutOrDefault(int? ms, int defaultMs) =>
|
||||
TimeSpan.FromMilliseconds(ms is int v and > 0 ? v : defaultMs);
|
||||
|
||||
private static T ParseEnum<T>(string? raw, string driverInstanceId, string field,
|
||||
string? tagName = null, T? fallback = null) where T : struct, Enum
|
||||
{
|
||||
|
||||
@@ -84,9 +84,9 @@ public static class FocasDriverFactoryExtensions
|
||||
{
|
||||
Enabled = dto.Probe?.Enabled ?? true,
|
||||
Interval = TimeSpan.FromMilliseconds(dto.Probe?.IntervalMs ?? 5_000),
|
||||
Timeout = TimeSpan.FromMilliseconds(dto.Probe?.TimeoutMs ?? 2_000),
|
||||
Timeout = PositiveTimeoutOrDefault(dto.Probe?.TimeoutMs, 2_000),
|
||||
},
|
||||
Timeout = TimeSpan.FromMilliseconds(dto.TimeoutMs ?? 2_000),
|
||||
Timeout = PositiveTimeoutOrDefault(dto.TimeoutMs, 2_000),
|
||||
FixedTree = BuildFixedTree(dto.FixedTree),
|
||||
AlarmProjection = BuildAlarmProjection(dto.AlarmProjection),
|
||||
HandleRecycle = BuildHandleRecycle(dto.HandleRecycle),
|
||||
@@ -96,6 +96,20 @@ public static class FocasDriverFactoryExtensions
|
||||
return new FocasDriver(options, driverInstanceId, clientFactory);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Maps an operator-supplied timeout (ms) to a positive <see cref="TimeSpan"/>, substituting
|
||||
/// <paramref name="defaultMs"/> for a null / zero / negative value. A non-positive
|
||||
/// <c>Timeout</c> would make <see cref="SynchronizedFocasClient"/> DISABLE its per-call
|
||||
/// wall-clock ceiling (falling back to the caller's long-lived poll token), reintroducing the
|
||||
/// exact unbounded-read-on-a-frozen-peer wedge the S7 R2-01 read-leg fix eliminated. Clamping
|
||||
/// here at the config boundary guarantees the deadline can never be authored away.
|
||||
/// </summary>
|
||||
/// <param name="ms">The operator-supplied timeout in milliseconds, or null.</param>
|
||||
/// <param name="defaultMs">The default applied when <paramref name="ms"/> is null or non-positive.</param>
|
||||
/// <returns>A strictly positive timeout.</returns>
|
||||
private static TimeSpan PositiveTimeoutOrDefault(int? ms, int defaultMs) =>
|
||||
TimeSpan.FromMilliseconds(ms is int v and > 0 ? v : defaultMs);
|
||||
|
||||
/// <summary>
|
||||
/// Builds the appropriate <see cref="IFocasClientFactory"/> based on the config DTO's backend selection.
|
||||
/// </summary>
|
||||
|
||||
+10
-6
@@ -9,11 +9,14 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Mapping;
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Only the nine numeric types are historizable on the gateway's analog write path. Two of them
|
||||
/// fall back to a wider historian type because the narrower one's write path is deferred upstream:
|
||||
/// <see cref="DriverDataType.UInt16"/> maps to <see cref="HistorianDataType.Uint4"/> (the historian's
|
||||
/// <c>UInt2</c> write path is not proven). String / DateTime / Reference are not historized in v1
|
||||
/// and throw <see cref="NotSupportedException"/>; callers that want to skip them without catching an
|
||||
/// exception should consult <see cref="IsHistorizable(DriverDataType)"/> first.
|
||||
/// fall back to a wider/adjacent historian type because the exact one's write path is not usable
|
||||
/// upstream: <see cref="DriverDataType.UInt16"/> maps to <see cref="HistorianDataType.Uint4"/> (the
|
||||
/// historian's <c>UInt2</c> write path is not proven), and <see cref="DriverDataType.Boolean"/> maps
|
||||
/// to <see cref="HistorianDataType.Int2"/> — the historian's <c>Int1</c> analog-tag creation path is
|
||||
/// server-degenerate (issue #441 / HistorianGateway #11: <c>EnsureTags(Int1)</c> stores an unusable
|
||||
/// stub), so Boolean historizes as a small 0/1 integer instead. String / DateTime / Reference are not
|
||||
/// historized in v1 and throw <see cref="NotSupportedException"/>; callers that want to skip them
|
||||
/// without catching an exception should consult <see cref="IsHistorizable(DriverDataType)"/> first.
|
||||
/// </remarks>
|
||||
internal static class HistorianTypeMapper
|
||||
{
|
||||
@@ -25,7 +28,8 @@ internal static class HistorianTypeMapper
|
||||
/// </exception>
|
||||
public static HistorianDataType ToHistorianDataType(DriverDataType dataType) => dataType switch
|
||||
{
|
||||
DriverDataType.Boolean => HistorianDataType.Int1,
|
||||
DriverDataType.Boolean => HistorianDataType.Int2, // Int1 analog-tag creation is server-degenerate (issue #441) → store 0/1
|
||||
|
||||
DriverDataType.Int16 => HistorianDataType.Int2,
|
||||
DriverDataType.Int32 => HistorianDataType.Int4,
|
||||
DriverDataType.Int64 => HistorianDataType.Int8,
|
||||
|
||||
@@ -87,6 +87,7 @@ public interface IS7PlcFactory
|
||||
internal sealed class S7PlcAdapter : IS7Plc
|
||||
{
|
||||
private readonly Plc _plc;
|
||||
private readonly TimeSpan _operationTimeout;
|
||||
|
||||
/// <summary>Initializes a new instance of the <see cref="S7PlcAdapter"/> class.</summary>
|
||||
/// <param name="cpuType">The CPU family.</param>
|
||||
@@ -98,9 +99,14 @@ internal sealed class S7PlcAdapter : IS7Plc
|
||||
public S7PlcAdapter(S7CpuType cpuType, string host, int port, short rack, short slot, TimeSpan timeout)
|
||||
{
|
||||
_plc = new Plc(S7CpuTypeMap.ToS7Net(cpuType), host, port, rack, slot);
|
||||
_operationTimeout = timeout;
|
||||
// S7netplus writes timeouts into the underlying TcpClient via ReadTimeout / WriteTimeout
|
||||
// (milliseconds). Set before OpenAsync so the handshake itself honours the bound — matches
|
||||
// the original inline construction in S7Driver.InitializeAsync.
|
||||
// the original inline construction in S7Driver.InitializeAsync. NOTE: these map to the
|
||||
// TcpClient's ReceiveTimeout / SendTimeout, which govern only SYNCHRONOUS socket calls; the
|
||||
// async read/write paths S7.Net actually uses ignore them, so every data-plane op below is
|
||||
// additionally bounded by S7OperationDeadline (a real wall-clock ceiling). OpenAsync is left
|
||||
// to S7Driver.EnsureConnectedAsync's own CancelAfter, so it is not double-bounded here.
|
||||
var ms = (int)timeout.TotalMilliseconds;
|
||||
_plc.ReadTimeout = ms;
|
||||
_plc.WriteTimeout = ms;
|
||||
@@ -117,23 +123,30 @@ internal sealed class S7PlcAdapter : IS7Plc
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<object?> ReadAsync(string address, CancellationToken cancellationToken) =>
|
||||
_plc.ReadAsync(address, cancellationToken)!;
|
||||
S7OperationDeadline.RunAsync(
|
||||
t => _plc.ReadAsync(address, t)!, _operationTimeout, $"read of '{address}'", cancellationToken);
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<byte[]> ReadBytesAsync(S7NetDataType area, int db, int startByteAdr, int count, CancellationToken cancellationToken) =>
|
||||
_plc.ReadBytesAsync(area, db, startByteAdr, count, cancellationToken);
|
||||
S7OperationDeadline.RunAsync(
|
||||
t => _plc.ReadBytesAsync(area, db, startByteAdr, count, t),
|
||||
_operationTimeout, $"block read of {area} db{db} @{startByteAdr}+{count}", cancellationToken);
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task WriteAsync(string address, object value, CancellationToken cancellationToken) =>
|
||||
_plc.WriteAsync(address, value, cancellationToken);
|
||||
S7OperationDeadline.RunAsync(
|
||||
t => _plc.WriteAsync(address, value, t), _operationTimeout, $"write of '{address}'", cancellationToken);
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task WriteBytesAsync(S7NetDataType area, int db, int startByteAdr, byte[] value, CancellationToken cancellationToken) =>
|
||||
_plc.WriteBytesAsync(area, db, startByteAdr, value, cancellationToken);
|
||||
S7OperationDeadline.RunAsync(
|
||||
t => _plc.WriteBytesAsync(area, db, startByteAdr, value, t),
|
||||
_operationTimeout, $"block write of {area} db{db} @{startByteAdr}", cancellationToken);
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task ReadStatusAsync(CancellationToken cancellationToken) =>
|
||||
_ = await _plc.ReadStatusAsync(cancellationToken).ConfigureAwait(false);
|
||||
public Task ReadStatusAsync(CancellationToken cancellationToken) =>
|
||||
S7OperationDeadline.RunAsync(
|
||||
t => _plc.ReadStatusAsync(t), _operationTimeout, "status read", cancellationToken);
|
||||
|
||||
/// <inheritdoc />
|
||||
// Plc implements IDisposable explicitly, so cast to invoke it; disposal tears down the TcpClient.
|
||||
|
||||
@@ -1287,7 +1287,9 @@ public sealed class S7Driver
|
||||
/// True when <paramref name="ex"/> (or any inner exception) is a socket-level / connection
|
||||
/// loss OR an ISO-on-TCP framing/desync fault that a reopen can repair — a
|
||||
/// <see cref="System.Net.Sockets.SocketException"/> / <see cref="System.IO.IOException"/> /
|
||||
/// <see cref="ObjectDisposedException"/>; an S7.Net framing violation
|
||||
/// <see cref="ObjectDisposedException"/>; a <see cref="TimeoutException"/> from a wire op that
|
||||
/// blew its <see cref="S7OperationDeadline"/> wall-clock ceiling (an unresponsive but still
|
||||
/// TCP-established peer — the READ-leg sibling of the connect-timeout fix, STAB-14); an S7.Net framing violation
|
||||
/// (<see cref="TPKTInvalidException"/> / <see cref="TPDUInvalidException"/> /
|
||||
/// <see cref="WrongNumberOfBytesException"/>); or a <see cref="PlcException"/> carrying
|
||||
/// <see cref="ErrorCode.ConnectionError"/> or <see cref="ErrorCode.WrongNumberReceivedBytes"/>.
|
||||
@@ -1314,6 +1316,13 @@ public sealed class S7Driver
|
||||
{
|
||||
if (e is System.Net.Sockets.SocketException or System.IO.IOException or ObjectDisposedException)
|
||||
return true;
|
||||
// A wire op that blew its wall-clock deadline (S7OperationDeadline — the async read/write
|
||||
// ceiling S7.Net's ignored socket ReadTimeout/WriteTimeout can't provide) means the
|
||||
// connection is unresponsive: discard + reopen. This is the READ-leg sibling of the
|
||||
// connect-timeout fix (STAB-14) — without it a frozen-but-established peer wedges the
|
||||
// poll loop forever instead of surfacing Bad + reconnecting.
|
||||
if (e is TimeoutException)
|
||||
return true;
|
||||
// ISO-on-TCP framing/desync surface (STAB-15a): the stream position is untrustworthy —
|
||||
// a half-read PDU left by a timeout/cancellation makes every later response mis-frame.
|
||||
// NOTE: deliberately NOT System.IO.InvalidDataException — ReinterpretRawValue throws it
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.S7;
|
||||
|
||||
/// <summary>
|
||||
/// Bounds a single S7 wire operation with a wall-clock deadline, surfacing an overrun as a
|
||||
/// <see cref="TimeoutException"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// S7.Net's <c>ReadTimeout</c> / <c>WriteTimeout</c> set the underlying
|
||||
/// <see cref="System.Net.Sockets.TcpClient"/>'s <c>ReceiveTimeout</c> / <c>SendTimeout</c>,
|
||||
/// which govern only <b>synchronous</b> socket calls — the <c>async</c> read/write paths
|
||||
/// S7.Net actually uses ignore them entirely. Without an explicit ceiling, a read on an
|
||||
/// established-but-frozen connection (a frozen PLC, a firewall DROP, or a cable pulled
|
||||
/// mid-flow — where the TCP session stays <i>open</i> but no reply ever arrives) blocks until
|
||||
/// the OS TCP stack gives up, which can be minutes. That silently wedges the driver's poll
|
||||
/// loop: no Bad tick, no reconnect — the STAB-14 "silently killed poll loop" failure, on the
|
||||
/// READ leg that the connect-timeout fix (<c>S7Driver.EnsureConnectedAsync</c>'s
|
||||
/// <c>CancelAfter</c>) never covered.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The timed-out operation surfaces as a <see cref="TimeoutException"/>, which
|
||||
/// <c>S7Driver.IsS7ConnectionFatal</c> classifies connection-fatal → the handle is marked
|
||||
/// dead → the next <c>EnsureConnectedAsync</c> reopens. The end-to-end proof is the live gate
|
||||
/// <c>S7_1500ConnectTimeoutOutageTests</c> (a real snap7 sim frozen with <c>docker pause</c>).
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal static class S7OperationDeadline
|
||||
{
|
||||
/// <summary>Runs <paramref name="operation"/> under a <paramref name="timeout"/> wall-clock ceiling.</summary>
|
||||
/// <typeparam name="T">The operation result type.</typeparam>
|
||||
/// <param name="operation">The wire operation; receives a token cancelled at the deadline.</param>
|
||||
/// <param name="timeout">The wall-clock ceiling for the operation.</param>
|
||||
/// <param name="what">A short description of the operation for the timeout message (e.g. <c>read of 'DB1.DBW0'</c>).</param>
|
||||
/// <param name="ct">The caller's cancellation token — its cancellation propagates as an <see cref="OperationCanceledException"/>, not a timeout.</param>
|
||||
/// <returns>The operation result when it completes within the deadline.</returns>
|
||||
/// <exception cref="TimeoutException">The operation did not complete within <paramref name="timeout"/>.</exception>
|
||||
public static async Task<T> RunAsync<T>(
|
||||
Func<CancellationToken, Task<T>> operation, TimeSpan timeout, string what, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(operation);
|
||||
using var opCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
|
||||
opCts.CancelAfter(timeout);
|
||||
try
|
||||
{
|
||||
// WaitAsync guarantees control returns at the deadline even if the underlying op ignores
|
||||
// its token; passing opCts.Token additionally lets an op that DOES honour cancellation
|
||||
// unwind cleanly instead of leaking a wedged socket read behind an abandoned task.
|
||||
return await operation(opCts.Token).WaitAsync(timeout, ct).ConfigureAwait(false);
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
// WaitAsync's own deadline fired (the op ignored the token).
|
||||
throw new TimeoutException(FormatMessage(what, timeout));
|
||||
}
|
||||
catch (OperationCanceledException) when (opCts.IsCancellationRequested && !ct.IsCancellationRequested)
|
||||
{
|
||||
// OUR deadline fired and the op honoured opCts.Token — not the caller's cancellation.
|
||||
throw new TimeoutException(FormatMessage(what, timeout));
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Runs a result-less <paramref name="operation"/> under a <paramref name="timeout"/> wall-clock ceiling.</summary>
|
||||
/// <param name="operation">The wire operation; receives a token cancelled at the deadline.</param>
|
||||
/// <param name="timeout">The wall-clock ceiling for the operation.</param>
|
||||
/// <param name="what">A short description of the operation for the timeout message.</param>
|
||||
/// <param name="ct">The caller's cancellation token.</param>
|
||||
/// <returns>A task that completes when the operation completes within the deadline.</returns>
|
||||
/// <exception cref="TimeoutException">The operation did not complete within <paramref name="timeout"/>.</exception>
|
||||
public static Task RunAsync(
|
||||
Func<CancellationToken, Task> operation, TimeSpan timeout, string what, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(operation);
|
||||
return RunAsync(
|
||||
async token => { await operation(token).ConfigureAwait(false); return true; },
|
||||
timeout, what, ct);
|
||||
}
|
||||
|
||||
private static string FormatMessage(string what, TimeSpan timeout) =>
|
||||
$"S7 {what} did not complete within {(int)timeout.TotalMilliseconds} ms.";
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
using Akka.Actor;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host;
|
||||
|
||||
/// <summary>
|
||||
/// Down-if-alone recovery watchdog (arch-review #459). The 2-node <c>keep-oldest</c> split-brain
|
||||
/// resolver downs the <b>lone survivor</b> when the OLDEST node crashes (the survivor is "the side
|
||||
/// without the oldest" → <c>DownReachable</c>), and <c>run-coordinated-shutdown-when-down = on</c>
|
||||
/// terminates that node's <see cref="ActorSystem"/>. Recovery is by <b>exit-and-rejoin</b>: the
|
||||
/// service supervisor (Windows <c>sc.exe failure … restart</c> / docker <c>restart: unless-stopped</c>)
|
||||
/// restarts the exited node and it re-forms / rejoins as a fresh incarnation.
|
||||
///
|
||||
/// <para>This watchdog closes the gap where the <see cref="ActorSystem"/> terminates but the .NET
|
||||
/// host process keeps running with a dead actor system (idling forever, never restarted). It watches
|
||||
/// <see cref="ActorSystem.WhenTerminated"/>; if the system terminates <b>outside</b> a normal host
|
||||
/// shutdown, it stops the application so the supervisor can restart the process. Mirrors the sister
|
||||
/// ScadaBridge project's proven pattern.</para>
|
||||
///
|
||||
/// <para>Registered <b>after</b> <c>AddAkka</c> so it starts after Akka's own hosted service has
|
||||
/// built the system; the <see cref="ActorSystem"/> is resolved lazily in <see cref="StartAsync"/>
|
||||
/// (never at construction) so it can't race Akka startup. A graceful host stop is distinguished from
|
||||
/// an unexpected SBR self-down via <see cref="_stopRequested"/> plus
|
||||
/// <see cref="IHostApplicationLifetime.ApplicationStopping"/>, so normal shutdown never logs a false
|
||||
/// alarm or double-triggers <see cref="IHostApplicationLifetime.StopApplication"/>.</para>
|
||||
/// </summary>
|
||||
public sealed class ActorSystemTerminationWatchdog : IHostedService
|
||||
{
|
||||
private readonly Func<ActorSystem> _actorSystemAccessor;
|
||||
private readonly IHostApplicationLifetime _lifetime;
|
||||
private readonly ILogger<ActorSystemTerminationWatchdog> _logger;
|
||||
private volatile bool _stopRequested;
|
||||
|
||||
/// <summary>Constructs the watchdog over a lazy <see cref="ActorSystem"/> accessor and the host lifetime.</summary>
|
||||
/// <param name="actorSystemAccessor">Lazy accessor (resolved in <see cref="StartAsync"/>, never at construction, so it can't race Akka startup).</param>
|
||||
/// <param name="lifetime">Host application lifetime used to stop the process on an unexpected self-down.</param>
|
||||
/// <param name="logger">Logger for the critical self-down diagnostic.</param>
|
||||
public ActorSystemTerminationWatchdog(
|
||||
Func<ActorSystem> actorSystemAccessor,
|
||||
IHostApplicationLifetime lifetime,
|
||||
ILogger<ActorSystemTerminationWatchdog> logger)
|
||||
{
|
||||
_actorSystemAccessor = actorSystemAccessor;
|
||||
_lifetime = lifetime;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
/// <summary>Wires the <see cref="ActorSystem.WhenTerminated"/> continuation that exits the host on an unexpected self-down.</summary>
|
||||
/// <param name="cancellationToken">Unused; the watchdog only registers a continuation.</param>
|
||||
/// <returns>A completed task.</returns>
|
||||
public Task StartAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
var system = _actorSystemAccessor();
|
||||
system.WhenTerminated.ContinueWith(
|
||||
_ =>
|
||||
{
|
||||
// Expected shutdown: our StopAsync ran, or the host is already stopping. Stay quiet.
|
||||
if (_stopRequested || _lifetime.ApplicationStopping.IsCancellationRequested)
|
||||
return;
|
||||
|
||||
_logger.LogCritical(
|
||||
"ActorSystem terminated outside host shutdown (SBR self-down / "
|
||||
+ "run-coordinated-shutdown-when-down). Stopping the host so the service supervisor "
|
||||
+ "restarts this node as a fresh incarnation (2-node keep-oldest exit-and-rejoin recovery).");
|
||||
_lifetime.StopApplication();
|
||||
},
|
||||
TaskContinuationOptions.ExecuteSynchronously);
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
/// <summary>Marks a graceful shutdown so the termination continuation stays silent and does not re-trigger stop.</summary>
|
||||
/// <param name="cancellationToken">Unused.</param>
|
||||
/// <returns>A completed task.</returns>
|
||||
public Task StopAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
_stopRequested = true;
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
@@ -296,6 +296,16 @@ builder.Services.AddAkka("otopcua", (ab, sp) =>
|
||||
ab.WithOtOpcUaRuntimeActors();
|
||||
});
|
||||
|
||||
// Down-if-alone recovery watchdog (#459). Registered AFTER AddAkka so it starts after Akka's own
|
||||
// hosted service has built the ActorSystem; it resolves the system lazily (never at construction) so
|
||||
// it can't race startup. On an unexpected SBR self-down it stops the host so the service supervisor
|
||||
// (sc.exe failure / docker restart: unless-stopped) restarts this node — the 2-node keep-oldest
|
||||
// exit-and-rejoin recovery path.
|
||||
builder.Services.AddHostedService(sp => new ActorSystemTerminationWatchdog(
|
||||
() => sp.GetRequiredService<ActorSystem>(),
|
||||
sp.GetRequiredService<IHostApplicationLifetime>(),
|
||||
sp.GetRequiredService<ILogger<ActorSystemTerminationWatchdog>>()));
|
||||
|
||||
if (hasAdmin)
|
||||
{
|
||||
// Auth + AdminUI surface only mounted on admin-role nodes. Driver-only nodes have no UI.
|
||||
|
||||
@@ -27,7 +27,11 @@ public sealed record AbServerProfile(
|
||||
public static class KnownProfiles
|
||||
{
|
||||
/// <summary>Gets all known server profiles.</summary>
|
||||
public static IReadOnlyList<AbServerProfile> All { get; } =
|
||||
/// <remarks>Expression-bodied (evaluated on access) so it observes the
|
||||
/// <c>static readonly</c> profile fields below after they are initialised;
|
||||
/// a <c>{ get; } =</c> initialiser here runs in textual order — before those
|
||||
/// fields — and would capture an all-null list.</remarks>
|
||||
public static IReadOnlyList<AbServerProfile> All =>
|
||||
[ControlLogix, CompactLogix, Micro800, GuardLogix];
|
||||
|
||||
/// <summary>Gets the ControlLogix profile.</summary>
|
||||
|
||||
@@ -17,6 +17,33 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests;
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class AbCipFactoryArrayTagTests
|
||||
{
|
||||
/// <summary>
|
||||
/// A non-positive <c>TimeoutMs</c> clamps to the 2 s default rather than reaching libplctag's
|
||||
/// <c>Tag.Timeout</c> setter (which throws <see cref="ArgumentOutOfRangeException"/> for a
|
||||
/// non-positive value, faulting every read/write). Driver timeout hardening.
|
||||
/// </summary>
|
||||
[Theory]
|
||||
[InlineData(0)]
|
||||
[InlineData(-1)]
|
||||
public void ParseOptions_clamps_non_positive_TimeoutMs_to_the_default(int timeoutMs)
|
||||
{
|
||||
var json = $$$"""{"TimeoutMs":{{{timeoutMs}}},"Probe":{"TimeoutMs":{{{timeoutMs}}}}}""";
|
||||
|
||||
var opts = AbCipDriverFactoryExtensions.ParseOptions("drv-1", json);
|
||||
|
||||
opts.Timeout.ShouldBe(TimeSpan.FromMilliseconds(2_000));
|
||||
opts.Timeout.ShouldBeGreaterThan(TimeSpan.Zero);
|
||||
opts.Probe.Timeout.ShouldBe(TimeSpan.FromMilliseconds(2_000));
|
||||
}
|
||||
|
||||
/// <summary>A positive <c>TimeoutMs</c> is honoured verbatim.</summary>
|
||||
[Fact]
|
||||
public void ParseOptions_honours_a_positive_TimeoutMs()
|
||||
{
|
||||
var opts = AbCipDriverFactoryExtensions.ParseOptions("drv-1", """{"TimeoutMs":500}""");
|
||||
opts.Timeout.ShouldBe(TimeSpan.FromMilliseconds(500));
|
||||
}
|
||||
|
||||
/// <summary>A driver-config tag declaring isArray/elementCount produces an array definition.</summary>
|
||||
[Fact]
|
||||
public void ParseOptions_threads_isArray_and_elementCount_into_tag_definition()
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
using libplctag;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.AbLegacy;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.AbLegacy.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Driver-timeout-hardening parity with AbCip: a read/write that fails with a non-zero libplctag
|
||||
/// status or a transport exception must EVICT the cached runtime so the next call re-creates a
|
||||
/// fresh handle — rather than returning the same failure forever until the probe loop happens to
|
||||
/// recycle it. Previously AbLegacy evicted on neither path (unlike AbCip).
|
||||
/// </summary>
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class AbLegacyEvictOnFailureTests
|
||||
{
|
||||
private const string Device = "ab://10.0.0.5/1,0";
|
||||
|
||||
// Probe disabled — the probe loop also calls _tagFactory.Create, which would perturb the
|
||||
// creation count these tests assert on.
|
||||
private static (AbLegacyDriver drv, FakeAbLegacyTagFactory factory) NewDriver(params AbLegacyTagDefinition[] tags)
|
||||
{
|
||||
var factory = new FakeAbLegacyTagFactory();
|
||||
var drv = new AbLegacyDriver(new AbLegacyDriverOptions
|
||||
{
|
||||
Devices = [new AbLegacyDeviceOptions(Device)],
|
||||
Tags = tags,
|
||||
Probe = new AbLegacyProbeOptions { Enabled = false },
|
||||
}, "drv-1", factory);
|
||||
return (drv, factory);
|
||||
}
|
||||
|
||||
/// <summary>A non-zero libplctag status on read evicts the runtime; the next read creates a fresh handle and recovers.</summary>
|
||||
[Fact]
|
||||
public async Task Read_nonzero_status_evicts_runtime_so_next_read_creates_fresh_handle()
|
||||
{
|
||||
var (drv, factory) = NewDriver(
|
||||
new AbLegacyTagDefinition("Counter", Device, "N7:0", AbLegacyDataType.Int));
|
||||
var callCount = 0;
|
||||
factory.Customise = p =>
|
||||
{
|
||||
callCount++;
|
||||
return callCount == 1
|
||||
? new FakeAbLegacyTag(p) { Status = (int)Status.ErrorBadConnection }
|
||||
: new FakeAbLegacyTag(p) { Status = 0, Value = 42 };
|
||||
};
|
||||
await drv.InitializeAsync("{}", CancellationToken.None);
|
||||
|
||||
// First read — bad status → runtime evicted, no reuse.
|
||||
var first = await drv.ReadAsync(["Counter"], CancellationToken.None);
|
||||
first.Single().StatusCode.ShouldNotBe(AbLegacyStatusMapper.Good);
|
||||
|
||||
// Second read — fresh handle, succeeds.
|
||||
var second = await drv.ReadAsync(["Counter"], CancellationToken.None);
|
||||
second.Single().StatusCode.ShouldBe(AbLegacyStatusMapper.Good);
|
||||
second.Single().Value.ShouldBe(42);
|
||||
callCount.ShouldBe(2); // one failed handle + one fresh handle
|
||||
}
|
||||
|
||||
/// <summary>A transport exception on read evicts the runtime; the next read creates a fresh handle and recovers.</summary>
|
||||
[Fact]
|
||||
public async Task Read_transport_exception_evicts_runtime_so_next_read_creates_fresh_handle()
|
||||
{
|
||||
var (drv, factory) = NewDriver(
|
||||
new AbLegacyTagDefinition("Counter", Device, "N7:0", AbLegacyDataType.Int));
|
||||
var callCount = 0;
|
||||
factory.Customise = p =>
|
||||
{
|
||||
callCount++;
|
||||
return callCount == 1
|
||||
? new FakeAbLegacyTag(p) { ThrowOnRead = true, Exception = new System.Net.Sockets.SocketException() }
|
||||
: new FakeAbLegacyTag(p) { Status = 0, Value = 7 };
|
||||
};
|
||||
await drv.InitializeAsync("{}", CancellationToken.None);
|
||||
|
||||
var first = await drv.ReadAsync(["Counter"], CancellationToken.None);
|
||||
first.Single().StatusCode.ShouldBe(AbLegacyStatusMapper.BadCommunicationError);
|
||||
|
||||
var second = await drv.ReadAsync(["Counter"], CancellationToken.None);
|
||||
second.Single().StatusCode.ShouldBe(AbLegacyStatusMapper.Good);
|
||||
second.Single().Value.ShouldBe(7);
|
||||
callCount.ShouldBe(2);
|
||||
}
|
||||
|
||||
/// <summary>A non-zero libplctag status on write evicts the runtime; the next write creates a fresh handle and recovers.</summary>
|
||||
[Fact]
|
||||
public async Task Write_nonzero_status_evicts_runtime_so_next_write_creates_fresh_handle()
|
||||
{
|
||||
var (drv, factory) = NewDriver(
|
||||
new AbLegacyTagDefinition("Counter", Device, "N7:0", AbLegacyDataType.Int));
|
||||
var callCount = 0;
|
||||
factory.Customise = p =>
|
||||
{
|
||||
callCount++;
|
||||
return callCount == 1
|
||||
? new FakeAbLegacyTag(p) { Status = (int)Status.ErrorBadConnection }
|
||||
: new FakeAbLegacyTag(p) { Status = 0 };
|
||||
};
|
||||
await drv.InitializeAsync("{}", CancellationToken.None);
|
||||
|
||||
var first = await drv.WriteAsync([new WriteRequest("Counter", (short)5)], CancellationToken.None);
|
||||
first.Single().StatusCode.ShouldNotBe(AbLegacyStatusMapper.Good);
|
||||
|
||||
var second = await drv.WriteAsync([new WriteRequest("Counter", (short)5)], CancellationToken.None);
|
||||
second.Single().StatusCode.ShouldBe(AbLegacyStatusMapper.Good);
|
||||
callCount.ShouldBe(2);
|
||||
}
|
||||
}
|
||||
+11
-1
@@ -415,7 +415,12 @@ class FocasMockServer:
|
||||
timer_type = self._block_u32(request_block, 8)
|
||||
key_map = {0: "power_on", 1: "operating", 2: "cutting", 3: "cycle"}
|
||||
seconds = int(self.store.snapshot()["timers"].get(key_map.get(timer_type, "power_on"), 0))
|
||||
return self._u32(seconds // 60) + self._u32((seconds % 60) * 1000)
|
||||
# cnc_rdtimer's minute+msec fields are LITTLE-endian on the wire — unlike the big-endian
|
||||
# block envelope and every other payload. Hardware-validated against a live FANUC 31i-B
|
||||
# (2026-06-25); the driver's FocasWireClient.ParseTimer decodes them little-endian. Encoding
|
||||
# them big-endian here (the old bug) made the driver read minute=60 as 0x3C000000. See
|
||||
# docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md.
|
||||
return self._u32_le(seconds // 60) + self._u32_le((seconds % 60) * 1000)
|
||||
|
||||
def _wire_alarms(self, request_block: bytes) -> bytes:
|
||||
requested = max(self._block_u32(request_block, 12), 1)
|
||||
@@ -481,6 +486,11 @@ class FocasMockServer:
|
||||
def _u32(self, value: int) -> bytes:
|
||||
return int(value).to_bytes(4, "big", signed=True)
|
||||
|
||||
def _u32_le(self, value: int) -> bytes:
|
||||
# Little-endian u32 — only cnc_rdtimer's minute/msec fields use this on real hardware
|
||||
# (see _wire_timer). Every other wire field is big-endian via _u32.
|
||||
return int(value).to_bytes(4, "little", signed=True)
|
||||
|
||||
def _ascii_fixed(self, value: str, length: int) -> bytes:
|
||||
return value.encode("ascii", errors="replace")[:length].ljust(length, b" ")
|
||||
|
||||
|
||||
@@ -29,6 +29,11 @@ public sealed class FocasSimFixture : IAsyncDisposable
|
||||
/// <summary>Gets the TCP port of the focas-mock simulator.</summary>
|
||||
public int Port { get; }
|
||||
|
||||
/// <summary>Gets the <c>focas://host:port</c> device URI the driver connects to — derived from
|
||||
/// <see cref="Host"/>/<see cref="Port"/> so tests honour <c>OTOPCUA_FOCAS_SIM_ENDPOINT</c> (a remote
|
||||
/// fixture on the Docker host) instead of assuming a localhost-colocated mock.</summary>
|
||||
public string DeviceUri => $"focas://{Host}:{Port}";
|
||||
|
||||
/// <summary>focas-mock profile stem the fixture should load (e.g. <c>fwlib30i64</c>,
|
||||
/// <c>ThirtyOne_i</c> — both resolve via the mock's alias table). Null when unset.</summary>
|
||||
public string? ExpectedProfile { get; }
|
||||
|
||||
+3
-1
@@ -23,7 +23,9 @@ public sealed class WireBackendCoverageTests
|
||||
/// <param name="fx">The FOCAS simulation fixture.</param>
|
||||
public WireBackendCoverageTests(FocasSimFixture fx) => _fx = fx;
|
||||
|
||||
private const string DeviceHost = "focas://127.0.0.1:8193";
|
||||
// Derived from the fixture endpoint (OTOPCUA_FOCAS_SIM_ENDPOINT) so the driver dials the SAME
|
||||
// mock the fixture probed — a remote fixture on the Docker host, not a hardcoded 127.0.0.1.
|
||||
private string DeviceHost => _fx.DeviceUri;
|
||||
|
||||
/// <summary>Verifies that user tag reads route via the wire backend.</summary>
|
||||
[Fact]
|
||||
|
||||
+3
-1
@@ -26,7 +26,9 @@ public sealed class WireBackendTests
|
||||
/// <param name="fx">The FOCAS simulator fixture.</param>
|
||||
public WireBackendTests(FocasSimFixture fx) => _fx = fx;
|
||||
|
||||
private const string DeviceHost = "focas://127.0.0.1:8193";
|
||||
// Derived from the fixture endpoint (OTOPCUA_FOCAS_SIM_ENDPOINT) so the driver dials the SAME
|
||||
// mock the fixture probed — a remote fixture on the Docker host, not a hardcoded 127.0.0.1.
|
||||
private string DeviceHost => _fx.DeviceUri;
|
||||
|
||||
/// <summary>Verifies that identity axes and dynamic data populate via the wire backend.</summary>
|
||||
[Fact]
|
||||
|
||||
@@ -57,6 +57,40 @@ public sealed class FocasFactoryConfigTests
|
||||
drv.Options.Devices[0].Series.ShouldBe(FocasCncSeries.Thirty_i);
|
||||
}
|
||||
|
||||
// ---- Driver timeout hardening: a non-positive TimeoutMs must not disable the per-call ceiling ----
|
||||
|
||||
/// <summary>
|
||||
/// A misconfigured <c>TimeoutMs: 0</c> (or negative) must clamp to the 2 s default, NOT map to
|
||||
/// <see cref="TimeSpan.Zero"/> — a zero/negative <c>Timeout</c> makes <c>SynchronizedFocasClient</c>
|
||||
/// disable its per-call wall-clock ceiling, reintroducing the unbounded-read-on-a-frozen-peer
|
||||
/// wedge the S7 R2-01 read-leg fix eliminated. Guards the factory clamp.
|
||||
/// </summary>
|
||||
[Theory]
|
||||
[InlineData(0)]
|
||||
[InlineData(-1)]
|
||||
[InlineData(-5000)]
|
||||
public void CreateInstance_clamps_non_positive_TimeoutMs_to_the_default(int timeoutMs)
|
||||
{
|
||||
var json = $$$"""{"Backend":"unimplemented","TimeoutMs":{{{timeoutMs}}},"Probe":{"TimeoutMs":{{{timeoutMs}}}}}""";
|
||||
|
||||
var drv = FocasDriverFactoryExtensions.CreateInstance("drv-1", json);
|
||||
|
||||
drv.Options.Timeout.ShouldBe(TimeSpan.FromMilliseconds(2_000));
|
||||
drv.Options.Timeout.ShouldBeGreaterThan(TimeSpan.Zero);
|
||||
drv.Options.Probe.Timeout.ShouldBe(TimeSpan.FromMilliseconds(2_000));
|
||||
}
|
||||
|
||||
/// <summary>A positive <c>TimeoutMs</c> is honoured verbatim (the clamp only rewrites non-positive values).</summary>
|
||||
[Fact]
|
||||
public void CreateInstance_honours_a_positive_TimeoutMs()
|
||||
{
|
||||
const string json = """{"Backend":"unimplemented","TimeoutMs":750}""";
|
||||
|
||||
var drv = FocasDriverFactoryExtensions.CreateInstance("drv-1", json);
|
||||
|
||||
drv.Options.Timeout.ShouldBe(TimeSpan.FromMilliseconds(750));
|
||||
}
|
||||
|
||||
/// <summary>Verifies that the AlarmProjection configuration section is mapped to driver options.</summary>
|
||||
[Fact]
|
||||
public void CreateInstance_maps_AlarmProjection_section_onto_options()
|
||||
|
||||
+13
-14
@@ -26,24 +26,23 @@ public sealed class GatewayTagProvisionerTests
|
||||
Assert.NotNull(fake.LastEnsureDefinitions);
|
||||
Assert.Equal(2, fake.LastEnsureDefinitions!.Count);
|
||||
Assert.Equal(HistorianDataType.Float, fake.LastEnsureDefinitions[0].DataType);
|
||||
Assert.Equal(HistorianDataType.Int1, fake.LastEnsureDefinitions[1].DataType);
|
||||
Assert.Equal(HistorianDataType.Int2, fake.LastEnsureDefinitions[1].DataType);
|
||||
Assert.Equal(2, result.Requested);
|
||||
Assert.Equal(0, result.Skipped);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// archreview 06/U-7 — pins the 0.2.0 Boolean→Int1 provisioning semantics at TWO levels. The
|
||||
/// 0.1.0→0.2.0 client bump made <c>HistorianTagDefinition.DataType</c> proto3-optional (explicit
|
||||
/// presence): under 0.1.0 the provisioner's <c>Int1</c> (wire 0) was indistinguishable from unset,
|
||||
/// so the gateway defaulted Boolean tags to Float; under 0.2.0 the same call carries explicit
|
||||
/// presence and provisions Int1 (the intended behavior). Asserting BOTH the mapped type AND
|
||||
/// <c>HasDataType</c> compile-pins the package floor (the <c>HasDataType</c>/<c>ClearDataType</c>
|
||||
/// members exist only on the 0.2.0 proto3-optional contract — a downgrade breaks the build) and
|
||||
/// assert-pins that the provisioner always sends explicit presence (a future contract change that
|
||||
/// stopped sending it would break the assert). This is a pin, not a fix — it passes on current code.
|
||||
/// Pins the Boolean→Int2 provisioning semantics at TWO levels. Boolean maps to <c>Int2</c> because
|
||||
/// the historian's <c>Int1</c> analog-tag creation path is server-degenerate — <c>EnsureTags(Int1)</c>
|
||||
/// stores an unusable stub (issue #441 / HistorianGateway #11) — so Boolean historizes as a small 0/1
|
||||
/// integer. Asserting BOTH the mapped type AND <c>HasDataType</c> compile-pins the package floor (the
|
||||
/// <c>HasDataType</c>/<c>ClearDataType</c> members exist only on the 0.2.0+ proto3-optional contract —
|
||||
/// a downgrade breaks the build) and assert-pins that the provisioner always sends explicit presence
|
||||
/// (a future contract change that stopped sending it would default the tag to the SDK's Float and
|
||||
/// break this assert). This is a pin, not a fix — it passes on current code.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Boolean_definition_carries_explicit_Int1_presence()
|
||||
public async Task Boolean_definition_carries_explicit_Int2_presence()
|
||||
{
|
||||
var fake = new FakeHistorianGatewayClient { EnsureTagsResult = new TagOperationResults() };
|
||||
var p = Provisioner(fake);
|
||||
@@ -54,9 +53,9 @@ public sealed class GatewayTagProvisionerTests
|
||||
|
||||
var defs = fake.LastEnsureDefinitions!;
|
||||
Assert.Single(defs);
|
||||
Assert.Equal(HistorianDataType.Int1, defs[0].DataType);
|
||||
// 0.2.0 proto3-optional presence witness — explicit presence is what flips the gateway from its
|
||||
// SDK-default Float to the requested Int1. Without this, a Boolean tag would provision as Float.
|
||||
Assert.Equal(HistorianDataType.Int2, defs[0].DataType);
|
||||
// proto3-optional presence witness — explicit presence is what flips the gateway from its
|
||||
// SDK-default Float to the requested Int2. Without this, a Boolean tag would provision as Float.
|
||||
Assert.True(defs[0].HasDataType);
|
||||
}
|
||||
|
||||
|
||||
+18
-15
@@ -255,25 +255,26 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// archreview 06/U-7 — Boolean provisioning leg on the 0.2.0 contract. Through the <b>real</b>
|
||||
/// issue #441 — Boolean historization leg. Through the <b>real</b>
|
||||
/// <see cref="GatewayTagProvisioner"/>, <c>EnsureTags</c> a <see cref="DriverDataType.Boolean"/>
|
||||
/// request for the dedicated Boolean sandbox tag (mapped to <c>Int1</c> with explicit proto3
|
||||
/// presence), assert it is ensured (not failed), then <c>WriteLiveValues</c> a <c>1.0</c> and read
|
||||
/// it back over a wide window. This retires "the live EnsureTags leg has never run on 0.2.0" for
|
||||
/// the Boolean path. The ±12h read window + timezone caveat mirror <see cref="Write_then_read_on_sandbox_tag"/>.
|
||||
/// request for the dedicated Boolean sandbox tag (mapped to <c>Int2</c> — the historian's
|
||||
/// <c>Int1</c> analog-tag creation path is server-degenerate, see issue #441 / HistorianGateway
|
||||
/// #11, so Boolean historizes as a small 0/1 integer), assert it is ensured (not failed), then
|
||||
/// <c>WriteLiveValues</c> a <c>1.0</c> and read it back over a wide window. The ±12h read window
|
||||
/// + timezone caveat mirror <see cref="Write_then_read_on_sandbox_tag"/>.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
[Trait("Category", "LiveIntegration")]
|
||||
public async Task EnsureTags_boolean_sandbox_provisions_as_Int1()
|
||||
public async Task EnsureTags_boolean_sandbox_provisions_as_Int2()
|
||||
{
|
||||
if (_fx.NotConfigured) Assert.Skip(_fx.SkipReason!);
|
||||
if (_fx.BoolSandboxTag is null)
|
||||
Assert.Skip("Skipped: set HISTGW_BOOL_SANDBOX_TAG to a writable Boolean sandbox tag (e.g. HistGW.LiveTest.BoolSandbox) to run the 0.2.0 Boolean EnsureTags leg.");
|
||||
Assert.Skip("Skipped: set HISTGW_BOOL_SANDBOX_TAG to a writable Boolean sandbox tag (e.g. HistGW.LiveTest.BoolSandbox) to run the Boolean historization leg.");
|
||||
|
||||
var ct = TestContext.Current.CancellationToken;
|
||||
var tag = _fx.BoolSandboxTag;
|
||||
|
||||
// EnsureTags (Boolean → Int1) through the REAL provisioner seam (the same code path deploys use).
|
||||
// EnsureTags (Boolean → Int2) through the REAL provisioner seam (the same code path deploys use).
|
||||
await using var writeClient = _fx.CreateClient();
|
||||
var provisioner = new GatewayTagProvisioner(writeClient, NullLogger<GatewayTagProvisioner>.Instance);
|
||||
var provision = await provisioner.EnsureTagsAsync(
|
||||
@@ -281,7 +282,7 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
|
||||
ct);
|
||||
|
||||
provision.Ensured.ShouldBe(1,
|
||||
"the Boolean sandbox tag must be ensured (needs the gateway on 0.2.0 contracts + an API key carrying historian:tags:write).");
|
||||
"the Boolean sandbox tag must be ensured as Int2 (needs the gateway on 0.2.0+ contracts + an API key carrying historian:tags:write).");
|
||||
provision.Failed.ShouldBe(0);
|
||||
|
||||
// WriteLiveValues a Boolean-as-1.0 and read it back (SQL live path can lag a flush cadence → poll).
|
||||
@@ -308,14 +309,16 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
|
||||
|
||||
hit.ShouldNotBeNull($"the written Boolean sample ({written}) should read back from '{tag}'.");
|
||||
TestContext.Current.SendDiagnosticMessage(
|
||||
$"Boolean round-trip: EnsureTags(Int1) + WriteLiveValues '{tag}'={written} → read back at {hit!.SourceTimestampUtc:O}.");
|
||||
$"Boolean round-trip: EnsureTags(Int2) + WriteLiveValues '{tag}'={written} → read back at {hit!.SourceTimestampUtc:O}.");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// archreview 06/U-7 retype probe — answers the assessment note's open question: what does the
|
||||
/// deployed AVEVA Historian do when <c>EnsureTags</c> (an <b>upsert</b>) is asked to change an
|
||||
/// existing tag's type? Against the <b>dedicated Boolean sandbox tag only</b> (never the shared
|
||||
/// Float write sandbox), EnsureTags first as <c>Float</c> then as <c>Int1</c> and assert the second
|
||||
/// Float write sandbox), EnsureTags first as <c>Float</c> then as <c>Int2</c> (both supported
|
||||
/// analog types — <c>Int2</c> is the type Boolean now historizes as; the previously-probed
|
||||
/// <c>Int1</c> is a documented server-degenerate exclusion, issue #441) and assert the second
|
||||
/// call's per-tag outcome is <em>deterministic and observable</em> — either <c>Success=true</c>
|
||||
/// (retype / tag-versioning accepted) or <c>Success=false</c> with a non-empty <c>Error</c> (mapped
|
||||
/// to the provisioner's <c>failed</c> tally). The assertion accepts both because the contract
|
||||
@@ -336,7 +339,7 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
|
||||
|
||||
await using var client = _fx.CreateClient();
|
||||
|
||||
// First establish the tag as Float, then ask EnsureTags to retype it to Int1 (Boolean).
|
||||
// First establish the tag as Float, then ask EnsureTags to retype it to Int2.
|
||||
await client.EnsureTagsAsync(
|
||||
new[]
|
||||
{
|
||||
@@ -357,9 +360,9 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
|
||||
new HistorianTagDefinition
|
||||
{
|
||||
TagName = tag,
|
||||
DataType = HistorianDataType.Int1,
|
||||
DataType = HistorianDataType.Int2,
|
||||
EngineeringUnit = string.Empty,
|
||||
Description = "OtOpcUa live validation retype probe (stage 2: Int1)",
|
||||
Description = "OtOpcUa live validation retype probe (stage 2: Int2)",
|
||||
StorageRateMs = 1000, // 0 makes the gateway's EnsureTags throw (storageRateMs must be > 0).
|
||||
},
|
||||
},
|
||||
@@ -372,7 +375,7 @@ public sealed class GatewayLiveIntegrationTests(GatewayLiveFixture fixture) : IC
|
||||
"or Success=false with a non-empty Error (mapped to the provisioner's failed tally).");
|
||||
|
||||
TestContext.Current.SendDiagnosticMessage(
|
||||
$"retype probe '{tag}' Float→Int1: Success={outcome.Success}, Error='{outcome.Error}'. " +
|
||||
$"retype probe '{tag}' Float→Int2: Success={outcome.Success}, Error='{outcome.Error}'. " +
|
||||
"(records the deployed historian's in-place analog-retype policy — feed this back into docs/Historian.md).");
|
||||
}
|
||||
}
|
||||
|
||||
+1
-1
@@ -8,7 +8,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.Mapping;
|
||||
public sealed class HistorianTypeMapperTests
|
||||
{
|
||||
[Theory]
|
||||
[InlineData(DriverDataType.Boolean, HistorianDataType.Int1)]
|
||||
[InlineData(DriverDataType.Boolean, HistorianDataType.Int2)] // Int1 analog-tag creation is server-degenerate (issue #441) → store 0/1
|
||||
[InlineData(DriverDataType.Int16, HistorianDataType.Int2)]
|
||||
[InlineData(DriverDataType.Int32, HistorianDataType.Int4)]
|
||||
[InlineData(DriverDataType.Int64, HistorianDataType.Int8)]
|
||||
|
||||
@@ -48,6 +48,31 @@ service + starting another. The integration tests discriminate by a
|
||||
separate `MODBUS_SIM_PROFILE` env var so they skip correctly when the
|
||||
wrong profile is live.
|
||||
|
||||
> **⚠️ Profile-cycling gotcha (bit us in the 2026-07 sweep — see
|
||||
> `archreview/plans/INTEGRATION-SWEEP-STATUS.md`).** Each profile is a
|
||||
> *separate compose service* selected by `--profile`, so a plain
|
||||
> `docker compose down` (no `--profile`) does **not** stop the
|
||||
> currently-running profile container — it keeps holding `:5020`. Start
|
||||
> another profile and the new container silently fails to bind the port
|
||||
> (compose still reports "Started"); every test then hits the **stale**
|
||||
> profile's data and fails with wrong values / Illegal-Data-Address. The
|
||||
> symptom is deceptive: reads succeed but return the previous profile's
|
||||
> seed (e.g. Mitsubishi's `D0` reads DL205's `0xCAFE`). Always tear the
|
||||
> old container down **by name / with its profile** before bringing up the
|
||||
> next, and verify the new container actually published `:5020`:
|
||||
>
|
||||
> ```bash
|
||||
> # reliable cycle (force-remove ALL modbus containers, then bring one up)
|
||||
> docker rm -f $(docker ps -aq --filter name=otopcua-modbus --filter name=otopcua-pymodbus)
|
||||
> docker compose --profile <profile> up -d --force-recreate
|
||||
> docker ps --filter name=otopcua --format '{{.Names}} {{.Ports}}' # MUST show 0.0.0.0:5020->5020
|
||||
> ```
|
||||
>
|
||||
> The `exception_injection` service uses a different container name
|
||||
> (`otopcua-modbus-exception-injector`, not `otopcua-pymodbus-*`), so a
|
||||
> name-filtered `docker rm` that only matches `otopcua-pymodbus` will miss
|
||||
> it — the `--force-recreate` above covers that case.
|
||||
|
||||
### Profile coverage matrix
|
||||
|
||||
The two general-purpose profiles cover disjoint test sets. A full pass
|
||||
|
||||
@@ -63,6 +63,40 @@ public sealed class S7DriverReconnectTests
|
||||
factory.Created[1].Disposed.ShouldBeFalse();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// R2-01 read-leg gate. A <see cref="TimeoutException"/> — what <c>S7OperationDeadline</c>
|
||||
/// raises when a wire read blows its wall-clock ceiling on an established-but-frozen peer
|
||||
/// (the async read S7.Net's ignored socket <c>ReadTimeout</c> can't bound) — is classified
|
||||
/// connection-fatal: the handle is marked dead and the NEXT read reopens a fresh connection,
|
||||
/// exactly like a socket drop. Before the fix a read-timeout was NOT fatal, so the wedged
|
||||
/// handle was reused and the poll loop never recovered (<c>Created.Count</c> pinned at 1).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Read_reopens_after_a_wire_operation_timeout()
|
||||
{
|
||||
var factory = new FakeS7PlcFactory();
|
||||
using var drv = new S7Driver(Options(), "s7-read-timeout-fatal", factory);
|
||||
await drv.InitializeAsync("{}", TestContext.Current.CancellationToken);
|
||||
factory.Created.Count.ShouldBe(1);
|
||||
|
||||
// The next read blows its deadline — surfaces as a TimeoutException, the shape
|
||||
// S7OperationDeadline raises for a frozen-but-established peer.
|
||||
factory.Created[0].NextReadThrows = new TimeoutException("S7 read of 'DB1.DBW0' did not complete within 250 ms.");
|
||||
|
||||
// Read #1 — timeout → Degraded, handle marked dead, no reopen yet.
|
||||
var bad = await drv.ReadAsync(["W0"], TestContext.Current.CancellationToken);
|
||||
bad[0].StatusCode.ShouldNotBe(0u);
|
||||
drv.GetHealth().State.ShouldBe(DriverState.Degraded);
|
||||
factory.Created.Count.ShouldBe(1);
|
||||
|
||||
// Read #2 — disposes the dead handle, opens a SECOND connection, reads Good, recovers.
|
||||
var recovered = await drv.ReadAsync(["W0"], TestContext.Current.CancellationToken);
|
||||
recovered[0].StatusCode.ShouldBe(0u);
|
||||
drv.GetHealth().State.ShouldBe(DriverState.Healthy);
|
||||
factory.Created.Count.ShouldBe(2);
|
||||
factory.Created[0].Disposed.ShouldBeTrue();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A protocol / data-address error (S7.Net <see cref="ErrorCode.ReadData"/>) is NOT a
|
||||
/// connection loss — the driver keeps the same connection and does not churn a reopen.
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
using System.Diagnostics;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.S7.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Unit tests for <see cref="S7OperationDeadline"/> — the wall-clock ceiling that bounds every
|
||||
/// S7 async wire op (R2-01 read-leg gate). Async S7.Net reads/writes ignore the socket
|
||||
/// ReadTimeout/WriteTimeout, so a frozen-but-established peer would otherwise block for minutes;
|
||||
/// these pin the deadline behaviour without a live PLC.
|
||||
/// </summary>
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class S7OperationDeadlineTests
|
||||
{
|
||||
private static readonly TimeSpan ShortTimeout = TimeSpan.FromMilliseconds(150);
|
||||
|
||||
/// <summary>An operation that completes within the deadline returns its value untouched.</summary>
|
||||
[Fact]
|
||||
public async Task Completes_within_deadline_returns_value()
|
||||
{
|
||||
var result = await S7OperationDeadline.RunAsync(
|
||||
_ => Task.FromResult(42), ShortTimeout, "read", TestContext.Current.CancellationToken);
|
||||
result.ShouldBe(42);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// An operation that ignores its token and never completes is cut at the deadline and
|
||||
/// surfaces a <see cref="TimeoutException"/> — the frozen-established-peer shape. Uses a bare
|
||||
/// <see cref="TaskCompletionSource"/> that never completes, so the token is genuinely ignored.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Token_ignoring_hang_times_out()
|
||||
{
|
||||
var neverCompletes = new TaskCompletionSource<int>().Task;
|
||||
var sw = Stopwatch.StartNew();
|
||||
|
||||
var ex = await Should.ThrowAsync<TimeoutException>(async () =>
|
||||
await S7OperationDeadline.RunAsync(
|
||||
_ => neverCompletes, ShortTimeout, "read of 'DB1.DBW0'", TestContext.Current.CancellationToken));
|
||||
|
||||
sw.Elapsed.ShouldBeLessThan(TimeSpan.FromSeconds(5));
|
||||
ex.Message.ShouldContain("did not complete within 150 ms");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// An operation that honours its token (the well-behaved async-read case) is cancelled at the
|
||||
/// deadline; the resulting cancellation is normalised to a <see cref="TimeoutException"/>,
|
||||
/// NOT surfaced as an <see cref="OperationCanceledException"/>.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Token_honouring_hang_times_out_as_timeout_not_cancel()
|
||||
{
|
||||
await Should.ThrowAsync<TimeoutException>(async () =>
|
||||
await S7OperationDeadline.RunAsync(
|
||||
async token =>
|
||||
{
|
||||
await Task.Delay(Timeout.Infinite, token);
|
||||
return 0;
|
||||
},
|
||||
ShortTimeout, "read", TestContext.Current.CancellationToken));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Caller cancellation (as opposed to the deadline firing) propagates as an
|
||||
/// <see cref="OperationCanceledException"/> — the deadline wrapper must not swallow genuine
|
||||
/// cancellation into a timeout.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Caller_cancellation_propagates_as_operation_cancelled()
|
||||
{
|
||||
using var cts = new CancellationTokenSource();
|
||||
cts.CancelAfter(50);
|
||||
|
||||
// A long deadline so the CALLER token fires first, not the internal deadline.
|
||||
await Should.ThrowAsync<OperationCanceledException>(async () =>
|
||||
await S7OperationDeadline.RunAsync(
|
||||
async token =>
|
||||
{
|
||||
await Task.Delay(Timeout.Infinite, token);
|
||||
return 0;
|
||||
},
|
||||
TimeSpan.FromSeconds(30), "read", cts.Token));
|
||||
}
|
||||
|
||||
/// <summary>The result-less overload also bounds a hang and surfaces a <see cref="TimeoutException"/>.</summary>
|
||||
[Fact]
|
||||
public async Task Resultless_overload_times_out()
|
||||
{
|
||||
var neverCompletes = new TaskCompletionSource().Task;
|
||||
await Should.ThrowAsync<TimeoutException>(async () =>
|
||||
await S7OperationDeadline.RunAsync(
|
||||
_ => neverCompletes, ShortTimeout, "write of 'DB1.DBW0'", TestContext.Current.CancellationToken));
|
||||
}
|
||||
}
|
||||
@@ -138,6 +138,59 @@ public sealed class ConfigComposerTests : ControlPlaneActorTestBase
|
||||
node.DeviceHost.ShouldBe("10.9.9.9:8193");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies that a <see cref="DriverInstance"/>'s per-instance <c>ResilienceConfig</c> override
|
||||
/// survives the real <see cref="ConfigComposer.SnapshotAndFlattenAsync"/> →
|
||||
/// <see cref="DeploymentArtifact.ParseDriverInstances(ReadOnlySpan{byte})"/> seam (arch-review
|
||||
/// follow-up #10 sub-gap). The composer serialises the whole entity, so the override rides the
|
||||
/// artifact into <c>DriverInstanceSpec.ResilienceConfig</c> — which <c>DriverHostActor</c> layers
|
||||
/// onto the driver's Polly pipeline. Guards the one leg the hand-rolled
|
||||
/// <c>DeploymentArtifactTests.ParseDriverInstances_carries_ResilienceConfig_onto_the_spec</c>
|
||||
/// cannot: if the composer ever stopped emitting the column (a projection, a <c>[JsonIgnore]</c>),
|
||||
/// authored resilience policy would silently revert to tier defaults in production while the
|
||||
/// hand-built artifact tests stayed green.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task ResilienceConfig_survives_ConfigComposer_to_ParseDriverInstances_round_trip()
|
||||
{
|
||||
const string resilienceJson =
|
||||
"{\"capabilityPolicies\":{\"Read\":{\"timeoutSeconds\":5,\"retryCount\":7,\"breakerFailureThreshold\":3}}}";
|
||||
|
||||
var f = NewInMemoryDbFactory();
|
||||
await using (var db = f.CreateDbContext())
|
||||
{
|
||||
db.ServerClusters.Add(NewCluster("c1"));
|
||||
db.Namespaces.Add(new Namespace
|
||||
{
|
||||
NamespaceId = "ns-eq", ClusterId = "c1",
|
||||
Kind = NamespaceKind.Equipment, NamespaceUri = "urn:eq",
|
||||
});
|
||||
db.DriverInstances.Add(new DriverInstance
|
||||
{
|
||||
DriverInstanceId = "drv-1", ClusterId = "c1", NamespaceId = "ns-eq",
|
||||
Name = "Modbus", DriverType = "Modbus", DriverConfig = "{}",
|
||||
ResilienceConfig = resilienceJson,
|
||||
});
|
||||
// A second instance with no override proves the null case rides the same real composer path.
|
||||
db.DriverInstances.Add(new DriverInstance
|
||||
{
|
||||
DriverInstanceId = "drv-2", ClusterId = "c1", NamespaceId = "ns-eq",
|
||||
Name = "Modbus2", DriverType = "Modbus", DriverConfig = "{}",
|
||||
});
|
||||
await db.SaveChangesAsync();
|
||||
}
|
||||
|
||||
await using var readDb = f.CreateDbContext();
|
||||
var artifact = await ConfigComposer.SnapshotAndFlattenAsync(readDb);
|
||||
var specs = DeploymentArtifact.ParseDriverInstances(artifact.Blob);
|
||||
|
||||
var withOverride = specs.Single(s => s.DriverInstanceId == "drv-1");
|
||||
withOverride.ResilienceConfig.ShouldBe(resilienceJson, "the authored override must ride the artifact byte-for-byte");
|
||||
|
||||
var withoutOverride = specs.Single(s => s.DriverInstanceId == "drv-2");
|
||||
withoutOverride.ResilienceConfig.ShouldBeNull("an unset override stays null ⇒ tier defaults at spawn");
|
||||
}
|
||||
|
||||
private static readonly DateTime FixedTimestamp = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
private static ServerCluster NewCluster(string id) => new()
|
||||
|
||||
+92
@@ -0,0 +1,92 @@
|
||||
using Akka.Actor;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Host;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
|
||||
|
||||
/// <summary>
|
||||
/// Unit coverage for the #459 down-if-alone recovery watchdog: an unexpected
|
||||
/// <see cref="ActorSystem"/> termination (SBR self-down) must stop the host so the supervisor
|
||||
/// restarts it, while a graceful host shutdown must stay silent and not re-trigger stop.
|
||||
/// </summary>
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class ActorSystemTerminationWatchdogTests
|
||||
{
|
||||
/// <summary>An unexpected ActorSystem termination stops the host so the supervisor restarts the node.</summary>
|
||||
[Fact]
|
||||
public async Task Unexpected_termination_stops_the_application()
|
||||
{
|
||||
var system = ActorSystem.Create("watchdog-unexpected");
|
||||
var lifetime = new FakeHostApplicationLifetime();
|
||||
var watchdog = new ActorSystemTerminationWatchdog(
|
||||
() => system, lifetime, NullLogger<ActorSystemTerminationWatchdog>.Instance);
|
||||
|
||||
await watchdog.StartAsync(CancellationToken.None);
|
||||
|
||||
// Simulate the SBR self-down: the ActorSystem terminates while the host is still running.
|
||||
await system.Terminate();
|
||||
|
||||
await WaitForAsync(() => lifetime.StopApplicationCalls > 0, TimeSpan.FromSeconds(5));
|
||||
lifetime.StopApplicationCalls.ShouldBe(1, "an out-of-band ActorSystem termination must stop the host once");
|
||||
}
|
||||
|
||||
/// <summary>A graceful host stop (StopAsync ran) leaves the termination continuation silent.</summary>
|
||||
[Fact]
|
||||
public async Task Graceful_stop_does_not_stop_the_application()
|
||||
{
|
||||
var system = ActorSystem.Create("watchdog-graceful");
|
||||
var lifetime = new FakeHostApplicationLifetime();
|
||||
var watchdog = new ActorSystemTerminationWatchdog(
|
||||
() => system, lifetime, NullLogger<ActorSystemTerminationWatchdog>.Instance);
|
||||
|
||||
await watchdog.StartAsync(CancellationToken.None);
|
||||
|
||||
// Host is shutting down normally: StopAsync runs, THEN the system terminates.
|
||||
await watchdog.StopAsync(CancellationToken.None);
|
||||
await system.Terminate();
|
||||
|
||||
// Give any continuation a chance to run, then assert it stayed silent.
|
||||
await Task.Delay(500, TestContext.Current.CancellationToken);
|
||||
lifetime.StopApplicationCalls.ShouldBe(0, "a graceful shutdown must not re-trigger StopApplication");
|
||||
}
|
||||
|
||||
/// <summary>Termination while the host is already stopping (ApplicationStopping fired) stays silent.</summary>
|
||||
[Fact]
|
||||
public async Task Termination_while_already_stopping_stays_silent()
|
||||
{
|
||||
var system = ActorSystem.Create("watchdog-already-stopping");
|
||||
var lifetime = new FakeHostApplicationLifetime();
|
||||
var watchdog = new ActorSystemTerminationWatchdog(
|
||||
() => system, lifetime, NullLogger<ActorSystemTerminationWatchdog>.Instance);
|
||||
|
||||
await watchdog.StartAsync(CancellationToken.None);
|
||||
|
||||
lifetime.TriggerStopping(); // the host is already tearing down (ApplicationStopping cancelled)
|
||||
await system.Terminate();
|
||||
|
||||
await Task.Delay(500, TestContext.Current.CancellationToken);
|
||||
lifetime.StopApplicationCalls.ShouldBe(0, "termination during an in-progress host shutdown must stay silent");
|
||||
}
|
||||
|
||||
private static async Task WaitForAsync(Func<bool> condition, TimeSpan timeout)
|
||||
{
|
||||
var deadline = DateTime.UtcNow + timeout;
|
||||
while (!condition() && DateTime.UtcNow < deadline)
|
||||
await Task.Delay(20);
|
||||
}
|
||||
|
||||
/// <summary>Minimal <see cref="IHostApplicationLifetime"/> that records <see cref="StopApplication"/> calls.</summary>
|
||||
private sealed class FakeHostApplicationLifetime : IHostApplicationLifetime
|
||||
{
|
||||
private readonly CancellationTokenSource _stopping = new();
|
||||
public int StopApplicationCalls { get; private set; }
|
||||
public CancellationToken ApplicationStarted => CancellationToken.None;
|
||||
public CancellationToken ApplicationStopping => _stopping.Token;
|
||||
public CancellationToken ApplicationStopped => CancellationToken.None;
|
||||
public void StopApplication() => StopApplicationCalls++;
|
||||
public void TriggerStopping() => _stopping.Cancel();
|
||||
}
|
||||
}
|
||||
@@ -23,6 +23,7 @@ public sealed class DeployHappyPathTests
|
||||
public async Task StartDeployment_seals_after_both_nodes_apply()
|
||||
{
|
||||
await using var harness = await TwoNodeClusterHarness.StartAsync();
|
||||
await harness.SeedDefaultClusterAsync();
|
||||
|
||||
await using var scope = harness.NodeA.Services.CreateAsyncScope();
|
||||
var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
|
||||
@@ -59,6 +60,7 @@ public sealed class DeployHappyPathTests
|
||||
public async Task Replaying_dispatch_to_same_revision_is_idempotent_no_op()
|
||||
{
|
||||
await using var harness = await TwoNodeClusterHarness.StartAsync();
|
||||
await harness.SeedDefaultClusterAsync();
|
||||
|
||||
await using var scope = harness.NodeA.Services.CreateAsyncScope();
|
||||
var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
|
||||
|
||||
@@ -218,8 +218,10 @@ public sealed class DriverReconnectE2eTests
|
||||
Name = "Reconnect E2E Cluster",
|
||||
Enterprise = "zb",
|
||||
Site = "central",
|
||||
NodeCount = 1,
|
||||
RedundancyMode = RedundancyMode.None,
|
||||
// Both harness nodes are Enabled driver members, so NodeCount must be 2 (Warm) to
|
||||
// satisfy the ClusterEnabledNodeCountMismatch validator + the SQL CHECK constraint.
|
||||
NodeCount = 2,
|
||||
RedundancyMode = RedundancyMode.Warm,
|
||||
CreatedBy = "test",
|
||||
});
|
||||
|
||||
@@ -240,6 +242,18 @@ public sealed class DriverReconnectE2eTests
|
||||
CreatedBy = "test",
|
||||
});
|
||||
|
||||
// The harness runs BOTH nodes as driver members, so the deployment records a
|
||||
// NodeDeploymentState for node B too — seed its ClusterNode row so the real-SQL FK
|
||||
// (FK_NodeDeploymentState_ClusterNode_NodeId) resolves. (In-memory ignores FKs.)
|
||||
db.ClusterNodes.Add(new ClusterNode
|
||||
{
|
||||
NodeId = harness.NodeBNodeId,
|
||||
ClusterId = ClusterId,
|
||||
Host = TwoNodeClusterHarness.LoopbackHost,
|
||||
ApplicationUri = "urn:zb:reconnect-e2e:node-b",
|
||||
CreatedBy = "test",
|
||||
});
|
||||
|
||||
db.DriverInstances.Add(new DriverInstance
|
||||
{
|
||||
DriverInstanceId = DriverId,
|
||||
|
||||
+16
-5
@@ -7,6 +7,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Validation;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
|
||||
@@ -28,19 +29,28 @@ public sealed class EquipmentNamespaceMaterializationTests
|
||||
{
|
||||
private static CancellationToken Ct => TestContext.Current.CancellationToken;
|
||||
|
||||
// Equipment.EquipmentId must equal DraftValidator.DeriveEquipmentId(EquipmentUuid) — the
|
||||
// 'EquipmentIdNotDerived' rule (EquipmentId = 'EQ-' + first 12 hex of the UUID). A fixed UUID
|
||||
// keeps the derived id stable for the assertions below.
|
||||
private static readonly Guid EquipmentUuid = Guid.Parse("11111111-1111-1111-1111-111111111111");
|
||||
private static readonly string EquipmentId = DraftValidator.DeriveEquipmentId(EquipmentUuid); // EQ-111111111111
|
||||
|
||||
/// <summary>Verifies a deployed Equipment namespace carries its signal into the composed artifact.</summary>
|
||||
[Fact]
|
||||
public async Task Deploying_an_equipment_namespace_carries_the_signal_into_the_artifact()
|
||||
{
|
||||
await using var harness = await TwoNodeClusterHarness.StartAsync();
|
||||
|
||||
// Seed the parent ServerCluster + ClusterNode rows the SQL FKs require (the c1 config below
|
||||
// FKs to ServerCluster; on the in-memory provider this is unnecessary — FKs aren't enforced).
|
||||
await harness.SeedDefaultClusterAsync("c1");
|
||||
await SeedEquipmentNamespaceAsync(harness);
|
||||
|
||||
await using var scope = harness.NodeA.Services.CreateAsyncScope();
|
||||
var client = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
|
||||
|
||||
var result = await client.StartDeploymentAsync(createdBy: "alice@test", Ct);
|
||||
result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted);
|
||||
result.Outcome.ShouldBe(StartDeploymentOutcome.Accepted, $"Deploy not accepted: {result.Message}");
|
||||
var deploymentId = result.DeploymentId!.Value.Value;
|
||||
|
||||
// The artifact is composed + persisted at dispatch time; assert on it without waiting for
|
||||
@@ -65,13 +75,13 @@ public sealed class EquipmentNamespaceMaterializationTests
|
||||
// ParseComposition extracted it as an EquipmentTag with FullName pulled from TagConfig.
|
||||
var tag = composition.EquipmentTags.ShouldHaveSingleItem();
|
||||
tag.TagId.ShouldBe("tag-speed");
|
||||
tag.EquipmentId.ShouldBe("eq-1");
|
||||
tag.EquipmentId.ShouldBe(EquipmentId);
|
||||
tag.Name.ShouldBe("Speed");
|
||||
tag.DataType.ShouldBe("Float");
|
||||
tag.FullName.ShouldBe("40001");
|
||||
|
||||
// The equipment folder browses by its friendly UNS Name, not the MachineCode.
|
||||
composition.EquipmentNodes.ShouldContain(e => e.EquipmentId == "eq-1" && e.DisplayName == "station-1");
|
||||
composition.EquipmentNodes.ShouldContain(e => e.EquipmentId == EquipmentId && e.DisplayName == "station-1");
|
||||
composition.UnsAreas.ShouldContain(a => a.UnsAreaId == "nw-area-filling" && a.DisplayName == "filling");
|
||||
composition.UnsLines.ShouldContain(l => l.UnsLineId == "nw-line-1" && l.DisplayName == "line-1");
|
||||
}
|
||||
@@ -95,12 +105,13 @@ public sealed class EquipmentNamespaceMaterializationTests
|
||||
db.UnsLines.Add(new UnsLine { UnsLineId = "nw-line-1", UnsAreaId = "nw-area-filling", Name = "line-1" });
|
||||
db.Equipment.Add(new Equipment
|
||||
{
|
||||
EquipmentId = "eq-1", DriverInstanceId = "drv-modbus", UnsLineId = "nw-line-1",
|
||||
EquipmentId = EquipmentId, EquipmentUuid = EquipmentUuid,
|
||||
DriverInstanceId = "drv-modbus", UnsLineId = "nw-line-1",
|
||||
Name = "station-1", MachineCode = "STATION_001",
|
||||
});
|
||||
db.Tags.Add(new Tag
|
||||
{
|
||||
TagId = "tag-speed", DriverInstanceId = "drv-modbus", EquipmentId = "eq-1",
|
||||
TagId = "tag-speed", DriverInstanceId = "drv-modbus", EquipmentId = EquipmentId,
|
||||
Name = "Speed", DataType = "Float", AccessLevel = TagAccessLevel.Read,
|
||||
TagConfig = "{\"FullName\":\"40001\"}",
|
||||
});
|
||||
|
||||
@@ -58,6 +58,7 @@ public sealed class FailoverDuringDeployTests
|
||||
// dispatch time — when only node A is Up, only one ApplyAck is expected and the
|
||||
// deployment seals without B ever participating.
|
||||
await using var harness = await TwoNodeClusterHarness.StartAsync();
|
||||
await harness.SeedDefaultClusterAsync();
|
||||
|
||||
await harness.StopNodeBAsync();
|
||||
await harness.WaitForClusterSizeAsync(1, TimeSpan.FromSeconds(20));
|
||||
|
||||
@@ -45,6 +45,7 @@ public sealed class FleetDiagnosticsRoundTripTests
|
||||
public async Task GetDiagnostics_after_deploy_reports_current_revision()
|
||||
{
|
||||
await using var harness = await TwoNodeClusterHarness.StartAsync();
|
||||
await harness.SeedDefaultClusterAsync();
|
||||
|
||||
await using var scope = harness.NodeA.Services.CreateAsyncScope();
|
||||
var adminOps = scope.ServiceProvider.GetRequiredService<IAdminOperationsClient>();
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
using System.Diagnostics;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.Auth.Abstractions.Roles;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Security.Ldap;
|
||||
using LdapTransport = ZB.MOM.WW.Auth.Abstractions.Ldap.LdapTransport;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
|
||||
|
||||
/// <summary>
|
||||
/// R2-08 03/S2 LIVE gate — the directory-outage circuit exercised against a REAL GLAuth server
|
||||
/// (the image + config PR #451 swapped in for the retired bitnami/openldap), driven through the
|
||||
/// production auth path (<see cref="OtOpcUaLdapAuthService"/> + <see cref="LdapOpcUaUserAuthenticator"/>).
|
||||
/// The offline <see cref="LdapAuthResilienceTests"/> prove the circuit logic with fakes + an
|
||||
/// unroutable-host blackhole; this proves the same behaviour against an actual directory that is
|
||||
/// paused mid-run (a real outage) and unpaused (recovery), and doubles as the first end-to-end
|
||||
/// verification that the GLAuth swap binds correctly.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Triple-gated so it never runs by accident and always restores the shared GLAuth:
|
||||
/// <list type="number">
|
||||
/// <item><c>GLAUTH_LIVE_HOST</c> — the GLAuth host (e.g. <c>10.100.0.35</c>); absent ⇒ skip;</item>
|
||||
/// <item><c>GLAUTH_OUTAGE_START_CMD</c> — pauses the GLAuth container (a real outage); and</item>
|
||||
/// <item><c>GLAUTH_OUTAGE_STOP_CMD</c> — unpauses it.</item>
|
||||
/// </list>
|
||||
/// A <c>try/finally</c> ALWAYS runs the stop command once the pause succeeded. Operator recipe:
|
||||
/// <code>
|
||||
/// ssh dohertj2@10.100.0.35 "cd ~/otopcua-harness && docker compose up -d ldap"
|
||||
/// export GLAUTH_LIVE_HOST=10.100.0.35
|
||||
/// export GLAUTH_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-harness-ldap-1"'
|
||||
/// export GLAUTH_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-harness-ldap-1"'
|
||||
/// dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapGlauthOutageLive"
|
||||
/// </code>
|
||||
/// </remarks>
|
||||
[Trait("Category", "Integration")]
|
||||
[Trait("Category", "LiveLdap")]
|
||||
public sealed class LdapGlauthOutageLiveTests
|
||||
{
|
||||
private const string HostEnvVar = "GLAUTH_LIVE_HOST";
|
||||
private const string PortEnvVar = "GLAUTH_LIVE_PORT";
|
||||
private const string StartCmdEnvVar = "GLAUTH_OUTAGE_START_CMD";
|
||||
private const string StopCmdEnvVar = "GLAUTH_OUTAGE_STOP_CMD";
|
||||
|
||||
private const int OutageThreshold = 3;
|
||||
private const int CooldownSeconds = 2;
|
||||
private static readonly TimeSpan AuthTimeout = TimeSpan.FromMilliseconds(2_500);
|
||||
|
||||
/// <summary>
|
||||
/// Healthy real binds succeed; pausing GLAuth makes consecutive binds fail bounded (never a
|
||||
/// multi-second SDK stall) and opens the circuit into a sub-second fast-deny; unpausing +
|
||||
/// cooldown lets a half-open probe close the circuit and binds succeed again.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Real_glauth_outage_opens_circuit_then_recovers_on_unpause()
|
||||
{
|
||||
var host = Environment.GetEnvironmentVariable(HostEnvVar);
|
||||
if (string.IsNullOrWhiteSpace(host))
|
||||
Assert.Skip($"Set {HostEnvVar} (+ {StartCmdEnvVar}/{StopCmdEnvVar}) to run the live GLAuth outage cycle.");
|
||||
var port = int.TryParse(Environment.GetEnvironmentVariable(PortEnvVar), out var p) ? p : 3894;
|
||||
var startCmd = Environment.GetEnvironmentVariable(StartCmdEnvVar);
|
||||
var stopCmd = Environment.GetEnvironmentVariable(StopCmdEnvVar);
|
||||
if (string.IsNullOrWhiteSpace(startCmd) || string.IsNullOrWhiteSpace(stopCmd))
|
||||
Assert.Skip($"Set both {StartCmdEnvVar} and {StopCmdEnvVar} (docker pause/unpause) to run this destructive live test.");
|
||||
|
||||
var ct = TestContext.Current.CancellationToken;
|
||||
var options = new LdapOptions
|
||||
{
|
||||
Enabled = true,
|
||||
DevStubMode = false,
|
||||
Server = host!,
|
||||
Port = port,
|
||||
Transport = LdapTransport.None,
|
||||
AllowInsecure = true,
|
||||
SearchBase = "dc=zb,dc=local",
|
||||
ServiceAccountDn = "cn=serviceaccount,dc=zb,dc=local",
|
||||
ServiceAccountPassword = "serviceaccount123",
|
||||
ConnectionTimeoutMs = 2_000,
|
||||
AuthTimeoutMs = (int)AuthTimeout.TotalMilliseconds,
|
||||
OutageFailureThreshold = OutageThreshold,
|
||||
OutageCooldownSeconds = CooldownSeconds,
|
||||
};
|
||||
var authenticator = BuildAuthenticator(options);
|
||||
|
||||
// ── 1. Healthy directory: real binds succeed, wrong creds deny (user-side, not an outage). ──
|
||||
var alice = await authenticator.AuthenticateUserNameAsync("alice", "alice123", ct);
|
||||
alice.Success.ShouldBeTrue("alice should bind against the live GLAuth");
|
||||
alice.Roles.ShouldNotBeEmpty("a healthy bind returns alice's mapped group roles");
|
||||
|
||||
var bob = await authenticator.AuthenticateUserNameAsync("bob", "bob123", ct);
|
||||
bob.Success.ShouldBeTrue("bob should bind against the live GLAuth");
|
||||
|
||||
var wrong = await authenticator.AuthenticateUserNameAsync("alice", "wrong-password", ct);
|
||||
wrong.Success.ShouldBeFalse("a wrong password is a user-side deny");
|
||||
|
||||
// ── 2. Outage: pause GLAuth. Consecutive binds fail BOUNDED, then the circuit fast-denies. ──
|
||||
var paused = false;
|
||||
try
|
||||
{
|
||||
await RunShellCommandAsync(startCmd!, ct);
|
||||
paused = true;
|
||||
|
||||
for (var i = 0; i < OutageThreshold; i++)
|
||||
{
|
||||
var sw = Stopwatch.StartNew();
|
||||
var outage = await authenticator.AuthenticateUserNameAsync("alice", "alice123", ct);
|
||||
sw.Stop();
|
||||
|
||||
outage.Success.ShouldBeFalse($"bind #{i + 1} during the outage must fail closed");
|
||||
// The whole flow is wall-clock bounded — never the SDK's default ~10-20 s stall.
|
||||
sw.Elapsed.ShouldBeLessThan(AuthTimeout + TimeSpan.FromSeconds(2),
|
||||
"the authenticator must bound a hung directory call, not stall on it");
|
||||
}
|
||||
|
||||
// Circuit is now open — the next call fast-denies WITHOUT a directory round-trip (sub-second).
|
||||
var blockedSw = Stopwatch.StartNew();
|
||||
var blocked = await authenticator.AuthenticateUserNameAsync("alice", "alice123", ct);
|
||||
blockedSw.Stop();
|
||||
|
||||
blocked.Success.ShouldBeFalse();
|
||||
blocked.Error.ShouldNotBeNull();
|
||||
blocked.Error!.ShouldContain("unavailable", Case.Insensitive,
|
||||
"an open outage circuit fast-denies 'backend unavailable'");
|
||||
blockedSw.Elapsed.ShouldBeLessThan(TimeSpan.FromSeconds(1),
|
||||
"an open circuit fast-denies without a directory call");
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (paused)
|
||||
await RunShellCommandAsync(stopCmd!, CancellationToken.None); // ALWAYS restore GLAuth
|
||||
}
|
||||
|
||||
// ── 3. Recovery: after the cooldown the half-open probe reaches the (restored) directory. ──
|
||||
await Task.Delay(TimeSpan.FromSeconds(CooldownSeconds) + TimeSpan.FromSeconds(1), ct);
|
||||
|
||||
var recovered = false;
|
||||
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
var probe = await authenticator.AuthenticateUserNameAsync("alice", "alice123", ct);
|
||||
if (probe.Success) { recovered = true; break; }
|
||||
await Task.Delay(TimeSpan.FromMilliseconds(500), ct);
|
||||
}
|
||||
recovered.ShouldBeTrue("the circuit must half-open and close once GLAuth is unpaused");
|
||||
}
|
||||
|
||||
private static LdapOpcUaUserAuthenticator BuildAuthenticator(LdapOptions options)
|
||||
{
|
||||
var ldap = new OtOpcUaLdapAuthService(Options.Create(options), NullLogger<OtOpcUaLdapAuthService>.Instance);
|
||||
return new LdapOpcUaUserAuthenticator(
|
||||
ldap, ScopeFactoryWith(new PassthroughMapper()), Options.Create(options),
|
||||
NullLogger<LdapOpcUaUserAuthenticator>.Instance);
|
||||
}
|
||||
|
||||
/// <summary>Builds an <see cref="IServiceScopeFactory"/> whose scopes resolve the supplied mapper.</summary>
|
||||
private static IServiceScopeFactory ScopeFactoryWith(IGroupRoleMapper<string> mapper)
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddScoped(_ => mapper);
|
||||
return services.BuildServiceProvider().GetRequiredService<IServiceScopeFactory>();
|
||||
}
|
||||
|
||||
/// <summary>Runs an SSH/docker one-liner through the login shell (verbatim, as in the S7 outage gate).</summary>
|
||||
private static async Task RunShellCommandAsync(string command, CancellationToken ct)
|
||||
{
|
||||
using var proc = new Process
|
||||
{
|
||||
StartInfo = new ProcessStartInfo("/bin/sh", $"-c \"{command.Replace("\"", "\\\"")}\"")
|
||||
{
|
||||
RedirectStandardOutput = true,
|
||||
RedirectStandardError = true,
|
||||
UseShellExecute = false,
|
||||
},
|
||||
};
|
||||
proc.Start();
|
||||
await proc.WaitForExitAsync(ct);
|
||||
proc.ExitCode.ShouldBe(0, $"outage command failed: {await proc.StandardError.ReadToEndAsync(ct)}");
|
||||
await Task.Delay(TimeSpan.FromSeconds(1), ct); // let the pause/unpause take effect
|
||||
}
|
||||
|
||||
/// <summary>Passes LDAP groups straight through as roles — the live gate asserts on the bind, not the mapping.</summary>
|
||||
private sealed class PassthroughMapper : IGroupRoleMapper<string>
|
||||
{
|
||||
public Task<GroupRoleMapping<string>> MapAsync(IReadOnlyList<string> groups, CancellationToken ct)
|
||||
=> Task.FromResult(new GroupRoleMapping<string>(groups, Scope: null));
|
||||
}
|
||||
}
|
||||
@@ -14,6 +14,8 @@ using ZB.MOM.WW.OtOpcUa.AdminUI.Clients;
|
||||
using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
|
||||
using ZB.MOM.WW.OtOpcUa.ControlPlane;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Health;
|
||||
@@ -36,9 +38,11 @@ namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
|
||||
/// <item><c>OTOPCUA_HARNESS_USE_SQL=1</c> → swap the in-memory DB for SQL Server on
|
||||
/// <c>localhost:14331</c> (see <c>docker-compose.yml</c>). Each harness gets a unique
|
||||
/// database name (<c>OtOpcUa_Harness_{guid}</c>) created via <c>EnsureCreated</c>
|
||||
/// and dropped via <c>EnsureDeleted</c> on dispose.</item>
|
||||
/// and dropped via <c>EnsureDeleted</c> on dispose. Override the SQL / LDAP host with
|
||||
/// <c>OTOPCUA_HARNESS_SQL_HOST</c> / <c>OTOPCUA_HARNESS_LDAP_HOST</c> to point at native
|
||||
/// fixtures on the shared Docker host (avoids the arm64-Mac emulated-mssql slowdown).</item>
|
||||
/// <item><c>OTOPCUA_HARNESS_USE_LDAP=1</c> → drop the stub and point <c>LdapAuthService</c>
|
||||
/// at OpenLDAP on <c>localhost:3894</c>.</item>
|
||||
/// at the GLAuth container on <c>localhost:3894</c> (see Docker/glauth/config.toml).</item>
|
||||
/// </list>
|
||||
///
|
||||
/// <para>Why not <c>WebApplicationFactory<Program></c>? Program.cs reads <c>OTOPCUA_ROLES</c>
|
||||
@@ -114,6 +118,53 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
|
||||
public Task<OtOpcUaConfigDbContext> CreateConfigDbContextAsync()
|
||||
=> NodeA.Services.GetRequiredService<IDbContextFactory<OtOpcUaConfigDbContext>>().CreateDbContextAsync();
|
||||
|
||||
/// <summary>
|
||||
/// Seeds a default <see cref="ServerCluster"/> plus a <see cref="ClusterNode"/> row for BOTH
|
||||
/// harness nodes (<see cref="NodeANodeId"/> / <see cref="NodeBNodeId"/>) so the real-SQL FK
|
||||
/// constraint <c>FK_NodeDeploymentState_ClusterNode_NodeId</c> is satisfied when a deployment
|
||||
/// records per-node state. The EF in-memory provider ignores FK constraints, so deploy E2E
|
||||
/// tests pass without this; against SQL Server each node's <c>NodeDeploymentState</c> INSERT
|
||||
/// fails without its parent <see cref="ClusterNode"/> row and the deployment never seals.
|
||||
/// No-op unless <c>OTOPCUA_HARNESS_USE_SQL=1</c> (in-memory needs no seeding). Call once, before
|
||||
/// <c>StartDeploymentAsync</c>, in tests that don't already seed their own cluster + both nodes.
|
||||
/// </summary>
|
||||
/// <param name="clusterId">Cluster id for the seeded rows. Defaults to <c>MAIN</c>.</param>
|
||||
public async Task SeedDefaultClusterAsync(string clusterId = "MAIN")
|
||||
{
|
||||
if (!Mode.UseSqlServer) return;
|
||||
await using var db = await CreateConfigDbContextAsync();
|
||||
db.ServerClusters.Add(new ServerCluster
|
||||
{
|
||||
ClusterId = clusterId,
|
||||
Name = "Harness Default Cluster",
|
||||
Enterprise = "zb",
|
||||
Site = "central",
|
||||
// 2-node cluster → Warm/Hot per CK_ServerCluster_RedundancyMode_NodeCount
|
||||
// (NodeCount=1↔None, NodeCount=2↔Warm|Hot). The harness runs two nodes.
|
||||
NodeCount = 2,
|
||||
RedundancyMode = RedundancyMode.Warm,
|
||||
CreatedBy = "harness",
|
||||
});
|
||||
db.ClusterNodes.AddRange(
|
||||
new ClusterNode
|
||||
{
|
||||
NodeId = NodeANodeId,
|
||||
ClusterId = clusterId,
|
||||
Host = LoopbackHost,
|
||||
ApplicationUri = "urn:zb:harness:node-a",
|
||||
CreatedBy = "harness",
|
||||
},
|
||||
new ClusterNode
|
||||
{
|
||||
NodeId = NodeBNodeId,
|
||||
ClusterId = clusterId,
|
||||
Host = LoopbackHost,
|
||||
ApplicationUri = "urn:zb:harness:node-b",
|
||||
CreatedBy = "harness",
|
||||
});
|
||||
await db.SaveChangesAsync();
|
||||
}
|
||||
|
||||
/// <summary>Boots both nodes and waits up to <paramref name="formationTimeout"/> for cluster convergence.</summary>
|
||||
/// <param name="formationTimeout">Maximum time to wait for cluster formation; defaults to 20 seconds if not provided.</param>
|
||||
/// <param name="driverFactory">Optional opt-in <see cref="IDriverFactory"/> both nodes register so deployed
|
||||
@@ -131,7 +182,12 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
|
||||
if (harness.Mode.UseSqlServer)
|
||||
{
|
||||
harness._sqlDbName = $"OtOpcUa_Harness_{Guid.NewGuid():N}";
|
||||
harness._sqlConnString = $"Server=localhost,14331;Database={harness._sqlDbName};User Id=sa;Password=OtOpcUa!Harness123;TrustServerCertificate=True;";
|
||||
// SQL host is overridable so the heavy 2-node E2E tests can target NATIVE-amd64 SQL on the
|
||||
// shared Docker host instead of the arm64 Mac's emulated mssql:2022 (no arm64 image → ~5-10×
|
||||
// slower → the deploy/failover E2E tests time out). Set OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331
|
||||
// with the harness sql container brought up there. Defaults to localhost,14331.
|
||||
var sqlHost = Environment.GetEnvironmentVariable("OTOPCUA_HARNESS_SQL_HOST") ?? "localhost,14331";
|
||||
harness._sqlConnString = $"Server={sqlHost};Database={harness._sqlDbName};User Id=sa;Password=OtOpcUa!Harness123;TrustServerCertificate=True;";
|
||||
await EnsureSqlSchemaCreatedAsync(harness._sqlConnString);
|
||||
}
|
||||
|
||||
@@ -295,13 +351,16 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
|
||||
// Bound section is Security:Ldap (see LdapOptions.SectionName); Transport replaces the
|
||||
// old UseTls bool and AllowInsecure replaces AllowInsecureLdap (Task 1.4).
|
||||
configOverrides["Security:Ldap:Enabled"] = "true";
|
||||
configOverrides["Security:Ldap:Server"] = "localhost";
|
||||
// Overridable alongside the SQL host so the GLAuth container can live on the Docker host too.
|
||||
configOverrides["Security:Ldap:Server"] = Environment.GetEnvironmentVariable("OTOPCUA_HARNESS_LDAP_HOST") ?? "localhost";
|
||||
configOverrides["Security:Ldap:Port"] = "3894";
|
||||
configOverrides["Security:Ldap:Transport"] = "None";
|
||||
configOverrides["Security:Ldap:AllowInsecure"] = "true";
|
||||
configOverrides["Security:Ldap:SearchBase"] = "dc=zb,dc=local";
|
||||
configOverrides["Security:Ldap:ServiceAccountDn"] = "cn=admin,dc=zb,dc=local";
|
||||
configOverrides["Security:Ldap:ServiceAccountPassword"] = "ldapadmin";
|
||||
// GLAuth's search-capable bind account (see Docker/glauth/config.toml) — matches the
|
||||
// shared GLAuth the data-plane binds against. Replaced the retired OpenLDAP cn=admin.
|
||||
configOverrides["Security:Ldap:ServiceAccountDn"] = "cn=serviceaccount,dc=zb,dc=local";
|
||||
configOverrides["Security:Ldap:ServiceAccountPassword"] = "serviceaccount123";
|
||||
}
|
||||
|
||||
builder.Configuration.AddInMemoryCollection(configOverrides);
|
||||
|
||||
@@ -6,8 +6,8 @@
|
||||
#
|
||||
# 1. EF behaviors that diverge between provider implementations — index uniqueness,
|
||||
# RowVersion concurrency, JSON column round-trips, EF migration application.
|
||||
# 2. Real LDAP binds against an OpenLDAP server with the dev users from
|
||||
# C:\publish\glauth\auth.md.
|
||||
# 2. Real LDAP binds against a GLAuth server (./glauth/config.toml) with dev users
|
||||
# alice (full access) / bob (read-only) and the cn=serviceaccount bind account.
|
||||
#
|
||||
# Activate by setting these env vars before running the suite:
|
||||
#
|
||||
@@ -43,15 +43,13 @@ services:
|
||||
retries: 20
|
||||
|
||||
ldap:
|
||||
# OpenLDAP image — same one docker-dev/ uses, just on a different port. Dev users
|
||||
# alice/bob match the GLAuth fixtures so AuthEndpoints contract tests share creds.
|
||||
image: bitnami/openldap:2.6
|
||||
environment:
|
||||
LDAP_ROOT: "dc=zb,dc=local"
|
||||
LDAP_ADMIN_USERNAME: "admin"
|
||||
LDAP_ADMIN_PASSWORD: "ldapadmin"
|
||||
LDAP_USERS: "alice,bob"
|
||||
LDAP_PASSWORDS: "alice123,bob123"
|
||||
LDAP_USER_DC: "ou=FleetAdmin"
|
||||
# GLAuth — the same LDAP server the rest of the project uses (docker-dev + the live
|
||||
# OPC UA data-plane auth, both against the shared GLAuth on :3893). Replaced the retired
|
||||
# bitnami/openldap:2.6 (Bitnami deprecated their free catalog in 2025). Seeded by
|
||||
# ./glauth/config.toml with a search-capable serviceaccount + dev users alice/bob.
|
||||
# Listens on :3893 in-container; mapped to host :3894 to run beside docker-dev's :3893.
|
||||
image: glauth/glauth:latest
|
||||
volumes:
|
||||
- ./glauth/config.toml:/app/config/config.cfg:ro
|
||||
ports:
|
||||
- "3894:1389"
|
||||
- "3894:3893"
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
# GLAuth config for the Host.IntegrationTests real-LDAP mode (OTOPCUA_HARNESS_USE_LDAP=1).
|
||||
#
|
||||
# Replaces the retired bitnami/openldap:2.6 image (Bitnami deprecated their free catalog
|
||||
# in 2025) with GLAuth — the same LDAP server the rest of the project uses (docker-dev +
|
||||
# the live OPC UA data-plane auth, both against the shared GLAuth on 10.100.0.35:3893).
|
||||
# Unifying on GLAuth removes the lone OpenLDAP outlier and the gone-image breakage.
|
||||
#
|
||||
# Mirrors scadaproj/infra/glauth/config.toml (source of truth) but trimmed to the harness's
|
||||
# needs: baseDN dc=zb,dc=local, a search-capable service account, and two dev users
|
||||
# (alice = full-access, bob = read-only). Bind DN form is GLAuth's cn=<name>,dc=zb,dc=local.
|
||||
#
|
||||
# Listens on :3893 inside the container; the compose maps host :3894 -> :3893 so this runs
|
||||
# side-by-side with docker-dev's shared GLAuth (:3893) without a port clash.
|
||||
|
||||
[ldap]
|
||||
enabled = true
|
||||
listen = "0.0.0.0:3893"
|
||||
|
||||
[ldaps]
|
||||
enabled = false
|
||||
|
||||
[backend]
|
||||
datastore = "config"
|
||||
baseDN = "dc=zb,dc=local"
|
||||
|
||||
[behaviors]
|
||||
# Tests may bind bad passwords on purpose (outage / wrong-cred paths) — don't lock out.
|
||||
LimitFailedBinds = false
|
||||
|
||||
# ── Groups (gidnumbers match scadaproj/infra/glauth so GroupToRole maps identically) ──
|
||||
[[groups]]
|
||||
name = "ReadOnly"
|
||||
gidnumber = 5601
|
||||
[[groups]]
|
||||
name = "WriteOperate"
|
||||
gidnumber = 5602
|
||||
[[groups]]
|
||||
name = "WriteTune"
|
||||
gidnumber = 5603
|
||||
[[groups]]
|
||||
name = "WriteConfigure"
|
||||
gidnumber = 5604
|
||||
[[groups]]
|
||||
name = "AlarmAck"
|
||||
gidnumber = 5605
|
||||
|
||||
# ── Users ─────────────────────────────────────────────────────────────
|
||||
# Service account the harness binds first to search for the user entry.
|
||||
# DN: cn=serviceaccount,dc=zb,dc=local password: serviceaccount123
|
||||
[[users]]
|
||||
name = "serviceaccount"
|
||||
uidnumber = 5999
|
||||
primarygroup = 5601
|
||||
passsha256 = "af29d0e5c9801ae98a999ed3915e1cf428a64b4b62b3cf221b6336cce0398419"
|
||||
[[users.capabilities]]
|
||||
action = "search"
|
||||
object = "*"
|
||||
|
||||
# alice — full access (member of every write/ack role group). password: alice123
|
||||
[[users]]
|
||||
name = "alice"
|
||||
givenname = "Alice"
|
||||
sn = "Admin"
|
||||
mail = "alice@zb.local"
|
||||
uidnumber = 5010
|
||||
primarygroup = 5604
|
||||
othergroups = [5601, 5602, 5603, 5605]
|
||||
passsha256 = "4e40e8ffe0ee32fa53e139147ed559229a5930f89c2204706fc174beb36210b3"
|
||||
|
||||
# bob — read-only. password: bob123
|
||||
[[users]]
|
||||
name = "bob"
|
||||
givenname = "Bob"
|
||||
sn = "Reader"
|
||||
mail = "bob@zb.local"
|
||||
uidnumber = 5011
|
||||
primarygroup = 5601
|
||||
passsha256 = "8d059c3640b97180dd2ee453e20d34ab0cb0f2eccbe87d01915a8e578a202b11"
|
||||
@@ -72,16 +72,15 @@ public sealed class DualEndpointTests
|
||||
// SDK 1.5.374 sync-style session-open path — mirrors src/Client/.../DefaultSessionFactory.cs
|
||||
// and DefaultApplicationConfigurationFactory.cs. The 1.5.378 telemetry/async overloads are
|
||||
// not available at this pinned version.
|
||||
var clientPki = TestClientSecurity.AllocatePkiRoot();
|
||||
try
|
||||
{
|
||||
var appConfig = new ApplicationConfiguration
|
||||
{
|
||||
ApplicationName = "OtOpcUa.DualEndpointClient",
|
||||
ApplicationUri = $"urn:OtOpcUa.DualEndpointClient.{Guid.NewGuid():N}",
|
||||
ApplicationType = ApplicationType.Client,
|
||||
SecurityConfiguration = new SecurityConfiguration
|
||||
{
|
||||
ApplicationCertificate = new CertificateIdentifier(),
|
||||
AutoAcceptUntrustedCertificates = true,
|
||||
},
|
||||
SecurityConfiguration = TestClientSecurity.Build(clientPki),
|
||||
ClientConfiguration = new ClientConfiguration { DefaultSessionTimeout = 60_000 },
|
||||
};
|
||||
await appConfig.ValidateAsync(ApplicationType.Client, default);
|
||||
@@ -107,6 +106,11 @@ public sealed class DualEndpointTests
|
||||
var value = await session.ReadValueAsync(VariableIds.Server_ServerArray, default);
|
||||
return (string[])value.Value;
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (Directory.Exists(clientPki)) Directory.Delete(clientPki, recursive: true);
|
||||
}
|
||||
}
|
||||
|
||||
private static int AllocateFreePort()
|
||||
{
|
||||
|
||||
+13
-10
@@ -215,9 +215,11 @@ public sealed class SubscriptionSurvivalTests
|
||||
sink.WriteValue(nodeIdAString, 11, Commons.OpcUa.OpcUaQuality.Good, DateTime.UtcNow);
|
||||
await WaitUntilAsync(() => { lock (gate) return receivedA.Any(v => Equals(v.Value, 11)); }, TimeSpan.FromSeconds(5));
|
||||
|
||||
// B is gone: a re-read returns BadNodeIdUnknown.
|
||||
var bRead = await session.ReadValueAsync(nodeIdB, ct);
|
||||
bRead.StatusCode.ShouldBe((StatusCode)StatusCodes.BadNodeIdUnknown);
|
||||
// B is gone: a re-read surfaces BadNodeIdUnknown. The 1.5.378 node-cache read path
|
||||
// throws ServiceResultException for an unknown node rather than returning a bad DataValue.
|
||||
var bReadEx = await Should.ThrowAsync<ServiceResultException>(
|
||||
async () => await session.ReadValueAsync(nodeIdB, ct));
|
||||
bReadEx.StatusCode.ShouldBe(StatusCodes.BadNodeIdUnknown);
|
||||
}
|
||||
finally
|
||||
{
|
||||
@@ -296,9 +298,11 @@ public sealed class SubscriptionSurvivalTests
|
||||
sink.WriteValue(nodeIdAString, 11, Commons.OpcUa.OpcUaQuality.Good, DateTime.UtcNow);
|
||||
await WaitUntilAsync(() => { lock (gate) return receivedA.Any(v => Equals(v.Value, 11)); }, TimeSpan.FromSeconds(5));
|
||||
|
||||
// B is gone.
|
||||
// B is gone. (1.5.378 node-cache read throws for an unknown node — see above.)
|
||||
var nodeIdB = new NodeId(Commons.OpcUa.EquipmentNodeIds.Variable("eq-1", "", "B"), ns);
|
||||
(await session.ReadValueAsync(nodeIdB, ct)).StatusCode.ShouldBe((StatusCode)StatusCodes.BadNodeIdUnknown);
|
||||
var bReadEx = await Should.ThrowAsync<ServiceResultException>(
|
||||
async () => await session.ReadValueAsync(nodeIdB, ct));
|
||||
bReadEx.StatusCode.ShouldBe(StatusCodes.BadNodeIdUnknown);
|
||||
|
||||
// C is browsable + readable.
|
||||
var nodeIdC = new NodeId(Commons.OpcUa.EquipmentNodeIds.Variable("eq-1", "", "C"), ns);
|
||||
@@ -331,11 +335,10 @@ public sealed class SubscriptionSurvivalTests
|
||||
ApplicationName = "OtOpcUa.SubscriptionSurvivalClient",
|
||||
ApplicationUri = $"urn:OtOpcUa.SubscriptionSurvivalClient.{Guid.NewGuid():N}",
|
||||
ApplicationType = ApplicationType.Client,
|
||||
SecurityConfiguration = new SecurityConfiguration
|
||||
{
|
||||
ApplicationCertificate = new CertificateIdentifier(),
|
||||
AutoAcceptUntrustedCertificates = true,
|
||||
},
|
||||
// Directory cert stores rooted at a throwaway temp PKI dir — ValidateAsync rejects a
|
||||
// bare SecurityConfiguration ("TrustedIssuerCertificates StorePath must be specified").
|
||||
// The returned session outlives this method, so the dir is left for the OS temp sweep.
|
||||
SecurityConfiguration = TestClientSecurity.Build(TestClientSecurity.AllocatePkiRoot()),
|
||||
ClientConfiguration = new ClientConfiguration { DefaultSessionTimeout = 60_000 },
|
||||
};
|
||||
await appConfig.ValidateAsync(ApplicationType.Client, ct);
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
using Opc.Ua;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests;
|
||||
|
||||
/// <summary>
|
||||
/// Builds the client-side <see cref="SecurityConfiguration"/> the integration tests use to
|
||||
/// open a real OPC UA session. The SDK's <c>ApplicationConfiguration.ValidateAsync</c> requires
|
||||
/// the trusted-issuer / trusted-peer / rejected stores to carry an explicit <c>StorePath</c>
|
||||
/// (it throws <c>TrustedIssuerCertificates StorePath must be specified</c> otherwise), so — unlike
|
||||
/// a bare <c>new SecurityConfiguration()</c> — each store is rooted at a throwaway PKI directory,
|
||||
/// mirroring <c>src/Client/.../DefaultApplicationConfigurationFactory.cs</c>.
|
||||
/// </summary>
|
||||
internal static class TestClientSecurity
|
||||
{
|
||||
/// <summary>Allocates a fresh throwaway client PKI root under the temp directory.</summary>
|
||||
/// <returns>An absolute path that does not yet exist; the SDK creates the stores under it.</returns>
|
||||
public static string AllocatePkiRoot() =>
|
||||
Path.Combine(Path.GetTempPath(), $"otopcua-client-pki-{Guid.NewGuid():N}");
|
||||
|
||||
/// <summary>Builds an auto-accepting client security config with Directory stores under <paramref name="pkiRoot"/>.</summary>
|
||||
/// <param name="pkiRoot">Root directory for the own / issuer / trusted / rejected stores.</param>
|
||||
/// <returns>A validated-ready <see cref="SecurityConfiguration"/>.</returns>
|
||||
public static SecurityConfiguration Build(string pkiRoot) => new()
|
||||
{
|
||||
ApplicationCertificate = new CertificateIdentifier
|
||||
{
|
||||
StoreType = CertificateStoreType.Directory,
|
||||
StorePath = Path.Combine(pkiRoot, "own"),
|
||||
},
|
||||
TrustedIssuerCertificates = new CertificateTrustList
|
||||
{
|
||||
StoreType = CertificateStoreType.Directory,
|
||||
StorePath = Path.Combine(pkiRoot, "issuer"),
|
||||
},
|
||||
TrustedPeerCertificates = new CertificateTrustList
|
||||
{
|
||||
StoreType = CertificateStoreType.Directory,
|
||||
StorePath = Path.Combine(pkiRoot, "trusted"),
|
||||
},
|
||||
RejectedCertificateStore = new CertificateTrustList
|
||||
{
|
||||
StoreType = CertificateStoreType.Directory,
|
||||
StorePath = Path.Combine(pkiRoot, "rejected"),
|
||||
},
|
||||
AutoAcceptUntrustedCertificates = true,
|
||||
};
|
||||
}
|
||||
Reference in New Issue
Block a user