docs: document OPC UA 2.0 compatibility and rollout

2026-11-10 11:47:41 -05:00
5 changed files with 445 additions and 170 deletions
@@ -22,6 +22,37 @@ Design references: [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](pl
 ---
 ## SDK 2.0.1 compatibility (#782)
 The server targets exact OPC Foundation SDK **2.0.1**, replacing **1.5.378.182**.
 The upgrade preserves existing application results, including the SQL backends'
 documented divergences from the old SDK below. Live `AggregateFilter` calculations
 use an application-private source port of the **37 legacy aggregate factories**,
 registered through the public SDK aggregate manager; they do not adopt SDK 2's new
 algorithms. Runtime and advertised aggregate defaults retain
 `TreatUncertainAsBad=false`. History backend capability sets remain separate from
 the live aggregate set; no backend gains aggregates or history operations through
 this upgrade.
 The isolated [original-SDK oracle](../scripts/validation/opcua-aggregate-baseline/README.md)
 generates **6,144 vectors** using the published 1.5.378.182 package, covering value
 kinds, statuses, timestamps/picoseconds, reverse reads and edge configurations.
 These are expected-output fixtures, not a claim that release validation passed.
 The candidate never loads the old SDK binary. Older references below to "the SDK"
 as an aggregate oracle mean that preserved baseline, rather than SDK 2's algorithms.
 SDK FILETIME-zero timestamps are translated back to the application's existing CLR
 unset-time sentinel at history boundaries. Opaque history continuation points remain
 session-owned, node/topology-bound and subject to current authorization; reconnect or
 rollback requires a fresh read. Request cancellation flows to the backend. No
 ConfigDb migration is planned; actual schema and serialization effects must be
 recorded at the [rollout gate](plans/2026-11-10-opcua-sdk-2-rollout.md).
 `HistorianGateway.Client` and `HistorianGateway.Contracts` are already **0.4.0**
 (on gateway master `16111a5e`, historiangw#119 closed). This package alignment is
 separate from the real gateway validation tracked by historiangw#78; it is not
 live AVEVA or backend-conformance evidence for SDK 2.
 ## Historized TagConfig schema
 A tag is historized by setting the **Historize this tag** checkbox and optional **Historian
@@ -1,185 +1,88 @@
 # OPC UA Client test fixture
-Coverage map + gap inventory for the OPC UA Client (gateway / aggregation)
+This is the fixture and command map for the driver/browser upgrade from SDK
-driver.
+**1.5.378.182** to exact **2.0.1** (#782). It describes available tests; execution,
 counts, skips and release decisions belong in [#782](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/782)
 and the linked [execution evidence](../plans/2026-11-10-opcua-sdk-2-evidence.md).
 A listed test or historical run is not candidate release evidence.
-**TL;DR:** Wire-level coverage now exists via
+## Fixture layers
 [opc-plc](https://github.com/Azure-Samples/iot-edge-opc-plc) — Microsoft
 Industrial IoT's OPC UA PLC simulator running in Docker (task #215). Real
 Secure Channel, real Session, real MonitoredItem exchange against an
 independent server implementation. Unit tests still carry the exhaustive
 capability matrix (cert auth / security policies / reconnect / failover /
 attribute mapping). Gaps remaining: upstream-server-specific quirks
 (historian aggregates, typed ConditionType events, SDK-publish-queue edge
 behavior under load) — opc-plc uses the same OPCFoundation stack internally
 so fully-independent-stack coverage needs `open62541/open62541` as a second
 image (follow-up).
-## What the fixture is
+| Layer | Repository location | Scope and limitation |
 |---|---|---|
 | Driver unit | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests/` | Faked session contracts: lifecycle/generations, value conversion, per-item errors, policy selection, credentials, namespace remapping, subscription cleanup, discovery, alarms and host-owned reinitialization. No real secure channel. |
 | Browser unit | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser.Tests/` | Browse attributes/pagination and certificate-identity ownership, including failure cleanup. No external server. |
 | In-process wire | `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/` | Real driver against OtOpcUa: `OpcUaClientDriverWireTests`, `OpcUaClientDriverStallWireTests`, `OpcUaClientDriverReconnectWireTests`. Real channels but both sides use this repository's SDK. |
 | External simulator | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests/` | `OpcUaClientSmokeTests`: real read, varied-type batch read and subscription against opc-plc. Does not prove all upstream products, secured authentication, write or alarm compatibility. |
 | Browser integration | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser.IntegrationTests/` | Separate fixture-backed browser tests; inspect fixture availability and individual results. |
-**Integration layer** (task #215):
+The SDK 2 unit boundary cases include `OpcUaClientSdkValueBoundaryTests` and
-`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests/` stands up
+`OpcUaClientContinuationPointReleaseTests`. They preserve CLR value types and
-`mcr.microsoft.com/iotedge/opc-plc:2.14.10` via `Docker/docker-compose.yml`
+null/empty buffers, reject unsupported matrix items without shifting batch results,
-on `opc.tcp://10.100.0.35:50000` (the shared Docker host; override via
+and release abandoned browse points. Certificate tests cover lifetime through
-`OPCUA_SIM_ENDPOINT`). `OpcPlcFixture` probes the port at
+session close; unit mocks alone do not establish trust-chain interoperability.
 collection init + skips tests with a clear message when the container's
 not running (matches the Modbus/pymodbus + S7/python-snap7 skip pattern).
 Docker is the launcher — no PowerShell wrapper needed because opc-plc
 ships pre-containerized. Compose-file flags: `--ut` (unsecured transport
 advertised), `--aa` (auto-accept client certs — opc-plc's cert trust store
 resets on each spin-up), `--alm` (alarm simulation for IAlarmSource
 follow-up coverage), `--pn=50000` (port).
-**Unit layer**:
+The in-process stall proxy holds bytes without closing the session, then resumes;
-`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests/` is still the primary
+its hard-close case refuses connections. These exercise stale notification rejection,
-coverage. Tests inject fakes through the driver's construction path; the
+quality transitions and explicit host reinitialization. The driver does not use
-OPCFoundation.NetStandard `Session` surface is wrapped behind an interface
+`SessionReconnectHandler` or `TransferSubscriptions`; historical transfer-based
-the tests mock.
+coverage descriptions do not apply to the current lifecycle (#670/#676).
-## What it actually covers
+The driver has **no upstream HistoryRead capability** (#655). Its tags can be
 recorded into the local server-wide historian; backend conformance and aggregate
 oracles are documented in [Historian.md](../Historian.md).
-### Integration (opc-plc Docker, task #215)
+## Running locally
- `OpcUaClientSmokeTests.Client_connects_and_reads_StepUp_node_through_real_OPC_UA_stack` —
+Run from the repository root. Build first and use a fresh result directory for
-  full Secure Channel + Session + `ns=3;s=StepUp` Read round-trip
+**each** project. The [runbook](../plans/2026-11-10-opcua-sdk-2-rollout.md#verification-commands)
- `OpcUaClientSmokeTests.Client_reads_batch_of_varied_types_from_live_simulator` —
+provides the result helper that preserves test exit status and rejects all-skipped
-  batch Read of UInt32 / Int32 / Boolean; asserts `bool`-specific Variant
+runs. For example, after defining that helper:
  decoding to catch a common attribute-mapping regression
 - `OpcUaClientSmokeTests.Client_subscribe_receives_StepUp_data_changes_from_live_server` —
  real `MonitoredItem` subscription against `ns=3;s=FastUInt1` (ticks every
  100 ms); asserts `OnDataChange` fires within 3 s of subscribe
-Wire-level surfaces verified: `IDriver` + `IReadable` + `ISubscribable` +
+```bash
-`IHostConnectivityProbe` (via the Secure Channel exchange).
+opc_test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests.csproj
 opc_test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser.Tests/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser.Tests.csproj
 opc_test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.csproj
 ```
-### In-process wire tests (no opc-plc)
+The simulator compose file is
 `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests/Docker/docker-compose.yml`.
 It launches `mcr.microsoft.com/iotedge/opc-plc:2.14.10` with `--ut --aa --alm --pn=50000`.
 Use the shared-host procedure in [infra/README.md](../../infra/README.md); coordinate
 shared fixture use before restarting containers. These dev flags do not establish
 production trust or secure-policy coverage.
-`tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/OpcUaClientDriverWireTests` drives the real driver
+`OpcPlcFixture` probes `OPCUA_SIM_ENDPOINT` (default `opc.tcp://10.100.0.35:50000`)
-against the in-process OtOpcUa server (`ServerEventRig`, authenticating with the rig's API-key credential):
+and skips when unavailable. With an available, reserved fixture:
 read (+ missing node), write through the node-write gateway (+ read-only refusal, + failed-write revert),
 subscribe/unsubscribe, host-status Running -> not-Running when the server stops, and discovery. These need
 no Docker. There is no HistoryRead leg: the driver has no history capability (#655).
-### Unit
+```bash
 export OPCUA_SIM_ENDPOINT=opc.tcp://10.100.0.35:50000
 opc_test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests.csproj
 ```
-The surface is broad because `OpcUaClientDriver` is the richest-capability
+A nonzero suite count does not prove every required scenario ran: inspect individual
-driver in the fleet (it's a gateway for another OPC UA server, so it
+TRX results and record required skips separately. No tests were run by preparing
-mirrors the full capability matrix):
+this documentation.
- `OpcUaClientDriverScaffoldTests` — `IDriver` lifecycle
+## Required compatibility evidence
 - `OpcUaClientReadWriteTests` — read + write lifecycle
 - `OpcUaClientSubscribeAndProbeTests` — monitored-item subscription + probe
  state transitions
 - `OpcUaClientDiscoveryTests` — `GetEndpoints` + endpoint selection
 - `OpcUaClientAttributeMappingTests` — OPC UA node attribute → driver value
  mapping
 - `OpcUaClientSecurityPolicyTests` — `SignAndEncrypt` / `Sign` / `None`
  policy negotiation contract
 - `OpcUaClientCertAuthTests` — cert store paths, revocation-list config
 - `OpcUaClientReconnectTests` — SDK reconnect hook + `TransferSubscriptions`
  across the disconnect boundary
 - `OpcUaClientFailoverTests` — primary → secondary session fallback per
  driver config
 - `OpcUaClientAlarmTests` — A&E severity bucket (1–1000 → Low / Medium /
  High / Critical), subscribe / unsubscribe / ack contract
-Capability surfaces whose contract is verified: `IDriver`, `ITagDiscovery`,
+Release validation must identify both application builds and SDK versions, endpoint
-`IReadable`, `IWritable`, `ISubscribable`, `IHostConnectivityProbe`,
+policy, token kind, certificate chain, configuration and executed capabilities.
-`IAlarmSource`.
+Use the baseline client against the candidate server and candidate client against
 the baseline server, plus the actual ScadaBridge adapter and external upstreams.
 For secured paths include trusted SignAndEncrypt, API-key allow/deny/revocation,
 certificate rejection and restart/rebind. Apply only operations the peer implements;
 a simulator need not implement OtOpcUa's API-key or historian contract.
-## What it does NOT cover
+For alarms verify typed EventFilter select clauses, result-field indexes, actual
 ConditionType/SourceNode/EventId identity, refresh, acknowledgements and no replay
 after an ambiguous transport failure. For all supported paths preserve existing
 operation statuses, locale behavior and null/empty filter meaning. Constructing an
 empty SDK 2 `ContentFilter` is an API adaptation, not evidence of a changed filter
 contract. Record any stricter protocol response and consumer impact before acceptance.
-### 1. Full real-stack exchange (unit tests only)
+opc-plc uses the OPC Foundation stack too. Independent upstream behavior, secured
-
+authentication variants, large publish queues, real alarm products, packet-loss
-The **unit** suite mocks `Session.ReadAsync`, `Session.CreateSubscription`,
+windows and performance require separate evidence. The rollout runbook links the
-`Session.AddItem`, etc. — no UA Secure Channel is opened. The **integration**
+pending gates; no simulator run substitutes for ScadaBridge, live gateway, Windows,
-suite (`OpcUaClientSmokeTests`, task #215) does open a real Secure Channel
+Linux or mixed-version cluster validation.
 against opc-plc and exercises Read + Subscribe end-to-end. What remains
 untested even in the integration suite: certificate validation under
 non-anonymous security policies, signing/encryption, nonce handling, chunk
 assembly, keep-alive cadence — all SDK-internal.
 ### 2. Subscription transfer across reconnect
 Contract test: "after a simulated reconnect, `TransferSubscriptions` is
 called with the right handles." Real behavior: SDK re-publishes against the
 new channel and some events can be lost depending on publish-queue state.
 The lossy window is not characterized.
 ### 3. Large-scale subscription stress
 100+ monitored items with heterogeneous publish intervals under a single
 session — the shape that breaks publish-queue-size tuning in the wild — is
 not exercised.
 ### 4. Real A&E events
 Alarm subscription is mocked via filtered monitored items; the actual
 `EventFilter` select-clause behavior against a server that exposes typed
 ConditionType events (non-base `BaseEventType`) is not verified.
 ### 6. Authentication variants
 - Anonymous, UserName/Password, X509 cert tokens — each is contract-tested
  but not exchanged against a server that actually enforces each.
 - LDAP-backed `UserName` (matching this repo's server-side
  `LdapOpcUaUserAuthenticator`) requires a live LDAP round-trip; not tested.
 ## When to trust OpcUaClient tests, when to reach for a server
 | Question | Unit tests | Integration (opc-plc) | Real upstream server |
 | --- | --- | --- | --- |
 | "Does severity 750 bucket as High?" | yes | - | yes |
 | "Does the driver call `TransferSubscriptions` after reconnect?" | yes | - | yes |
 | "Does a real OPC UA read round-trip work?" | no | yes | yes |
 | "Does a real OPC UA subscribe deliver changes?" | no | yes | yes |
 | "Does write round-trip work against a live server?" | no | no (not yet exercised) | yes (required) |
 | "Does event-filter-based alarm subscription return ConditionType events?" | no | no | yes (required) |
 | "Does history read from AVEVA Historian return correct aggregates?" | no | no | yes (required) |
 | "Does the SDK's publish queue lose notifications under load?" | no | no | yes (stress) |
 ## Follow-up candidates
 The easiest win here is to **wire the client driver tests against this
 repo's own server**. The v2 integration test project
 `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/DualEndpointTests.cs`
 (the v2 replacement for the retired v1 `OpcUaServerIntegrationTests`) already
 stands up a real OPC UA server on a non-default port with a seeded
 FakeDriver. An `OpcUaClientLiveLoopbackTests` that connects the client
 driver to that server would give:
 - Real Secure Channel negotiation
 - Real Session / Subscription / MonitoredItem exchange
 - Real read/write round-trip
 - Real certificate validation (the integration test already sets up PKI)
 It wouldn't cover upstream-server-specific quirks (AVEVA Historian, Kepware,
 Prosys), but it would cover 80% of the SDK surface the driver sits on top
 of.
 Beyond that:
 1. **Prosys OPC UA Simulation Server** — free, Windows-available, scriptable.
 2. **UaExpert Server-Side Simulator** — Unified Automation's sample server;
   good coverage of typed ConditionType events.
 3. **Dedicated historian integration lab** — only path for
   historian-specific coverage.
 ## Key fixture / config files
 - `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests/` — unit tests with
  mocked `Session`
 - `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests/OpcPlcFixture.cs`
  — collection fixture; parses `OPCUA_SIM_ENDPOINT` (default
  `opc.tcp://10.100.0.35:50000`), TCP-probes at collection init, records
  `SkipReason` when unreachable
 - `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests/OpcUaClientSmokeTests.cs`
  — wire-level test suite (3 `[Fact]` methods: read, batch read, subscribe)
 - `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests/Docker/docker-compose.yml`
  — `mcr.microsoft.com/iotedge/opc-plc:2.14.10` with `--ut --aa --alm --pn=50000`
 - `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriver.cs` — ctor +
  session-factory seam tests mock through; implements `IAlarmSource`;
  does NOT implement `IRediscoverable`
 - `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/DualEndpointTests.cs` —
  the v2 dual-endpoint integration harness a future loopback client test could
  piggyback on (v1 `OpcUaServerIntegrationTests.cs` retired with the v1 server project)
@@ -9,8 +9,9 @@ and for the test-harness map read [OpcUaClient-Test-Fixture.md](OpcUaClient-Test
 A **remote OPC UA server**. This driver runs the *opposite* direction from the
 usual "server exposes PLC data" flow: it acts as an OPC UA **client**, opens a
 `Session` against an upstream server, and re-exposes that server's address space
-through the local OtOpcUa server. Browse, read, write, subscribe, alarm, and
+through the local OtOpcUa server. Browse, read, write, subscribe and alarm
-history calls are passed through to the upstream endpoint.
+operations use the upstream endpoint. HistoryRead uses the local server-wide
 historian instead; see [History](#history-not-supported-gitea-655).
 It is built on the OPC Foundation UA .NET Standard reference SDK and runs
 in-process in the OtOpcUa server's .NET 10 AnyCPU host — pure managed, no
@@ -20,6 +21,31 @@ out-of-process isolation.
 > a remote OPC UA endpoint by hand, point the general-purpose
 > [Client CLI](../Client.CLI.md) at it directly.
 ## SDK 2.0.1 compatibility (#782)
 The driver, shared session layer and AdminUI browser target exact SDK **2.0.1**
 (previous **1.5.378.182**). Their existing abstractions and host-owned reconnect
 lifecycle remain in place. SDK immutable values are detached at the boundary into
 existing CLR scalars and one-dimensional arrays, preserving null/empty ByteStrings,
 UTC timestamps, Guid values, numeric widths, item order and per-item status. Matrix
 payloads remain unsupported and fail per item; this upgrade adds no matrix or
 upstream-history capability.
 Certificate-based identities retain their loaded certificate until their owning
 session closes, including failure cleanup. The browser releases abandoned browse
 continuation points. Endpoint policy selection, namespace-URI remapping, old-session
 notification rejection and the existing refusal to replay non-idempotent writes or
 alarm acknowledgements remain compatibility requirements.
 No application status-code, locale-selection or event-filter semantic change is
 approved by this upgrade. SDK value wrappers and empty-filter construction are API
 adaptations; callers must continue to inspect operation and select-clause results
 by index. Confirm actual client/server combinations with the
 [fixture map](OpcUaClient-Test-Fixture.md) and
 [rollout gates](../plans/2026-11-10-opcua-sdk-2-rollout.md) before deployment.
 Historical SDK 1.5 observations elsewhere on this page describe their original
 measurements; they are not fresh SDK 2.0.1 interoperability evidence.
 ## Project split
 | Project | Target | Role |
@@ -365,7 +391,7 @@ They are not part of `DriverHealth` or the
 ## Testing
 - **Unit tests** — `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests/`
-  cover the session lifecycle, namespace remapping, alarm/history passthrough,
+  cover the session lifecycle, namespace remapping, alarm dispatch,
  and config binding against a faked SDK session.
 - **Integration fixture** — exercises the driver against a reference OPC UA
  server (opc-plc) on the shared docker host; see
@@ -0,0 +1,284 @@
 # OPC UA SDK 2.0.1 rollout and rollback
 This operator runbook accompanies [#782](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/782),
 the [implementation plan](2026-11-10-opcua-sdk-2-upgrade.md) and
 [execution evidence](2026-11-10-opcua-sdk-2-evidence.md). Gitea is the single
 status system. This draft was checked against source `a21104d2`; it does not
 approve an artifact or record a deployment. The coordinator must attach final
 commit-specific evidence in #782 before advancing a gate. All placeholders below
 are **pending evidence**, not passing results.
 The target is exact OPC Foundation SDK **2.0.1**; the known-good SDK baseline is
 **1.5.378.182** (application baseline `83ed880e4471344deaf008da9bd028c7e170bdc7`).
 Restore the actual archived known-good application as a unit during rollback.
 Package downgrades alone cannot reverse the application/API migration.
 ## Operating constraints
 **The remote-execution hold on wonder boxes vd03/vq03/vq04 remains binding.**
 No WinRM, psbridge or replacement remote-exec channel may be used until the user
 clears a channel. UNC/SMB file copying and permitted SQL access do not authorize
 service control, PowerShell execution or deployment on those hosts. Steps requiring
 a command on a held host remain pending. No deployment was performed for this draft.
 Run the service commands below only from an approved local operator session on an
 authorized Windows node. Record the node, account, channel and maintenance window
 in #782 first. Linux runtime/service handling requires its own approved environment
 and smoke record; this runbook supplies no invented Linux service-unit name.
 Do not use `scripts/install/Refresh-Services.ps1` as a blanket upgrade action: it
 also touches gateway/retired sidecar paths outside this SDK rollout.
 ## Preserved contract
 - Existing raw/UNS namespace URIs, identifier forms, endpoint profiles, API-key/anonymous
  admission, ACL flag behavior, role and key revocation gates remain the contract.
  Source uses public transport peer evidence for miss budgets, not an IP admission rule.
 - Scalar leniency, checked numeric-array conversion and pass-through driver behavior
  stay as before. SDK immutable values are detached at boundaries; no new matrix support.
 - No application status-code, locale or event-filter semantic change is approved.
  SDK status wrappers and explicit empty `ContentFilter` construction are API changes,
  not evidence of changed protocol values. Record any stricter response, field-level
  filter error or localization difference with a consumer-impact decision before go.
 - Auditing stays opt-in. Ordinary SDK automatic audit delivery retains its type
  restrictions. Only the application's rejected-write audit gets a private, temporary
  receive-permission context after current source `Read` ACL and audit/encryption checks;
  no shared role metadata or general audit access is widened. See [security](../security.md).
 - The private source port retains the **37 legacy aggregate factories** and existing
  application results rather than adopting SDK 2 algorithms. Original-SDK fixtures
  contain **6,144 vectors**. `TreatUncertainAsBad=false` stays explicit; historian
  backend aggregate sets and existing documented differences remain unchanged.
  SDK FILETIME-zero maps to the existing CLR unset-time sentinel at history boundaries.
 - No new historian backend, companion specification, PubSub, ManagedSession, AOT or
  cluster rolling-upgrade capability is claimed. HistorianGateway Client/Contracts
  **0.4.0** is already on gateway master `16111a5e` (historiangw#119 closed);
  the real gateway gate historiangw#78 remains separate from package alignment.
 ## Release record and preflight
 Fill this record in #782 with links to immutable logs/manifests; do not paste keys,
 passwords, connection strings, private certificates or unredacted environment dumps.
 | Required evidence | Record to supply before release |
 |---|---|
 | Candidate source and graph | Pending: full commit, clean/dirty state, SDK/runtime/OS, resolved package graph showing exact 2.0.1 |
 | Known-good recovery set | Pending: deployed application version, full binary/config archive SHA-256, protected PKI backup location and permission record |
 | Candidate artifacts | Pending: Windows and Linux publish directories/archives, RID/self-contained choice, SHA-256 manifest, dependencies and script-worker contents |
 | G2 compilation | Pending: restore/build logs and warning/error counts at the final SHA |
 | G3 behavior/security | Pending: project TRX/counts, required case results, positive and negative authorization, audit privacy, CP lifecycle and 6,144-vector comparison |
 | Interoperability | Pending: baseline/candidate client-server pairs, actual ScadaBridge adapter, upstream driver/browser products and mixed-version cluster gate |
 | Historian | Pending: configured SqlServer/QuestDB/Gateway capability-specific results; historiangw#78 link/evidence for live gateway |
 | G4 resources/performance | Pending: baseline comparison, at least three measured runs, two-hour soak, Windows/Linux start-stop-twice smoke with preserved PKI |
 | Review/operations | Pending: independent security/release review, authorized channel/window, named operator, canary/rollback results and decision |
 Before staging:
 1. Inventory actual service executable/arguments, roles, endpoints, advertised host,
   application URI, raw/UNS namespace identities, certificate thumbprints, active/standby
   state and connected client builds. Record configured queues, request/transport quotas,
   history caps and expected load. Establish performance budgets and stop thresholds.
 2. Archive complete known-good binaries and node configuration, preserving access controls.
   Securely back up PKI own/private keys, peer/issuer chains, CRLs and their permissions.
   Verify recovery access under the service identity, private-key presence, expiry,
   application URI/hostname match and CA/key-usage suitability. Retain existing certificates.
 3. Inventory ConfigDb, LocalDb, historian/outbox and secret-provider state/locations. No
   ConfigDb migration is planned. Record the actual schema/serialization diff, including
   Akka/deployment contracts; if persistence changes appear, reopen the design gate and
   define data rollback before deployment. Never overwrite live historian/outbox data
   with an old binary backup or rewrite persisted sessions/subscriptions.
 4. Publish into fresh directories, hash artifacts, stage separately from the live install
   and validate graph contents. Keep baseline and candidate ports, PKI, config and stores
   separate in the isolated rig; never run both against the same mutable fixture.
 5. Confirm required positive/negative checks, reviewers and recovery artifact access.
   Missing evidence, inaccessible environments or an uncleared command channel means hold.
 ## Verification commands
 These commands are recipes, not execution results. Run from the repository root in
 Bash with .NET 10. Record the final SHA and logs in a fresh evidence directory; inspect
 exit status before continuing. Do not pipe a failing build/test into a successful logger.
 ```bash
 git rev-parse HEAD
 git status --short
 dotnet --info
 dotnet restore ZB.MOM.WW.OtOpcUa.slnx
 dotnet build ZB.MOM.WW.OtOpcUa.slnx -c Release --no-restore
 ```
 Define the helper once after a successful build. Each invocation creates its own
 fresh TRX directory. It preserves both test failure and zero-executed-test failure.
 A successful guard still requires inspection of required cases and skips.
 ```bash
 opc_test() {
  local opc_project="$1"
  local opc_results
  opc_results=$(mktemp -d "${TMPDIR:-/tmp}/opcua2-test.XXXXXXXX") || return
  local opc_test_status=0
  dotnet test "$opc_project" -c Release --no-build --logger trx \
    --results-directory "$opc_results" || opc_test_status=$?
  local opc_count_status=0
  MIN_EXECUTED=1 bash scripts/ci/assert-not-all-skipped.sh "$opc_results"/*.trx \
    || opc_count_status=$?
  printf 'TRX directory: %s\n' "$opc_results"
  if [ "$opc_test_status" -ne 0 ]; then return "$opc_test_status"; fi
  return "$opc_count_status"
 }
 opc_test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests.csproj
 opc_test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.csproj
 opc_test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.csproj
 opc_test tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests.csproj
 opc_test tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests.csproj
 ```
 Driver/browser and simulator commands are in the
 [fixture map](../drivers/OpcUaClient-Test-Fixture.md#running-locally). Backend and
 live historian prerequisites are in [Historian.md](../Historian.md) and
 [infra/README.md](../../infra/README.md). Live tests can write sandbox data: use
 only designated sandbox tags and approved credentials through the existing secret
 provider/environment. A skipped live/DB fixture never satisfies its gate.
 Generate original aggregate evidence only with the isolated
 [oracle procedure](../../scripts/validation/opcua-aggregate-baseline/README.md),
 a new output path and original 1.5.378.182 binary; never derive expectations from
 the candidate's calculators.
 Also run the full CI-equivalent test leg after integration; it does not replace
 fixture-specific or live gates:
 ```bash
 opc_solution_test() {
  local opc_results
  opc_results=$(mktemp -d "${TMPDIR:-/tmp}/opcua2-solution.XXXXXXXX") || return
  local opc_test_status=0
  dotnet test ZB.MOM.WW.OtOpcUa.slnx -c Release --no-build \
    -m:1 -p:TestTfmsInParallel=false --settings scripts/ci/ci.runsettings \
    --filter 'Category!=E2E&Category!=LiveIntegration' \
    --logger trx --results-directory "$opc_results" || opc_test_status=$?
  local opc_count_status=0
  MIN_EXECUTED=12000 bash scripts/ci/assert-not-all-skipped.sh "$opc_results"/*.trx \
    || opc_count_status=$?
  printf 'TRX directory: %s\n' "$opc_results"
  if [ "$opc_test_status" -ne 0 ]; then return "$opc_test_status"; fi
  return "$opc_count_status"
 }
 opc_solution_test
 ```
 Publish the Host project to a fresh operator-selected `$OPC_PUBLISH_DIR` using the
 approved `$OPC_RID` and deployment's self-contained choice. The Windows install
 expects `OtOpcUa.Host.exe`; [ServiceHosting.md](../ServiceHosting.md) also describes
 script-worker packaging. Example self-contained Windows artifact command:
 ```bash
 dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj \
  -c Release -r win-x64 --self-contained true -o "$OPC_PUBLISH_DIR"
 ```
 Repeat with the approved Linux RID/framework settings in its validation environment;
 cross-publishing is not runtime/install evidence. Hash the complete staged artifact
 and retain a manifest outside it (PowerShell 7, where `$Candidate` and `$Manifest`
 are approved absolute local paths and the manifest is outside the candidate):
 ```powershell
 Get-ChildItem -LiteralPath $Candidate -Recurse -File |
    Get-FileHash -Algorithm SHA256 |
    Export-Csv -LiteralPath $Manifest -NoTypeInformation
 ```
 ## Sequence and stop/go decision
 Progress only after the preceding stage's recorded checks and explicit gate decision:
 **isolated rig → ScadaBridge/upstream interoperability → DEV canary → QA pair →
 production canary**. Attach source/artifact identity, start/end times, operator,
 checks, observations and go/hold/rollback decision for each stage to #782.
 A standby-first pair restart is allowed only after the application's mixed-version
 Akka serialization, deployment/config and failover gate succeeds. Verify standby
 health, security, data, history and events; exercise failover before upgrading its
 peer. OPC UA wire interoperability alone does not prove cluster rolling safety.
 Without that gate, arrange a maintenance window and coordinated pair restart;
 record expected service interruption and recovery checks.
 The following Windows commands are **local-session steps only**. Bind `$Install`
 to the inspected current binary directory, `$Candidate` to the reviewed staged
 artifact and `$Backup` to a new protected directory. Keep PKI/data stores outside
 binary replacement; if they are inside the install tree, define exclusions and a
 separate protected backup before using these commands. Save configuration outside
 `$Candidate`, then apply the reviewed node configuration to the staging directory;
 never overwrite it with publisher defaults or expose secrets in command output.
 ```powershell
 Get-CimInstance Win32_Service -Filter "Name='OtOpcUaHost'" |
    Select-Object Name, State, PathName, StartName
 Stop-Service -Name OtOpcUaHost -ErrorAction Stop
 (Get-Service -Name OtOpcUaHost).WaitForStatus('Stopped', [TimeSpan]::FromSeconds(60))
 robocopy $Install $Backup /E /COPYALL /DCOPY:DAT /R:1 /W:1
 if ($LASTEXITCODE -ge 8) { throw 'Known-good backup failed' }
 # After checking the backup and candidate manifest, swap complete binary directories.
 # Use sibling paths on the same volume; the service PathName must still resolve to $Install.
 Move-Item -LiteralPath $Install -Destination $PreviousInstall -ErrorAction Stop
 Move-Item -LiteralPath $Candidate -Destination $Install -ErrorAction Stop
 Start-Service -Name OtOpcUaHost -ErrorAction Stop
 (Get-Service -Name OtOpcUaHost).WaitForStatus('Running', [TimeSpan]::FromSeconds(60))
 ```
 `$PreviousInstall` must be a fresh reviewed sibling path. Handle any failure between
 moves by restoring that complete directory before starting. Service `Running` is
 only a process check. Run the substantive checks below after every start and failover:
 - Check `/healthz`, `/health/ready` and role-appropriate `/health/active` at the
  recorded HTTP base URL; compare ServiceLevel and active/standby state to baseline.
 - Confirm certificate thumbprints, trust/issuer chains, endpoint policies/application
  URI and namespace-URI/NodeId identity. Prove trusted SignAndEncrypt works and
  untrusted/revoked/inappropriate certificates and wrong keys fail without downgrade.
 - Exercise raw and UNS browse/read, scalar/array writes on sandbox nodes, subscriptions,
  quality and source/server timestamps. Verify current legitimate ACL allows as well
  as denies, key expiry/revocation and audience-specific event/audit filtering.
 - Verify alarms/refresh/ack and no missing or duplicate events; verify configured raw,
  modified, processed and event history, paging/release/reauthentication and expected
  unsupported operations. Verify recorder/outbox progress without duplicate replay.
 - Observe reconnects, stale values, legitimate-call denials, continuation-point errors,
  session/item/certificate-handle growth, shutdown, queue depths and latency.
 Use [Client.CLI.md](../Client.CLI.md) for exact connect/browse/read/subscribe/history
 and alarm syntax. Supply credentials through the approved operator mechanism; do
 not record command lines containing API-key passwords in release evidence.
 Stop promotion and restore the known-good application for any authorization regression,
 missing/duplicate alarm or history data, changed identifiers, insecure fallback,
 restart failure or sustained performance-budget miss. Compare like hardware/config/load:
 at least three warmed measurements, provisional limits ≤10% median throughput loss
 and ≤15% p95 latency/allocation/RSS growth, plus at least two hours of lifecycle soak.
 Any deviation needs investigation and a recorded acceptance decision. Do not lower
 thresholds or count absent environments as passed.
 ## Rollback
 1. Freeze promotion; record the trigger and affected nodes/clients in #782. Preserve
   candidate logs and manifests with secrets redacted. Stop the candidate through the
   approved local channel; the remote-exec hold still applies during rollback.
 2. With the service stopped, move the candidate install to a fresh quarantine path,
   restore the complete `$PreviousInstall` (or verified `$Backup`) to the inspected
   service path, and restore the matching node configuration and preserved PKI with
   original permissions. Check hashes/thumbprints before starting. Do not delete or
   roll back live historian, outbox or ConfigDb data without a reviewed data strategy.
 3. Start `OtOpcUaHost` locally with the same `Start-Service`/wait commands above.
   On Linux, use only the environment's approved recorded supervisor procedure.
   Verify health, endpoint/namespace/certificate identity and active/standby state.
 4. Reconnect clients as needed. In-memory sessions, subscriptions and continuation
   points do not survive process rollback; discard old points and begin new reads.
   Rebind namespaces by URI. Never replay ambiguous non-idempotent writes or alarm
   acknowledgements. Confirm API-key writes/ack, subscriptions, alarm counts and history
   continuity with the same positive and negative checks used at rollout.
 5. Record recovery time, checks, data reconciliation and final hold/go decision.
   If rolling source back, revert package and application commits together; never
   mix old package binaries with SDK 2 application assemblies. Keep candidate evidence
   for diagnosis and link follow-up issues from #782.
 ## Results handoff
 Pending: coordinator to attach the final SHA, artifact hashes, TRX counts/skips,
 interoperability/performance/runtime evidence, reviews and per-environment operator
 records in #782. This document makes no claim of a passing build, test, ScadaBridge,
 Windows/Linux smoke, live backend, soak or deployment. Required inaccessible steps
 remain open gates unless formally separated into linked issues with accurate scope.
@@ -36,6 +36,37 @@ Transport security and OPC UA authentication are per-node concerns configured in
 ## Transport Security
 ### SDK 2.0.1 compatibility (#782)
 The application targets exact OPC Foundation SDK **2.0.1**, replacing **1.5.378.182**.
 The upgrade retains the existing host, security profiles, anonymous-admission setting,
 API-key authentication and ACL flag semantics. It adds no companion-specification or
 PubSub capability. Release evidence and operator gates are linked from the
 [rollout runbook](plans/2026-11-10-opcua-sdk-2-rollout.md); source changes alone do not establish release approval.
 SDK role augmentation preserves the application's principal, API-key id and application
 roles. A rejected API key must remain rejected rather than falling through to another
 identity provider. Per-IP authentication miss accounting uses the public transport-observed
 `SecureChannelContext.PeerAddress` during activation; an unknown peer retains the existing
 process-wide budget fallback. This is accounting evidence, not a client-supplied address or
 a new IP admission policy.
 SDK event receive checks compose with application ACL filtering. For the application's
 rejected-write `AuditWriteUpdateEvent` only, a temporary validation context supplies
 `ReceiveEvents` to the recipient's SDK roles after the current source variable, effective
 identity, source `Read` grant and audit/encryption guard are checked. It expires before
 queueing; shared role-permission metadata is not widened. Prior permission callback
 errors veto delivery. SDK-generated audits retain SDK defaults. This does not enable
 auditing or grant a general audit feed.
 Certificate configuration uses the SDK certificate manager, and certificate ownership
 lasts through the owning host/session's lifetime. Preserve the existing application
 certificate, private key, issuer/peer stores and filesystem permissions during rollout;
 verify key access and CA usage under the service identity. Do not rotate certificates,
 enable auto-accept, add `None`, or relax revocation checks to get the candidate running.
 Continuation points remain session-owned and bound to node/topology and current authority;
 restart requires fresh sessions and fresh points.
 ### Overview
 The OtOpcUa Server supports configurable OPC UA transport security profiles that control how data is protected on the wire between OPC UA clients and the server.