lmxopcua/docs/plans/2026-11-10-opcua-sdk-2-rollout.md

# OPC UA SDK 2.0.1 rollout and rollback

This operator runbook accompanies [#782](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/782),
the [implementation plan](2026-11-10-opcua-sdk-2-upgrade.md) and
[execution evidence](2026-11-10-opcua-sdk-2-evidence.md). Gitea is the single
status system. This runbook was reconciled with SDK2/HG1.0 checkout `f2a8ce13` and the
commit-specific local receipts linked below. It records local validation, not
operator release approval or a fleet deployment. Receipt revisions vary; use each
report's full source/artifact identity rather than treating every result as a test
of this documentation commit. Open failures and operator gates still prevent a
release go decision.

The target is exact OPC Foundation SDK **2.0.1**; the known-good SDK baseline is
**1.5.378.182** (application baseline `83ed880e4471344deaf008da9bd028c7e170bdc7`).
Restore the actual archived known-good application as a unit during rollback.
Package downgrades alone cannot reverse the application/API migration.

## Operating constraints

**The remote-execution hold on wonder boxes vd03/vq03/vq04 remains binding.**
No WinRM, psbridge or replacement remote-exec channel may be used until the user
clears a channel. UNC/SMB file copying and permitted SQL access do not authorize
service control, PowerShell execution or deployment on those hosts. Steps requiring
a command on a held host remain pending. No fleet deployment was performed.

Run the service commands below only from an approved local operator session on an
authorized Windows node. Record the node, account, channel and maintenance window
in #782 first. Linux runtime/service handling requires its own approved environment
and smoke record; this runbook supplies no invented Linux service-unit name.
Do not use `scripts/install/Refresh-Services.ps1` as a blanket upgrade action: it
also touches gateway/retired sidecar paths outside this SDK rollout.

## Preserved contract

- Existing raw/UNS namespace URIs, identifier forms, endpoint profiles, API-key/anonymous
  admission, ACL flag behavior, role and key revocation gates remain the contract.
  Source uses public transport peer evidence for miss budgets, not an IP admission rule.
- Scalar leniency, checked numeric-array conversion and pass-through driver behavior
  stay as before. SDK immutable values are detached at boundaries; no new matrix support.
- No application status-code, locale or event-filter semantic change is approved.
  SDK status wrappers and explicit empty `ContentFilter` construction are API changes,
  not evidence of changed protocol values. Record any stricter response, field-level
  filter error or localization difference with a consumer-impact decision before go.
- Auditing stays opt-in. Ordinary SDK automatic audit delivery retains its type
  restrictions. Only the application's rejected-write audit gets a private, temporary
  receive-permission context after current source `Read` ACL and audit/encryption checks;
  no shared role metadata or general audit access is widened. See [security](../security.md).
- The private source port retains the **37 legacy aggregate factories** and existing
  application results rather than adopting SDK 2 algorithms. Original-SDK fixtures
  contain **6,144 vectors**. `TreatUncertainAsBad=false` stays explicit; historian
  backend aggregate sets and existing documented differences remain unchanged.
  SDK FILETIME-zero maps to the existing CLR unset-time sentinel at history boundaries.
- No new historian backend, companion specification, PubSub, ManagedSession, AOT or
  cluster rolling-upgrade capability is claimed. HistorianGateway Client/Contracts
  **1.0.0** is on OtOpcUa master `230cef4e` and merged into this candidate
  (historiangw#131 closed after published-package/TLS verification). The preceding
  0.4.0 alignment closed #119. The live gateway gate historiangw#78 remains separate.
- Certificate creation correction `bd3143b7` uses the SDK's **12-month** default
  instead of treating 2048 as months. The separate RSA minimum stays 2048 bits;
  existing certificates/private-key bytes and PKI identity are retained.

## Release record and preflight

The following local results have recorded evidence. They do not substitute for
unexecuted operator or duration gates. Keep secrets out of logs and release records.

| Local validation | Recorded result and receipt |
|---|---|
| G2 compilation and graph | Whole-solution Release builds pass with **0 warnings/errors**; production Foundation packages are exact 2.0.1, with no runtime migration analyzer. [Build/package evidence](2026-11-10-opcua-sdk-2-evidence.md#solution-performance-and-package-evidence). |
| Security and application contracts | **17/17** final security probes; **187** wire cases; **324** Shared tests; **1115** server units with one existing skip. Seven certificate-manager/API checks are separate from network handshakes. [Case scope and provenance](2026-11-10-opcua-sdk-2-evidence.md#local-compatibility-and-security-evidence). |
| Original aggregate results | **6,144 vectors pass directly and through the production manager**. Fixture SHA-256 `2ec2a1ac97f05d199c18da55157f8bbe0276853c3ba73ebf1b509c535c5a7dd8`; [original-SDK provenance and comparison scope](2026-11-10-opcua-sdk-2-evidence.md#preserved-aggregate-algorithms-and-independent-oracles). |
| Separate-process compatibility | **62/62** baseline/candidate Shared, driver and actual CLI cases pass. Strict mutual DER trust/encrypted API-key coverage is separate from CLI autoaccept. [Matrix receipts](../../scripts/validation/opcua-sdk-compatibility/evidence/2026-11-10/). |
| Actual ScadaBridge adapter | **18 None/API-key cases pass**, nine against each server. Encrypted attempts fail against both due to the preexisting own-certificate configuration gap, [ScadaBridge#121](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/121); secure interoperability is blocked. [Consumer receipts](../../scripts/validation/scadabridge-sdk-compatibility/). |
| Local historian backends | SQL Server **195/195** on each version; QuestDB **233 passed, 0 failed, 1 existing opt-in race-stress skip** on each. Case outcomes match. [Backend matrix](evidence/2026-11-10-opcua2/backend-matrix.md). This is not live AVEVA evidence. |
| Bounded performance subset | Three warmed measurements per version: median rebuild **17→15 ms**, removal **81→80 ms**, equal visit counts. [Benchmark scope/load caveats](evidence/2026-11-10-opcua2/lock-benchmarks.md); these are not full wire-throughput, p95 or steady-RSS approval. |
| Publish/package and Linux smoke | **Four** self-contained Linux ARM64/Windows x64 Host/CLI publishes pass. Actual Linux full Host passes **two start/stop cycles**, strict security/health checks and unchanged PKI. [Fresh HG1.0 artifact hashes and runtime receipts](evidence/2026-11-10-opcua2/publish-smoke-hg100.md). Both Hosts contain Client/Contracts1.0.0; Foundation remains2.0.1. Windows execution is still pending. |
| Production client lifecycle | **7,200.984 seconds,5,420 cycles,43,360 verified manager disposals**, zero weak product survivors and PKI descriptors across543 samples. [Exact source/runtime/counter receipts](../../scripts/validation/opcua-client-lifecycle/evidence/2026-11-10-full/verification.json). This is a client lifecycle pass, not a server RSS or full G4 pass. |
| Source review | Independent reviews covered identity, ownership, aggregate preservation, hierarchy and filtered audit paths. [Review scope](2026-11-10-opcua-sdk-2-evidence.md#integrated-implementation-and-source-reviews). Source review is not release approval. |

The final corrected CI-equivalent run at `75057d8c` completed with **63 TRX,
19,124 total, 18,489 executed, 18,478 passed, 11 failed and 635 existing skips**.
Test exit **1**; execution guard **0** at floor12,000. It is not green: the eleven
failures are ten PercentGood cases and one ShadowMode250ms assertion, independently
reproduced on baseline. [Final counters/failures/project hashes](evidence/2026-11-10-opcua2/solution-tests4.json).
CI3's extra five simulator-advertisement failures were corrected by a directly
reachable matching-port fixture; the baseline Runtime test-ordering race was fixed
through correlated completion barriers (#805, master52d1373e). Both corrections
pass in the final full run. The retained CI3 report and original SDK1.5 NAT parity
are in the [main evidence](2026-11-10-opcua-sdk-2-evidence.md#ci3-investigation-and-corrected-final-verification);
preexisting browser NAT behavior remains #806.
The ten PercentGood failures are tracked in
[#796](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/796), and the separately
reproduced baseline ScriptWorker timing failure is
[#798](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/798).
The opaque scripted-alarm narrow-scope denial is a separate preexisting functionality
gap, [#797](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/797), not a new
SDK incompatibility or an approved synthetic-ID workaround.

**The server resource gate failed.** The Mac restart soak failed twice around
17 minutes at the unchanged RSS cap. The same-cap self-contained Linux server
workload also failed at **248 cycles / 903.5 seconds**: RSS **353,931,264 bytes**
exceeded **351,850,496 bytes**. Its retained result is
`/tmp/opcua2-server-linux-full-20261110/task/results/soak.trx`.
[#799](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/799) owns investigation;
no cap waiver or two-hour server pass is recorded. The distinct actual-production
Shared/driver/reinitialize/browser Linux lifecycle run passed its actual two-hour
duration and cleanup checks, as recorded above; it does not replace this failed
server workload. See [long-run scope](2026-11-10-opcua-sdk-2-evidence.md#resource-finding-and-long-run-scope)
for instrumentation limitations and final-result updates.
The fixed candidate saturation diagnostic later met its selected generated-factory
criterion, but retained the cold RSS rejection and exit1. The original-SDK adapter
records partial wire coverage and unsupported ownership/process-memory parity;
it supplies no approved performance denominator. See [scoped saturation result](evidence/2026-11-10-opcua2/saturation-candidate.md)
and [original adapter limitations](../../scripts/validation/opcua-lifecycle-memory/baseline-canonical/README.md).
Astra's final source/security review is closed; release remains on HOLD.


| Operator release field | Still required before its release decision |
|---|---|
| Recovery set and candidate selection | Actual deployed known-good application/config archive hashes, protected PKI/permission backup, selected complete candidate artifact and matching source/manifest. Local artifact manifests above are available; a fleet recovery set is not inferred. |
| Windows runtime/service | Approved-node installation, service account/ACL checks, preserved-PKI two-start/stop and protocol checks: [#800](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/800). Cross-publishing does not satisfy this. |
| Live AVEVA/consumer/pair | Licensed gateway/backend, remaining secure consumer work and mixed-version Akka/config/failover validation: [#801](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/801), with historiangw#78 and ScadaBridge#121 linked as applicable. No rolling-safe claim. |
| Resource/performance acceptance | Resolve #799, record completed required-duration runs and remaining representative load/budget evidence; no shortened run or changed cap substitutes. |
| Rollout/rollback authorization and results | Named operator, cleared channel, environment/window, actual schema/serialization assessment, canary/rollback checks and final review/go decision in #782. The wonder-box hold remains binding. |

Before staging:

1. Inventory actual service executable/arguments, roles, endpoints, advertised host,
   application URI, raw/UNS namespace identities, certificate thumbprints, active/standby
   state and connected client builds. Record configured queues, request/transport quotas,
   history caps and expected load. Establish performance budgets and stop thresholds.
2. Archive complete known-good binaries and node configuration, preserving access controls.
   Securely back up PKI own/private keys, peer/issuer chains, CRLs and their permissions.
   Verify recovery access under the service identity, private-key presence, expiry,
   application URI/hostname match and CA/key-usage suitability. Retain existing certificates.
3. Inventory ConfigDb, LocalDb, historian/outbox and secret-provider state/locations. No
   ConfigDb migration is planned. Record the actual schema/serialization diff, including
   Akka/deployment contracts; if persistence changes appear, reopen the design gate and
   define data rollback before deployment. Never overwrite live historian/outbox data
   with an old binary backup or rewrite persisted sessions/subscriptions.
4. Publish into fresh directories, hash artifacts, stage separately from the live install
   and validate graph contents. Keep baseline and candidate ports, PKI, config and stores
   separate in the isolated rig; never run both against the same mutable fixture.
5. Confirm required positive/negative checks, reviewers and recovery artifact access.
   Missing evidence, inaccessible environments or an uncleared command channel means hold.

## Verification commands

These commands are recipes, not execution results. Run from the repository root in
Bash with .NET 10. Record the final SHA and logs in a fresh evidence directory; inspect
exit status before continuing. Do not pipe a failing build/test into a successful logger.

```bash
git rev-parse HEAD
git status --short
dotnet --info
dotnet restore ZB.MOM.WW.OtOpcUa.slnx
dotnet build ZB.MOM.WW.OtOpcUa.slnx -c Release --no-restore
```

Define the helper once after a successful build. Each invocation creates its own
fresh TRX directory. It preserves both test failure and zero-executed-test failure.
A successful guard still requires inspection of required cases and skips.

```bash
opc_test() {
  local opc_project="$1"
  local opc_results
  opc_results=$(mktemp -d "${TMPDIR:-/tmp}/opcua2-test.XXXXXXXX") || return
  local opc_test_status=0
  dotnet test "$opc_project" -c Release --no-build --logger trx \
    --results-directory "$opc_results" || opc_test_status=$?
  local opc_count_status=0
  MIN_EXECUTED=1 bash scripts/ci/assert-not-all-skipped.sh "$opc_results"/*.trx \
    || opc_count_status=$?
  printf 'TRX directory: %s\n' "$opc_results"
  if [ "$opc_test_status" -ne 0 ]; then return "$opc_test_status"; fi
  return "$opc_count_status"
}
opc_test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests.csproj
opc_test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.csproj
opc_test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.csproj
opc_test tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests.csproj
opc_test tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests.csproj
```

Driver/browser and simulator commands are in the
[fixture map](../drivers/OpcUaClient-Test-Fixture.md#running-locally). Backend and
live historian prerequisites are in [Historian.md](../Historian.md) and
[infra/README.md](../../infra/README.md). Live tests can write sandbox data: use
only designated sandbox tags and approved credentials through the existing secret
provider/environment. A skipped live/DB fixture never satisfies its gate.
Generate original aggregate evidence only with the isolated
[oracle procedure](../../scripts/validation/opcua-aggregate-baseline/README.md),
a new output path and original 1.5.378.182 binary; never derive expectations from
the candidate's calculators.

Also run the full CI-equivalent test leg after integration; it does not replace
fixture-specific or live gates:

```bash
opc_solution_test() {
  local opc_results
  opc_results=$(mktemp -d "${TMPDIR:-/tmp}/opcua2-solution.XXXXXXXX") || return
  local opc_test_status=0
  dotnet test ZB.MOM.WW.OtOpcUa.slnx -c Release --no-build \
    -m:1 -p:TestTfmsInParallel=false --settings scripts/ci/ci.runsettings \
    --filter 'Category!=E2E&Category!=LiveIntegration' \
    --logger trx --results-directory "$opc_results" || opc_test_status=$?
  local opc_count_status=0
  MIN_EXECUTED=12000 bash scripts/ci/assert-not-all-skipped.sh "$opc_results"/*.trx \
    || opc_count_status=$?
  printf 'TRX directory: %s\n' "$opc_results"
  if [ "$opc_test_status" -ne 0 ]; then return "$opc_test_status"; fi
  return "$opc_count_status"
}
opc_solution_test
```

Publish the Host project to a fresh operator-selected `$OPC_PUBLISH_DIR` using the
approved `$OPC_RID` and deployment's self-contained choice. The Windows install
expects `OtOpcUa.Host.exe`; [ServiceHosting.md](../ServiceHosting.md) also describes
script-worker packaging. Example self-contained Windows artifact command:

```bash
dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj \
  -c Release -r win-x64 --self-contained true -o "$OPC_PUBLISH_DIR"
```

Repeat with the approved Linux RID/framework settings in its validation environment;
cross-publishing is not runtime/install evidence. Hash the complete staged artifact
and retain a manifest outside it (PowerShell 7, where `$Candidate` and `$Manifest`
are approved absolute local paths and the manifest is outside the candidate):

```powershell
Get-ChildItem -LiteralPath $Candidate -Recurse -File |
    Get-FileHash -Algorithm SHA256 |
    Export-Csv -LiteralPath $Manifest -NoTypeInformation
```

## Sequence and stop/go decision

Progress only after the preceding stage's recorded checks and explicit gate decision:
**isolated rig → ScadaBridge/upstream interoperability → DEV canary → QA pair →
production canary**. Attach source/artifact identity, start/end times, operator,
checks, observations and go/hold/rollback decision for each stage to #782.

A standby-first pair restart is allowed only after the application's mixed-version
Akka serialization, deployment/config and failover gate succeeds. Verify standby
health, security, data, history and events; exercise failover before upgrading its
peer. OPC UA wire interoperability alone does not prove cluster rolling safety.
Without that gate, arrange a maintenance window and coordinated pair restart;
record expected service interruption and recovery checks.

The following Windows commands are **local-session steps only**. Bind `$Install`
to the inspected current binary directory, `$Candidate` to the reviewed staged
artifact and `$Backup` to a new protected directory. Keep PKI/data stores outside
binary replacement; if they are inside the install tree, define exclusions and a
separate protected backup before using these commands. Save configuration outside
`$Candidate`, then apply the reviewed node configuration to the staging directory;
never overwrite it with publisher defaults or expose secrets in command output.

```powershell
Get-CimInstance Win32_Service -Filter "Name='OtOpcUaHost'" |
    Select-Object Name, State, PathName, StartName
Stop-Service -Name OtOpcUaHost -ErrorAction Stop
(Get-Service -Name OtOpcUaHost).WaitForStatus('Stopped', [TimeSpan]::FromSeconds(60))
robocopy $Install $Backup /E /COPYALL /DCOPY:DAT /R:1 /W:1
if ($LASTEXITCODE -ge 8) { throw 'Known-good backup failed' }
# After checking the backup and candidate manifest, swap complete binary directories.
# Use sibling paths on the same volume; the service PathName must still resolve to $Install.
Move-Item -LiteralPath $Install -Destination $PreviousInstall -ErrorAction Stop
Move-Item -LiteralPath $Candidate -Destination $Install -ErrorAction Stop
Start-Service -Name OtOpcUaHost -ErrorAction Stop
(Get-Service -Name OtOpcUaHost).WaitForStatus('Running', [TimeSpan]::FromSeconds(60))
```

`$PreviousInstall` must be a fresh reviewed sibling path. Handle any failure between
moves by restoring that complete directory before starting. Service `Running` is
only a process check. Run the substantive checks below after every start and failover:

- Check `/healthz`, `/health/ready` and role-appropriate `/health/active` at the
  recorded HTTP base URL; compare ServiceLevel and active/standby state to baseline.
- Confirm certificate thumbprints, trust/issuer chains, endpoint policies/application
  URI and namespace-URI/NodeId identity. Prove trusted SignAndEncrypt works and
  untrusted/revoked/inappropriate certificates and wrong keys fail without downgrade.
- Exercise raw and UNS browse/read, scalar/array writes on sandbox nodes, subscriptions,
  quality and source/server timestamps. Verify current legitimate ACL allows as well
  as denies, key expiry/revocation and audience-specific event/audit filtering.
- Verify alarms/refresh/ack and no missing or duplicate events; verify configured raw,
  modified, processed and event history, paging/release/reauthentication and expected
  unsupported operations. Verify recorder/outbox progress without duplicate replay.
- Observe reconnects, stale values, legitimate-call denials, continuation-point errors,
  session/item/certificate-handle growth, shutdown, queue depths and latency.

Use [Client.CLI.md](../Client.CLI.md) for exact connect/browse/read/subscribe/history
and alarm syntax. Supply credentials through the approved operator mechanism; do
not record command lines containing API-key passwords in release evidence.

Stop promotion and restore the known-good application for any authorization regression,
missing/duplicate alarm or history data, changed identifiers, insecure fallback,
restart failure or sustained performance-budget miss. Compare like hardware/config/load:
at least three warmed measurements, provisional limits ≤10% median throughput loss
and ≤15% p95 latency/allocation/RSS growth, plus at least two hours of lifecycle soak.
Any deviation needs investigation and a recorded acceptance decision. Do not lower
thresholds or count absent environments as passed.

## Rollback

1. Freeze promotion; record the trigger and affected nodes/clients in #782. Preserve
   candidate logs and manifests with secrets redacted. Stop the candidate through the
   approved local channel; the remote-exec hold still applies during rollback.
2. With the service stopped, move the candidate install to a fresh quarantine path,
   restore the complete `$PreviousInstall` (or verified `$Backup`) to the inspected
   service path, and restore the matching node configuration and preserved PKI with
   original permissions. Check hashes/thumbprints before starting. Do not delete or
   roll back live historian, outbox or ConfigDb data without a reviewed data strategy.
3. Start `OtOpcUaHost` locally with the same `Start-Service`/wait commands above.
   On Linux, use only the environment's approved recorded supervisor procedure.
   Verify health, endpoint/namespace/certificate identity and active/standby state.
4. Reconnect clients as needed. In-memory sessions, subscriptions and continuation
   points do not survive process rollback; discard old points and begin new reads.
   Rebind namespaces by URI. Never replay ambiguous non-idempotent writes or alarm
   acknowledgements. Confirm API-key writes/ack, subscriptions, alarm counts and history
   continuity with the same positive and negative checks used at rollout.
5. Record recovery time, checks, data reconciliation and final hold/go decision.
   If rolling source back, revert package and application commits together; never
   mix old package binaries with SDK 2 application assemblies. Keep candidate evidence
   for diagnosis and link follow-up issues from #782.

## Results handoff

Local build, security/wire, aggregate, backend, compatibility, publish and Linux
smoke evidence is linked above with its actual scope and revisions. The server
RSS gate is failed (#799), full CI retains the 11 baseline failures (#796/#798),
the client lifecycle run completed two hours with cleanup checks passed, secure
ScadaBridge interoperability is blocked (ScadaBridge#121), and
Windows/live AVEVA/pair/operator gates remain pending (#800/#801). No fleet rollout
or rollback has been executed. The final post-HG1.0 corrected full run retains
only the eleven independently reproduced baseline failures; earlier failed runs
remain recorded. The coordinator records remaining decisions in #782 and the main
evidence; none is inferred from a
passing subset or cross-published artifact.