lmxopcua/docs/plans/2026-11-10-opcua-sdk-2-rollout.md

OPC UA SDK 2.0.1 rollout and rollback

This operator runbook accompanies #782, the implementation plan and execution evidence. Gitea is the single status system. This runbook was reconciled with SDK2/HG1.0 checkout f2a8ce13 and the commit-specific local receipts linked below. It records local validation, not operator release approval or a fleet deployment. Receipt revisions vary; use each report's full source/artifact identity rather than treating every result as a test of this documentation commit. Open failures and operator gates still prevent a release go decision.

The target is exact OPC Foundation SDK 2.0.1; the known-good SDK baseline is 1.5.378.182 (application baseline 83ed880e4471344deaf008da9bd028c7e170bdc7). Restore the actual archived known-good application as a unit during rollback. Package downgrades alone cannot reverse the application/API migration.

Operating constraints

The remote-execution hold on wonder boxes vd03/vq03/vq04 remains binding. No WinRM, psbridge or replacement remote-exec channel may be used until the user clears a channel. UNC/SMB file copying and permitted SQL access do not authorize service control, PowerShell execution or deployment on those hosts. Steps requiring a command on a held host remain pending. No fleet deployment was performed.

Run the service commands below only from an approved local operator session on an authorized Windows node. Record the node, account, channel and maintenance window in #782 first. Linux runtime/service handling requires its own approved environment and smoke record; this runbook supplies no invented Linux service-unit name. Do not use scripts/install/Refresh-Services.ps1 as a blanket upgrade action: it also touches gateway/retired sidecar paths outside this SDK rollout.

Preserved contract

• Existing raw/UNS namespace URIs, identifier forms, endpoint profiles, API-key/anonymous admission, ACL flag behavior, role and key revocation gates remain the contract. Source uses public transport peer evidence for miss budgets, not an IP admission rule.
• Scalar leniency, checked numeric-array conversion and pass-through driver behavior stay as before. SDK immutable values are detached at boundaries; no new matrix support.
• No application status-code, locale or event-filter semantic change is approved. SDK status wrappers and explicit empty ContentFilter construction are API changes, not evidence of changed protocol values. Record any stricter response, field-level filter error or localization difference with a consumer-impact decision before go.
• Auditing stays opt-in. Ordinary SDK automatic audit delivery retains its type restrictions. Only the application's rejected-write audit gets a private, temporary receive-permission context after current source Read ACL and audit/encryption checks; no shared role metadata or general audit access is widened. See security.
• The private source port retains the 37 legacy aggregate factories and existing application results rather than adopting SDK 2 algorithms. Original-SDK fixtures contain 6,144 vectors. TreatUncertainAsBad=false stays explicit; historian backend aggregate sets and existing documented differences remain unchanged. SDK FILETIME-zero maps to the existing CLR unset-time sentinel at history boundaries.
• No new historian backend, companion specification, PubSub, ManagedSession, AOT or cluster rolling-upgrade capability is claimed. HistorianGateway Client/Contracts 1.0.0 is on OtOpcUa master 230cef4e and merged into this candidate (historiangw#131 closed after published-package/TLS verification). The preceding 0.4.0 alignment closed #119. The live gateway gate historiangw#78 remains separate.
• Certificate creation correction bd3143b7 uses the SDK's 12-month default instead of treating 2048 as months. The separate RSA minimum stays 2048 bits; existing certificates/private-key bytes and PKI identity are retained.

Release record and preflight

The following local results have recorded evidence. They do not substitute for unexecuted operator or duration gates. Keep secrets out of logs and release records.

Local validation	Recorded result and receipt
G2 compilation and graph	Whole-solution Release builds pass with 0 warnings/errors; production Foundation packages are exact 2.0.1, with no runtime migration analyzer. Build/package evidence.
Security and application contracts	17/17 final security probes; 187 wire cases; 324 Shared tests; 1115 server units with one existing skip. Seven certificate-manager/API checks are separate from network handshakes. Case scope and provenance.
Original aggregate results	6,144 vectors pass directly and through the production manager. Fixture SHA-256 2ec2a1ac97f05d199c18da55157f8bbe0276853c3ba73ebf1b509c535c5a7dd8; original-SDK provenance and comparison scope.
Separate-process compatibility	62/62 baseline/candidate Shared, driver and actual CLI cases pass. Strict mutual DER trust/encrypted API-key coverage is separate from CLI autoaccept. Matrix receipts.
Actual ScadaBridge adapter	18 None/API-key cases pass, nine against each server. Encrypted attempts fail against both due to the preexisting own-certificate configuration gap, ScadaBridge#121; secure interoperability is blocked. Consumer receipts.
Local historian backends	SQL Server 195/195 on each version; QuestDB 233 passed, 0 failed, 1 existing opt-in race-stress skip on each. Case outcomes match. Backend matrix. This is not live AVEVA evidence.
Bounded performance subset	Three warmed measurements per version: median rebuild 17→15 ms, removal 81→80 ms, equal visit counts. Benchmark scope/load caveats; these are not full wire-throughput, p95 or steady-RSS approval.
Publish/package and Linux smoke	Four self-contained Linux ARM64/Windows x64 Host/CLI publishes pass. Actual Linux full Host passes two start/stop cycles, strict security/health checks and unchanged PKI. Fresh HG1.0 artifact hashes and runtime receipts. Both Hosts contain Client/Contracts1.0.0; Foundation remains2.0.1. Windows execution is still pending.
Production client lifecycle	7,200.984 seconds,5,420 cycles,43,360 verified manager disposals, zero weak product survivors and PKI descriptors across543 samples. Exact source/runtime/counter receipts. This is a client lifecycle pass, not a server RSS or full G4 pass.
Source review	Independent reviews covered identity, ownership, aggregate preservation, hierarchy and filtered audit paths. Review scope. Source review is not release approval.

The final corrected CI-equivalent run at 75057d8c completed with 63 TRX, 19,124 total, 18,489 executed, 18,478 passed, 11 failed and 635 existing skips. Test exit 1; execution guard 0 at floor12,000. It is not green: the eleven failures are ten PercentGood cases and one ShadowMode250ms assertion, independently reproduced on baseline. Final counters/failures/project hashes. CI3's extra five simulator-advertisement failures were corrected by a directly reachable matching-port fixture; the baseline Runtime test-ordering race was fixed through correlated completion barriers (#805, master52d1373e). Both corrections pass in the final full run. The retained CI3 report and original SDK1.5 NAT parity are in the main evidence; preexisting browser NAT behavior remains #806. The ten PercentGood failures are tracked in #796, and the separately reproduced baseline ScriptWorker timing failure is #798. The opaque scripted-alarm narrow-scope denial is a separate preexisting functionality gap, #797, not a new SDK incompatibility or an approved synthetic-ID workaround.

The server resource gate failed. The Mac restart soak failed twice around 17 minutes at the unchanged RSS cap. The same-cap self-contained Linux server workload also failed at 248 cycles / 903.5 seconds: RSS 353,931,264 bytes exceeded 351,850,496 bytes. Its retained result is /tmp/opcua2-server-linux-full-20261110/task/results/soak.trx. #799 owns investigation; no cap waiver or two-hour server pass is recorded. The distinct actual-production Shared/driver/reinitialize/browser Linux lifecycle run passed its actual two-hour duration and cleanup checks, as recorded above; it does not replace this failed server workload. See long-run scope for instrumentation limitations and final-result updates. The fixed candidate saturation diagnostic later met its selected generated-factory criterion, but retained the cold RSS rejection and exit1. The original-SDK adapter records partial wire coverage and unsupported ownership/process-memory parity; it supplies no approved performance denominator. See scoped saturation result and original adapter limitations. Astra's final source/security review is closed; release remains on HOLD.

Operator release field	Still required before its release decision
Recovery set and candidate selection	Actual deployed known-good application/config archive hashes, protected PKI/permission backup, selected complete candidate artifact and matching source/manifest. Local artifact manifests above are available; a fleet recovery set is not inferred.
Windows runtime/service	Approved-node installation, service account/ACL checks, preserved-PKI two-start/stop and protocol checks: #800. Cross-publishing does not satisfy this.
Live AVEVA/consumer/pair	Licensed gateway/backend, remaining secure consumer work and mixed-version Akka/config/failover validation: #801, with historiangw#78 and ScadaBridge#121 linked as applicable. No rolling-safe claim.
Resource/performance acceptance	Resolve #799, record completed required-duration runs and remaining representative load/budget evidence; no shortened run or changed cap substitutes.
Rollout/rollback authorization and results	Named operator, cleared channel, environment/window, actual schema/serialization assessment, canary/rollback checks and final review/go decision in #782. The wonder-box hold remains binding.

Before staging:

1. Inventory actual service executable/arguments, roles, endpoints, advertised host, application URI, raw/UNS namespace identities, certificate thumbprints, active/standby state and connected client builds. Record configured queues, request/transport quotas, history caps and expected load. Establish performance budgets and stop thresholds.
2. Archive complete known-good binaries and node configuration, preserving access controls. Securely back up PKI own/private keys, peer/issuer chains, CRLs and their permissions. Verify recovery access under the service identity, private-key presence, expiry, application URI/hostname match and CA/key-usage suitability. Retain existing certificates.
3. Inventory ConfigDb, LocalDb, historian/outbox and secret-provider state/locations. No ConfigDb migration is planned. Record the actual schema/serialization diff, including Akka/deployment contracts; if persistence changes appear, reopen the design gate and define data rollback before deployment. Never overwrite live historian/outbox data with an old binary backup or rewrite persisted sessions/subscriptions.
4. Publish into fresh directories, hash artifacts, stage separately from the live install and validate graph contents. Keep baseline and candidate ports, PKI, config and stores separate in the isolated rig; never run both against the same mutable fixture.
5. Confirm required positive/negative checks, reviewers and recovery artifact access. Missing evidence, inaccessible environments or an uncleared command channel means hold.

Verification commands

These commands are recipes, not execution results. Run from the repository root in Bash with .NET 10. Record the final SHA and logs in a fresh evidence directory; inspect exit status before continuing. Do not pipe a failing build/test into a successful logger.

git rev-parse HEAD
git status --short
dotnet --info
dotnet restore ZB.MOM.WW.OtOpcUa.slnx
dotnet build ZB.MOM.WW.OtOpcUa.slnx -c Release --no-restore

Define the helper once after a successful build. Each invocation creates its own fresh TRX directory. It preserves both test failure and zero-executed-test failure. A successful guard still requires inspection of required cases and skips.

opc_test() {
  local opc_project="$1"
  local opc_results
  opc_results=$(mktemp -d "${TMPDIR:-/tmp}/opcua2-test.XXXXXXXX") || return
  local opc_test_status=0
  dotnet test "$opc_project" -c Release --no-build --logger trx \
    --results-directory "$opc_results" || opc_test_status=$?
  local opc_count_status=0
  MIN_EXECUTED=1 bash scripts/ci/assert-not-all-skipped.sh "$opc_results"/*.trx \
    || opc_count_status=$?
  printf 'TRX directory: %s\n' "$opc_results"
  if [ "$opc_test_status" -ne 0 ]; then return "$opc_test_status"; fi
  return "$opc_count_status"
}
opc_test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests.csproj
opc_test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.csproj
opc_test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.csproj
opc_test tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests.csproj
opc_test tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests.csproj

Driver/browser and simulator commands are in the fixture map. Backend and live historian prerequisites are in Historian.md and infra/README.md. Live tests can write sandbox data: use only designated sandbox tags and approved credentials through the existing secret provider/environment. A skipped live/DB fixture never satisfies its gate. Generate original aggregate evidence only with the isolated oracle procedure, a new output path and original 1.5.378.182 binary; never derive expectations from the candidate's calculators.

Also run the full CI-equivalent test leg after integration; it does not replace fixture-specific or live gates:

opc_solution_test() {
  local opc_results
  opc_results=$(mktemp -d "${TMPDIR:-/tmp}/opcua2-solution.XXXXXXXX") || return
  local opc_test_status=0
  dotnet test ZB.MOM.WW.OtOpcUa.slnx -c Release --no-build \
    -m:1 -p:TestTfmsInParallel=false --settings scripts/ci/ci.runsettings \
    --filter 'Category!=E2E&Category!=LiveIntegration' \
    --logger trx --results-directory "$opc_results" || opc_test_status=$?
  local opc_count_status=0
  MIN_EXECUTED=12000 bash scripts/ci/assert-not-all-skipped.sh "$opc_results"/*.trx \
    || opc_count_status=$?
  printf 'TRX directory: %s\n' "$opc_results"
  if [ "$opc_test_status" -ne 0 ]; then return "$opc_test_status"; fi
  return "$opc_count_status"
}
opc_solution_test

Publish the Host project to a fresh operator-selected $OPC_PUBLISH_DIR using the approved $OPC_RID and deployment's self-contained choice. The Windows install expects OtOpcUa.Host.exe; ServiceHosting.md also describes script-worker packaging. Example self-contained Windows artifact command:

dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj \
  -c Release -r win-x64 --self-contained true -o "$OPC_PUBLISH_DIR"

Repeat with the approved Linux RID/framework settings in its validation environment; cross-publishing is not runtime/install evidence. Hash the complete staged artifact and retain a manifest outside it (PowerShell 7, where $Candidate and $Manifest are approved absolute local paths and the manifest is outside the candidate):

Get-ChildItem -LiteralPath $Candidate -Recurse -File |
    Get-FileHash -Algorithm SHA256 |
    Export-Csv -LiteralPath $Manifest -NoTypeInformation

Sequence and stop/go decision

Progress only after the preceding stage's recorded checks and explicit gate decision: isolated rig → ScadaBridge/upstream interoperability → DEV canary → QA pair → production canary. Attach source/artifact identity, start/end times, operator, checks, observations and go/hold/rollback decision for each stage to #782.

A standby-first pair restart is allowed only after the application's mixed-version Akka serialization, deployment/config and failover gate succeeds. Verify standby health, security, data, history and events; exercise failover before upgrading its peer. OPC UA wire interoperability alone does not prove cluster rolling safety. Without that gate, arrange a maintenance window and coordinated pair restart; record expected service interruption and recovery checks.

The following Windows commands are local-session steps only. Bind $Install to the inspected current binary directory, $Candidate to the reviewed staged artifact and $Backup to a new protected directory. Keep PKI/data stores outside binary replacement; if they are inside the install tree, define exclusions and a separate protected backup before using these commands. Save configuration outside $Candidate, then apply the reviewed node configuration to the staging directory; never overwrite it with publisher defaults or expose secrets in command output.

Get-CimInstance Win32_Service -Filter "Name='OtOpcUaHost'" |
    Select-Object Name, State, PathName, StartName
Stop-Service -Name OtOpcUaHost -ErrorAction Stop
(Get-Service -Name OtOpcUaHost).WaitForStatus('Stopped', [TimeSpan]::FromSeconds(60))
robocopy $Install $Backup /E /COPYALL /DCOPY:DAT /R:1 /W:1
if ($LASTEXITCODE -ge 8) { throw 'Known-good backup failed' }
# After checking the backup and candidate manifest, swap complete binary directories.
# Use sibling paths on the same volume; the service PathName must still resolve to $Install.
Move-Item -LiteralPath $Install -Destination $PreviousInstall -ErrorAction Stop
Move-Item -LiteralPath $Candidate -Destination $Install -ErrorAction Stop
Start-Service -Name OtOpcUaHost -ErrorAction Stop
(Get-Service -Name OtOpcUaHost).WaitForStatus('Running', [TimeSpan]::FromSeconds(60))

$PreviousInstall must be a fresh reviewed sibling path. Handle any failure between moves by restoring that complete directory before starting. Service Running is only a process check. Run the substantive checks below after every start and failover:

• Check /healthz, /health/ready and role-appropriate /health/active at the recorded HTTP base URL; compare ServiceLevel and active/standby state to baseline.
• Confirm certificate thumbprints, trust/issuer chains, endpoint policies/application URI and namespace-URI/NodeId identity. Prove trusted SignAndEncrypt works and untrusted/revoked/inappropriate certificates and wrong keys fail without downgrade.
• Exercise raw and UNS browse/read, scalar/array writes on sandbox nodes, subscriptions, quality and source/server timestamps. Verify current legitimate ACL allows as well as denies, key expiry/revocation and audience-specific event/audit filtering.
• Verify alarms/refresh/ack and no missing or duplicate events; verify configured raw, modified, processed and event history, paging/release/reauthentication and expected unsupported operations. Verify recorder/outbox progress without duplicate replay.
• Observe reconnects, stale values, legitimate-call denials, continuation-point errors, session/item/certificate-handle growth, shutdown, queue depths and latency.

Use Client.CLI.md for exact connect/browse/read/subscribe/history and alarm syntax. Supply credentials through the approved operator mechanism; do not record command lines containing API-key passwords in release evidence.

Stop promotion and restore the known-good application for any authorization regression, missing/duplicate alarm or history data, changed identifiers, insecure fallback, restart failure or sustained performance-budget miss. Compare like hardware/config/load: at least three warmed measurements, provisional limits ≤10% median throughput loss and ≤15% p95 latency/allocation/RSS growth, plus at least two hours of lifecycle soak. Any deviation needs investigation and a recorded acceptance decision. Do not lower thresholds or count absent environments as passed.

Rollback

1. Freeze promotion; record the trigger and affected nodes/clients in #782. Preserve candidate logs and manifests with secrets redacted. Stop the candidate through the approved local channel; the remote-exec hold still applies during rollback.
2. With the service stopped, move the candidate install to a fresh quarantine path, restore the complete $PreviousInstall (or verified $Backup) to the inspected service path, and restore the matching node configuration and preserved PKI with original permissions. Check hashes/thumbprints before starting. Do not delete or roll back live historian, outbox or ConfigDb data without a reviewed data strategy.
3. Start OtOpcUaHost locally with the same Start-Service/wait commands above. On Linux, use only the environment's approved recorded supervisor procedure. Verify health, endpoint/namespace/certificate identity and active/standby state.
4. Reconnect clients as needed. In-memory sessions, subscriptions and continuation points do not survive process rollback; discard old points and begin new reads. Rebind namespaces by URI. Never replay ambiguous non-idempotent writes or alarm acknowledgements. Confirm API-key writes/ack, subscriptions, alarm counts and history continuity with the same positive and negative checks used at rollout.
5. Record recovery time, checks, data reconciliation and final hold/go decision. If rolling source back, revert package and application commits together; never mix old package binaries with SDK 2 application assemblies. Keep candidate evidence for diagnosis and link follow-up issues from #782.

Results handoff

Local build, security/wire, aggregate, backend, compatibility, publish and Linux smoke evidence is linked above with its actual scope and revisions. The server RSS gate is failed (#799), full CI retains the 11 baseline failures (#796/#798), the client lifecycle run completed two hours with cleanup checks passed, secure ScadaBridge interoperability is blocked (ScadaBridge#121), and Windows/live AVEVA/pair/operator gates remain pending (#800/#801). No fleet rollout or rollback has been executed. The final post-HG1.0 corrected full run retains only the eleven independently reproduced baseline failures; earlier failed runs remain recorded. The coordinator records remaining decisions in #782 and the main evidence; none is inferred from a passing subset or cross-published artifact.