Files
lmxopcua/docs/plans/2026-11-10-opcua-sdk-2-rollout.md
T
2026-11-10 11:48:19 -05:00

17 KiB

OPC UA SDK 2.0.1 rollout and rollback

This operator runbook accompanies #782, the implementation plan and execution evidence. Gitea is the single status system. This draft was checked against source a21104d2; it does not approve an artifact or record a deployment. The coordinator must attach final commit-specific evidence in #782 before advancing a gate. All placeholders below are pending evidence, not passing results.

The target is exact OPC Foundation SDK 2.0.1; the known-good SDK baseline is 1.5.378.182 (application baseline 83ed880e4471344deaf008da9bd028c7e170bdc7). Restore the actual archived known-good application as a unit during rollback. Package downgrades alone cannot reverse the application/API migration.

Operating constraints

The remote-execution hold on wonder boxes vd03/vq03/vq04 remains binding. No WinRM, psbridge or replacement remote-exec channel may be used until the user clears a channel. UNC/SMB file copying and permitted SQL access do not authorize service control, PowerShell execution or deployment on those hosts. Steps requiring a command on a held host remain pending. No deployment was performed for this draft.

Run the service commands below only from an approved local operator session on an authorized Windows node. Record the node, account, channel and maintenance window in #782 first. Linux runtime/service handling requires its own approved environment and smoke record; this runbook supplies no invented Linux service-unit name. Do not use scripts/install/Refresh-Services.ps1 as a blanket upgrade action: it also touches gateway/retired sidecar paths outside this SDK rollout.

Preserved contract

  • Existing raw/UNS namespace URIs, identifier forms, endpoint profiles, API-key/anonymous admission, ACL flag behavior, role and key revocation gates remain the contract. Source uses public transport peer evidence for miss budgets, not an IP admission rule.
  • Scalar leniency, checked numeric-array conversion and pass-through driver behavior stay as before. SDK immutable values are detached at boundaries; no new matrix support.
  • No application status-code, locale or event-filter semantic change is approved. SDK status wrappers and explicit empty ContentFilter construction are API changes, not evidence of changed protocol values. Record any stricter response, field-level filter error or localization difference with a consumer-impact decision before go.
  • Auditing stays opt-in. Ordinary SDK automatic audit delivery retains its type restrictions. Only the application's rejected-write audit gets a private, temporary receive-permission context after current source Read ACL and audit/encryption checks; no shared role metadata or general audit access is widened. See security.
  • The private source port retains the 37 legacy aggregate factories and existing application results rather than adopting SDK 2 algorithms. Original-SDK fixtures contain 6,144 vectors. TreatUncertainAsBad=false stays explicit; historian backend aggregate sets and existing documented differences remain unchanged. SDK FILETIME-zero maps to the existing CLR unset-time sentinel at history boundaries.
  • No new historian backend, companion specification, PubSub, ManagedSession, AOT or cluster rolling-upgrade capability is claimed. HistorianGateway Client/Contracts 0.4.0 is already on gateway master 16111a5e (historiangw#119 closed); the real gateway gate historiangw#78 remains separate from package alignment.

Release record and preflight

Fill this record in #782 with links to immutable logs/manifests; do not paste keys, passwords, connection strings, private certificates or unredacted environment dumps.

Required evidence Record to supply before release
Candidate source and graph Pending: full commit, clean/dirty state, SDK/runtime/OS, resolved package graph showing exact 2.0.1
Known-good recovery set Pending: deployed application version, full binary/config archive SHA-256, protected PKI backup location and permission record
Candidate artifacts Pending: Windows and Linux publish directories/archives, RID/self-contained choice, SHA-256 manifest, dependencies and script-worker contents
G2 compilation Pending: restore/build logs and warning/error counts at the final SHA
G3 behavior/security Pending: project TRX/counts, required case results, positive and negative authorization, audit privacy, CP lifecycle and 6,144-vector comparison
Interoperability Pending: baseline/candidate client-server pairs, actual ScadaBridge adapter, upstream driver/browser products and mixed-version cluster gate
Historian Pending: configured SqlServer/QuestDB/Gateway capability-specific results; historiangw#78 link/evidence for live gateway
G4 resources/performance Pending: baseline comparison, at least three measured runs, two-hour soak, Windows/Linux start-stop-twice smoke with preserved PKI
Review/operations Pending: independent security/release review, authorized channel/window, named operator, canary/rollback results and decision

Before staging:

  1. Inventory actual service executable/arguments, roles, endpoints, advertised host, application URI, raw/UNS namespace identities, certificate thumbprints, active/standby state and connected client builds. Record configured queues, request/transport quotas, history caps and expected load. Establish performance budgets and stop thresholds.
  2. Archive complete known-good binaries and node configuration, preserving access controls. Securely back up PKI own/private keys, peer/issuer chains, CRLs and their permissions. Verify recovery access under the service identity, private-key presence, expiry, application URI/hostname match and CA/key-usage suitability. Retain existing certificates.
  3. Inventory ConfigDb, LocalDb, historian/outbox and secret-provider state/locations. No ConfigDb migration is planned. Record the actual schema/serialization diff, including Akka/deployment contracts; if persistence changes appear, reopen the design gate and define data rollback before deployment. Never overwrite live historian/outbox data with an old binary backup or rewrite persisted sessions/subscriptions.
  4. Publish into fresh directories, hash artifacts, stage separately from the live install and validate graph contents. Keep baseline and candidate ports, PKI, config and stores separate in the isolated rig; never run both against the same mutable fixture.
  5. Confirm required positive/negative checks, reviewers and recovery artifact access. Missing evidence, inaccessible environments or an uncleared command channel means hold.

Verification commands

These commands are recipes, not execution results. Run from the repository root in Bash with .NET 10. Record the final SHA and logs in a fresh evidence directory; inspect exit status before continuing. Do not pipe a failing build/test into a successful logger.

git rev-parse HEAD
git status --short
dotnet --info
dotnet restore ZB.MOM.WW.OtOpcUa.slnx
dotnet build ZB.MOM.WW.OtOpcUa.slnx -c Release --no-restore

Define the helper once after a successful build. Each invocation creates its own fresh TRX directory. It preserves both test failure and zero-executed-test failure. A successful guard still requires inspection of required cases and skips.

opc_test() {
  local opc_project="$1"
  local opc_results
  opc_results=$(mktemp -d "${TMPDIR:-/tmp}/opcua2-test.XXXXXXXX") || return
  local opc_test_status=0
  dotnet test "$opc_project" -c Release --no-build --logger trx \
    --results-directory "$opc_results" || opc_test_status=$?
  local opc_count_status=0
  MIN_EXECUTED=1 bash scripts/ci/assert-not-all-skipped.sh "$opc_results"/*.trx \
    || opc_count_status=$?
  printf 'TRX directory: %s\n' "$opc_results"
  if [ "$opc_test_status" -ne 0 ]; then return "$opc_test_status"; fi
  return "$opc_count_status"
}
opc_test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests.csproj
opc_test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.csproj
opc_test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.csproj
opc_test tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests.csproj
opc_test tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests.csproj

Driver/browser and simulator commands are in the fixture map. Backend and live historian prerequisites are in Historian.md and infra/README.md. Live tests can write sandbox data: use only designated sandbox tags and approved credentials through the existing secret provider/environment. A skipped live/DB fixture never satisfies its gate. Generate original aggregate evidence only with the isolated oracle procedure, a new output path and original 1.5.378.182 binary; never derive expectations from the candidate's calculators.

Also run the full CI-equivalent test leg after integration; it does not replace fixture-specific or live gates:

opc_solution_test() {
  local opc_results
  opc_results=$(mktemp -d "${TMPDIR:-/tmp}/opcua2-solution.XXXXXXXX") || return
  local opc_test_status=0
  dotnet test ZB.MOM.WW.OtOpcUa.slnx -c Release --no-build \
    -m:1 -p:TestTfmsInParallel=false --settings scripts/ci/ci.runsettings \
    --filter 'Category!=E2E&Category!=LiveIntegration' \
    --logger trx --results-directory "$opc_results" || opc_test_status=$?
  local opc_count_status=0
  MIN_EXECUTED=12000 bash scripts/ci/assert-not-all-skipped.sh "$opc_results"/*.trx \
    || opc_count_status=$?
  printf 'TRX directory: %s\n' "$opc_results"
  if [ "$opc_test_status" -ne 0 ]; then return "$opc_test_status"; fi
  return "$opc_count_status"
}
opc_solution_test

Publish the Host project to a fresh operator-selected $OPC_PUBLISH_DIR using the approved $OPC_RID and deployment's self-contained choice. The Windows install expects OtOpcUa.Host.exe; ServiceHosting.md also describes script-worker packaging. Example self-contained Windows artifact command:

dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj \
  -c Release -r win-x64 --self-contained true -o "$OPC_PUBLISH_DIR"

Repeat with the approved Linux RID/framework settings in its validation environment; cross-publishing is not runtime/install evidence. Hash the complete staged artifact and retain a manifest outside it (PowerShell 7, where $Candidate and $Manifest are approved absolute local paths and the manifest is outside the candidate):

Get-ChildItem -LiteralPath $Candidate -Recurse -File |
    Get-FileHash -Algorithm SHA256 |
    Export-Csv -LiteralPath $Manifest -NoTypeInformation

Sequence and stop/go decision

Progress only after the preceding stage's recorded checks and explicit gate decision: isolated rig → ScadaBridge/upstream interoperability → DEV canary → QA pair → production canary. Attach source/artifact identity, start/end times, operator, checks, observations and go/hold/rollback decision for each stage to #782.

A standby-first pair restart is allowed only after the application's mixed-version Akka serialization, deployment/config and failover gate succeeds. Verify standby health, security, data, history and events; exercise failover before upgrading its peer. OPC UA wire interoperability alone does not prove cluster rolling safety. Without that gate, arrange a maintenance window and coordinated pair restart; record expected service interruption and recovery checks.

The following Windows commands are local-session steps only. Bind $Install to the inspected current binary directory, $Candidate to the reviewed staged artifact and $Backup to a new protected directory. Keep PKI/data stores outside binary replacement; if they are inside the install tree, define exclusions and a separate protected backup before using these commands. Save configuration outside $Candidate, then apply the reviewed node configuration to the staging directory; never overwrite it with publisher defaults or expose secrets in command output.

Get-CimInstance Win32_Service -Filter "Name='OtOpcUaHost'" |
    Select-Object Name, State, PathName, StartName
Stop-Service -Name OtOpcUaHost -ErrorAction Stop
(Get-Service -Name OtOpcUaHost).WaitForStatus('Stopped', [TimeSpan]::FromSeconds(60))
robocopy $Install $Backup /E /COPYALL /DCOPY:DAT /R:1 /W:1
if ($LASTEXITCODE -ge 8) { throw 'Known-good backup failed' }
# After checking the backup and candidate manifest, swap complete binary directories.
# Use sibling paths on the same volume; the service PathName must still resolve to $Install.
Move-Item -LiteralPath $Install -Destination $PreviousInstall -ErrorAction Stop
Move-Item -LiteralPath $Candidate -Destination $Install -ErrorAction Stop
Start-Service -Name OtOpcUaHost -ErrorAction Stop
(Get-Service -Name OtOpcUaHost).WaitForStatus('Running', [TimeSpan]::FromSeconds(60))

$PreviousInstall must be a fresh reviewed sibling path. Handle any failure between moves by restoring that complete directory before starting. Service Running is only a process check. Run the substantive checks below after every start and failover:

  • Check /healthz, /health/ready and role-appropriate /health/active at the recorded HTTP base URL; compare ServiceLevel and active/standby state to baseline.
  • Confirm certificate thumbprints, trust/issuer chains, endpoint policies/application URI and namespace-URI/NodeId identity. Prove trusted SignAndEncrypt works and untrusted/revoked/inappropriate certificates and wrong keys fail without downgrade.
  • Exercise raw and UNS browse/read, scalar/array writes on sandbox nodes, subscriptions, quality and source/server timestamps. Verify current legitimate ACL allows as well as denies, key expiry/revocation and audience-specific event/audit filtering.
  • Verify alarms/refresh/ack and no missing or duplicate events; verify configured raw, modified, processed and event history, paging/release/reauthentication and expected unsupported operations. Verify recorder/outbox progress without duplicate replay.
  • Observe reconnects, stale values, legitimate-call denials, continuation-point errors, session/item/certificate-handle growth, shutdown, queue depths and latency.

Use Client.CLI.md for exact connect/browse/read/subscribe/history and alarm syntax. Supply credentials through the approved operator mechanism; do not record command lines containing API-key passwords in release evidence.

Stop promotion and restore the known-good application for any authorization regression, missing/duplicate alarm or history data, changed identifiers, insecure fallback, restart failure or sustained performance-budget miss. Compare like hardware/config/load: at least three warmed measurements, provisional limits ≤10% median throughput loss and ≤15% p95 latency/allocation/RSS growth, plus at least two hours of lifecycle soak. Any deviation needs investigation and a recorded acceptance decision. Do not lower thresholds or count absent environments as passed.

Rollback

  1. Freeze promotion; record the trigger and affected nodes/clients in #782. Preserve candidate logs and manifests with secrets redacted. Stop the candidate through the approved local channel; the remote-exec hold still applies during rollback.
  2. With the service stopped, move the candidate install to a fresh quarantine path, restore the complete $PreviousInstall (or verified $Backup) to the inspected service path, and restore the matching node configuration and preserved PKI with original permissions. Check hashes/thumbprints before starting. Do not delete or roll back live historian, outbox or ConfigDb data without a reviewed data strategy.
  3. Start OtOpcUaHost locally with the same Start-Service/wait commands above. On Linux, use only the environment's approved recorded supervisor procedure. Verify health, endpoint/namespace/certificate identity and active/standby state.
  4. Reconnect clients as needed. In-memory sessions, subscriptions and continuation points do not survive process rollback; discard old points and begin new reads. Rebind namespaces by URI. Never replay ambiguous non-idempotent writes or alarm acknowledgements. Confirm API-key writes/ack, subscriptions, alarm counts and history continuity with the same positive and negative checks used at rollout.
  5. Record recovery time, checks, data reconciliation and final hold/go decision. If rolling source back, revert package and application commits together; never mix old package binaries with SDK 2 application assemblies. Keep candidate evidence for diagnosis and link follow-up issues from #782.

Results handoff

Pending: coordinator to attach the final SHA, artifact hashes, TRX counts/skips, interoperability/performance/runtime evidence, reviews and per-environment operator records in #782. This document makes no claim of a passing build, test, ScadaBridge, Windows/Linux smoke, live backend, soak or deployment. Required inaccessible steps remain open gates unless formally separated into linked issues with accurate scope.