docs: document OPC UA 2.0 compatibility and rollout
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,284 @@
|
||||
# OPC UA SDK 2.0.1 rollout and rollback
|
||||
|
||||
This operator runbook accompanies [#782](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/782),
|
||||
the [implementation plan](2026-11-10-opcua-sdk-2-upgrade.md) and
|
||||
[execution evidence](2026-11-10-opcua-sdk-2-evidence.md). Gitea is the single
|
||||
status system. This draft was checked against source `a21104d2`; it does not
|
||||
approve an artifact or record a deployment. The coordinator must attach final
|
||||
commit-specific evidence in #782 before advancing a gate. All placeholders below
|
||||
are **pending evidence**, not passing results.
|
||||
|
||||
The target is exact OPC Foundation SDK **2.0.1**; the known-good SDK baseline is
|
||||
**1.5.378.182** (application baseline `83ed880e4471344deaf008da9bd028c7e170bdc7`).
|
||||
Restore the actual archived known-good application as a unit during rollback.
|
||||
Package downgrades alone cannot reverse the application/API migration.
|
||||
|
||||
## Operating constraints
|
||||
|
||||
**The remote-execution hold on wonder boxes vd03/vq03/vq04 remains binding.**
|
||||
No WinRM, psbridge or replacement remote-exec channel may be used until the user
|
||||
clears a channel. UNC/SMB file copying and permitted SQL access do not authorize
|
||||
service control, PowerShell execution or deployment on those hosts. Steps requiring
|
||||
a command on a held host remain pending. No deployment was performed for this draft.
|
||||
|
||||
Run the service commands below only from an approved local operator session on an
|
||||
authorized Windows node. Record the node, account, channel and maintenance window
|
||||
in #782 first. Linux runtime/service handling requires its own approved environment
|
||||
and smoke record; this runbook supplies no invented Linux service-unit name.
|
||||
Do not use `scripts/install/Refresh-Services.ps1` as a blanket upgrade action: it
|
||||
also touches gateway/retired sidecar paths outside this SDK rollout.
|
||||
|
||||
## Preserved contract
|
||||
|
||||
- Existing raw/UNS namespace URIs, identifier forms, endpoint profiles, API-key/anonymous
|
||||
admission, ACL flag behavior, role and key revocation gates remain the contract.
|
||||
Source uses public transport peer evidence for miss budgets, not an IP admission rule.
|
||||
- Scalar leniency, checked numeric-array conversion and pass-through driver behavior
|
||||
stay as before. SDK immutable values are detached at boundaries; no new matrix support.
|
||||
- No application status-code, locale or event-filter semantic change is approved.
|
||||
SDK status wrappers and explicit empty `ContentFilter` construction are API changes,
|
||||
not evidence of changed protocol values. Record any stricter response, field-level
|
||||
filter error or localization difference with a consumer-impact decision before go.
|
||||
- Auditing stays opt-in. Ordinary SDK automatic audit delivery retains its type
|
||||
restrictions. Only the application's rejected-write audit gets a private, temporary
|
||||
receive-permission context after current source `Read` ACL and audit/encryption checks;
|
||||
no shared role metadata or general audit access is widened. See [security](../security.md).
|
||||
- The private source port retains the **37 legacy aggregate factories** and existing
|
||||
application results rather than adopting SDK 2 algorithms. Original-SDK fixtures
|
||||
contain **6,144 vectors**. `TreatUncertainAsBad=false` stays explicit; historian
|
||||
backend aggregate sets and existing documented differences remain unchanged.
|
||||
SDK FILETIME-zero maps to the existing CLR unset-time sentinel at history boundaries.
|
||||
- No new historian backend, companion specification, PubSub, ManagedSession, AOT or
|
||||
cluster rolling-upgrade capability is claimed. HistorianGateway Client/Contracts
|
||||
**0.4.0** is already on gateway master `16111a5e` (historiangw#119 closed);
|
||||
the real gateway gate historiangw#78 remains separate from package alignment.
|
||||
|
||||
## Release record and preflight
|
||||
|
||||
Fill this record in #782 with links to immutable logs/manifests; do not paste keys,
|
||||
passwords, connection strings, private certificates or unredacted environment dumps.
|
||||
|
||||
| Required evidence | Record to supply before release |
|
||||
|---|---|
|
||||
| Candidate source and graph | Pending: full commit, clean/dirty state, SDK/runtime/OS, resolved package graph showing exact 2.0.1 |
|
||||
| Known-good recovery set | Pending: deployed application version, full binary/config archive SHA-256, protected PKI backup location and permission record |
|
||||
| Candidate artifacts | Pending: Windows and Linux publish directories/archives, RID/self-contained choice, SHA-256 manifest, dependencies and script-worker contents |
|
||||
| G2 compilation | Pending: restore/build logs and warning/error counts at the final SHA |
|
||||
| G3 behavior/security | Pending: project TRX/counts, required case results, positive and negative authorization, audit privacy, CP lifecycle and 6,144-vector comparison |
|
||||
| Interoperability | Pending: baseline/candidate client-server pairs, actual ScadaBridge adapter, upstream driver/browser products and mixed-version cluster gate |
|
||||
| Historian | Pending: configured SqlServer/QuestDB/Gateway capability-specific results; historiangw#78 link/evidence for live gateway |
|
||||
| G4 resources/performance | Pending: baseline comparison, at least three measured runs, two-hour soak, Windows/Linux start-stop-twice smoke with preserved PKI |
|
||||
| Review/operations | Pending: independent security/release review, authorized channel/window, named operator, canary/rollback results and decision |
|
||||
|
||||
Before staging:
|
||||
|
||||
1. Inventory actual service executable/arguments, roles, endpoints, advertised host,
|
||||
application URI, raw/UNS namespace identities, certificate thumbprints, active/standby
|
||||
state and connected client builds. Record configured queues, request/transport quotas,
|
||||
history caps and expected load. Establish performance budgets and stop thresholds.
|
||||
2. Archive complete known-good binaries and node configuration, preserving access controls.
|
||||
Securely back up PKI own/private keys, peer/issuer chains, CRLs and their permissions.
|
||||
Verify recovery access under the service identity, private-key presence, expiry,
|
||||
application URI/hostname match and CA/key-usage suitability. Retain existing certificates.
|
||||
3. Inventory ConfigDb, LocalDb, historian/outbox and secret-provider state/locations. No
|
||||
ConfigDb migration is planned. Record the actual schema/serialization diff, including
|
||||
Akka/deployment contracts; if persistence changes appear, reopen the design gate and
|
||||
define data rollback before deployment. Never overwrite live historian/outbox data
|
||||
with an old binary backup or rewrite persisted sessions/subscriptions.
|
||||
4. Publish into fresh directories, hash artifacts, stage separately from the live install
|
||||
and validate graph contents. Keep baseline and candidate ports, PKI, config and stores
|
||||
separate in the isolated rig; never run both against the same mutable fixture.
|
||||
5. Confirm required positive/negative checks, reviewers and recovery artifact access.
|
||||
Missing evidence, inaccessible environments or an uncleared command channel means hold.
|
||||
|
||||
## Verification commands
|
||||
|
||||
These commands are recipes, not execution results. Run from the repository root in
|
||||
Bash with .NET 10. Record the final SHA and logs in a fresh evidence directory; inspect
|
||||
exit status before continuing. Do not pipe a failing build/test into a successful logger.
|
||||
|
||||
```bash
|
||||
git rev-parse HEAD
|
||||
git status --short
|
||||
dotnet --info
|
||||
dotnet restore ZB.MOM.WW.OtOpcUa.slnx
|
||||
dotnet build ZB.MOM.WW.OtOpcUa.slnx -c Release --no-restore
|
||||
```
|
||||
|
||||
Define the helper once after a successful build. Each invocation creates its own
|
||||
fresh TRX directory. It preserves both test failure and zero-executed-test failure.
|
||||
A successful guard still requires inspection of required cases and skips.
|
||||
|
||||
```bash
|
||||
opc_test() {
|
||||
local opc_project="$1"
|
||||
local opc_results
|
||||
opc_results=$(mktemp -d "${TMPDIR:-/tmp}/opcua2-test.XXXXXXXX") || return
|
||||
local opc_test_status=0
|
||||
dotnet test "$opc_project" -c Release --no-build --logger trx \
|
||||
--results-directory "$opc_results" || opc_test_status=$?
|
||||
local opc_count_status=0
|
||||
MIN_EXECUTED=1 bash scripts/ci/assert-not-all-skipped.sh "$opc_results"/*.trx \
|
||||
|| opc_count_status=$?
|
||||
printf 'TRX directory: %s\n' "$opc_results"
|
||||
if [ "$opc_test_status" -ne 0 ]; then return "$opc_test_status"; fi
|
||||
return "$opc_count_status"
|
||||
}
|
||||
opc_test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests.csproj
|
||||
opc_test tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.csproj
|
||||
opc_test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.csproj
|
||||
opc_test tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests.csproj
|
||||
opc_test tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests.csproj
|
||||
```
|
||||
|
||||
Driver/browser and simulator commands are in the
|
||||
[fixture map](../drivers/OpcUaClient-Test-Fixture.md#running-locally). Backend and
|
||||
live historian prerequisites are in [Historian.md](../Historian.md) and
|
||||
[infra/README.md](../../infra/README.md). Live tests can write sandbox data: use
|
||||
only designated sandbox tags and approved credentials through the existing secret
|
||||
provider/environment. A skipped live/DB fixture never satisfies its gate.
|
||||
Generate original aggregate evidence only with the isolated
|
||||
[oracle procedure](../../scripts/validation/opcua-aggregate-baseline/README.md),
|
||||
a new output path and original 1.5.378.182 binary; never derive expectations from
|
||||
the candidate's calculators.
|
||||
|
||||
Also run the full CI-equivalent test leg after integration; it does not replace
|
||||
fixture-specific or live gates:
|
||||
|
||||
```bash
|
||||
opc_solution_test() {
|
||||
local opc_results
|
||||
opc_results=$(mktemp -d "${TMPDIR:-/tmp}/opcua2-solution.XXXXXXXX") || return
|
||||
local opc_test_status=0
|
||||
dotnet test ZB.MOM.WW.OtOpcUa.slnx -c Release --no-build \
|
||||
-m:1 -p:TestTfmsInParallel=false --settings scripts/ci/ci.runsettings \
|
||||
--filter 'Category!=E2E&Category!=LiveIntegration' \
|
||||
--logger trx --results-directory "$opc_results" || opc_test_status=$?
|
||||
local opc_count_status=0
|
||||
MIN_EXECUTED=12000 bash scripts/ci/assert-not-all-skipped.sh "$opc_results"/*.trx \
|
||||
|| opc_count_status=$?
|
||||
printf 'TRX directory: %s\n' "$opc_results"
|
||||
if [ "$opc_test_status" -ne 0 ]; then return "$opc_test_status"; fi
|
||||
return "$opc_count_status"
|
||||
}
|
||||
opc_solution_test
|
||||
```
|
||||
|
||||
Publish the Host project to a fresh operator-selected `$OPC_PUBLISH_DIR` using the
|
||||
approved `$OPC_RID` and deployment's self-contained choice. The Windows install
|
||||
expects `OtOpcUa.Host.exe`; [ServiceHosting.md](../ServiceHosting.md) also describes
|
||||
script-worker packaging. Example self-contained Windows artifact command:
|
||||
|
||||
```bash
|
||||
dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj \
|
||||
-c Release -r win-x64 --self-contained true -o "$OPC_PUBLISH_DIR"
|
||||
```
|
||||
|
||||
Repeat with the approved Linux RID/framework settings in its validation environment;
|
||||
cross-publishing is not runtime/install evidence. Hash the complete staged artifact
|
||||
and retain a manifest outside it (PowerShell 7, where `$Candidate` and `$Manifest`
|
||||
are approved absolute local paths and the manifest is outside the candidate):
|
||||
|
||||
```powershell
|
||||
Get-ChildItem -LiteralPath $Candidate -Recurse -File |
|
||||
Get-FileHash -Algorithm SHA256 |
|
||||
Export-Csv -LiteralPath $Manifest -NoTypeInformation
|
||||
```
|
||||
|
||||
## Sequence and stop/go decision
|
||||
|
||||
Progress only after the preceding stage's recorded checks and explicit gate decision:
|
||||
**isolated rig → ScadaBridge/upstream interoperability → DEV canary → QA pair →
|
||||
production canary**. Attach source/artifact identity, start/end times, operator,
|
||||
checks, observations and go/hold/rollback decision for each stage to #782.
|
||||
|
||||
A standby-first pair restart is allowed only after the application's mixed-version
|
||||
Akka serialization, deployment/config and failover gate succeeds. Verify standby
|
||||
health, security, data, history and events; exercise failover before upgrading its
|
||||
peer. OPC UA wire interoperability alone does not prove cluster rolling safety.
|
||||
Without that gate, arrange a maintenance window and coordinated pair restart;
|
||||
record expected service interruption and recovery checks.
|
||||
|
||||
The following Windows commands are **local-session steps only**. Bind `$Install`
|
||||
to the inspected current binary directory, `$Candidate` to the reviewed staged
|
||||
artifact and `$Backup` to a new protected directory. Keep PKI/data stores outside
|
||||
binary replacement; if they are inside the install tree, define exclusions and a
|
||||
separate protected backup before using these commands. Save configuration outside
|
||||
`$Candidate`, then apply the reviewed node configuration to the staging directory;
|
||||
never overwrite it with publisher defaults or expose secrets in command output.
|
||||
|
||||
```powershell
|
||||
Get-CimInstance Win32_Service -Filter "Name='OtOpcUaHost'" |
|
||||
Select-Object Name, State, PathName, StartName
|
||||
Stop-Service -Name OtOpcUaHost -ErrorAction Stop
|
||||
(Get-Service -Name OtOpcUaHost).WaitForStatus('Stopped', [TimeSpan]::FromSeconds(60))
|
||||
robocopy $Install $Backup /E /COPYALL /DCOPY:DAT /R:1 /W:1
|
||||
if ($LASTEXITCODE -ge 8) { throw 'Known-good backup failed' }
|
||||
# After checking the backup and candidate manifest, swap complete binary directories.
|
||||
# Use sibling paths on the same volume; the service PathName must still resolve to $Install.
|
||||
Move-Item -LiteralPath $Install -Destination $PreviousInstall -ErrorAction Stop
|
||||
Move-Item -LiteralPath $Candidate -Destination $Install -ErrorAction Stop
|
||||
Start-Service -Name OtOpcUaHost -ErrorAction Stop
|
||||
(Get-Service -Name OtOpcUaHost).WaitForStatus('Running', [TimeSpan]::FromSeconds(60))
|
||||
```
|
||||
|
||||
`$PreviousInstall` must be a fresh reviewed sibling path. Handle any failure between
|
||||
moves by restoring that complete directory before starting. Service `Running` is
|
||||
only a process check. Run the substantive checks below after every start and failover:
|
||||
|
||||
- Check `/healthz`, `/health/ready` and role-appropriate `/health/active` at the
|
||||
recorded HTTP base URL; compare ServiceLevel and active/standby state to baseline.
|
||||
- Confirm certificate thumbprints, trust/issuer chains, endpoint policies/application
|
||||
URI and namespace-URI/NodeId identity. Prove trusted SignAndEncrypt works and
|
||||
untrusted/revoked/inappropriate certificates and wrong keys fail without downgrade.
|
||||
- Exercise raw and UNS browse/read, scalar/array writes on sandbox nodes, subscriptions,
|
||||
quality and source/server timestamps. Verify current legitimate ACL allows as well
|
||||
as denies, key expiry/revocation and audience-specific event/audit filtering.
|
||||
- Verify alarms/refresh/ack and no missing or duplicate events; verify configured raw,
|
||||
modified, processed and event history, paging/release/reauthentication and expected
|
||||
unsupported operations. Verify recorder/outbox progress without duplicate replay.
|
||||
- Observe reconnects, stale values, legitimate-call denials, continuation-point errors,
|
||||
session/item/certificate-handle growth, shutdown, queue depths and latency.
|
||||
|
||||
Use [Client.CLI.md](../Client.CLI.md) for exact connect/browse/read/subscribe/history
|
||||
and alarm syntax. Supply credentials through the approved operator mechanism; do
|
||||
not record command lines containing API-key passwords in release evidence.
|
||||
|
||||
Stop promotion and restore the known-good application for any authorization regression,
|
||||
missing/duplicate alarm or history data, changed identifiers, insecure fallback,
|
||||
restart failure or sustained performance-budget miss. Compare like hardware/config/load:
|
||||
at least three warmed measurements, provisional limits ≤10% median throughput loss
|
||||
and ≤15% p95 latency/allocation/RSS growth, plus at least two hours of lifecycle soak.
|
||||
Any deviation needs investigation and a recorded acceptance decision. Do not lower
|
||||
thresholds or count absent environments as passed.
|
||||
|
||||
## Rollback
|
||||
|
||||
1. Freeze promotion; record the trigger and affected nodes/clients in #782. Preserve
|
||||
candidate logs and manifests with secrets redacted. Stop the candidate through the
|
||||
approved local channel; the remote-exec hold still applies during rollback.
|
||||
2. With the service stopped, move the candidate install to a fresh quarantine path,
|
||||
restore the complete `$PreviousInstall` (or verified `$Backup`) to the inspected
|
||||
service path, and restore the matching node configuration and preserved PKI with
|
||||
original permissions. Check hashes/thumbprints before starting. Do not delete or
|
||||
roll back live historian, outbox or ConfigDb data without a reviewed data strategy.
|
||||
3. Start `OtOpcUaHost` locally with the same `Start-Service`/wait commands above.
|
||||
On Linux, use only the environment's approved recorded supervisor procedure.
|
||||
Verify health, endpoint/namespace/certificate identity and active/standby state.
|
||||
4. Reconnect clients as needed. In-memory sessions, subscriptions and continuation
|
||||
points do not survive process rollback; discard old points and begin new reads.
|
||||
Rebind namespaces by URI. Never replay ambiguous non-idempotent writes or alarm
|
||||
acknowledgements. Confirm API-key writes/ack, subscriptions, alarm counts and history
|
||||
continuity with the same positive and negative checks used at rollout.
|
||||
5. Record recovery time, checks, data reconciliation and final hold/go decision.
|
||||
If rolling source back, revert package and application commits together; never
|
||||
mix old package binaries with SDK 2 application assemblies. Keep candidate evidence
|
||||
for diagnosis and link follow-up issues from #782.
|
||||
|
||||
## Results handoff
|
||||
|
||||
Pending: coordinator to attach the final SHA, artifact hashes, TRX counts/skips,
|
||||
interoperability/performance/runtime evidence, reviews and per-environment operator
|
||||
records in #782. This document makes no claim of a passing build, test, ScadaBridge,
|
||||
Windows/Linux smoke, live backend, soak or deployment. Required inaccessible steps
|
||||
remain open gates unless formally separated into linked issues with accurate scope.
|
||||
Reference in New Issue
Block a user