Files
lmxopcua/docs/deploy/2026-11-13-next-vd03-qa-to-c9640f5c.md
T
Joseph Doherty 0c19f26c81
v2-ci / unit-tests (push) Has been cancelled
fix(install): cut the swap's stopped window to the swapped-in folder (#905)
Root cause: the protected-DACL coverage scan, not the ACL writes. Both -Reapply and
-Verify walked every denied folder (all of E:\ApiInstall on vd03, nested roots again)
with Get-ChildItem -Recurse + Get-Acl per item + a pipeline per item for -Skip, about
3.4 ms an item on PS 5.1, all while the service was stopped. An unchanged deny is read,
not rewritten; the re-applied install-folder deny propagates over ~1,400 files in 0.5 s.

- Find-OtOpcUaDenyCoverageGap: .NET enumeration, -Skip folders pruned, DACL-only
  GetAccessControl(Access) (about 0.15 ms an item); reparse points not followed.
- Get-OtOpcUaCoverageWalkRoots: each denied folder walked once (outermost only).
- Install-ScriptWorkerIsolation.ps1 -Scope All|InstallRoot|Outside (with -Reapply,
  -Verify, -PlanOnly). InstallRoot = the folder ACEs a swap drops; Outside = the rest,
  including the firewall rule (its query costs 3-5 s; its exe path does not change).
  The two partition the plan. Prints the scan time.
- Invoke-OtOpcUaInstallSwap.ps1: -Reapply/-Verify -Scope Outside before the stop (a
  failure stops before anything moves), -Scope InstallRoot inside the window; preflight
  checks the candidate carries the worker exe; prints the stopped window and its parts.
  -NoStart, -NoCarry and the pre-#901 refusal unchanged.

windev, vd03-like tree (17,851 items): stopped window 153.1 s -> 6.0 s (PS 5.1),
101.6 s -> 6.2 s (PS 7); the Outside part takes ~23 s with the old build serving.
Dry-run harness 101 -> 123 passed; Windows test 72 passed on PS 5.1 and PS 7.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 03:51:42 -05:00

30 KiB
Raw Blame History

Next deploy: DEV vd03 6c4d2990 and QA vq03/vq04 fea7de0a → master c9640f5c

EXECUTED 2026-11-13 (DEV vd03 07:33Z, QA pair 07:47Z): record 2026-11-13-vd03-qa-to-c9640f5c.md. Deviations there: longer downtime from the in-swap -Verify coverage walk; the §5 #903 revert cannot return the chip to "In sync" (RowVersion is in the revision hash); QA Secondary swapped before the Primary stopped.

Status when written: PREPARED, NOT EXECUTED (2026-11-13). This pack covers both tiers, plus the read-only ConfigDb preflight, which was run over direct SQL from the Mac on 2026-11-13. Nothing was built, staged or run on any wonder box. No blocker found; no issue filed.

Target: origin/master c9640f5c (c9640f5ced383e2a722702409adbc62298731994, #904). If master moves before the window, either pin to c9640f5c (as the 6c4d2990 run pinned) or re-run §1–§3 for the new range.

Templates: 2026-11-13-vd03-fea7de0a-to-6c4d2990.md (vd03 layout and scripts), qa-upgrade-b8b4c18f-to-fea7de0a.md (QA window, canary, #890/#891 denies, "Worker isolation across the next swap"), vd03-upgrade-b8b4c18f-to-master.md §e/§f and "Worker isolation and box-wide denies".

1. Commit ranges and what changes

vd03: 6c4d2990..c9640f5c (9 commits)

Commit Change Effect on vd03 (one fused node, one FOCAS driver, 22 tags, runner InProcess)
efaf3435 #900 HistorianGateway FailedPrecondition + AGGREGATE_NOT_SUPPORTED_FOR_TAG maps to BadAggregateNotSupported, with no health failure. Adds Grpc.StatusProto 2.76.0. Inert today. vd03's HG ea9f21c predates historiangw#135 (a220ec0) and still answers Internal, which keeps the current handling. It takes effect when HG ≥ a220ec0 is deployed.
9375c9e2 docs (the 6c4d2990 record) —
d5288388 #895 review, c9640f5c #904 Galaxy alarm reverse map: re-areaed, subtag-provider and pre-subscription references. A picker-form ack waits up to 2 s for the feed snapshot. Not reachable (no Galaxy driver).
2470f82e #872 Galaxy status rows Pending/Warning → Uncertain; detail 8001 → BadTypeMismatch. A Pending write reply is now "not committed". Not reachable.
b85e004b #902 AdminUI: MonacoEditor.DisposeAsync on a gone circuit logs at Debug, not [ERR]. Fixes the "Observed" [ERR] from the 6c4d2990 record.
35576485 #903 AdminUI /deployments: the drift chip is never a cached "In sync". The first render shows "Checking for changes...". A failed check shows "Drift check failed". Fixes the stale "In sync" chip from the 6c4d2990 record.
2cb8a2c3 #901 Installer records its settings and gains -Reapply/-Verify. New scripts/install/Invoke-OtOpcUaInstallSwap.ps1. Changes the deploy procedure (§4). No runtime change.
f0c6cf96 #834 Running Galaxy/OpcUaClient drivers refuse out-of-namespace credential refs at resolve time (the warn-only path is gone). The preflight logs one Error per finding, and the /raw banner is alert-danger. FOCAS has no credential fields. Preflight: 0 findings (§3). Only the boot line wording changes (§5).

QA: fea7de0a..c9640f5c (46 commits: the 37 above 6c4d2990 plus the 9 vd03 commits)

The vd03 rows above, plus everything vd03 already got on 2026-11-13 (fea7de0a..6c4d2990, 37 commits):

  • Historian.
    • ServerHistorian:Gateway:EnableNativeTimeWeighted now defaults to true (#694, 53978ca6). TimeAverage/Total are advertised and served natively (12 aggregates). QA HG fa9cf31 includes 9f711c1, so it supports this.
    • Alarm history goes out in SendEvents batches of ≤200, with a per-event fallback (#746).
    • A Boolean node refuses TimeAverage/Total up front with BadAggregateNotSupported (#897). A String/DateTime node refuses the numeric-only aggregates.
    • historianReadOnly: true TagConfig plus the new ServerHistorian:ProvisionTags setting (#899).
    • Float/Double/Byte/SByte historized tags are now provisioned (#898). See the upgrade hazard in §3.
  • Address space (#896): a UNS reference to an alarm-bearing raw tag no longer gets a UNS value Variable. The alarm surfaces only as the condition on the equipment folder.
  • Galaxy (#895): picker-authored native alarms resolve and ack. Galaxy spans and galaxy.events.* counters are exported (#892).
  • Telemetry 0.1.0 → 0.3.1 (scadaproj#55/#56).
    • The OTel Resource carries service.version (the git SHA), site.id (the Cluster id) and node.role.
    • With Exporter=Otlp, Serilog logs go over OTLP too. QA has OtOpcUa__Telemetry__Exporter=Otlp and OtlpEndpoint=http://127.0.0.1:4317 since 2026-11-12, so logs will start to reach the local ZbOtelCollector.
    • 0.3 refuses a scheme-less host:port when Exporter=Otlp. The recorded value has http://, so it passes; re-check it in the on-box preflight.
  • AdminUI #761 (fa8c3d1a): a bodyless POST /api/deployments returned 404 on fused admin+driver nodes. It now deploys.
  • Drivers.
    • AB Legacy MicroLogix function files are refused locally with BadNotSupported, not reported as a comms fault (#592).
    • Heidenhain gains the Controller/Family, Status/*StateName and Tool/AxisName leaves, PLC INPUT_DWORD/OUTPUT_DWORD, and new refusal-code mappings (#829/#853). Not reachable on QA.
  • Installer #890 (f75ee7d0): -DenyPaths. Already applied on QA from staged scripts on 2026-11-12.
  • Other commits are docs, tests or docker-dev only (#893).

What a ScadaBridge consumer would notice

On DEV and QA: nothing. ScadaBridge on both tiers reads no OtOpcUa data; its only data connection is MxGateway (QA pack §verification, vd03 fea7de0a section). For an OtOpcUa-bound ScadaBridge elsewhere:

  • Galaxy.
    • Pending/Warning rows now read Uncertain, not Good/Bad (#872).
    • A Galaxy write that MXAccess answers Pending is reported not committed. The node reverts and the later commit arrives as a data change.
    • 8001 now reads BadTypeMismatch.
  • Alarm-bearing references (#896): a UNS binding to a reference on an alarm-bearing tag now gets BadNodeIdUnknown. Before, it read BadWaitingForInitialData forever. Bind the alarm as an event subscription.
  • HistoryRead-Processed: TimeAverage/Total on a Boolean node now answers BadAggregateNotSupported (#897). With HG ≥ a220ec0, the same holds for a numeric node backed by a discrete AVEVA tag (#900). TimeAverage/Total are advertised again (QA range).
  • Credential refs (#834): a Galaxy or OpcUaClient driver with an out-of-namespace credential reference no longer connects. It is not a ScadaBridge-visible wire change, but its nodes go Bad.
  • No proto, NodeId-scheme or ServiceLevel-semantics change. Contracts are unchanged.

2. Migrations and settings

  • ConfigDb migrations: none in either range. The only file touched under src/Core/ZB.MOM.WW.OtOpcUa.Configuration is Validation/DraftValidator.cs (#899 HistorianReadOnlyInvalid deploy rule, already on vd03). The repo head stays 20261022102459_EnableSnapshotIsolation (9 migrations). Both DBs are already there (§3).
  • New optional setting: ServerHistorian:ProvisionTags = CreateOrUpdate (default, the previous behaviour) | CreateOnly (Gateway backend only) | Off. An unknown value, or CreateOnly on a SQL backend, fails host start. Neither tier sets it today (vd03 record). Leave it unset: there are 0 historized tags on either tier.
  • Changed default: ServerHistorian:Gateway:EnableNativeTimeWeighted false → true (QA only; vd03 already has it). The kill switch is ServerHistorian__Gateway__EnableNativeTimeWeighted=false. It is not needed: QA HG fa9cf31 supports native time-weighted reads.
  • Telemetry (QA): no new keys. 0.3 reads the same OtOpcUa:Telemetry section and adds optional SiteId/NodeRole/ServiceVersion/ResourceAttributes:* overrides.
  • Required env: no new required variable. The existing required ones stay: Security__ApiKeys__Pepper (≥16 chars) and the JWT key on both tiers, and LocalDb__Replication__ApiKey, identical on both QA nodes.
  • Packages: Grpc.StatusProto 2.76.0 (new). Telemetry(.Serilog) is 0.3.1 (already on vd03). LocalDb 0.3.7, MxGateway.Client 0.4.0 and HistorianGateway.Client 1.0.0 are unchanged.
  • Peer services: none to upgrade first. HG ea9f21c (vd03) and fa9cf31 (QA) serve every RPC this build uses; #900 is capability-tolerant. mxgw 76a71bc8 is unaffected, because neither tier has a Galaxy driver.

3. Read-only preflight (run 2026-11-13 from the Mac, direct SQL)

Run as wwadmin with sqlcmd -C -I, the password passed in SQLCMDPASSWORD, against OtOpcUaConfig on wonder-app-vd03.zmr.zimmer.com,1433 (DEV) and wonder-sql-vq03.zmr.zimmer.com,1433 (QA). The queries were SELECTs only, plus the existing script's local table variables. Two passes:

  1. scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql unchanged (15 checks). It already reproduces DriverCredentialPolicy / CredentialReferenceDeployChecks on the draft rows (checks 6/7).
  2. Ad-hoc SELECTs for what the script does not cover:
    • the deployed side of CredentialReferencePreflight.ScanAsync: the latest Status=2 (Sealed) ArtifactBlob, DECOMPRESSed and searched;
    • #898/#899/#897/#896 over dbo.Tag.TagConfig (isHistorized, historianTagname, historianReadOnly, alarm) and dbo.VirtualTag.Historize.
Check DEV vd03 QA (shared ConfigDb on wonder-sql-vq03)
__EFMigrationsHistory 9 rows, head 20261022102459_EnableSnapshotIsolation = repo head. Snapshot isolation ON, compat 160. Same: 9 rows, same head, snapshot ON, compat 160.
Inventory 1 driver (FOCAS focas-z-34184), 1 device, 22 tags (8 Double, 8 Int32, 6 String), 0 VT, 0 scripted alarms, 2 UNS refs, 10 deployments 1 driver (Calculation script-runner-canary), 1 device, 1 tag (Int32), 1 VT (Int32), 0 scripted alarms, 0 UNS refs, 3 deployments
Latest Sealed deployment 595cb909, rev fa9a2e28, 2026-11-13 06:34Z (gzip, 13,655 B JSON) 5905003e, rev 82dd0598, 2026-11-13 00:05Z (gzip, 4,146 B JSON)
#834 draft: script checks 6 (#815/#836 OutOfScope) and 7 (#835/#849 OtherInstance) ok, 0 rows ok, 0 rows
#834 draft: credential-ruled driver types (Galaxy / OpcUaClient / Heidenhain) and any secret:/env: text in DriverConfig/DeviceConfig none. FOCAS has no credential fields, and neither config holds a reference. none. Calculation has no credential fields; no reference.
#834 deployed: latest Sealed artifact contains "Galaxy" / "OpcUaClient" / "Heidenhain" / secret: / env: 0 / 0 / 0 / 0 / 0 0 / 0 / 0 / 0 / 0
→ #834 verdict No finding. The boot preflight will log "every stored and deployed reference is namespaced". No finding.
#898 historized Float/Double/Byte/SByte tags (first start would EnsureTags) 0 (0 tags with isHistorized: true at all; no historianTagname) 0 (no historized tag; the VT has Historize=0)
#899 tags with a historianReadOnly key / a non-boolean value (HistorianReadOnlyInvalid) 0 / 0 0 / 0
#899 ProvisionTags default Nothing to provision, so the default CreateOrUpdate is harmless. Same.
#897 historized Boolean tags/VTs (TimeAverage/Total now refused up front) 0 (no Boolean tags at all). Aggregates are chosen per client request, not stored in config; no historized Boolean node can receive one. 0
#896 UNS references to alarm-bearing tags 0 (no tag has an alarm object) 0
Other script checks (#783, Calc trigger, #791 ×2, #792, #848, MTConnect, #781 Sql, #803 Heidenhain, UNS collisions) all ok all ok
Script check 0 "Environment" attention: a stale expectation. The script was written for b8b4c18f, and the migration it asks for is already applied. Not an issue. Same.
Script check 13 (FOCAS renames, info) 9 info rows. These are the b8b4c18f→master FOCAS address notes, already live since the 2026-11-12 deploy. ok

Verdict: no blocker on either tier; no issue filed. Re-run both passes just before the window if the configuration has changed since 2026-11-13. On QA the canary is the only configuration.

Pack correction found during preflight: QA records 12 deny paths, not 11

The 2026-11-12 OTel collector install (scadaproj infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md, 21:46–21:47) re-ran the installer on vq03/vq04 and added E:\ApiInstall\OtelCollector (DenyPathRecord + SensitiveDirDeny changed). The QA pack's 11-path list is therefore one short. If the one-time full-argument run (§4) used the 11-path list, it would remove the collector deny. §4 uses the 12-path list. As before, the rule is to use exactly what reg query shows.

The same record also shows that the OtOpcUa Primary failed over to vq04 during the restarts at 21:49–21:50. Read ServiceLevel before the window to know which node is Primary now.

4. Build, stage, one-time isolation record, swap

Build (Mac, isolated worktree at c9640f5c; not done here)

git worktree add ../OtOpcUa-wt-c9640f5c c9640f5c && cd ../OtOpcUa-wt-c9640f5c
dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true -o ../publish-c9640f5c
# expect ~1396+ files, OtOpcUa.Host.dll ProductVersion 1.0.0+c9640f5c…, script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe present;
# deps.json: Telemetry(.Serilog) 0.3.1, LocalDb 0.3.7, MxGateway.Client 0.4.0, HistorianGateway.Client 1.0.0, Grpc.StatusProto 2.76.0
cd ../publish-c9640f5c && zip -qr ../otopcua-c9640f5c.zip . && shasum -a 256 ../otopcua-c9640f5c.zip > ../otopcua-c9640f5c.zip.sha256

Copy the zip and .sha256 over SMB into the staging folders below, and check the hash on the box before expanding. Stage the three install scripts from c9640f5c next to them. Their SHA-256 values at c9640f5c:

File SHA-256
scripts/install/Install-ScriptWorkerIsolation.ps1 7d4078cb93a42b362ab50ba5b79bc293bf92001d5444f58d5c6733e0d51c4914
scripts/install/ScriptWorkerIsolation.psm1 ae9d84fd0077a730ecaaf01b8d476ee83755c2d3fb163e752379c276f227d902
scripts/install/Invoke-OtOpcUaInstallSwap.ps1 9498fd8092c3b2ea95f7c7f8275b704d6bed99dfeea0dcfadfbab1045d79e88d

All remote steps below are on-box deploy steps (winrm skill, deploy scope). Run them in an elevated Windows PowerShell, from the staged install\ folder. Never run Install-Services.ps1, Uninstall-Services.ps1 or Refresh-Services.ps1 (vd03 pack §e step 5).

DEV vd03 (E:\ApiInstall\OtOpcUa, data in E:\ApiInstall\OtOpcUa-data, runner InProcess)

Staging: E:\ApiInstall\_stage-otopcua-c9640f5c\ (zip, .sha256, install\). Backups go to E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-c9640f5c\. Create that folder with a protected DACL (SYSTEM + Administrators only, #891).

  1. Prepare (service up).
    • reg export HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaHost into the backup folder.
    • Check the hash, then expand to E:\ApiInstall\OtOpcUa.new-c9640f5c and check ProductVersion.
    • BACKUP DATABASE [OtOpcUaConfig] TO DISK = N'OtOpcUaConfig-pre-20261113-otopcua-c9640f5c.bak' WITH COPY_ONLY, CHECKSUM. There is no migration, so this is only a configuration fallback.
    • Read the service env (names and count only; expect 28). Check that OtOpcUa__Telemetry__OtlpEndpoint has a scheme.
  2. One-time #901 settings record (service can stay up; vd03 was isolated by the pre-#901 installer, so Invoke-OtOpcUaInstallSwap.ps1 refuses until the record exists):
    reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths
    # expect exactly: E:\ApiInstall, E:\ApiInstall\OtOpcUa\pki, E:\ApiInstall\OtOpcUa-data, C:\ProgramData\ZB\kek, E:\psrem-stage
    .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -HostServiceAccount LocalSystem `
      -DenyPaths E:\ApiInstall,E:\ApiInstall\OtOpcUa\pki,E:\ApiInstall\OtOpcUa-data,C:\ProgramData\ZB\kek,E:\psrem-stage -WhatIf
    .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -HostServiceAccount LocalSystem `
      -DenyPaths E:\ApiInstall,E:\ApiInstall\OtOpcUa\pki,E:\ApiInstall\OtOpcUa-data,C:\ProgramData\ZB\kek,E:\psrem-stage
    .\Install-ScriptWorkerIsolation.ps1 -Verify          # VERIFY OK, exit 0
    reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths   # same 5 paths
    
    • No -ConfigureHostService: the vd03 runner stays InProcess (vd03 pack §"Worker isolation and box-wide denies").
    • Expect SettingsRecord changed, DenyPathRecord unchanged, and every other step unchanged. The reisolate.ps1 run after the 6c4d2990 swap restored them.
    • The "protected DACL" WARNs under E:\psrem-stage and the locked backups are pre-existing and expected.
    • If reg query lists anything other than the 5 paths, stop and use the recorded list.
  3. Swap.
    • Stop-Service OtOpcUaHost.
    • robocopy E:\ApiInstall\OtOpcUa-data <backup>\OtOpcUa-data /E /COPY:DAT (stopped, so the SQLite triplets are consistent).
    • Then run:
    .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.new-c9640f5c `
        -PreviousInstall E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c
    
    • The default -Carry (appsettings*.json, pki, logs) is right for vd03.
    • The script tolerates an already-stopped service. It moves the folders, carries the box files, runs -Reapply (expect WorkerFolderAcl, InstallRootDeny and the pki SensitiveDirDeny changed), then -Verify, then starts.
    • Gate: the script must finish without throwing. A failed -Verify leaves the service stopped: fix the problem, re-run -Reapply/-Verify, then start.
    • After the start, run .\Install-ScriptWorkerIsolation.ps1 -Verify once more and record its output (exit 0).

QA vq03 + vq04 (stop both, swap both, start both)

Layout: E:\ApiInstall\OtOpcUa\{app,data,pki}, service OtOpcUaHost (LocalSystem), runner OutOfProcess (Calculation + VirtualTag). Staging: E:\psrem-stage\otopcua-c9640f5c\ (zip, .sha256, install\). Backups go to E:\psrem-stage\otopcua-c9640f5c\backup\, created with a protected DACL (#891).

  1. Prepare (both nodes, services up).
    • Expand to E:\ApiInstall\OtOpcUa\app.new-c9640f5c (check the hash and ProductVersion).
    • reg export …\OtOpcUaHost backup\OtOpcUaHost-pre-c9640f5c.reg.
    • Read the env names and count: expect 48 on vq03 and 47 on vq04, including Scripting__Runner*, Scripting__Worker__*, the pepper, LocalDb__Replication__ApiKey and OtOpcUa__Telemetry__*. Check that OtlpEndpoint has a scheme.
    • Once: BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION on wonder-sql-vq03 → OtOpcUaConfig_pre-c9640f5c_20261113.bak, then RESTORE VERIFYONLY.
    • Read ServiceLevel on both nodes to learn which one is Primary (250). It may be vq04 since the 2026-11-12 OTel restarts.
  2. One-time #901 settings record (both nodes, services up). QA was isolated by the pre-#901 installer (DenyPaths only):
    reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths
    # expect these 12 (the 11 #891 paths + the OTel collector folder added 2026-11-12):
    $deny = 'E:\ApiInstall\OtOpcUa','E:\ApiInstall\OtOpcUa\pki','E:\ApiInstall\OtOpcUa\data','C:\ProgramData\ZB\kek',
            'E:\ApiInstall\ScadaBridge','E:\ApiInstall\HistorianGateway','E:\ApiInstall\MxAccessGateway','E:\ApiInstall\Overview',
            'E:\ApiInstall\btdbmanager','E:\psrem-stage','E:\psrem','E:\ApiInstall\OtelCollector'
    .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny -WhatIf
    .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny
    .\Install-ScriptWorkerIsolation.ps1 -Verify          # VERIFY OK, exit 0
    reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths   # same 12 paths
    
    • If reg query differs from the 12 above, use exactly the recorded list. A missing entry in $deny removes that deny.
    • -ConfigureHostService stays, because QA's runner flip used it.
    • Expect SettingsRecord changed and DenyPathRecord unchanged.
    • HostServiceEnv may report changed as a reorder only, as in the #890 run, if the OTel entries were appended after the Scripting__Worker__* pair. Confirm the env count is unchanged (48/47) and that no value changed (compare SHA-256 prefixes, never print values). Every other step should be unchanged.
    • The vq04 kek\historiangw.kek/mxgateway.kek and locked .reg protected-DACL WARNs are expected.
  3. Stop both. Stop the Secondary (240) first, then the Primary (250). On each node, while stopped, run robocopy data backup\data-pre-c9640f5c /E /COPY:DAT and the same for pki.
  4. Swap each node (services stopped):
    .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.new-c9640f5c `
        -PreviousInstall E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-c9640f5c -Carry 'appsettings*.json','logs' -NoStart
    
    • pki\ and data\ sit beside app\ and are not moved.
    • Expect the re-apply to report WorkerFolderAcl and InstallRootDeny changed, then -Verify OK.
    • Gate: the script returns without throwing on both nodes. If -Verify fails on either node, do not start either; fix it, then run -Reapply and -Verify.
  5. Start both. Start the node that was Primary first; it waits for its peer, about 40 s last time. Then start the other node.
  6. On each node, after ready: .\Install-ScriptWorkerIsolation.ps1 -Verify → VERIFY OK, exit 0. Record the output.

5. Post-deploy gates

Run these from the Mac over HTTP / opc.tcp. Read logs over SMB (logs\otopcua-YYYYMMDD.log). The Windows-listed size of an open log reads 0; read its contents.

Gate DEV vd03 QA vq03 + vq04
Isolation -Verify exit 0 after the swap and after the start (§4) same, both nodes
Version live OtOpcUa.Host.dll (SMB copy) ProductVersion 1.0.0+c9640f5c…; SHA-256 equals the published build same, both nodes
Health /healthz, /health/ready, /health/active all 200 on :9000 /health/ready 200 on both, with 5/5 components including Out-of-process script runner ready.; /health/active 200 on the Primary, 503 on the Secondary
ServiceLevel 250 250 / 240, stable over repeated fresh-session reads
Failover (QA) — Stop the Primary: the other node reaches 250 and /health/active 200. Restart the old Primary: 240, never above, no failback.
LocalDb sync (QA) — localdb-replication connected, oplog backlog 0 on both, before and after the failover
Canary (QA) — ns=2;s=script-runner-canary/Device1/canary-calc-seconds and ns=3;s=EQ-ec3979550d90/canary-vt-second are Good and advancing about once a second on both endpoints. otopcua_script_worker_up = 1 per shard. The #780 runaway drill (while(true) VT, expect 2 kills + quarantine, siblings Good, 250/240 unchanged, then delete and redeploy back to rev 82dd0598…) is optional.
Driver data FOCAS raw: 22 variables, 21 Good, Program/BlockCount BadNoData (as before) the canary above
Boot log roles=[admin,driver]; Credential-reference preflight (lmxopcua#815/#834): every stored and deployed reference is namespaced. (the new wording; any Error line from this preflight is a stop); Historian gateway probe: the gateway is reachable; no FTL same, both nodes; also no Telemetry endpoint FATAL (0.3 check)
#903 drift chip On /deployments (login disabled on DEV): the first render shows "Checking for changes...", then "In sync". Make a harmless draft change, for example a tag description on /raw, with no deploy. Reload /deployments: it must show "Configuration drift" at once. Revert the change: the chip returns to "In sync", and the revision is still fa9a2e28…. Optional. The QA AdminUI needs the localhost forwarder plus a temporary role-mapping row (QA pack §2). If you do it, make the change on the canary and revert it. Delete the temporary row afterwards.
#902 Monaco Open /scripts/new (the Monaco editor) and close the browser tab without saving. Then search the log after that time for MonacoEditor / Unhandled exception in circuit: expect no [ERR] (it is Debug now). Same if the AdminUI is opened; otherwise n/a
Historian writes HG http://wonder-app-vd03…:5220/metrics historian_gateway_grpc_requests_total: snapshot before the stop and about 2 min after ready. Only HistorianStatus/Probe should grow. No EnsureTags, AddTagExtendedProperties, WriteLiveValues, AddHistoricalValues, SendEvent(s). Same on vq03's HG :5220/metrics; both QA nodes point at 10.220.158.2:5221
OTLP logs (QA) — (vd03 already sends logs) New: the collector's file exporter on each box starts receiving OtOpcUa log records. Check over SMB, read-only. Not a gate.
Overview :5320 shows OtOpcUa Up vq03/vq04 :5320: OtOpcUa Up, 2 members / 0 unreachable, 5/5 checks
ScadaBridge /health/ready 200 (no OtOpcUa link) /health/ready 200 on both (no OtOpcUa link)

6. Rollback

There is no schema undo in either range: no migration. The ConfigDb .bak exists only for a configuration restore.

DEV vd03 → 6c4d2990:

.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c `
    -PreviousInstall E:\ApiInstall\OtOpcUa.failed-c9640f5c -NoCarry -NoStart
  • The script stops the service, swaps the complete previous folder back (its own appsettings*, pki, logs), then runs -Reapply + -Verify. The service stays stopped.
  • Then mirror <backup>\OtOpcUa-data back over E:\ApiInstall\OtOpcUa-data and Start-Service OtOpcUaHost.
  • Run -Verify again, then check health and ServiceLevel 250.
  • The registry is unchanged by this deploy, so no import is needed.
  • Fallback: E:\ApiInstall\_stage-otopcua-6c4d2990\rollback.ps1 is one generation older and targets fea7de0a. Do not use it for this rollback.

QA → fea7de0a: stop both, then on each node:

.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-c9640f5c `
    -PreviousInstall E:\ApiInstall\OtOpcUa\app.failed-c9640f5c -NoCarry -NoStart
  • After the swap, mirror backup\data-pre-c9640f5c back over data\.
  • Import backup\OtOpcUaHost-pre-c9640f5c.reg only if step 2's -ConfigureHostService changed the env. A reorder is harmless.
  • Run -Verify on each node (exit 0), then start both, the old Primary first.
  • The isolation record and denies stay valid for fea7de0a. The installer works independently of the host build.
  • DB fallback: OtOpcUaConfig_pre-c9640f5c_20261113.bak.
  • SDK tooling caveat: not applicable. Both builds are on SDK 2.0.1.

Isolation step only (§4 step 2): the settings record is additive. Re-running the installer with the same arguments is idempotent. -Uninstall removes everything, including the recorded denies; do not run it as part of a binary rollback.

7. Downtime estimates (from the previous records)

Tier Previous Estimate for this pack
vd03 fea7de0a about 10 s (stop 2 s); 6c4d2990 about 15 s (stop 6 s), stop → /health/ready about 20–40 s. The data robocopy and the in-script -Reapply/-Verify now run while the service is stopped. The step-2 record write happens with the service up.
QA pair fea7de0a window about 2 min pair outage (16:48:13 → 16:50:14), of which about 70 s was the isolation install + uninstall drill. The first node took 39 s to ready because it waits for its peer; the second took 8 s. about 1–1.5 min pair outage: stops about 1–2 s each, backup + swap + -Reapply/-Verify per node, then about 40 s + 8 s to ready. One-node restarts (the OTel record) took up to 32 s to stop a Secondary with its peer up. Stopping both together avoids that graceful-leave wait.
QA failover gate 30 s graceful leave, then the new Primary at 250 on its first sample same

After lmxopcua#905 (for the next pack that uses Invoke-OtOpcUaInstallSwap.ps1). The 101 s (vd03) and 50–60 s per QA node were the protected-DACL coverage scan (Get-Acl per item over every denied folder, in both -Reapply and -Verify, while stopped), not the ACL writes. The swap now runs -Reapply/-Verify -Scope Outside (the full scan included) before the stop and only -Scope InstallRoot inside the window. windev, a vd03-like tree of 17,851 items under the denied folder: stopped window 153.1 s → 6.0 s (Windows PowerShell 5.1) and 101.6 s → 6.2 s (PowerShell 7); the pre-stop part takes about 23 s with the old build still serving. Estimate for the next vd03 swap: stop (2–6 s) + the data\ backup + about 6 s + start-up to /health/ready (about 8–10 s), i.e. about 20–30 s. QA: about 10 s of swap per node inside the pair outage instead of 50–60 s, and the Secondary-first order (deviation 3) is no longer needed to hide it. The script prints stopped window (…): N s [stop, move, carry, isolation]; record that line.