Root cause: the protected-DACL coverage scan, not the ACL writes. Both -Reapply and -Verify walked every denied folder (all of E:\ApiInstall on vd03, nested roots again) with Get-ChildItem -Recurse + Get-Acl per item + a pipeline per item for -Skip, about 3.4 ms an item on PS 5.1, all while the service was stopped. An unchanged deny is read, not rewritten; the re-applied install-folder deny propagates over ~1,400 files in 0.5 s. - Find-OtOpcUaDenyCoverageGap: .NET enumeration, -Skip folders pruned, DACL-only GetAccessControl(Access) (about 0.15 ms an item); reparse points not followed. - Get-OtOpcUaCoverageWalkRoots: each denied folder walked once (outermost only). - Install-ScriptWorkerIsolation.ps1 -Scope All|InstallRoot|Outside (with -Reapply, -Verify, -PlanOnly). InstallRoot = the folder ACEs a swap drops; Outside = the rest, including the firewall rule (its query costs 3-5 s; its exe path does not change). The two partition the plan. Prints the scan time. - Invoke-OtOpcUaInstallSwap.ps1: -Reapply/-Verify -Scope Outside before the stop (a failure stops before anything moves), -Scope InstallRoot inside the window; preflight checks the candidate carries the worker exe; prints the stopped window and its parts. -NoStart, -NoCarry and the pre-#901 refusal unchanged. windev, vd03-like tree (17,851 items): stopped window 153.1 s -> 6.0 s (PS 5.1), 101.6 s -> 6.2 s (PS 7); the Outside part takes ~23 s with the old build serving. Dry-run harness 101 -> 123 passed; Windows test 72 passed on PS 5.1 and PS 7. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
30 KiB
Next deploy: DEV vd03 6c4d2990 and QA vq03/vq04 fea7de0a → master c9640f5c
EXECUTED 2026-11-13 (DEV vd03 07:33Z, QA pair 07:47Z): record
2026-11-13-vd03-qa-to-c9640f5c.md. Deviations there: longer downtime from the in-swap-Verifycoverage walk; the §5 #903 revert cannot return the chip to "In sync" (RowVersionis in the revision hash); QA Secondary swapped before the Primary stopped.Status when written: PREPARED, NOT EXECUTED (2026-11-13). This pack covers both tiers, plus the read-only ConfigDb preflight, which was run over direct SQL from the Mac on 2026-11-13. Nothing was built, staged or run on any wonder box. No blocker found; no issue filed.
Target:
origin/masterc9640f5c(c9640f5ced383e2a722702409adbc62298731994, #904). If master moves before the window, either pin toc9640f5c(as the6c4d2990run pinned) or re-run §1–§3 for the new range.Templates:
2026-11-13-vd03-fea7de0a-to-6c4d2990.md(vd03 layout and scripts),qa-upgrade-b8b4c18f-to-fea7de0a.md(QA window, canary, #890/#891 denies, "Worker isolation across the next swap"),vd03-upgrade-b8b4c18f-to-master.md§e/§f and "Worker isolation and box-wide denies".
1. Commit ranges and what changes
vd03: 6c4d2990..c9640f5c (9 commits)
| Commit | Change | Effect on vd03 (one fused node, one FOCAS driver, 22 tags, runner InProcess) |
|---|---|---|
efaf3435 #900 |
HistorianGateway FailedPrecondition + AGGREGATE_NOT_SUPPORTED_FOR_TAG maps to BadAggregateNotSupported, with no health failure. Adds Grpc.StatusProto 2.76.0. |
Inert today. vd03's HG ea9f21c predates historiangw#135 (a220ec0) and still answers Internal, which keeps the current handling. It takes effect when HG ≥ a220ec0 is deployed. |
9375c9e2 |
docs (the 6c4d2990 record) |
— |
d5288388 #895 review, c9640f5c #904 |
Galaxy alarm reverse map: re-areaed, subtag-provider and pre-subscription references. A picker-form ack waits up to 2 s for the feed snapshot. | Not reachable (no Galaxy driver). |
2470f82e #872 |
Galaxy status rows Pending/Warning → Uncertain; detail 8001 → BadTypeMismatch. A Pending write reply is now "not committed". |
Not reachable. |
b85e004b #902 |
AdminUI: MonacoEditor.DisposeAsync on a gone circuit logs at Debug, not [ERR]. |
Fixes the "Observed" [ERR] from the 6c4d2990 record. |
35576485 #903 |
AdminUI /deployments: the drift chip is never a cached "In sync". The first render shows "Checking for changes...". A failed check shows "Drift check failed". |
Fixes the stale "In sync" chip from the 6c4d2990 record. |
2cb8a2c3 #901 |
Installer records its settings and gains -Reapply/-Verify. New scripts/install/Invoke-OtOpcUaInstallSwap.ps1. |
Changes the deploy procedure (§4). No runtime change. |
f0c6cf96 #834 |
Running Galaxy/OpcUaClient drivers refuse out-of-namespace credential refs at resolve time (the warn-only path is gone). The preflight logs one Error per finding, and the /raw banner is alert-danger. |
FOCAS has no credential fields. Preflight: 0 findings (§3). Only the boot line wording changes (§5). |
QA: fea7de0a..c9640f5c (46 commits: the 37 above 6c4d2990 plus the 9 vd03 commits)
The vd03 rows above, plus everything vd03 already got on 2026-11-13 (fea7de0a..6c4d2990, 37 commits):
- Historian.
ServerHistorian:Gateway:EnableNativeTimeWeightednow defaults totrue(#694,53978ca6). TimeAverage/Total are advertised and served natively (12 aggregates). QA HGfa9cf31includes9f711c1, so it supports this.- Alarm history goes out in
SendEventsbatches of ≤200, with a per-event fallback (#746). - A Boolean node refuses TimeAverage/Total up front with
BadAggregateNotSupported(#897). A String/DateTime node refuses the numeric-only aggregates. historianReadOnly: trueTagConfig plus the newServerHistorian:ProvisionTagssetting (#899).- Float/Double/Byte/SByte historized tags are now provisioned (#898). See the upgrade hazard in §3.
- Address space (#896): a UNS reference to an alarm-bearing raw tag no longer gets a UNS value Variable. The alarm surfaces only as the condition on the equipment folder.
- Galaxy (#895): picker-authored native alarms resolve and ack. Galaxy spans and
galaxy.events.*counters are exported (#892). - Telemetry 0.1.0 → 0.3.1 (scadaproj#55/#56).
- The OTel Resource carries
service.version(the git SHA),site.id(the Cluster id) andnode.role. - With
Exporter=Otlp, Serilog logs go over OTLP too. QA hasOtOpcUa__Telemetry__Exporter=OtlpandOtlpEndpoint=http://127.0.0.1:4317since 2026-11-12, so logs will start to reach the localZbOtelCollector. - 0.3 refuses a scheme-less
host:portwhenExporter=Otlp. The recorded value hashttp://, so it passes; re-check it in the on-box preflight.
- The OTel Resource carries
- AdminUI #761 (
fa8c3d1a): a bodylessPOST /api/deploymentsreturned 404 on fused admin+driver nodes. It now deploys. - Drivers.
- AB Legacy MicroLogix function files are refused locally with
BadNotSupported, not reported as a comms fault (#592). - Heidenhain gains the
Controller/Family,Status/*StateNameandTool/AxisNameleaves, PLCINPUT_DWORD/OUTPUT_DWORD, and new refusal-code mappings (#829/#853). Not reachable on QA.
- AB Legacy MicroLogix function files are refused locally with
- Installer #890 (
f75ee7d0):-DenyPaths. Already applied on QA from staged scripts on 2026-11-12. - Other commits are docs, tests or docker-dev only (#893).
What a ScadaBridge consumer would notice
On DEV and QA: nothing. ScadaBridge on both tiers reads no OtOpcUa data; its only data connection is MxGateway
(QA pack §verification, vd03 fea7de0a section). For an OtOpcUa-bound ScadaBridge elsewhere:
- Galaxy.
- Pending/Warning rows now read Uncertain, not Good/Bad (#872).
- A Galaxy write that MXAccess answers Pending is reported not committed. The node reverts and the later commit arrives as a data change.
- 8001 now reads
BadTypeMismatch.
- Alarm-bearing references (#896): a UNS binding to a reference on an alarm-bearing tag now gets
BadNodeIdUnknown. Before, it readBadWaitingForInitialDataforever. Bind the alarm as an event subscription. - HistoryRead-Processed: TimeAverage/Total on a Boolean node now answers
BadAggregateNotSupported(#897). With HG ≥a220ec0, the same holds for a numeric node backed by a discrete AVEVA tag (#900). TimeAverage/Total are advertised again (QA range). - Credential refs (#834): a Galaxy or OpcUaClient driver with an out-of-namespace credential reference no longer connects. It is not a ScadaBridge-visible wire change, but its nodes go Bad.
- No proto, NodeId-scheme or ServiceLevel-semantics change. Contracts are unchanged.
2. Migrations and settings
- ConfigDb migrations: none in either range. The only file touched under
src/Core/ZB.MOM.WW.OtOpcUa.ConfigurationisValidation/DraftValidator.cs(#899HistorianReadOnlyInvaliddeploy rule, already on vd03). The repo head stays20261022102459_EnableSnapshotIsolation(9 migrations). Both DBs are already there (§3). - New optional setting:
ServerHistorian:ProvisionTags=CreateOrUpdate(default, the previous behaviour) |CreateOnly(Gateway backend only) |Off. An unknown value, orCreateOnlyon a SQL backend, fails host start. Neither tier sets it today (vd03 record). Leave it unset: there are 0 historized tags on either tier. - Changed default:
ServerHistorian:Gateway:EnableNativeTimeWeightedfalse→true(QA only; vd03 already has it). The kill switch isServerHistorian__Gateway__EnableNativeTimeWeighted=false. It is not needed: QA HGfa9cf31supports native time-weighted reads. - Telemetry (QA): no new keys. 0.3 reads the same
OtOpcUa:Telemetrysection and adds optionalSiteId/NodeRole/ServiceVersion/ResourceAttributes:*overrides. - Required env: no new required variable. The existing required ones stay:
Security__ApiKeys__Pepper(≥16 chars) and the JWT key on both tiers, andLocalDb__Replication__ApiKey, identical on both QA nodes. - Packages:
Grpc.StatusProto 2.76.0(new). Telemetry(.Serilog) is 0.3.1 (already on vd03). LocalDb 0.3.7, MxGateway.Client 0.4.0 and HistorianGateway.Client 1.0.0 are unchanged. - Peer services: none to upgrade first. HG
ea9f21c(vd03) andfa9cf31(QA) serve every RPC this build uses; #900 is capability-tolerant. mxgw76a71bc8is unaffected, because neither tier has a Galaxy driver.
3. Read-only preflight (run 2026-11-13 from the Mac, direct SQL)
Run as wwadmin with sqlcmd -C -I, the password passed in SQLCMDPASSWORD, against OtOpcUaConfig on
wonder-app-vd03.zmr.zimmer.com,1433 (DEV) and wonder-sql-vq03.zmr.zimmer.com,1433 (QA). The queries were SELECTs only,
plus the existing script's local table variables. Two passes:
scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sqlunchanged (15 checks). It already reproducesDriverCredentialPolicy/CredentialReferenceDeployCheckson the draft rows (checks 6/7).- Ad-hoc SELECTs for what the script does not cover:
- the deployed side of
CredentialReferencePreflight.ScanAsync: the latestStatus=2(Sealed)ArtifactBlob,DECOMPRESSed and searched; - #898/#899/#897/#896 over
dbo.Tag.TagConfig(isHistorized,historianTagname,historianReadOnly,alarm) anddbo.VirtualTag.Historize.
- the deployed side of
| Check | DEV vd03 | QA (shared ConfigDb on wonder-sql-vq03) |
|---|---|---|
__EFMigrationsHistory |
9 rows, head 20261022102459_EnableSnapshotIsolation = repo head. Snapshot isolation ON, compat 160. |
Same: 9 rows, same head, snapshot ON, compat 160. |
| Inventory | 1 driver (FOCAS focas-z-34184), 1 device, 22 tags (8 Double, 8 Int32, 6 String), 0 VT, 0 scripted alarms, 2 UNS refs, 10 deployments |
1 driver (Calculation script-runner-canary), 1 device, 1 tag (Int32), 1 VT (Int32), 0 scripted alarms, 0 UNS refs, 3 deployments |
| Latest Sealed deployment | 595cb909, rev fa9a2e28, 2026-11-13 06:34Z (gzip, 13,655 B JSON) |
5905003e, rev 82dd0598, 2026-11-13 00:05Z (gzip, 4,146 B JSON) |
| #834 draft: script checks 6 (#815/#836 OutOfScope) and 7 (#835/#849 OtherInstance) | ok, 0 rows | ok, 0 rows |
#834 draft: credential-ruled driver types (Galaxy / OpcUaClient / Heidenhain) and any secret:/env: text in DriverConfig/DeviceConfig |
none. FOCAS has no credential fields, and neither config holds a reference. | none. Calculation has no credential fields; no reference. |
#834 deployed: latest Sealed artifact contains "Galaxy" / "OpcUaClient" / "Heidenhain" / secret: / env: |
0 / 0 / 0 / 0 / 0 | 0 / 0 / 0 / 0 / 0 |
| → #834 verdict | No finding. The boot preflight will log "every stored and deployed reference is namespaced". | No finding. |
#898 historized Float/Double/Byte/SByte tags (first start would EnsureTags) |
0 (0 tags with isHistorized: true at all; no historianTagname) |
0 (no historized tag; the VT has Historize=0) |
#899 tags with a historianReadOnly key / a non-boolean value (HistorianReadOnlyInvalid) |
0 / 0 | 0 / 0 |
#899 ProvisionTags default |
Nothing to provision, so the default CreateOrUpdate is harmless. |
Same. |
| #897 historized Boolean tags/VTs (TimeAverage/Total now refused up front) | 0 (no Boolean tags at all). Aggregates are chosen per client request, not stored in config; no historized Boolean node can receive one. | 0 |
| #896 UNS references to alarm-bearing tags | 0 (no tag has an alarm object) |
0 |
| Other script checks (#783, Calc trigger, #791 ×2, #792, #848, MTConnect, #781 Sql, #803 Heidenhain, UNS collisions) | all ok | all ok |
| Script check 0 "Environment" | attention: a stale expectation. The script was written for b8b4c18f, and the migration it asks for is already applied. Not an issue. |
Same. |
| Script check 13 (FOCAS renames, info) | 9 info rows. These are the b8b4c18f→master FOCAS address notes, already live since the 2026-11-12 deploy. | ok |
Verdict: no blocker on either tier; no issue filed. Re-run both passes just before the window if the configuration has changed since 2026-11-13. On QA the canary is the only configuration.
Pack correction found during preflight: QA records 12 deny paths, not 11
The 2026-11-12 OTel collector install (scadaproj infra/otel-collector/wonder/2026-11-12-wonder-otel-collector.md,
21:46–21:47) re-ran the installer on vq03/vq04 and added E:\ApiInstall\OtelCollector (DenyPathRecord +
SensitiveDirDeny changed). The QA pack's 11-path list is therefore one short. If the one-time full-argument run
(§4) used the 11-path list, it would remove the collector deny. §4 uses the 12-path list. As before, the rule is to
use exactly what reg query shows.
The same record also shows that the OtOpcUa Primary failed over to vq04 during the restarts at 21:49–21:50. Read ServiceLevel before the window to know which node is Primary now.
4. Build, stage, one-time isolation record, swap
Build (Mac, isolated worktree at c9640f5c; not done here)
git worktree add ../OtOpcUa-wt-c9640f5c c9640f5c && cd ../OtOpcUa-wt-c9640f5c
dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true -o ../publish-c9640f5c
# expect ~1396+ files, OtOpcUa.Host.dll ProductVersion 1.0.0+c9640f5c…, script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe present;
# deps.json: Telemetry(.Serilog) 0.3.1, LocalDb 0.3.7, MxGateway.Client 0.4.0, HistorianGateway.Client 1.0.0, Grpc.StatusProto 2.76.0
cd ../publish-c9640f5c && zip -qr ../otopcua-c9640f5c.zip . && shasum -a 256 ../otopcua-c9640f5c.zip > ../otopcua-c9640f5c.zip.sha256
Copy the zip and .sha256 over SMB into the staging folders below, and check the hash on the box before expanding.
Stage the three install scripts from c9640f5c next to them. Their SHA-256 values at c9640f5c:
| File | SHA-256 |
|---|---|
scripts/install/Install-ScriptWorkerIsolation.ps1 |
7d4078cb93a42b362ab50ba5b79bc293bf92001d5444f58d5c6733e0d51c4914 |
scripts/install/ScriptWorkerIsolation.psm1 |
ae9d84fd0077a730ecaaf01b8d476ee83755c2d3fb163e752379c276f227d902 |
scripts/install/Invoke-OtOpcUaInstallSwap.ps1 |
9498fd8092c3b2ea95f7c7f8275b704d6bed99dfeea0dcfadfbab1045d79e88d |
All remote steps below are on-box deploy steps (winrm skill, deploy scope). Run them in an elevated Windows
PowerShell, from the staged install\ folder. Never run Install-Services.ps1, Uninstall-Services.ps1 or
Refresh-Services.ps1 (vd03 pack §e step 5).
DEV vd03 (E:\ApiInstall\OtOpcUa, data in E:\ApiInstall\OtOpcUa-data, runner InProcess)
Staging: E:\ApiInstall\_stage-otopcua-c9640f5c\ (zip, .sha256, install\). Backups go to
E:\ApiInstall\OtOpcUa-backups\pre-20261113-otopcua-c9640f5c\. Create that folder with a protected DACL (SYSTEM +
Administrators only, #891).
- Prepare (service up).
reg export HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaHostinto the backup folder.- Check the hash, then expand to
E:\ApiInstall\OtOpcUa.new-c9640f5cand checkProductVersion. BACKUP DATABASE [OtOpcUaConfig] TO DISK = N'OtOpcUaConfig-pre-20261113-otopcua-c9640f5c.bak' WITH COPY_ONLY, CHECKSUM. There is no migration, so this is only a configuration fallback.- Read the service env (names and count only; expect 28). Check that
OtOpcUa__Telemetry__OtlpEndpointhas a scheme.
- One-time #901 settings record (service can stay up; vd03 was isolated by the pre-#901 installer, so
Invoke-OtOpcUaInstallSwap.ps1refuses until the record exists):reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths # expect exactly: E:\ApiInstall, E:\ApiInstall\OtOpcUa\pki, E:\ApiInstall\OtOpcUa-data, C:\ProgramData\ZB\kek, E:\psrem-stage .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -HostServiceAccount LocalSystem ` -DenyPaths E:\ApiInstall,E:\ApiInstall\OtOpcUa\pki,E:\ApiInstall\OtOpcUa-data,C:\ProgramData\ZB\kek,E:\psrem-stage -WhatIf .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -HostServiceAccount LocalSystem ` -DenyPaths E:\ApiInstall,E:\ApiInstall\OtOpcUa\pki,E:\ApiInstall\OtOpcUa-data,C:\ProgramData\ZB\kek,E:\psrem-stage .\Install-ScriptWorkerIsolation.ps1 -Verify # VERIFY OK, exit 0 reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths # same 5 paths- No
-ConfigureHostService: the vd03 runner staysInProcess(vd03 pack §"Worker isolation and box-wide denies"). - Expect
SettingsRecordchanged,DenyPathRecordunchanged, and every other step unchanged. Thereisolate.ps1run after the6c4d2990swap restored them. - The "protected DACL" WARNs under
E:\psrem-stageand the locked backups are pre-existing and expected. - If
reg querylists anything other than the 5 paths, stop and use the recorded list.
- No
- Swap.
Stop-Service OtOpcUaHost.robocopy E:\ApiInstall\OtOpcUa-data <backup>\OtOpcUa-data /E /COPY:DAT(stopped, so the SQLite triplets are consistent).- Then run:
.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.new-c9640f5c ` -PreviousInstall E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c- The default
-Carry(appsettings*.json,pki,logs) is right for vd03. - The script tolerates an already-stopped service. It moves the folders, carries the box files, runs
-Reapply(expectWorkerFolderAcl,InstallRootDenyand thepkiSensitiveDirDenychanged), then-Verify, then starts. - Gate: the script must finish without throwing. A failed
-Verifyleaves the service stopped: fix the problem, re-run-Reapply/-Verify, then start. - After the start, run
.\Install-ScriptWorkerIsolation.ps1 -Verifyonce more and record its output (exit 0).
QA vq03 + vq04 (stop both, swap both, start both)
Layout: E:\ApiInstall\OtOpcUa\{app,data,pki}, service OtOpcUaHost (LocalSystem), runner OutOfProcess (Calculation +
VirtualTag). Staging: E:\psrem-stage\otopcua-c9640f5c\ (zip, .sha256, install\). Backups go to
E:\psrem-stage\otopcua-c9640f5c\backup\, created with a protected DACL (#891).
- Prepare (both nodes, services up).
- Expand to
E:\ApiInstall\OtOpcUa\app.new-c9640f5c(check the hash and ProductVersion). reg export …\OtOpcUaHost backup\OtOpcUaHost-pre-c9640f5c.reg.- Read the env names and count: expect 48 on vq03 and 47 on vq04, including
Scripting__Runner*,Scripting__Worker__*, the pepper,LocalDb__Replication__ApiKeyandOtOpcUa__Telemetry__*. Check thatOtlpEndpointhas a scheme. - Once:
BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSIONon wonder-sql-vq03 →OtOpcUaConfig_pre-c9640f5c_20261113.bak, thenRESTORE VERIFYONLY. - Read ServiceLevel on both nodes to learn which one is Primary (250). It may be vq04 since the 2026-11-12 OTel restarts.
- Expand to
- One-time #901 settings record (both nodes, services up). QA was isolated by the pre-#901 installer
(DenyPaths only):
reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths # expect these 12 (the 11 #891 paths + the OTel collector folder added 2026-11-12): $deny = 'E:\ApiInstall\OtOpcUa','E:\ApiInstall\OtOpcUa\pki','E:\ApiInstall\OtOpcUa\data','C:\ProgramData\ZB\kek', 'E:\ApiInstall\ScadaBridge','E:\ApiInstall\HistorianGateway','E:\ApiInstall\MxAccessGateway','E:\ApiInstall\Overview', 'E:\ApiInstall\btdbmanager','E:\psrem-stage','E:\psrem','E:\ApiInstall\OtelCollector' .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny -WhatIf .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny .\Install-ScriptWorkerIsolation.ps1 -Verify # VERIFY OK, exit 0 reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths # same 12 paths- If
reg querydiffers from the 12 above, use exactly the recorded list. A missing entry in$denyremoves that deny. -ConfigureHostServicestays, because QA's runner flip used it.- Expect
SettingsRecordchanged andDenyPathRecordunchanged. HostServiceEnvmay report changed as a reorder only, as in the #890 run, if the OTel entries were appended after theScripting__Worker__*pair. Confirm the env count is unchanged (48/47) and that no value changed (compare SHA-256 prefixes, never print values). Every other step should be unchanged.- The vq04
kek\historiangw.kek/mxgateway.kekand locked.regprotected-DACL WARNs are expected.
- If
- Stop both. Stop the Secondary (240) first, then the Primary (250). On each node, while stopped, run
robocopy data backup\data-pre-c9640f5c /E /COPY:DATand the same forpki. - Swap each node (services stopped):
.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.new-c9640f5c ` -PreviousInstall E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-c9640f5c -Carry 'appsettings*.json','logs' -NoStartpki\anddata\sit besideapp\and are not moved.- Expect the re-apply to report
WorkerFolderAclandInstallRootDenychanged, then-VerifyOK. - Gate: the script returns without throwing on both nodes. If
-Verifyfails on either node, do not start either; fix it, then run-Reapplyand-Verify.
- Start both. Start the node that was Primary first; it waits for its peer, about 40 s last time. Then start the other node.
- On each node, after ready:
.\Install-ScriptWorkerIsolation.ps1 -Verify→VERIFY OK, exit 0. Record the output.
5. Post-deploy gates
Run these from the Mac over HTTP / opc.tcp. Read logs over SMB (logs\otopcua-YYYYMMDD.log). The Windows-listed size of
an open log reads 0; read its contents.
| Gate | DEV vd03 | QA vq03 + vq04 |
|---|---|---|
| Isolation | -Verify exit 0 after the swap and after the start (§4) |
same, both nodes |
| Version | live OtOpcUa.Host.dll (SMB copy) ProductVersion 1.0.0+c9640f5c…; SHA-256 equals the published build |
same, both nodes |
| Health | /healthz, /health/ready, /health/active all 200 on :9000 |
/health/ready 200 on both, with 5/5 components including Out-of-process script runner ready.; /health/active 200 on the Primary, 503 on the Secondary |
| ServiceLevel | 250 | 250 / 240, stable over repeated fresh-session reads |
| Failover (QA) | — | Stop the Primary: the other node reaches 250 and /health/active 200. Restart the old Primary: 240, never above, no failback. |
| LocalDb sync (QA) | — | localdb-replication connected, oplog backlog 0 on both, before and after the failover |
| Canary (QA) | — | ns=2;s=script-runner-canary/Device1/canary-calc-seconds and ns=3;s=EQ-ec3979550d90/canary-vt-second are Good and advancing about once a second on both endpoints. otopcua_script_worker_up = 1 per shard. The #780 runaway drill (while(true) VT, expect 2 kills + quarantine, siblings Good, 250/240 unchanged, then delete and redeploy back to rev 82dd0598…) is optional. |
| Driver data | FOCAS raw: 22 variables, 21 Good, Program/BlockCount BadNoData (as before) |
the canary above |
| Boot log | roles=[admin,driver]; Credential-reference preflight (lmxopcua#815/#834): every stored and deployed reference is namespaced. (the new wording; any Error line from this preflight is a stop); Historian gateway probe: the gateway is reachable; no FTL |
same, both nodes; also no Telemetry endpoint FATAL (0.3 check) |
| #903 drift chip | On /deployments (login disabled on DEV): the first render shows "Checking for changes...", then "In sync". Make a harmless draft change, for example a tag description on /raw, with no deploy. Reload /deployments: it must show "Configuration drift" at once. Revert the change: the chip returns to "In sync", and the revision is still fa9a2e28…. |
Optional. The QA AdminUI needs the localhost forwarder plus a temporary role-mapping row (QA pack §2). If you do it, make the change on the canary and revert it. Delete the temporary row afterwards. |
| #902 Monaco | Open /scripts/new (the Monaco editor) and close the browser tab without saving. Then search the log after that time for MonacoEditor / Unhandled exception in circuit: expect no [ERR] (it is Debug now). |
Same if the AdminUI is opened; otherwise n/a |
| Historian writes | HG http://wonder-app-vd03…:5220/metrics historian_gateway_grpc_requests_total: snapshot before the stop and about 2 min after ready. Only HistorianStatus/Probe should grow. No EnsureTags, AddTagExtendedProperties, WriteLiveValues, AddHistoricalValues, SendEvent(s). |
Same on vq03's HG :5220/metrics; both QA nodes point at 10.220.158.2:5221 |
| OTLP logs (QA) | — (vd03 already sends logs) | New: the collector's file exporter on each box starts receiving OtOpcUa log records. Check over SMB, read-only. Not a gate. |
| Overview | :5320 shows OtOpcUa Up |
vq03/vq04 :5320: OtOpcUa Up, 2 members / 0 unreachable, 5/5 checks |
| ScadaBridge | /health/ready 200 (no OtOpcUa link) |
/health/ready 200 on both (no OtOpcUa link) |
6. Rollback
There is no schema undo in either range: no migration. The ConfigDb .bak exists only for a configuration restore.
DEV vd03 → 6c4d2990:
.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-c9640f5c `
-PreviousInstall E:\ApiInstall\OtOpcUa.failed-c9640f5c -NoCarry -NoStart
- The script stops the service, swaps the complete previous folder back (its own
appsettings*,pki,logs), then runs-Reapply+-Verify. The service stays stopped. - Then mirror
<backup>\OtOpcUa-databack overE:\ApiInstall\OtOpcUa-dataandStart-Service OtOpcUaHost. - Run
-Verifyagain, then check health and ServiceLevel 250. - The registry is unchanged by this deploy, so no import is needed.
- Fallback:
E:\ApiInstall\_stage-otopcua-6c4d2990\rollback.ps1is one generation older and targetsfea7de0a. Do not use it for this rollback.
QA → fea7de0a: stop both, then on each node:
.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.bak.20261113-pre-c9640f5c `
-PreviousInstall E:\ApiInstall\OtOpcUa\app.failed-c9640f5c -NoCarry -NoStart
- After the swap, mirror
backup\data-pre-c9640f5cback overdata\. - Import
backup\OtOpcUaHost-pre-c9640f5c.regonly if step 2's-ConfigureHostServicechanged the env. A reorder is harmless. - Run
-Verifyon each node (exit 0), then start both, the old Primary first. - The isolation record and denies stay valid for
fea7de0a. The installer works independently of the host build. - DB fallback:
OtOpcUaConfig_pre-c9640f5c_20261113.bak. - SDK tooling caveat: not applicable. Both builds are on SDK 2.0.1.
Isolation step only (§4 step 2): the settings record is additive. Re-running the installer with the same arguments
is idempotent. -Uninstall removes everything, including the recorded denies; do not run it as part of a binary
rollback.
7. Downtime estimates (from the previous records)
| Tier | Previous | Estimate for this pack |
|---|---|---|
| vd03 | fea7de0a about 10 s (stop 2 s); 6c4d2990 about 15 s (stop 6 s), stop → /health/ready |
about 20–40 s. The data robocopy and the in-script -Reapply/-Verify now run while the service is stopped. The step-2 record write happens with the service up. |
| QA pair | fea7de0a window about 2 min pair outage (16:48:13 → 16:50:14), of which about 70 s was the isolation install + uninstall drill. The first node took 39 s to ready because it waits for its peer; the second took 8 s. |
about 1–1.5 min pair outage: stops about 1–2 s each, backup + swap + -Reapply/-Verify per node, then about 40 s + 8 s to ready. One-node restarts (the OTel record) took up to 32 s to stop a Secondary with its peer up. Stopping both together avoids that graceful-leave wait. |
| QA failover gate | 30 s graceful leave, then the new Primary at 250 on its first sample | same |
After lmxopcua#905 (for the next pack that uses Invoke-OtOpcUaInstallSwap.ps1). The 101 s (vd03) and 50–60 s per
QA node were the protected-DACL coverage scan (Get-Acl per item over every denied folder, in both -Reapply and
-Verify, while stopped), not the ACL writes. The swap now runs -Reapply/-Verify -Scope Outside (the full scan
included) before the stop and only -Scope InstallRoot inside the window. windev, a vd03-like tree of 17,851 items
under the denied folder: stopped window 153.1 s → 6.0 s (Windows PowerShell 5.1) and 101.6 s → 6.2 s (PowerShell 7);
the pre-stop part takes about 23 s with the old build still serving. Estimate for the next vd03 swap: stop (2–6 s) +
the data\ backup + about 6 s + start-up to /health/ready (about 8–10 s), i.e. about 20–30 s. QA: about 10 s of
swap per node inside the pair outage instead of 50–60 s, and the Secondary-first order (deviation 3) is no longer needed
to hide it. The script prints stopped window (…): N s [stop, move, carry, isolation]; record that line.