lmxopcua/docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md

QA wonder-app-vq03/vq04 upgrade b8b4c18f → master fea7de0a (SDK 2.0.1), with the DEV rehearsal

Status: EXECUTED 2026-11-12. Follow-up run the same evening: the #890 re-apply, a script-runner canary, the live #794 step 5 check and the #780 runaway drill (last section). DEV vd03 runs fea7de0a (from 5fb4b4db). The QA pair vq03 + vq04 runs fea7de0a (from b8b4c18f) with the out-of-process script runner on (Calculation + VirtualTag seams) and the ScriptWorker isolation installed. Tracking: #801 (SDK 2 live gate), #793 (ConfigDb preflight), #794 (worker isolation install), #780 (P3 runner flip). Template: vd03-upgrade-b8b4c18f-to-master.md.

Channel: WinRM + SMB from the Mac (the winrm skill), used only for the on-box steps of this deploy, which the owner requested on 2026-11-12. Health, OPC UA, ScadaBridge and Overview checks ran from the Mac over HTTP / opc.tcp.

Build

• Source: origin/master fea7de0a (fea7de0aefd905522d187dad495c3c8163a5ec03), built in an isolated worktree.
• dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true on the Mac: 0 warnings, 1396 files, OtOpcUa.Host.exe ProductVersion 1.0.0+fea7de0a…, self-contained script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe.
• Packages in deps.json: LocalDb + LocalDb.Replication 0.3.7, MxGateway.Client 0.4.0, HistorianGateway.Client 1.0.0.
• Zip otopcua-fea7de0a.zip, SHA-256 170E7392…8F8024; the hash was checked on every box before expanding.
• 5fb4b4db..fea7de0a: 39 commits, no ConfigDb migration, no new required setting.

DEV: wonder-app-vd03 (5fb4b4db → fea7de0a)

Same layout and scripts as §(h) of the vd03 pack (single fused node, E:\ApiInstall\OtOpcUa, data in E:\ApiInstall\OtOpcUa-data).

• Prepare (service up): registry export; BACKUP DATABASE … COPY_ONLY → OtOpcUaConfig-pre-20261112-otopcua-fea7de0a.bak (instance default backup dir); the ConfigDb was already at 20261022102459_EnableSnapshotIsolation, snapshot ON (9 migrations). The pepper was already present (26 env vars, unchanged). w32tm source USWAP-VDC-ZMR1.
• Swap: stop 2 s, OtOpcUa-data robocopied to OtOpcUa-backups\pre-20261112-otopcua-fea7de0a, previous install at E:\ApiInstall\OtOpcUa.bak.20261112-pre-fea7de0a, box appsettings*/pki/logs carried. Downtime about 10 s. Rollback: E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1.
• Verified:
  • /healthz, /health/ready (opcua-endpoint, script-runner InProcess, configdb, config-publish, akka) and /health/active all 200.
  • ServiceLevel 250 on the single node (was 255 on 5fb4b4db, #884/#885).
  • Raw + UNS browse identical to before (ns=2/ns=3 NodeIds).
  • 23/24 variables Good, the same set as before. The one non-Good is Program/BlockCount, BadNoData, as before. parts-count (raw and UNS) Good, Spindle/S1/MotorSpeedPct Good, Program/Name Good.
  • AdminUI /, /hosts, /fleet, /deployments, /raw all 200 (login disabled on DEV).
  • Boot log: roles=[admin,driver], the #815 preflight line, no RegisterServer noise (#876), no new ERR. The repeated domain … is not listed in the server certificate ERR comes from CLI clients dialling the FQDN against the CN=OtOpcUa certificate. It was there before the upgrade too (36 lines earlier the same day).
  • ScadaBridge on vd03: /health/ready 200. It has no OPC UA link to OtOpcUa (MxGateway only), so it is unaffected.
• Not run on DEV: a post-deploy AdminUI deploy (no configuration change) and Galaxy alarm history (vd03 has no Galaxy driver).

QA preflight (#793)

Run on the box with the service's own SQL connection (wwadmin → wonder-sql-vq03, OtOpcUaConfig), the Mac login being unable to open the DB.

• ConfigDb checks (scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql, sqlcmd -I): all 15 checks ok, including #783 VirtualTagNoEffectiveTrigger, #791 VT→VT and VirtualTagDependencyCycle, #792/E2 forbidden script APIs, and the credential/MTConnect/Sql/Heidenhain/UNS/FOCAS rules.
  • The QA ConfigDb is empty: 0 deployments, 0 driver instances, 0 tags, 0 virtual tags, 0 scripts, 0 scripted alarms, 0 API keys, 0 NodeAcl rows.
  • Two ClusterNode rows: 10.220.158.2:4053 (vq03) and 10.220.158.3:4053 (vq04), cluster DEV, ServiceLevelBase 200.
  • Migrations 8 (last AddApiKeyTable), snapshot isolation OFF, compat 160.
• PKI / CA KeyUsage (#782): each node has only its own self-signed CN=OtOpcUa certificate (CA:FALSE). vq03 is 06498CB2…, SAN urn:wonder-app-vq03:OtOpcUa + IP 10.220.158.2. vq04 is CE4859A4…, SAN urn:wonder-app-vq04:OtOpcUa
  • IP 10.220.158.3. Both are valid to 2027-08-11. There are no CA or issuer certificates, so nothing for the stricter rule to reject. ScadaBridge QA has no OPC UA connection to OtOpcUa (its one data connection is MxGateway), so there is no trust chain to check.
• Service environment: 41 entries on vq03, 40 on vq04 (vq04 has no PeerAddress). Compared by SHA-256 prefix only:
  • Security__ApiKeys__Pepper is already set (44 chars) and identical on both nodes.
  • LocalDb__Replication__ApiKey is already set (44 chars) and byte-identical on both nodes. This is what LocalDb 0.3.7 requires; it lives in the service registry environment, not in a file.
  • The JWT key is 64 bytes, with the default Issuer/Audience.
  • ServerHistorian is enabled → http://10.220.158.2:5221 (vq03's HistorianGateway d8d890c), UseTls=false.
  • No OpcUa:EnabledSecurityProfiles anywhere, so #879 changes nothing here.
• Clocks (#873): vq03 → USWAP-VDC-ZMR3, vq04 → USSBN-VDC-ZMR1, both synced within minutes.
• HistorianGateway: not upgraded. Master uses only RPCs that d8d890c serves (ReadRaw/Aggregate/AtTime/Events, WriteLiveValues, SendEvent, EnsureTags, Probe). It sends its own event GUIDs and does not use the new store-forward fields or string backfill (docs/Historian.md). The preflight boot logged Historian gateway probe: the gateway is reachable against d8d890c with Client 1.0.0 over h2c. QA has no historized tags. historiangw#78 stays its own deploy.
• mxaccessgw: not upgraded. The #877 stream-attached ack is capability-negotiated (stream-events-attached-ack; SupportsStreamAttachedAck defaults false → the old best-effort wait). The #749 operator password is an additive proto field that the old gateway ignores. QA OtOpcUa has no Galaxy driver at all, so 4f931e4 stays.
• Isolated preflight boot on vq04 (live service running): the candidate ran from a copy with the node's real service environment, using its own ports (HTTP 19000, Akka 14053 seeded to itself, OPC UA 14840, sync 19101), a clone OtOpcUaConfig_pf (snapshot migration applied) and copies of data\/pki\.
  • A: /health/ready 200 in 11 s, all five components Healthy.
  • B (negative control): a 5-character pepper → exit 1, FATAL … Security:ApiKeys:Pepper must be at least 16 characters.
  • The clone was dropped afterwards.
  • Finding: the first A run also overrode OpcUa:PublicHostname=127.0.0.1. SDK 2.0.1 then refused to start the endpoint (The certificate with subject CN=OtOpcUa in the configuration is invalid), because 127.0.0.1 is not in the certificate SAN. /health/ready went 503 (opcua-endpoint Unhealthy) while /health/active stayed 200. So OpcUa:PublicHostname must stay a name or IP in the certificate SAN, and only readiness shows a dead endpoint.

QA window (stop both, swap both, start both; owner decision, no mixed-SDK rolling restart)

Layout: E:\ApiInstall\OtOpcUa\{app,data,pki}, service OtOpcUaHost (LocalSystem), staging E:\psrem-stage\otopcua-fea7de0a.

1. Prepare (services up):
   • Candidate expanded to app.new-fea7de0a (1396 files, hash ok) and the box's five appsettings*.json carried into it (SHA-identical; the publisher's copies were not used).
   • Isolation -PlanOnly: 7 steps.
   • Registry export backup\OtOpcUaHost-pre-fea7de0a.reg on each node.
   • BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION → OtOpcUaConfig_pre-fea7de0a_20261112.bak (default backup dir on wonder-sql-vq03), RESTORE VERIFYONLY ok.
   • Migration 20261022102459_EnableSnapshotIsolation applied to the live DB (snapshot ON, 9 migrations), while b8b4c18f was still running. The old build does not use it.
2. Stop + swap (16:48:13–16:48:20 EST): vq03 (Secondary) stopped in 1 s, then vq04 (Primary) in 1 s.
   • data\ and pki\ robocopied to backup\{data,pki}-pre-fea7de0a (stopped, so the SQLite triplets are consistent).
   • app → app.bak.20261112-pre-fea7de0a, app.new-fea7de0a → app, logs\ carried; appsettings matched.
3. #794 isolation on the new app\ (the swap drops folder ACEs), both nodes, services stopped. Run with Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService:
   • -WhatIf: 7 operations, nothing changed.
   • Apply: 7/7 changed. Second apply: 7/7 unchanged.
   • Rollback drill: -Uninstall: 6 changed (the PE cache is kept). Second uninstall: 6/6 unchanged.
   • Re-apply: 6 changed + PeCacheDir unchanged. Second re-apply: 7/7 unchanged.
   • State verified on both nodes:
     • sc qc OtOpcUaScriptWorker: DISABLED, NT SERVICE\OtOpcUaScriptWorker. showsid: S-1-5-80-1846135827-…-3023753135.
     • C:\ProgramData\OtOpcUa\script-pe-cache: SYSTEM and Administrators full, no Users. Its pe\ adds the identity (OI)(CI)(M).
     • app\: identity (OI)(CI)(DENY)(S,RD). app\script-worker: identity (OI)(CI)(RX) (explicit, ahead of the inherited deny).
     • Firewall rule OtOpcUa-ScriptWorker-BlockOutbound: Outbound / Block / Any / enabled, program E:\ApiInstall\OtOpcUa\app\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe.
     • Service env gained Scripting__Worker__Identity and Scripting__Worker__PeCacheDir, with the other entries intact.
4. #780 P3 flip (registry env):
   • Scripting__Runner=OutOfProcess and Scripting__Runner__Seams__0=Calculation.
   • After the start below, widened to Scripting__Runner__Seams__1=VirtualTag. ScriptedAlarm is not enabled.
   • Env count is 46 on vq03 and 45 on vq04.
5. Start vq04 (16:49:34 → ready 16:50:14, 39 s: it waits for its peer), then vq03 (ready 16:50:23 in 8 s). Pair outage 16:48:13 → 16:50:14, about 2 min, of which about 70 s was the isolation drill.
6. Cleanup: the clone DB and the preflight copy were removed. The staging dir and backups stay on each node.

QA verification (#801)

Check	Result
Version	OtOpcUa.Host.exe 1.0.0+fea7de0a… on both.
ServiceLevel on the real pair	vq04 250 (Primary), vq03 240 (Secondary), stable over repeated fresh-session reads. ApplicationUri per node unchanged.
Health	Both /health/ready 200 with opcua-endpoint, script-runner (Out-of-process script runner ready.), configdb, config-publish and akka Healthy. /health/active 200 on the Primary, 503 on the Secondary (by design).
LocalDb sync (0.3.7, ApiKey enforced)	localdb-replication: connected, oplog backlog 0 on both nodes, after the window and again after the failover. Negative control: a sync call from the Mac with no bearer token and one with a bogus token both get grpc-status 7 LocalDb sync authentication failed on both nodes, and each node logs Rejected a LocalDb sync call.
Restart Secondary (to pick up the VirtualTag seam)	vq03 down → 100 → 240 within about 8 s of start. vq04 held 250 throughout.
Failover	Stop-Service on the Primary vq04 at 16:53:42. The graceful leave took 30 s, and vq03 became oldest at 16:54:13 (OldestChanged … -> myself). From the first sample: vq03 250, /health/active 200.
Old Primary restart	vq04 started 16:55:32: 100 → 240 at its first sample after ready, and never above 240 (no 255, #885). No failback: vq03 stays 250 / active 200, vq04 240 / active 503.
AdminUI	Login posture unchanged: / → 302 /login, /login 200, /api/deployments 302.
Boot logs	roles=[admin,driver], the #815 preflight line, Historian gateway probe: the gateway is reachable. No FTL. The ERR lines are bootstrap-time Akka Removed (vq04 before join) and the CLI certificate-domain noise.
Overview (vq03 + vq04 :5320)	OtOpcUa Up, 2 instances, 2 members / 0 unreachable, 5/5 checks healthy (now including opcua-endpoint and script-runner).
ScadaBridge QA	/health/ready 200 on both nodes before and after. Its only data connection is MxGateway, so it reads no OtOpcUa data on QA. The consumer check is not applicable here.
AVEVA history via HG	Not applicable on QA: no Galaxy driver, no historized tags. The real-AVEVA gate passed separately against vd03's gateway (#801 comment, 008f0b36).

Not exercisable on QA (empty configuration):

• No Calculation, VirtualTag or scripted-alarm exists, so no worker has started on either node. The following were therefore not observed: the restricted-token worker under the real service session, the window-station grant, the PE-cache fill, the firewall drop log, and "Calculation tags stay Good".
• The runaway drill was skipped: running it needs a script in the configuration, and authoring QA configuration was out of scope for this deploy.
• script-runner reports Out-of-process script runner ready on both nodes.

Isolation coverage note. The installer denies the worker identity on InstallRoot (app\) only. On QA the siblings E:\ApiInstall\OtOpcUa\pki (including own\private\*.pfx) and data\ (secrets.db, LocalDb) inherit BUILTIN\Users:ReadAndExecute. Users is one of the worker token's restricting SIDs, so those folders are not covered by the deny. File-level ACLs were not inspected. Follow-up: #890.

#890 fix: re-applied on vq03 and vq04 later the same day (see "Canary, live isolation check and runaway drill" below, which also adds the KEK folder to the list). The original instructions follow. The installer now takes -DenyPaths and records them. Once the scripts at that commit are staged on each node, run the following in an elevated Windows PowerShell 5.1 session. The services can stay up: only folder ACLs change for the worker identity, and no worker runs on QA today.

cd <staged scripts\install>
$deny = 'E:\ApiInstall\OtOpcUa','E:\ApiInstall\OtOpcUa\pki','E:\ApiInstall\OtOpcUa\data'
.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny -WhatIf
.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny
.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny

Before running it, check Secrets__SqlitePath, its key file and LocalDb__Path in the service environment. Add any folder that sits outside E:\ApiInstall\OtOpcUa to $deny.

Expected results:

• First apply: 11 steps, 5 changed. They are DenyPathRecord, InstallRootDeny (mask 0x1 → 0xD0157, read + write) and the three SensitiveDirDeny. The other 6 steps are unchanged.
• Second apply: 11/11 unchanged.
• Check:
  • icacls E:\ApiInstall\OtOpcUa, …\pki and …\data and …\app each show NT SERVICE\OtOpcUaScriptWorker:(OI)(CI)(DENY)(W,D,WDAC,WO,RD,DC), which is what windev's PowerShell 5.1 renders for 0xD0157.
  • reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths lists the three folders.
  • The coverage: line reports the number of protected items the deny does not reach. Inspect each one WARNed (for example a pki\own\private file the SDK wrote with its own DACL).

Rollback stays … -ConfigureHostService -Uninstall. It now also removes the deny from the recorded folders.

Configuration set (service registry environment; no file changes, no secrets added)

Node	Added
vq03, vq04	Scripting__Worker__Identity=NT SERVICE\OtOpcUaScriptWorker, Scripting__Worker__PeCacheDir=C:\ProgramData\OtOpcUa\script-pe-cache (installer), Scripting__Runner=OutOfProcess, Scripting__Runner__Seams__0=Calculation, Scripting__Runner__Seams__1=VirtualTag
vq03, vq04	Kept, already present and identical: Security__ApiKeys__Pepper, LocalDb__Replication__ApiKey
ConfigDb (QA)	EnableSnapshotIsolation applied
vd03	No change

Rollback

• Runner only: set Scripting__Runner=InProcess (or remove the four Scripting__Runner* entries) and restart the node. To remove the isolation, run Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -Uninstall. After the #890 re-apply, this also removes the recorded -DenyPaths denies, so there is no need to repeat them.

• QA binaries: run E:\psrem-stage\otopcua-fea7de0a\rollback.ps1 on both nodes (stop both first), then start both. The script does three things:

  1. app → app.failed-fea7de0a, and app.bak.20261112-pre-fea7de0a → app;
  2. mirrors backup\data-pre-fea7de0a back over data\;
  3. imports backup\OtOpcUaHost-pre-fea7de0a.reg, which restores the pre-deploy environment, without the Scripting entries.

  The isolation ACEs, firewall rule and anchor service are harmless to b8b4c18f. No schema undo is needed. OtOpcUaConfig_pre-fea7de0a_20261112.bak is the DB fallback.

• DEV: E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1 (→ 5fb4b4db).

• SDK 1.5 tooling caveat after a rollback: see the rollout runbook §Rollback item 4 (#880).

Canary, live isolation check and runaway drill (2026-11-12, 18:49–19:07 EST)

Owner request the same day: "add the canary tag and finish QA". WinRM was used only for the on-box steps below (installer, SQL role grant, live-token probe, log check of the drill). Everything else ran over HTTP / opc.tcp / SMB from the Mac. No service restart; the pair stayed at vq03 250 (Primary) / vq04 240 throughout.

1. #890 re-apply (both nodes, services up)

Scripts scripts/install/{Install-ScriptWorkerIsolation.ps1,ScriptWorkerIsolation.psm1} at f75ee7d0 (SHA-256 E0C11AEA…, F9425E4C…), staged over SMB to E:\psrem-stage\otopcua-890-f75ee7d0\install, hashes checked on each box.

• Path check (service env): Secrets__SqlitePath=E:\ApiInstall\OtOpcUa\data\secrets.db and LocalDb__Path=E:\ApiInstall\OtOpcUa\data\otopcua-localdb.db are inside. Secrets__MasterKey__FilePath=C:\ProgramData\ZB\kek\otopcua.kek is outside, so C:\ProgramData\ZB\kek was added. That folder is already protected (SYSTEM + Administrators only), so the deny there is defence in depth.
• $deny = 'E:\ApiInstall\OtOpcUa','E:\ApiInstall\OtOpcUa\pki','E:\ApiInstall\OtOpcUa\data','C:\ProgramData\ZB\kek', then -WhatIf (12 operations, nothing changed) → apply → apply.
• First apply: 12 steps, 7 changed: DenyPathRecord, InstallRootDeny (0x1 → 0xD0157), four SensitiveDirDeny, and HostServiceEnv. The last one is a reorder only: the earlier P3 flip had appended the Scripting__Runner* entries after Scripting__Worker__Identity/PeCacheDir, and the step rewrites the list with its two entries last. Same values, env count unchanged (46 / 45). Second apply: 12/12 unchanged on both nodes.
• Coverage: vq03 5 denied folder(s); 0 protected item(s). vq04 WARNs that C:\ProgramData\ZB\kek\historiangw.kek and mxgateway.kek have protected DACLs. The live probe below shows the worker cannot read either.
• State, both nodes: icacls on E:\ApiInstall\OtOpcUa, pki, data, app and C:\ProgramData\ZB\kek shows NT SERVICE\OtOpcUaScriptWorker:(OI)(CI)(DENY)(W,D,WDAC,WO,RD,DC); app\script-worker keeps the explicit (OI)(CI)(RX) ahead of the inherited deny. reg query …\OtOpcUaScriptWorker\Parameters /v DenyPaths lists the four folders. Anchor DISABLED, firewall rule Outbound/Block on the worker exe. /health/ready 200 on both afterwards.

2. Canary configuration (authored in the AdminUI, deployed with "Deploy current configuration")

The QA AdminUI cannot be used from a browser over plain HTTP (its auth cookie is Secure), and QA had no LDAP group → role grant. For this run only: a local forwarder localhost:19003 → wonder-app-vq03:9000 (Chrome treats http://localhost as secure), and a temporary LdapGroupRoleMapping row (SCADA-Admins → Administrator, system-wide, on-box SQL with the service's own connection) for the box GLAuth test login. The row was deleted at the end (QA again has 0 role mappings; the same login now gets AccessDenied).

Object	Value
Scripts	ScriptRunnerCanary-Calc (SC-82e11d9323ff): return (int)DateTime.UtcNow.TimeOfDay.TotalSeconds; · ScriptRunnerCanary-VT (SC-c2ec9b4c4c53): return DateTime.UtcNow.Second;
Raw	Calculation driver script-runner-canary (under the QA cluster) → Device1 → tag canary-calc-seconds, Int32, Read, {"scriptId":"SC-82e11d9323ff","changeTriggered":true,"timerIntervalMs":1000} → ns=2;s=script-runner-canary/Device1/canary-calc-seconds
UNS	area canary (area-canary) → line script-runner (line-canary) → equipment script-runner-canary (EQ-ec3979550d90, MachineCode SCRIPT_RUNNER_CANARY) → virtual tag canary-vt-second (VT-script-runner-canary), Int32, timer 1000 ms → ns=3;s=EQ-ec3979550d90/canary-vt-second
Deployment	bcaba714 rev 82dd0598c37f… Sealed, both nodes in sync (the first deployment ever on QA)

Result: both values Good and advancing once a second on both endpoints (opc.tcp://10.220.158.2:4840 and .3:4840, anonymous, read by the OtOpcUa CLI). Both nodes run the drivers, so no failover was needed to see it. Workers started on the first evaluation: otopcua_script_worker_up{shard=0,1} = 1, runner_state 0, two PE entries in C:\ProgramData\OtOpcUa\script-pe-cache\pe\ on each node (listed over SMB).

Kept on QA as a runner monitor (the design and #780 say nothing about removing it): a Bad or frozen canary-calc-seconds / canary-vt-second, or otopcua_script_worker_up = 0, means the out-of-process runner is sick.

3. #794 step 5: the live worker under the real service session (both nodes)

Probe: an admin PowerShell on the box duplicated the live worker's own token (opened under the host's LocalSystem token, since an admin cannot open it directly) and impersonated it for file opens. The same reads were then repeated without impersonation as the positive control (all READ-OK).

Check	vq03 and vq04 (identical)
Worker processes	2 per node, …\app\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe, parent = OtOpcUa.Host.exe
Token	user NT AUTHORITY\SYSTEM, restricted = True, privileges only SeChangeNotifyPrivilege, restricting SIDs NT SERVICE\OtOpcUaScriptWorker, Everyone, BUILTIN\Users, NT AUTHORITY\RESTRICTED (host token: unrestricted, 28 privileges)
Window-station grant	works under the real service session: both shards run and evaluate (no ERROR_DLL_INIT_FAILED)
#890 paths, read	DENIED: pki\own\private\*.pfx, pki\trusted\certs\*.der, data\secrets.db, data\otopcua-localdb.db, app\appsettings.json, app\OtOpcUa.Host.dll, app\logs\scripts-*.log, app.bak.…\appsettings.json, C:\ProgramData\ZB\kek\{otopcua,historiangw,mxgateway}.kek, script-pe-cache\pe-cache.key
#890 paths, write	DENIED: create a file in pki\trusted\certs, data, app, app\script-worker
Positive controls	READ-OK: app\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.dll, C:\Windows\System32\kernel32.dll
Network	0 TCP / 0 UDP endpoints owned by either worker pid; rule OtOpcUa-ScriptWorker-BlockOutbound Outbound/Block/Any/enabled and its program equals the running image path
PE cache	filled on first use (above); after the drill's worker restarts otopcua_script_compile{cache="hit"} = 4 on each node

Not done: the dropped-packet firewall log. The worker has no code path that opens a socket, so the log would be empty whether or not the rule worked. Turning on profile-wide logging was not worth it for that.

Finding (outside #890's scope, filed as #891): the worker token can read some Users-readable files outside OtOpcUa's folders: E:\psrem-stage\otopcua-fea7de0a\backup\OtOpcUaHost-pre-fea7de0a.reg (a registry export of the OtOpcUaHost environment, which holds secrets), and E:\ApiInstall\ScadaBridge\app\appsettings.json / E:\ApiInstall\HistorianGateway\app\appsettings.json. C:\ProgramData\ZB\qa-build\* is denied (protected ACL). Those folders belong to other apps or to deploy staging, so no deny was added there in this run. Follow-up: #891.

4. #780 runaway drill (live gate: "a QA flip with the same drill")

A second script ScriptRunnerCanary-RunawayDrill = while (true) { } bound to VT canary-runaway-drill (Int32, timer 1000 ms) on the canary equipment, deployed as e746cc5f (rev 43bd8917…). Seams: Calculation + VirtualTag.

Observed	vq03	vq04
Drill VT	BadInternalError from the first sample (00:02:59Z)	same
Kills	19:03:03.32 and 19:03:08.31 EST: exceeded its hard deadline (4000 ms); killing ScriptWorker[0] → ended: "KilledDeadline" … culprit 1:VT-script-runner-canary-runaway, 1 evaluation(s) … in flight	19:03:03.32 and 19:03:08.27, same lines
Quarantine	Script "1:VT-script-runner-canary-runaway" quarantined: "it exceeded its hard deadline and its worker was killed (2 time(s) in the last hour)…"; otopcua_script_quarantined_total{reason="killed"} 1, worker_exits_total{reason="killed_deadline",shard="0"} 2, run_refused 4	same
Shard 0 restart	back in about 0.5 s each time	same
Siblings	canary-vt-second 77 updates, canary-calc-seconds 88 updates over the drill, all Good	74 / 88, all Good
Node	/health/ready 200, runner_state 0, /health/active 200	/health/ready 200, /health/active 503 (Secondary)
ServiceLevel	250 on all 21 samples	240 on all 21 samples

Cleanup: the drill VT and its script were deleted in the AdminUI and the configuration redeployed as 5905003e. Its revision is 82dd0598…, the same as the canary-only deployment bcaba714. The drill node now reads BadNodeIdUnknown and both canary values are Good on both nodes.

ScriptedAlarm seam: left off. No canary scripted alarm was added: alarms go last (§14 #10), and the alarm drill already passed on docker-dev (§12.4).

Final QA state (19:07 EST)

• vq03 250 Primary, /health/active 200. vq04 240 Secondary, 503. Both /health/ready 200 with 5/5 components. Overview: OtOpcUa Up, 2 members / 0 unreachable.
• Scripting__Runner=OutOfProcess, seams Calculation + VirtualTag. Isolation with the #890 denies on both nodes.
• ConfigDb: the canary only (1 driver, 1 device, 1 tag, 1 VT, 2 scripts, 1 area/line/equipment, 0 scripted alarms, 0 role mappings, 0 API keys). Three sealed deployments; 5905003e is current.
• ScadaBridge, mxgw and HistorianGateway on QA were not touched.
• Rollback of this run: remove the canary by deleting its rows in the AdminUI and deploying (or restore OtOpcUaConfig_pre-fea7de0a_20261112.bak). The isolation rollback is unchanged: … -ConfigureHostService -Uninstall removes the four recorded denies.

Box-wide worker denies and backup lockdown (lmxopcua#891, 2026-11-12, owner-approved)

Owner approval: the owner approved locking down the registry exports and adding the denies for the other apps' folders. Done on both vq03 and vq04 with the f75ee7d0 installer staged at E:\psrem-stage\otopcua-890-f75ee7d0\install. Order: one -WhatIf run, then the apply, then a re-apply that reported all items unchanged. No service was restarted.

1. Registry exports locked down. These six deploy-backup .reg files per box now have a protected DACL allowing only SYSTEM and BUILTIN\Administrators (FullControl); every other ACE was removed:

• hg-d8d890c\backup\HistorianGateway-svc-*.reg
• mxgw-4f931e4\backup\MxAccessGw-servicekey-*.reg (2 files)
• otopcua-b8b4c18f\backup\OtOpcUaHost-service*.reg (2 files)
• otopcua-fea7de0a\backup\OtOpcUaHost-pre-fea7de0a.reg

The files themselves were not deleted, so they still serve as rollback copies.

2. Box-wide denies. -DenyPaths now lists:

• E:\ApiInstall\OtOpcUa, \pki, \data, and C:\ProgramData\ZB\kek (all from before);
• E:\ApiInstall\ScadaBridge, HistorianGateway, MxAccessGateway, Overview and btdbmanager;
• E:\psrem-stage and E:\psrem.

icacls shows NT SERVICE\OtOpcUaScriptWorker:(OI)(CI)(DENY)(W,D,WDAC,WO,RD,DC) on each of these folders, and reg query shows the 11 recorded paths. The deny names only the worker identity, so the other apps' service accounts are not affected.

3. Installer warnings, all checked.

• The six locked .reg files have protected DACLs on purpose.
• On vq04, C:\ProgramData\ZB\kek\historiangw.kek and mxgateway.kek have protected DACLs that allow only SYSTEM and Administrators.

None of these grants anything to Everyone, BUILTIN\Users or RESTRICTED. A restricted worker token fails the restricting-SID check on all of them, so they need no deny of their own.

4. After the change. /health/ready returns 200 on both nodes for OtOpcUa :9000, Overview :5320, ScadaBridge :8085 and mxgw :5130.

Deploy procedure from now on. Write registry exports and appsettings backups into a backup folder that you create with a protected ACL (SYSTEM + Administrators only), and run the installer with the box-wide -DenyPaths list above. Any new app folder under E:\ApiInstall must be added to the list when it is installed.

Rollback: re-run the installer with the previous four-path list to remove the new denies, or use -Uninstall. To restore the .reg ACLs, run icacls <file> /reset.