lmxopcua/docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md

# QA wonder-app-vq03/vq04 upgrade `b8b4c18f` → master `fea7de0a` (SDK 2.0.1), with the DEV rehearsal

> **Status: EXECUTED 2026-11-12.** Follow-up run the same evening: the #890 re-apply, a script-runner canary, the live #794 step 5 check and the #780 runaway drill (last section). DEV vd03 runs `fea7de0a` (from `5fb4b4db`). The QA pair vq03 + vq04 runs
> `fea7de0a` (from `b8b4c18f`) with the out-of-process script runner on (Calculation + VirtualTag seams) and the
> ScriptWorker isolation installed. Tracking: #801 (SDK 2 live gate), #793 (ConfigDb preflight), #794 (worker
> isolation install), #780 (P3 runner flip). Template: [`vd03-upgrade-b8b4c18f-to-master.md`](vd03-upgrade-b8b4c18f-to-master.md).

Channel: WinRM + SMB from the Mac (the `winrm` skill), used only for the on-box steps of this deploy, which the owner
requested on 2026-11-12. Health, OPC UA, ScadaBridge and Overview checks ran from the Mac over HTTP / opc.tcp.

## Build

- Source: `origin/master` **`fea7de0a`** (`fea7de0aefd905522d187dad495c3c8163a5ec03`), built in an isolated worktree.
- `dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true`
  on the Mac: 0 warnings, **1396 files**, `OtOpcUa.Host.exe` ProductVersion `1.0.0+fea7de0a…`, self-contained
  `script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`.
- Packages in `deps.json`: LocalDb + LocalDb.Replication **0.3.7**, MxGateway.Client **0.4.0**, HistorianGateway.Client **1.0.0**.
- Zip `otopcua-fea7de0a.zip`, SHA-256 `170E7392…8F8024`; the hash was checked on every box before expanding.
- `5fb4b4db..fea7de0a`: 39 commits, **no ConfigDb migration**, no new required setting.

## DEV: wonder-app-vd03 (`5fb4b4db` → `fea7de0a`)

Same layout and scripts as §(h) of the vd03 pack (single fused node, `E:\ApiInstall\OtOpcUa`, data in `E:\ApiInstall\OtOpcUa-data`).

- **Prepare (service up):** registry export; `BACKUP DATABASE … COPY_ONLY` →
  `OtOpcUaConfig-pre-20261112-otopcua-fea7de0a.bak` (instance default backup dir); the ConfigDb was already at
  `20261022102459_EnableSnapshotIsolation`, snapshot ON (9 migrations). The pepper was already present (26 env vars, unchanged).
  w32tm source USWAP-VDC-ZMR1.
- **Swap:** stop 2 s, `OtOpcUa-data` robocopied to `OtOpcUa-backups\pre-20261112-otopcua-fea7de0a`, previous install at
  `E:\ApiInstall\OtOpcUa.bak.20261112-pre-fea7de0a`, box `appsettings*`/`pki`/`logs` carried. **Downtime about 10 s.**
  Rollback: `E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1`.
- **Verified:**
  - `/healthz`, `/health/ready` (opcua-endpoint, script-runner InProcess, configdb, config-publish, akka) and `/health/active` all 200.
  - **ServiceLevel 250** on the single node (was 255 on `5fb4b4db`, #884/#885).
  - Raw + UNS browse identical to before (ns=2/ns=3 NodeIds).
  - 23/24 variables Good, the same set as before. The one non-Good is `Program/BlockCount`, BadNoData, as before.
    `parts-count` (raw and UNS) Good, `Spindle/S1/MotorSpeedPct` Good, `Program/Name` Good.
  - AdminUI `/`, `/hosts`, `/fleet`, `/deployments`, `/raw` all 200 (login disabled on DEV).
  - Boot log: `roles=[admin,driver]`, the #815 preflight line, no `RegisterServer` noise (#876), no new ERR. The repeated
    `domain … is not listed in the server certificate` ERR comes from CLI clients dialling the FQDN against the
    `CN=OtOpcUa` certificate. It was there before the upgrade too (36 lines earlier the same day).
  - ScadaBridge on vd03: `/health/ready` 200. It has no OPC UA link to OtOpcUa (MxGateway only), so it is unaffected.
- Not run on DEV: a post-deploy AdminUI deploy (no configuration change) and Galaxy alarm history (vd03 has no Galaxy driver).

## QA preflight (#793)

Run on the box with the service's own SQL connection (`wwadmin` → `wonder-sql-vq03`, `OtOpcUaConfig`), the Mac
login being unable to open the DB.

- **ConfigDb checks** (`scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql`, `sqlcmd -I`): **all 15 checks
  `ok`**, including #783 VirtualTagNoEffectiveTrigger, #791 VT→VT and VirtualTagDependencyCycle, #792/E2 forbidden
  script APIs, and the credential/MTConnect/Sql/Heidenhain/UNS/FOCAS rules.
  - **The QA ConfigDb is empty:** 0 deployments, 0 driver instances, 0 tags, 0 virtual tags, 0 scripts, 0 scripted
    alarms, 0 API keys, 0 NodeAcl rows.
  - Two ClusterNode rows: `10.220.158.2:4053` (vq03) and `10.220.158.3:4053` (vq04), cluster `DEV`, ServiceLevelBase 200.
  - Migrations 8 (last `AddApiKeyTable`), snapshot isolation **OFF**, compat 160.
- **PKI / CA KeyUsage (#782):** each node has only its own self-signed `CN=OtOpcUa` certificate (CA:FALSE). vq03 is
  `06498CB2…`, SAN `urn:wonder-app-vq03:OtOpcUa` + IP 10.220.158.2. vq04 is `CE4859A4…`, SAN `urn:wonder-app-vq04:OtOpcUa`
  + IP 10.220.158.3. Both are valid to 2027-08-11. There are no CA or issuer certificates, so nothing for the stricter
  rule to reject. ScadaBridge QA has no OPC UA connection to OtOpcUa (its one data connection is `MxGateway`), so
  there is no trust chain to check.
- **Service environment:** 41 entries on vq03, 40 on vq04 (vq04 has no `PeerAddress`). Compared by SHA-256 prefix only:
  - `Security__ApiKeys__Pepper` is already set (44 chars) and identical on both nodes.
  - **`LocalDb__Replication__ApiKey`** is already set (44 chars) and **byte-identical on both nodes**. This is what
    LocalDb 0.3.7 requires; it lives in the service registry environment, not in a file.
  - The JWT key is 64 bytes, with the default Issuer/Audience.
  - `ServerHistorian` is enabled → `http://10.220.158.2:5221` (vq03's HistorianGateway `d8d890c`), `UseTls=false`.
  - No `OpcUa:EnabledSecurityProfiles` anywhere, so #879 changes nothing here.
- **Clocks (#873):** vq03 → USWAP-VDC-ZMR3, vq04 → USSBN-VDC-ZMR1, both synced within minutes.
- **HistorianGateway: not upgraded.** Master uses only RPCs that `d8d890c` serves (ReadRaw/Aggregate/AtTime/Events,
  WriteLiveValues, SendEvent, EnsureTags, Probe). It sends its own event GUIDs and does not use the new store-forward
  fields or string backfill (`docs/Historian.md`). The preflight boot logged `Historian gateway probe: the gateway is reachable`
  against `d8d890c` with Client 1.0.0 over h2c. QA has no historized tags. historiangw#78 stays its own deploy.
- **mxaccessgw: not upgraded.** The #877 stream-attached ack is capability-negotiated (`stream-events-attached-ack`;
  `SupportsStreamAttachedAck` defaults false → the old best-effort wait). The #749 operator password is an additive
  proto field that the old gateway ignores. QA OtOpcUa has no Galaxy driver at all, so `4f931e4` stays.
- **Isolated preflight boot on vq04** (live service running): the candidate ran from a copy with the node's real
  service environment, using its own ports (HTTP 19000, Akka 14053 seeded to itself, OPC UA 14840, sync 19101), a
  clone `OtOpcUaConfig_pf` (snapshot migration applied) and copies of `data\`/`pki\`.
  - **A:** `/health/ready` 200 in 11 s, all five components Healthy.
  - **B (negative control):** a 5-character pepper → exit 1, `FATAL … Security:ApiKeys:Pepper must be at least 16 characters`.
  - The clone was dropped afterwards.
  - **Finding:** the first A run also overrode `OpcUa:PublicHostname=127.0.0.1`. SDK 2.0.1 then refused to start the
    endpoint (`The certificate with subject CN=OtOpcUa in the configuration is invalid`), because 127.0.0.1 is not in
    the certificate SAN. `/health/ready` went 503 (opcua-endpoint Unhealthy) while **`/health/active` stayed 200**. So
    `OpcUa:PublicHostname` must stay a name or IP in the certificate SAN, and only readiness shows a dead endpoint.

## QA window (stop both, swap both, start both; owner decision, no mixed-SDK rolling restart)

Layout: `E:\ApiInstall\OtOpcUa\{app,data,pki}`, service `OtOpcUaHost` (LocalSystem), staging `E:\psrem-stage\otopcua-fea7de0a`.

1. **Prepare (services up):**
   - Candidate expanded to `app.new-fea7de0a` (1396 files, hash ok) and the box's five `appsettings*.json` carried into it
     (SHA-identical; the publisher's copies were not used).
   - Isolation `-PlanOnly`: 7 steps.
   - Registry export `backup\OtOpcUaHost-pre-fea7de0a.reg` on each node.
   - `BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION` → `OtOpcUaConfig_pre-fea7de0a_20261112.bak` (default
     backup dir on wonder-sql-vq03), `RESTORE VERIFYONLY` ok.
   - **Migration `20261022102459_EnableSnapshotIsolation` applied to the live DB** (snapshot ON, 9 migrations), while
     `b8b4c18f` was still running. The old build does not use it.
2. **Stop + swap** (16:48:13–16:48:20 EST): vq03 (Secondary) stopped in 1 s, then vq04 (Primary) in 1 s.
   - `data\` and `pki\` robocopied to `backup\{data,pki}-pre-fea7de0a` (stopped, so the SQLite triplets are consistent).
   - `app` → `app.bak.20261112-pre-fea7de0a`, `app.new-fea7de0a` → `app`, `logs\` carried; appsettings matched.
3. **#794 isolation on the new `app\`** (the swap drops folder ACEs), both nodes, services stopped. Run with
   `Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService`:
   - `-WhatIf`: 7 operations, nothing changed.
   - Apply: 7/7 `changed`. **Second apply: 7/7 `unchanged`.**
   - Rollback drill: `-Uninstall`: 6 changed (the PE cache is kept). **Second uninstall: 6/6 `unchanged`.**
   - Re-apply: 6 changed + PeCacheDir unchanged. **Second re-apply: 7/7 `unchanged`.**
   - State verified on both nodes:
     - `sc qc OtOpcUaScriptWorker`: DISABLED, `NT SERVICE\OtOpcUaScriptWorker`. `showsid`: `S-1-5-80-1846135827-…-3023753135`.
     - `C:\ProgramData\OtOpcUa\script-pe-cache`: SYSTEM and Administrators full, no Users. Its `pe\` adds the identity `(OI)(CI)(M)`.
     - `app\`: identity `(OI)(CI)(DENY)(S,RD)`. `app\script-worker`: identity `(OI)(CI)(RX)` (explicit, ahead of the inherited deny).
     - Firewall rule `OtOpcUa-ScriptWorker-BlockOutbound`: Outbound / Block / Any / enabled, program
       `E:\ApiInstall\OtOpcUa\app\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`.
     - Service env gained `Scripting__Worker__Identity` and `Scripting__Worker__PeCacheDir`, with the other entries intact.
4. **#780 P3 flip** (registry env):
   - `Scripting__Runner=OutOfProcess` and `Scripting__Runner__Seams__0=Calculation`.
   - After the start below, widened to `Scripting__Runner__Seams__1=VirtualTag`. **ScriptedAlarm is not enabled.**
   - Env count is 46 on vq03 and 45 on vq04.
5. **Start** vq04 (16:49:34 → ready 16:50:14, 39 s: it waits for its peer), then vq03 (ready 16:50:23 in 8 s).
   **Pair outage 16:48:13 → 16:50:14, about 2 min**, of which about 70 s was the isolation drill.
6. Cleanup: the clone DB and the preflight copy were removed. The staging dir and backups stay on each node.

## QA verification (#801)

| Check | Result |
|---|---|
| Version | `OtOpcUa.Host.exe` `1.0.0+fea7de0a…` on both. |
| ServiceLevel on the real pair | **vq04 250 (Primary), vq03 240 (Secondary)**, stable over repeated fresh-session reads. ApplicationUri per node unchanged. |
| Health | Both `/health/ready` 200 with opcua-endpoint, script-runner (**`Out-of-process script runner ready.`**), configdb, config-publish and akka Healthy. `/health/active` 200 on the Primary, 503 on the Secondary (by design). |
| LocalDb sync (0.3.7, ApiKey enforced) | `localdb-replication`: **connected, oplog backlog 0** on both nodes, after the window and again after the failover. **Negative control:** a sync call from the Mac with no bearer token and one with a bogus token both get `grpc-status 7 LocalDb sync authentication failed` on both nodes, and each node logs `Rejected a LocalDb sync call`. |
| Restart Secondary (to pick up the VirtualTag seam) | vq03 down → 100 → **240** within about 8 s of start. vq04 held 250 throughout. |
| **Failover** | `Stop-Service` on the Primary vq04 at 16:53:42. The graceful leave took 30 s, and vq03 became oldest at 16:54:13 (`OldestChanged … -> myself`). From the first sample: **vq03 250, `/health/active` 200**. |
| Old Primary restart | vq04 started 16:55:32: **100 → 240** at its first sample after ready, and never above 240 (no 255, #885). **No failback**: vq03 stays 250 / active 200, vq04 240 / active 503. |
| AdminUI | Login posture unchanged: `/` → 302 `/login`, `/login` 200, `/api/deployments` 302. |
| Boot logs | `roles=[admin,driver]`, the #815 preflight line, `Historian gateway probe: the gateway is reachable`. No FTL. The ERR lines are bootstrap-time Akka `Removed` (vq04 before join) and the CLI certificate-domain noise. |
| Overview (vq03 + vq04 `:5320`) | OtOpcUa **Up**, 2 instances, 2 members / 0 unreachable, 5/5 checks healthy (now including opcua-endpoint and script-runner). |
| ScadaBridge QA | `/health/ready` 200 on both nodes before and after. Its only data connection is `MxGateway`, so **it reads no OtOpcUa data on QA**. The consumer check is not applicable here. |
| AVEVA history via HG | **Not applicable on QA**: no Galaxy driver, no historized tags. The real-AVEVA gate passed separately against vd03's gateway (#801 comment, `008f0b36`). |

**Not exercisable on QA (empty configuration):**
- No Calculation, VirtualTag or scripted-alarm exists, so no worker has started on either node. The following were
  therefore not observed: the restricted-token worker under the real service session, the window-station grant, the
  PE-cache fill, the firewall drop log, and "Calculation tags stay Good".
- The runaway drill was **skipped**: running it needs a script in the configuration, and authoring QA configuration
  was out of scope for this deploy.
- `script-runner` reports `Out-of-process script runner ready` on both nodes.

**Isolation coverage note.** The installer denies the worker identity on `InstallRoot` (`app\`) only. On QA the
siblings `E:\ApiInstall\OtOpcUa\pki` (including `own\private\*.pfx`) and `data\` (`secrets.db`, LocalDb) inherit
`BUILTIN\Users:ReadAndExecute`. Users is one of the worker token's restricting SIDs, so those folders are not covered
by the deny. File-level ACLs were not inspected. Follow-up: #890.

**#890 fix: re-applied on vq03 and vq04 later the same day** (see "Canary, live isolation check and runaway drill" below, which also adds the KEK folder to the list). The original instructions follow.
The installer now takes `-DenyPaths` and records them. Once the scripts at that commit are staged on each node, run the
following in an elevated Windows PowerShell 5.1 session. The services can stay up: only folder ACLs change for the worker
identity, and no worker runs on QA today.

```powershell
cd <staged scripts\install>
$deny = 'E:\ApiInstall\OtOpcUa','E:\ApiInstall\OtOpcUa\pki','E:\ApiInstall\OtOpcUa\data'
.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny -WhatIf
.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny
.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny
```

Before running it, check `Secrets__SqlitePath`, its key file and `LocalDb__Path` in the service environment. Add any folder
that sits outside `E:\ApiInstall\OtOpcUa` to `$deny`.

Expected results:
- **First apply: 11 steps, 5 `changed`.** They are `DenyPathRecord`, `InstallRootDeny` (mask `0x1` → `0xD0157`, read + write)
  and the three `SensitiveDirDeny`. The other 6 steps are `unchanged`.
- **Second apply: 11/11 `unchanged`.**
- **Check:**
  - `icacls E:\ApiInstall\OtOpcUa`, `…\pki` and `…\data` and `…\app` each show
    `NT SERVICE\OtOpcUaScriptWorker:(OI)(CI)(DENY)(W,D,WDAC,WO,RD,DC)`, which is what windev's PowerShell 5.1 renders for
    `0xD0157`.
  - `reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths` lists the three folders.
  - The `coverage:` line reports the number of protected items the deny does not reach. Inspect each one WARNed (for
    example a `pki\own\private` file the SDK wrote with its own DACL).

Rollback stays `… -ConfigureHostService -Uninstall`. It now also removes the deny from the recorded folders.

## Configuration set (service registry environment; no file changes, no secrets added)

| Node | Added |
|---|---|
| vq03, vq04 | `Scripting__Worker__Identity=NT SERVICE\OtOpcUaScriptWorker`, `Scripting__Worker__PeCacheDir=C:\ProgramData\OtOpcUa\script-pe-cache` (installer), `Scripting__Runner=OutOfProcess`, `Scripting__Runner__Seams__0=Calculation`, `Scripting__Runner__Seams__1=VirtualTag` |
| vq03, vq04 | Kept, already present and identical: `Security__ApiKeys__Pepper`, `LocalDb__Replication__ApiKey` |
| ConfigDb (QA) | `EnableSnapshotIsolation` applied |
| vd03 | No change |

## Rollback

- **Runner only:** set `Scripting__Runner=InProcess` (or remove the four `Scripting__Runner*` entries) and restart the
  node. To remove the isolation, run `Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app
  -HostServiceAccount LocalSystem -ConfigureHostService -Uninstall`. After the #890 re-apply, this also removes the recorded
  `-DenyPaths` denies, so there is no need to repeat them.
- **QA binaries:** run `E:\psrem-stage\otopcua-fea7de0a\rollback.ps1` on **both** nodes (stop both first), then start both.
  The script does three things:
  1. `app` → `app.failed-fea7de0a`, and `app.bak.20261112-pre-fea7de0a` → `app`;
  2. mirrors `backup\data-pre-fea7de0a` back over `data\`;
  3. imports `backup\OtOpcUaHost-pre-fea7de0a.reg`, which restores the pre-deploy environment, without the Scripting entries.

  The isolation ACEs, firewall rule and anchor service are harmless to `b8b4c18f`. No schema undo is needed.
  `OtOpcUaConfig_pre-fea7de0a_20261112.bak` is the DB fallback.
- **DEV:** `E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1` (→ `5fb4b4db`).
- SDK 1.5 tooling caveat after a rollback: see the rollout runbook §Rollback item 4 (#880).

## Canary, live isolation check and runaway drill (2026-11-12, 18:49–19:07 EST)

Owner request the same day: "add the canary tag and finish QA". WinRM was used only for the on-box steps below
(installer, SQL role grant, live-token probe, log check of the drill). Everything else ran over HTTP / opc.tcp / SMB
from the Mac. No service restart; the pair stayed at **vq03 250 (Primary) / vq04 240** throughout.

### 1. #890 re-apply (both nodes, services up)

Scripts `scripts/install/{Install-ScriptWorkerIsolation.ps1,ScriptWorkerIsolation.psm1}` at `f75ee7d0` (SHA-256 `E0C11AEA…`,
`F9425E4C…`), staged over SMB to `E:\psrem-stage\otopcua-890-f75ee7d0\install`, hashes checked on each box.

- **Path check (service env):** `Secrets__SqlitePath=E:\ApiInstall\OtOpcUa\data\secrets.db` and
  `LocalDb__Path=E:\ApiInstall\OtOpcUa\data\otopcua-localdb.db` are inside. **`Secrets__MasterKey__FilePath=C:\ProgramData\ZB\kek\otopcua.kek`
  is outside**, so `C:\ProgramData\ZB\kek` was added. That folder is already protected (SYSTEM + Administrators only),
  so the deny there is defence in depth.
- `$deny = 'E:\ApiInstall\OtOpcUa','E:\ApiInstall\OtOpcUa\pki','E:\ApiInstall\OtOpcUa\data','C:\ProgramData\ZB\kek'`, then
  `-WhatIf` (12 operations, nothing changed) → apply → apply.
- **First apply: 12 steps, 7 `changed`:** `DenyPathRecord`, `InstallRootDeny` (`0x1` → `0xD0157`), four `SensitiveDirDeny`,
  and **`HostServiceEnv`**. The last one is a reorder only: the earlier P3 flip had appended the `Scripting__Runner*`
  entries after `Scripting__Worker__Identity`/`PeCacheDir`, and the step rewrites the list with its two entries last.
  Same values, env count unchanged (46 / 45). **Second apply: 12/12 `unchanged`** on both nodes.
- **Coverage:** vq03 `5 denied folder(s); 0 protected item(s)`. vq04 WARNs that `C:\ProgramData\ZB\kek\historiangw.kek` and
  `mxgateway.kek` have protected DACLs. The live probe below shows the worker cannot read either.
- **State, both nodes:** `icacls` on `E:\ApiInstall\OtOpcUa`, `pki`, `data`, `app` and `C:\ProgramData\ZB\kek` shows
  `NT SERVICE\OtOpcUaScriptWorker:(OI)(CI)(DENY)(W,D,WDAC,WO,RD,DC)`; `app\script-worker` keeps the explicit `(OI)(CI)(RX)`
  ahead of the inherited deny. `reg query …\OtOpcUaScriptWorker\Parameters /v DenyPaths` lists the four folders. Anchor
  DISABLED, firewall rule Outbound/Block on the worker exe. `/health/ready` 200 on both afterwards.

### 2. Canary configuration (authored in the AdminUI, deployed with "Deploy current configuration")

The QA AdminUI cannot be used from a browser over plain HTTP (its auth cookie is `Secure`), and QA had no LDAP
group → role grant. For this run only: a local forwarder `localhost:19003 → wonder-app-vq03:9000` (Chrome treats
`http://localhost` as secure), and a **temporary** `LdapGroupRoleMapping` row (`SCADA-Admins` → `Administrator`,
system-wide, on-box SQL with the service's own connection) for the box GLAuth test login. **The row was deleted at the
end** (QA again has 0 role mappings; the same login now gets AccessDenied).

| Object | Value |
|---|---|
| Scripts | `ScriptRunnerCanary-Calc` (`SC-82e11d9323ff`): `return (int)DateTime.UtcNow.TimeOfDay.TotalSeconds;` · `ScriptRunnerCanary-VT` (`SC-c2ec9b4c4c53`): `return DateTime.UtcNow.Second;` |
| Raw | Calculation driver `script-runner-canary` (under the QA cluster) → `Device1` → tag `canary-calc-seconds`, Int32, Read, `{"scriptId":"SC-82e11d9323ff","changeTriggered":true,"timerIntervalMs":1000}` → `ns=2;s=script-runner-canary/Device1/canary-calc-seconds` |
| UNS | area `canary` (`area-canary`) → line `script-runner` (`line-canary`) → equipment `script-runner-canary` (`EQ-ec3979550d90`, MachineCode `SCRIPT_RUNNER_CANARY`) → virtual tag `canary-vt-second` (`VT-script-runner-canary`), Int32, timer 1000 ms → `ns=3;s=EQ-ec3979550d90/canary-vt-second` |
| Deployment | `bcaba714` rev `82dd0598c37f…` Sealed, both nodes in sync (the first deployment ever on QA) |

**Result:** both values **Good and advancing once a second on both endpoints** (`opc.tcp://10.220.158.2:4840` and
`.3:4840`, anonymous, read by the OtOpcUa CLI). Both nodes run the drivers, so no failover was needed to see it.
Workers started on the first evaluation: `otopcua_script_worker_up{shard=0,1}` = 1, `runner_state` 0, two PE entries
in `C:\ProgramData\OtOpcUa\script-pe-cache\pe\` on each node (listed over SMB).

**Kept on QA as a runner monitor** (the design and #780 say nothing about removing it): a Bad or frozen
`canary-calc-seconds` / `canary-vt-second`, or `otopcua_script_worker_up` = 0, means the out-of-process runner is sick.

### 3. #794 step 5: the live worker under the real service session (both nodes)

Probe: an admin PowerShell on the box duplicated the **live worker's own token** (opened under the host's LocalSystem
token, since an admin cannot open it directly) and impersonated it for file opens. The same reads were then repeated
without impersonation as the positive control (all READ-OK).

| Check | vq03 and vq04 (identical) |
|---|---|
| Worker processes | 2 per node, `…\app\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`, parent = `OtOpcUa.Host.exe` |
| Token | user `NT AUTHORITY\SYSTEM`, **restricted = True**, privileges **only `SeChangeNotifyPrivilege`**, restricting SIDs `NT SERVICE\OtOpcUaScriptWorker, Everyone, BUILTIN\Users, NT AUTHORITY\RESTRICTED` (host token: unrestricted, 28 privileges) |
| Window-station grant | works under the real service session: both shards run and evaluate (no `ERROR_DLL_INIT_FAILED`) |
| **#890 paths, read** | **DENIED:** `pki\own\private\*.pfx`, `pki\trusted\certs\*.der`, `data\secrets.db`, `data\otopcua-localdb.db`, `app\appsettings.json`, `app\OtOpcUa.Host.dll`, `app\logs\scripts-*.log`, `app.bak.…\appsettings.json`, `C:\ProgramData\ZB\kek\{otopcua,historiangw,mxgateway}.kek`, `script-pe-cache\pe-cache.key` |
| **#890 paths, write** | **DENIED:** create a file in `pki\trusted\certs`, `data`, `app`, `app\script-worker` |
| Positive controls | READ-OK: `app\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.dll`, `C:\Windows\System32\kernel32.dll` |
| Network | 0 TCP / 0 UDP endpoints owned by either worker pid; rule `OtOpcUa-ScriptWorker-BlockOutbound` Outbound/Block/Any/enabled and its program equals the running image path |
| PE cache | filled on first use (above); after the drill's worker restarts `otopcua_script_compile{cache="hit"}` = 4 on each node |

Not done: the dropped-packet firewall log. The worker has no code path that opens a socket, so the log would be empty
whether or not the rule worked. Turning on profile-wide logging was not worth it for that.

**Finding (outside #890's scope, filed as #891):** the worker token **can** read some Users-readable files outside
OtOpcUa's folders: `E:\psrem-stage\otopcua-fea7de0a\backup\OtOpcUaHost-pre-fea7de0a.reg` (a registry export of the
OtOpcUaHost environment, which holds secrets), and `E:\ApiInstall\ScadaBridge\app\appsettings.json` /
`E:\ApiInstall\HistorianGateway\app\appsettings.json`. `C:\ProgramData\ZB\qa-build\*` is denied (protected ACL). Those
folders belong to other apps or to deploy staging, so no deny was added there in this run. Follow-up: #891.

### 4. #780 runaway drill (live gate: "a QA flip with the same drill")

A second script `ScriptRunnerCanary-RunawayDrill` = `while (true) { }` bound to VT `canary-runaway-drill` (Int32, timer
1000 ms) on the canary equipment, deployed as `e746cc5f` (rev `43bd8917…`). Seams: Calculation + VirtualTag.

| Observed | vq03 | vq04 |
|---|---|---|
| Drill VT | `BadInternalError` from the first sample (00:02:59Z) | same |
| Kills | 19:03:03.32 and 19:03:08.31 EST: `exceeded its hard deadline (4000 ms); killing ScriptWorker[0]` → `ended: "KilledDeadline" … culprit 1:VT-script-runner-canary-runaway, 1 evaluation(s) … in flight` | 19:03:03.32 and 19:03:08.27, same lines |
| Quarantine | `Script "1:VT-script-runner-canary-runaway" quarantined: "it exceeded its hard deadline and its worker was killed (2 time(s) in the last hour)…"`; `otopcua_script_quarantined_total{reason="killed"}` 1, `worker_exits_total{reason="killed_deadline",shard="0"}` 2, `run_refused` 4 | same |
| Shard 0 restart | back in about 0.5 s each time | same |
| Siblings | `canary-vt-second` 77 updates, `canary-calc-seconds` 88 updates over the drill, **all Good** | 74 / 88, all Good |
| Node | `/health/ready` 200, `runner_state` 0, `/health/active` 200 | `/health/ready` 200, `/health/active` 503 (Secondary) |
| ServiceLevel | **250** on all 21 samples | **240** on all 21 samples |

**Cleanup:** the drill VT and its script were deleted in the AdminUI and the configuration redeployed as `5905003e`. Its
revision is **`82dd0598…`, the same as the canary-only deployment `bcaba714`**. The drill node now reads `BadNodeIdUnknown`
and both canary values are Good on both nodes.

**ScriptedAlarm seam: left off.** No canary scripted alarm was added: alarms go last (§14 #10), and the alarm drill
already passed on docker-dev (§12.4).

### Final QA state (19:07 EST)

- vq03 **250** Primary, `/health/active` 200. vq04 **240** Secondary, 503. Both `/health/ready` 200 with 5/5 components.
  Overview: OtOpcUa Up, 2 members / 0 unreachable.
- `Scripting__Runner=OutOfProcess`, seams Calculation + VirtualTag. Isolation with the #890 denies on both nodes.
- ConfigDb: the canary only (1 driver, 1 device, 1 tag, 1 VT, 2 scripts, 1 area/line/equipment, 0 scripted alarms, 0 role
  mappings, 0 API keys). Three sealed deployments; `5905003e` is current.
- ScadaBridge, mxgw and HistorianGateway on QA were not touched.
- **Rollback of this run:** remove the canary by deleting its rows in the AdminUI and deploying (or restore
  `OtOpcUaConfig_pre-fea7de0a_20261112.bak`). The isolation rollback is unchanged: `… -ConfigureHostService -Uninstall`
  removes the four recorded denies.

## Box-wide worker denies and backup lockdown (lmxopcua#891, 2026-11-12, owner-approved)

**Owner approval:** the owner approved locking down the registry exports and adding the denies for the other apps' folders. Done on both vq03 and vq04 with the `f75ee7d0` installer staged at `E:\psrem-stage\otopcua-890-f75ee7d0\install`. Order: one `-WhatIf` run, then the apply, then a re-apply that reported all items `unchanged`. No service was restarted.

**1. Registry exports locked down.** These six deploy-backup `.reg` files per box now have a protected DACL allowing only `SYSTEM` and `BUILTIN\Administrators` (FullControl); every other ACE was removed:
- `hg-d8d890c\backup\HistorianGateway-svc-*.reg`
- `mxgw-4f931e4\backup\MxAccessGw-servicekey-*.reg` (2 files)
- `otopcua-b8b4c18f\backup\OtOpcUaHost-service*.reg` (2 files)
- `otopcua-fea7de0a\backup\OtOpcUaHost-pre-fea7de0a.reg`

The files themselves were not deleted, so they still serve as rollback copies.

**2. Box-wide denies.** `-DenyPaths` now lists:
- `E:\ApiInstall\OtOpcUa`, `\pki`, `\data`, and `C:\ProgramData\ZB\kek` (all from before);
- `E:\ApiInstall\ScadaBridge`, `HistorianGateway`, `MxAccessGateway`, `Overview` and `btdbmanager`;
- `E:\psrem-stage` and `E:\psrem`.

`icacls` shows `NT SERVICE\OtOpcUaScriptWorker:(OI)(CI)(DENY)(W,D,WDAC,WO,RD,DC)` on each of these folders, and `reg query` shows the 11 recorded paths. The deny names only the worker identity, so the other apps' service accounts are not affected.

**3. Installer warnings, all checked.**
- The six locked `.reg` files have protected DACLs on purpose.
- On vq04, `C:\ProgramData\ZB\kek\historiangw.kek` and `mxgateway.kek` have protected DACLs that allow only `SYSTEM` and `Administrators`.

None of these grants anything to `Everyone`, `BUILTIN\Users` or `RESTRICTED`. A restricted worker token fails the restricting-SID check on all of them, so they need no deny of their own.

**4. After the change.** `/health/ready` returns 200 on both nodes for OtOpcUa `:9000`, Overview `:5320`, ScadaBridge `:8085` and mxgw `:5130`.

**Deploy procedure from now on.** Write registry exports and appsettings backups into a backup folder that you create with a protected ACL (SYSTEM + Administrators only), and run the installer with the box-wide `-DenyPaths` list above. Any new app folder under `E:\ApiInstall` must be added to the list when it is installed.

**Rollback:** re-run the installer with the previous four-path list to remove the new denies, or use `-Uninstall`. To restore the `.reg` ACLs, run `icacls <file> /reset`.