fix(install): worker isolation survives a folder swap: recorded settings, -Reapply, -Verify, swap script (#901)

An upgrade that swaps the install folder dropped the explicit ScriptWorker isolation ACEs on script-worker\, the
install folder and a copied-back pki\ (vd03 2026-11-13). Re-applying needed the operator to repeat the arguments.

- Install-ScriptWorkerIsolation.ps1 records its effective settings (InstallRoot, HostServiceAccount, PeCacheDir,
  HostServiceName, ConfigureHostService) beside DenyPaths under ...\Services\OtOpcUaScriptWorker\Parameters (new plan
  step SettingsRecord) and locks that key to SYSTEM + Administrators (protected DACL; no secrets recorded).
- -Reapply: re-apply from the record, no arguments; prints a summary line (changed/unchanged).
- -Verify: check only, [ok]/[MISSING] per step, exit 1 when anything is missing. -Uninstall also works from the record.
  A node isolated by a pre-#901 installer (DenyPaths only) must run the full-argument install once; both modes say so.
- Invoke-OtOpcUaInstallSwap.ps1: preflight -> stop -> move live/candidate -> carry appsettings/pki/logs ->
  -Reapply -> -Verify -> start; a failed -Verify leaves the service stopped. Also the rollback (-NoCarry).
- Tests: dry-run harness 101/101 (record + argument rules; record-key SDDL mutation turns 2 red). The Windows test adds
  swap -> Verify fails (exactly the 3 lost ACEs) -> Reapply (3 changed) -> Verify ok -> Reapply 12/12 unchanged, PE-cache
  and firewall negative controls, the swap script, and the pre-#901 refusal; 52/52 on windev under 5.1 and 7, with a
  scratch local host account and with SYSTEM. Evidence: scripts/validation/opcua-windows-service/evidence/2026-11-13-windev-901-reapply-verify.
- Docs: ServiceHosting isolation section; vd03 pack (swap step, rollback, post-deploy gate, one-time full-argument run),
  the 2026-11-13 vd03 record, and the QA pack (QA swap preceded the isolation install; ACLs checked intact over SMB on
  vq03/vq04 2026-11-13; next-deploy procedure).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

docs/deploy/2026-11-13-vd03-fea7de0a-to-6c4d2990.md

@@ -56,6 +56,10 @@ Staged in `E:\ApiInstall\_stage-otopcua-6c4d2990\` (`prepare.ps1`, `swap.ps1`, `
   (`E:\ApiInstall\OtOpcUa`) and `SensitiveDirDeny` (`…\OtOpcUa\pki`) applied as **changed**; the immediate re-apply
   reported **every step unchanged**. The runner stays `InProcess`; no restart needed. The same 6 pre-existing
   "protected DACL" warnings under `E:\psrem-stage` as before.
+  **Follow-up: lmxopcua#901.** The re-apply was a manual step with recorded arguments. From #901 the installer records its
+  settings and has `-Reapply` / `-Verify`, and `scripts/install/Invoke-OtOpcUaInstallSwap.ps1` runs both after the swap
+  (service left stopped if `-Verify` fails). Next vd03 deploy: run the full-argument install once to write the record, then
+  swap with the script and gate on `-Verify` ([`vd03-upgrade-b8b4c18f-to-master.md`](vd03-upgrade-b8b4c18f-to-master.md) §e/§f).
 
 ## Verification
 
@@ -81,7 +85,8 @@ Details: [evidence, second run](../plans/evidence/2026-11-13-historian-700-vd03/
 
 `E:\ApiInstall\_stage-otopcua-6c4d2990\rollback.ps1`: stop; move the live folder to `E:\ApiInstall\OtOpcUa.failed-6c4d2990`;
 restore `E:\ApiInstall\OtOpcUa.bak.20261113-pre-6c4d2990` (`fea7de0a`); restore `OtOpcUa-data` from the pre-deploy backup;
-start; re-apply the worker isolation. No schema undo (no migration in range). The ConfigDb `.bak` is there if the
+start; re-apply the worker isolation. (From #901 the re-apply comes before the start, as `-Reapply` then `-Verify`:
+`Invoke-OtOpcUaInstallSwap.ps1 -Candidate <bak> -PreviousInstall <failed> -NoCarry` does it.) No schema undo (no migration in range). The ConfigDb `.bak` is there if the
 configuration itself must go back.
 
 ## Observed (not deploy blockers)

docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md

@@ -5,6 +5,10 @@
 > ScriptWorker isolation installed. Tracking: #801 (SDK 2 live gate), #793 (ConfigDb preflight), #794 (worker
 > isolation install), #780 (P3 runner flip). Template: [`vd03-upgrade-b8b4c18f-to-master.md`](vd03-upgrade-b8b4c18f-to-master.md).
 >
+> **lmxopcua#901 (2026-11-13):** a folder swap drops the worker-isolation folder ACEs. The installer now records its
+> settings and has `-Reapply` / `-Verify`; `scripts/install/Invoke-OtOpcUaInstallSwap.ps1` runs both after the swap. QA's
+> ACLs were checked intact over SMB; see "Worker isolation across the next swap" at the end for the next QA deploy.
+>
 > **DEV vd03 moved on 2026-11-13:** `fea7de0a` → `6c4d2990` (#899 read-only historian bindings; #700 checks). Record:
 > [`2026-11-13-vd03-fea7de0a-to-6c4d2990.md`](2026-11-13-vd03-fea7de0a-to-6c4d2990.md). QA is unchanged.
 
@@ -207,6 +211,8 @@ Rollback stays `… -ConfigureHostService -Uninstall`. It now also removes the d
   3. imports `backup\OtOpcUaHost-pre-fea7de0a.reg`, which restores the pre-deploy environment, without the Scripting entries.
 
   The isolation ACEs, firewall rule and anchor service are harmless to `b8b4c18f`. No schema undo is needed.
+  **The rollback swaps `app\` back, which drops the folder ACEs (lmxopcua#901):** re-apply with `-Reapply` and gate with
+  `-Verify` before starting (see "Worker isolation across the next swap" below).
   `OtOpcUaConfig_pre-fea7de0a_20261112.bak` is the DB fallback.
 - **DEV:** `E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1` (→ `5fb4b4db`).
 - SDK 1.5 tooling caveat after a rollback: see the rollout runbook §Rollback item 4 (#880).
@@ -351,4 +357,52 @@ None of these grants anything to `Everyone`, `BUILTIN\Users` or `RESTRICTED`. A
 
 **Deploy procedure from now on.** Write registry exports and appsettings backups into a backup folder that you create with a protected ACL (SYSTEM + Administrators only), and run the installer with the box-wide `-DenyPaths` list above. Any new app folder under `E:\ApiInstall` must be added to the list when it is installed.
 
+## Worker isolation across the next swap (lmxopcua#901)
+
+**Was the 2026-11-12 QA swap before or after the isolation?** Before. The window swapped `app\` at 16:48:13–16:48:20 and
+installed the #794 isolation on the new `app\` afterwards (QA window step 3, services stopped); the #890 and #891 re-applies
+ran later the same evening, with no further swap. So nothing on QA was lost to a swap. **Checked 2026-11-13 from the Mac,
+read-only over SMB** (a DACL query per folder, no remote exec), both vq03 and vq04 identical:
+
+| Folder | Worker identity ACEs |
+|---|---|
+| `E:\ApiInstall\OtOpcUa` | explicit Deny `0xD0157` (OI)(CI) |
+| `…\OtOpcUa\app` | explicit Deny `0xD0157` + the inherited one |
+| `…\app\script-worker` | explicit Allow `0x1200A9` (RX) (OI)(CI) ahead of the inherited deny |
+| `…\OtOpcUa\pki`, `…\OtOpcUa\data` | explicit Deny `0xD0157` + inherited |
+| `C:\ProgramData\ZB\kek` (protected) | explicit Deny `0xD0157` |
+| `C:\ProgramData\OtOpcUa\script-pe-cache\pe` | explicit Allow `0x1301BF` (Modify) |
+| `E:\ApiInstall\ScadaBridge`, `E:\psrem-stage` (#891 box-wide) | explicit Deny `0xD0157` |
+
+Controls: `C:\Windows` and the PE-cache root show no worker ACE (as designed). The anchor service, the firewall rule and
+the host env entries are not visible over SMB; `-Verify` checks them at the next deploy.
+
+**Next QA deploy (both nodes, stop-both / start-both):**
+
+1. **One time, before the first swap with the #901 scripts:** QA was isolated by the older installer, which recorded only
+   `DenyPaths`. Run the full-argument install once from the staged `scripts\install` to write the settings record (and
+   lock its key to SYSTEM + Administrators). The `-DenyPaths` list is the #891 box-wide one:
+   ```powershell
+   $deny = 'E:\ApiInstall\OtOpcUa','E:\ApiInstall\OtOpcUa\pki','E:\ApiInstall\OtOpcUa\data','C:\ProgramData\ZB\kek',
+           'E:\ApiInstall\ScadaBridge','E:\ApiInstall\HistorianGateway','E:\ApiInstall\MxAccessGateway','E:\ApiInstall\Overview',
+           'E:\ApiInstall\btdbmanager','E:\psrem-stage','E:\psrem'
+   .\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -DenyPaths $deny
+   .\Install-ScriptWorkerIsolation.ps1 -Verify     # VERIFY OK
+   ```
+   Check the list against `reg query HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters /v DenyPaths`
+   first and use exactly what is recorded. Expect `SettingsRecord` changed, every other step unchanged.
+2. **Swap each node with the gate built in**, services stopped, then start both:
+   ```powershell
+   .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.new-<sha> `
+       -PreviousInstall E:\ApiInstall\OtOpcUa\app.bak.<yyyymmdd>-pre-<sha> -Carry 'appsettings*.json','logs' -NoStart
+   ```
+   It stops the service, swaps, carries the box files, runs `-Reapply` and `-Verify`, and throws (service still stopped) if
+   `-Verify` fails. On QA `pki\` and `data\` sit beside `app\`, so only `app\` and its `script-worker\` lose their explicit
+   ACEs; expect the re-apply to report `WorkerFolderAcl` and `InstallRootDeny` changed. If the swap is done by hand instead,
+   run `-Reapply` then `-Verify` right after it, **before** starting the service.
+3. **Post-deploy gate:** `Install-ScriptWorkerIsolation.ps1 -Verify` on each node → `VERIFY OK`, exit 0. Record the output.
+4. **Rollback** swaps the old `app\` back, so it loses the ACEs the same way: `Invoke-OtOpcUaInstallSwap.ps1 -Candidate
+   …\app.bak.<…> -PreviousInstall …\app.failed-<sha> -NoCarry -NoStart`, then restore `data\`/the registry as before, then
+   start both. Without the script: `-Reapply` + `-Verify` after the folder moves.
+
 **Rollback:** re-run the installer with the previous four-path list to remove the new denies, or use `-Uninstall`. To restore the `.reg` ACLs, run `icacls <file> /reset`.

docs/deploy/vd03-upgrade-b8b4c18f-to-master.md

@@ -207,7 +207,21 @@ The source is `docs/plans/2026-11-10-opcua-sdk-2-rollout.md` (§Sequence, §Roll
    - `Move-Item` the live install to a fresh sibling `$PreviousInstall`, then `Move-Item` the candidate into
      `E:\ApiInstall\OtOpcUa`.
    - Copy back the box's `appsettings*.json`, `pki\` and `data\`.
+   - **Re-apply and verify the worker isolation before the start (lmxopcua#901).** The swap drops the explicit
+     isolation ACEs on `script-worker\`, `E:\ApiInstall\OtOpcUa` and the copied-back `pki\`. Run
+     `Install-ScriptWorkerIsolation.ps1 -Reapply`, then `-Verify` (exit 1 = do not start; fix and re-run).
    - `Start-Service` and wait for Running.
+
+   **From #901 on, use `scripts/install/Invoke-OtOpcUaInstallSwap.ps1`** for the stop / move / carry / re-apply / verify /
+   start sequence (it leaves the service stopped when `-Verify` fails):
+   `.\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.new-<sha> -PreviousInstall E:\ApiInstall\OtOpcUa.bak.<yyyymmdd>-pre-<sha>`
+   (default `-Carry` = `appsettings*.json`, `pki`, `logs`; vd03's `data\` lives outside the install in `E:\ApiInstall\OtOpcUa-data`).
+   **One-time step at the first deploy with the #901 installer:** vd03 was isolated by the older installer, which recorded only
+   `DenyPaths`, so the swap script refuses until the record is written. Before the swap, run the full-argument install once
+   (§"Worker isolation and box-wide denies" below; it writes the settings record and locks the key):
+   `.\Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -HostServiceAccount LocalSystem -DenyPaths E:\ApiInstall,E:\ApiInstall\OtOpcUa\pki,E:\ApiInstall\OtOpcUa-data,C:\ProgramData\ZB\kek,E:\psrem-stage`
+   (add `-ConfigureHostService` only if the vd03 runner flip is part of that deploy). Expect `DenyPathRecord` unchanged and
+   `SettingsRecord` changed; then `-Verify` must print `VERIFY OK`.
 5. **Never run `Install-Services.ps1`, `Uninstall-Services.ps1` or `Refresh-Services.ps1` here.**
    - `sc delete` wipes the 23 environment variables.
    - `Install-Services.ps1` rewrites the `Environment` value with only 2–3 entries.
@@ -216,7 +230,9 @@ The source is `docs/plans/2026-11-10-opcua-sdk-2-rollout.md` (§Sequence, §Roll
 6. **Rollback.**
    - Stop the service and move the candidate to a quarantine path.
    - Restore `$PreviousInstall` (or the robocopy backup), including its `appsettings*.json`, `pki\` and `data\`.
-   - Check the certificate thumbprint, then start the service.
+   - Check the certificate thumbprint, re-apply and verify the worker isolation (`-Reapply`, `-Verify`; or run the
+     rollback as `Invoke-OtOpcUaInstallSwap.ps1 -Candidate <previous install> -PreviousInstall <quarantine path> -NoCarry`),
+     then start the service.
    - Clients reconnect. Do not replay writes or acks.
 
    No schema undo is needed (§a). Remove the pepper only if it got in the way; `b8b4c18f` ignores it. The
@@ -228,6 +244,9 @@ The source is `docs/plans/2026-11-10-opcua-sdk-2-rollout.md` (§Sequence, §Roll
 
 ## (f) Post-deploy checks
 
+- **Worker isolation gate (lmxopcua#901), first.** `Install-ScriptWorkerIsolation.ps1 -Verify` from the staged
+  `scripts\install` (elevated) must print `VERIFY OK` and exit 0. Exit 1 lists each `[MISSING]` step: run `-Reapply`,
+  then `-Verify` again. Do this even when the runner is `InProcess` (vd03 today): the denies also cover the box-wide folders.
 - **Boot log** `logs\otopcua-YYYYMMDD.log`. Read its contents; the listed size shows 0 while the service holds
   it open.
   - Expect `OtOpcUa.Host starting with roles=[admin,driver]`, with no roles-disagreement warning.
@@ -407,3 +426,8 @@ None of them grants anything to Everyone, Users or RESTRICTED.
 **After:** `/health/ready` returns 200 for OtOpcUa `:9000`, Overview `:5320`, ScadaBridge `:8085`, mxgw `:5130` and HG `:5220`.
 
 **Rollback:** `-Uninstall`, which removes the recorded denies, the anchor service, the firewall rule and the PE cache ACLs. To undo the lockdown on a backup file, run `icacls <file> /reset`.
+
+**Every later swap (lmxopcua#901).** The 2026-11-13 upgrade (`6c4d2990`) showed that the folder swap drops `WorkerFolderAcl`,
+`InstallRootDeny` and the `pki\` `SensitiveDirDeny`; re-running the line above restored them. From the #901 installer on, the
+settings are recorded, so a swap is followed by `-Reapply` + `-Verify` with no arguments, or the whole swap runs through
+`Invoke-OtOpcUaInstallSwap.ps1` (§e step 4). The first #901 run on vd03 must be the full-argument line above (one time).