Upgrade folder swap drops the script-worker isolation ACLs; deploy must re-apply and verify them #901

New Issue

2026-11-13T01:38:04-05:00

dohertj2 commented

2026-11-13 01:38:04 -05:00

Found on the vd03 upgrade fea7de0a → 6c4d2990 (2026-11-13, record docs/deploy/2026-11-13-vd03-fea7de0a-to-6c4d2990.md, 9375c9e2).

What happened. The deploy swapped the install folder (rename the old one, move the new build into place). That dropped three of the #794/#890/#891 worker-isolation permissions: the ones on script-worker\, on the install folder and on pki\. These are set on the folders themselves, and a freshly moved-in folder doesn't carry them. Re-running the isolation installer with the recorded arguments restored them; a second run reported every step unchanged.

Why it matters. Every OtOpcUa upgrade on a box with isolation installed silently weakens it: the QA pair vq03/vq04 and vd03. On QA the out-of-process runner is live, so after a swap the worker token can read paths it is meant to be denied, until someone notices.

To do.

Make the upgrade path re-apply isolation automatically. For example, the deploy pack / upgrade script runs the isolation installer with the recorded arguments after the swap, or the installer's settings are stored on the box so a re-apply needs no arguments.
Add a post-deploy verification step that fails when an expected deny or restriction is missing, using the installer's own check mode if it has one.
Update docs/deploy/ packs and the vd03/QA runbooks.
Check whether the 2026-11-12 QA swap was done before or after isolation was installed. Record it either way.

Found on the vd03 upgrade `fea7de0a` → `6c4d2990` (2026-11-13, record `docs/deploy/2026-11-13-vd03-fea7de0a-to-6c4d2990.md`, `9375c9e2`). **What happened.** The deploy swapped the install folder (rename the old one, move the new build into place). That dropped three of the #794/#890/#891 worker-isolation permissions: the ones on `script-worker\`, on the install folder and on `pki\`. These are set on the folders themselves, and a freshly moved-in folder doesn't carry them. Re-running the isolation installer with the recorded arguments restored them; a second run reported every step unchanged. **Why it matters.** Every OtOpcUa upgrade on a box with isolation installed silently weakens it: the QA pair vq03/vq04 and vd03. On QA the out-of-process runner is live, so after a swap the worker token can read paths it is meant to be denied, until someone notices. **To do.** - Make the upgrade path re-apply isolation automatically. For example, the deploy pack / upgrade script runs the isolation installer with the recorded arguments after the swap, or the installer's settings are stored on the box so a re-apply needs no arguments. - Add a post-deploy verification step that fails when an expected deny or restriction is missing, using the installer's own check mode if it has one. - Update `docs/deploy/` packs and the vd03/QA runbooks. - Check whether the 2026-11-12 QA swap was done before or after isolation was installed. Record it either way.

dohertj2 added the type:bug priority:p2 labels 2026-11-13 01:38:04 -05:00

dohertj2 referenced this issue from a commit

2026-11-13 02:06:21 -05:00

fix(install): worker isolation survives a folder swap: recorded settings, -Reapply, -Verify, swap script (#901)

dohertj2 commented

2026-11-13 02:06:37 -05:00

Fixed on master in 2cb8a2c3 (code + docs only; nothing deployed, nothing changed on vd03 or QA).

What changed

Install-ScriptWorkerIsolation.ps1 now records its effective settings (InstallRoot, HostServiceAccount, PeCacheDir, HostServiceName, ConfigureHostService) beside DenyPaths under HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters (new step SettingsRecord), and sets that key's DACL protected, SYSTEM + Administrators only. No secrets are recorded.
-Reapply: re-applies from the record with no other arguments and prints summary: N step(s), X changed, Y unchanged.
-Verify: check only. Prints [ok]/[MISSING] per step (anchor service, record + key ACL, PE cache ACL, pe\, script-worker\, install-folder deny, every -DenyPaths deny, the firewall rule and the worker exe it names, and the host env entries when they are recorded). Exits 1 with VERIFY FAILED: … when anything is missing. -Uninstall also works from the record now.
New scripts/install/Invoke-OtOpcUaInstallSwap.ps1 runs preflight → stop → move live/candidate → carry appsettings*.json/pki/logs (QA: 'appsettings*.json','logs') → -Reapply → -Verify → start. If -Verify fails, the service stays stopped. With -NoCarry it is also the rollback, and -NoStart covers the QA pair window. It refuses to run, before changing anything, on a node whose isolation was installed by a pre-#901 installer, which recorded only DenyPaths.
Docs updated: docs/ServiceHosting.md (isolation section), the vd03 pack (§e swap + rollback, §f post-deploy gate, one-time step), the 2026-11-13 vd03 record, and the QA pack (new section "Worker isolation across the next swap").

Tests

Dry-run harness, macOS: 101/101. A mutation of the record-key SDDL turns 2 assertions red.
Test-ScriptWorkerIsolation.Windows.ps1 on windev: 52/52 under Windows PowerShell 5.1 and under PowerShell 7, run once with a scratch local host account and once with SYSTEM. Scratch tree, scratch services; everything was cleaned up afterwards. Transcript (5.1):

Verify (installed)                 exit 0; 12 ok
Verify (after the swap)            exit 1; MISSING: WorkerFolderAcl <scratch>\install\script-worker | InstallRootDeny <scratch>\install | SensitiveDirDeny <scratch>\install\pki
Reapply (no arguments)             exit 0; 9 unchanged, 3 changed
Verify (after Reapply)             exit 0; 12 ok
Reapply again                      exit 0; 12 unchanged
Verify (cache ACL + rule broken)   exit 1; MISSING: PeCacheDir … | FirewallBlockOutbound …
Invoke-OtOpcUaInstallSwap          exit 0; VERIFY OK after its Reapply
Invoke-OtOpcUaInstallSwap (pre-#901) exit 1; refused: settings not recorded

Evidence: scripts/validation/opcua-windows-service/evidence/2026-11-13-windev-901-reapply-verify/RESULTS.md.

QA: was the 2026-11-12 swap before or after the isolation install? Before. The window swapped app\ at 16:48. The #794 isolation went onto the new app\ after that (window step 3), and the #890/#891 re-applies came later that evening with no further swap. Checked 2026-11-13, read-only over SMB from the Mac (one DACL query per folder, no remote exec). vq03 and vq04 are identical:

E:\ApiInstall\OtOpcUa, …\app, …\pki, …\data, C:\ProgramData\ZB\kek, E:\ApiInstall\ScadaBridge and E:\psrem-stage each carry the explicit worker Deny 0xD0157 (OI)(CI).
…\app\script-worker has its explicit Allow 0x1200A9; script-pe-cache\pe has its explicit Modify 0x1301BF.
Controls: C:\Windows and the PE-cache root show no worker ACE.

vd03 also reads intact after the 2026-11-13 manual re-apply. The anchor service, the firewall rule and the env entries can't be seen over SMB; -Verify checks them at the next deploy.

What the next deploy must do (vd03 and QA)

One time: both boxes were isolated by the pre-#901 installer, so first run the full-argument install from the new staged scripts. The arguments are in each pack: vd03 is the 5-path line in the vd03 pack, QA is the 11-path #891 list in the QA pack. Check it against reg query …\OtOpcUaScriptWorker\Parameters /v DenyPaths first. Expect SettingsRecord to show changed and every other step unchanged, then -Verify → VERIFY OK.
Swap with Invoke-OtOpcUaInstallSwap.ps1. If you swap by hand instead, run -Reapply and then -Verify before the start.
Post-deploy gate: Install-ScriptWorkerIsolation.ps1 -Verify → exit 0, and record the output.
A rollback swaps the folder back and loses the ACEs the same way, so run the same re-apply and verify (-NoCarry).

Fixed on master in `2cb8a2c3` (code + docs only; nothing deployed, nothing changed on vd03 or QA). **What changed** - `Install-ScriptWorkerIsolation.ps1` now records its effective settings (`InstallRoot`, `HostServiceAccount`, `PeCacheDir`, `HostServiceName`, `ConfigureHostService`) beside `DenyPaths` under `HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters` (new step `SettingsRecord`), and sets that key's DACL protected, SYSTEM + Administrators only. No secrets are recorded. - `-Reapply`: re-applies from the record with no other arguments and prints `summary: N step(s), X changed, Y unchanged`. - `-Verify`: check only. Prints `[ok]`/`[MISSING]` per step (anchor service, record + key ACL, PE cache ACL, `pe\`, `script-worker\`, install-folder deny, every `-DenyPaths` deny, the firewall rule and the worker exe it names, and the host env entries when they are recorded). Exits **1** with `VERIFY FAILED: …` when anything is missing. `-Uninstall` also works from the record now. - New `scripts/install/Invoke-OtOpcUaInstallSwap.ps1` runs preflight → stop → move live/candidate → carry `appsettings*.json`/`pki`/`logs` (QA: `'appsettings*.json','logs'`) → `-Reapply` → `-Verify` → start. If `-Verify` fails, the service stays **stopped**. With `-NoCarry` it is also the rollback, and `-NoStart` covers the QA pair window. It refuses to run, before changing anything, on a node whose isolation was installed by a pre-#901 installer, which recorded only `DenyPaths`. - Docs updated: `docs/ServiceHosting.md` (isolation section), the vd03 pack (§e swap + rollback, §f post-deploy gate, one-time step), the 2026-11-13 vd03 record, and the QA pack (new section "Worker isolation across the next swap"). **Tests** - Dry-run harness, macOS: 101/101. A mutation of the record-key SDDL turns 2 assertions red. - `Test-ScriptWorkerIsolation.Windows.ps1` on windev: **52/52** under Windows PowerShell 5.1 and under PowerShell 7, run once with a scratch local host account and once with SYSTEM. Scratch tree, scratch services; everything was cleaned up afterwards. Transcript (5.1): ``` Verify (installed) exit 0; 12 ok Verify (after the swap) exit 1; MISSING: WorkerFolderAcl <scratch>\install\script-worker | InstallRootDeny <scratch>\install | SensitiveDirDeny <scratch>\install\pki Reapply (no arguments) exit 0; 9 unchanged, 3 changed Verify (after Reapply) exit 0; 12 ok Reapply again exit 0; 12 unchanged Verify (cache ACL + rule broken) exit 1; MISSING: PeCacheDir … | FirewallBlockOutbound … Invoke-OtOpcUaInstallSwap exit 0; VERIFY OK after its Reapply Invoke-OtOpcUaInstallSwap (pre-#901) exit 1; refused: settings not recorded ``` Evidence: `scripts/validation/opcua-windows-service/evidence/2026-11-13-windev-901-reapply-verify/RESULTS.md`. **QA: was the 2026-11-12 swap before or after the isolation install? Before.** The window swapped `app\` at 16:48. The #794 isolation went onto the new `app\` after that (window step 3), and the #890/#891 re-applies came later that evening with no further swap. **Checked 2026-11-13, read-only over SMB from the Mac** (one DACL query per folder, no remote exec). vq03 and vq04 are identical: - `E:\ApiInstall\OtOpcUa`, `…\app`, `…\pki`, `…\data`, `C:\ProgramData\ZB\kek`, `E:\ApiInstall\ScadaBridge` and `E:\psrem-stage` each carry the explicit worker Deny `0xD0157` (OI)(CI). - `…\app\script-worker` has its explicit Allow `0x1200A9`; `script-pe-cache\pe` has its explicit Modify `0x1301BF`. - Controls: `C:\Windows` and the PE-cache root show no worker ACE. vd03 also reads intact after the 2026-11-13 manual re-apply. The anchor service, the firewall rule and the env entries can't be seen over SMB; `-Verify` checks them at the next deploy. **What the next deploy must do (vd03 and QA)** 1. **One time:** both boxes were isolated by the pre-#901 installer, so first run the full-argument install from the new staged scripts. The arguments are in each pack: vd03 is the 5-path line in the vd03 pack, QA is the 11-path #891 list in the QA pack. Check it against `reg query …\OtOpcUaScriptWorker\Parameters /v DenyPaths` first. Expect `SettingsRecord` to show changed and every other step unchanged, then `-Verify` → `VERIFY OK`. 2. Swap with `Invoke-OtOpcUaInstallSwap.ps1`. If you swap by hand instead, run `-Reapply` and then `-Verify` **before** the start. 3. Post-deploy gate: `Install-ScriptWorkerIsolation.ps1 -Verify` → exit 0, and record the output. 4. A rollback swaps the folder back and loses the ACEs the same way, so run the same re-apply and verify (`-NoCarry`).

dohertj2 closed this issue

2026-11-13 02:06:39 -05:00

dohertj2 referenced this issue from a commit

2026-11-13 02:57:07 -05:00

docs(deploy): vd03 + QA vq03/vq04 deployed to c9640f5c (#901 #902 #903 #834 #872 #904)

dohertj2 commented

2026-11-13 02:57:31 -05:00

Deployed to DEV vd03 + QA vq03/vq04 in c9640f5c on 2026-11-13.

One-time full-argument install on each node wrote the settings record: SettingsRecord changed, every other step unchanged (QA: plus a reorder-only HostServiceEnv; sorted env SHA unchanged). DenyPaths unchanged (vd03 5, QA 12 incl. E:\ApiInstall\OtelCollector).
Swaps via Invoke-OtOpcUaInstallSwap.ps1: -Reapply changed exactly WorkerFolderAcl + InstallRootDeny (+ SensitiveDirDeny pki on vd03), then the gate:
- vd03: VERIFY OK: 13/13 step(s) in place. after the swap and again after the start (exit 0).
- vq03 / vq04 (-NoStart): VERIFY OK: 21/21 step(s) in place. after each swap, after each start, and after the failover restart (exit 0).
Note: the coverage walk inside -Reapply/-Verify runs with the service stopped. On vd03 (deny on all of E:\ApiInstall, 48 WARN items) that made the stop→ready window ~101 s; ~50–60 s per QA node.

Record: docs/deploy/2026-11-13-vd03-qa-to-c9640f5c.md (c0ba450d).

Deployed to DEV vd03 + QA vq03/vq04 in `c9640f5c` on 2026-11-13. - One-time full-argument install on each node wrote the settings record: **`SettingsRecord` changed**, every other step unchanged (QA: plus a reorder-only `HostServiceEnv`; sorted env SHA unchanged). DenyPaths unchanged (vd03 5, QA 12 incl. `E:\ApiInstall\OtelCollector`). - Swaps via `Invoke-OtOpcUaInstallSwap.ps1`: `-Reapply` changed exactly `WorkerFolderAcl` + `InstallRootDeny` (+ `SensitiveDirDeny` `pki` on vd03), then the gate: - vd03: `VERIFY OK: 13/13 step(s) in place.` after the swap and again after the start (exit 0). - vq03 / vq04 (`-NoStart`): `VERIFY OK: 21/21 step(s) in place.` after each swap, after each start, and after the failover restart (exit 0). - Note: the coverage walk inside `-Reapply`/`-Verify` runs with the service stopped. On vd03 (deny on all of `E:\ApiInstall`, 48 WARN items) that made the stop→ready window ~101 s; ~50–60 s per QA node. Record: `docs/deploy/2026-11-13-vd03-qa-to-c9640f5c.md` (`c0ba450d`).

dohertj2 referenced this issue

2026-11-13 03:01:45 -05:00

Install swap: -Reapply/-Verify folder walk runs while the service is stopped (vd03 downtime ~101 s vs 20–40 s estimate) #905

dohertj2 referenced this issue

2026-11-13 03:16:15 -05:00

Implement the out-of-process script runner (design #750) — code on master; QA deploy (#793/#794) + P3 flip remain (no shadow week) #780

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#901