fix(install): cut the swap's stopped window to the swapped-in folder (#905)
v2-ci / unit-tests (push) Has been cancelled

Root cause: the protected-DACL coverage scan, not the ACL writes. Both -Reapply and
-Verify walked every denied folder (all of E:\ApiInstall on vd03, nested roots again)
with Get-ChildItem -Recurse + Get-Acl per item + a pipeline per item for -Skip, about
3.4 ms an item on PS 5.1, all while the service was stopped. An unchanged deny is read,
not rewritten; the re-applied install-folder deny propagates over ~1,400 files in 0.5 s.

- Find-OtOpcUaDenyCoverageGap: .NET enumeration, -Skip folders pruned, DACL-only
  GetAccessControl(Access) (about 0.15 ms an item); reparse points not followed.
- Get-OtOpcUaCoverageWalkRoots: each denied folder walked once (outermost only).
- Install-ScriptWorkerIsolation.ps1 -Scope All|InstallRoot|Outside (with -Reapply,
  -Verify, -PlanOnly). InstallRoot = the folder ACEs a swap drops; Outside = the rest,
  including the firewall rule (its query costs 3-5 s; its exe path does not change).
  The two partition the plan. Prints the scan time.
- Invoke-OtOpcUaInstallSwap.ps1: -Reapply/-Verify -Scope Outside before the stop (a
  failure stops before anything moves), -Scope InstallRoot inside the window; preflight
  checks the candidate carries the worker exe; prints the stopped window and its parts.
  -NoStart, -NoCarry and the pre-#901 refusal unchanged.

windev, vd03-like tree (17,851 items): stopped window 153.1 s -> 6.0 s (PS 5.1),
101.6 s -> 6.2 s (PS 7); the Outside part takes ~23 s with the old build serving.
Dry-run harness 101 -> 123 passed; Windows test 72 passed on PS 5.1 and PS 7.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 03:51:41 -05:00
parent f817c77e02
commit 0c19f26c81
8 changed files with 475 additions and 58 deletions
+26 -5
View File
@@ -138,25 +138,46 @@ Two argument-free modes use the record:
|---|---|
| `Install-ScriptWorkerIsolation.ps1 -Reapply` | Re-applies with the recorded settings; one `[changed]`/`[unchanged]` line per step and a `summary:` line. After a swap: the three folder ACEs `changed`, the rest `unchanged`; a second run all `unchanged`. |
| `Install-ScriptWorkerIsolation.ps1 -Verify` | Check only. `[ok]`/`[MISSING]` per step (anchor service, record + key ACL, PE cache ACL, `pe\`, `script-worker\`, install-folder deny, every `-DenyPaths` deny, firewall rule and the worker exe it names, the host env entries when recorded). **Exit 1** with `VERIFY FAILED: …` when anything is missing, exit 0 with `VERIFY OK`. Protected-DACL coverage gaps are WARNed, not failed. |
| `… -Reapply -Scope InstallRoot` / `-Verify -Scope InstallRoot` | (lmxopcua#905) Only what a swap can undo: the ACEs on the install folder, `script-worker\` and a denied folder inside it (vd03's `pki\`). The coverage scan walks the install folder only. |
| `… -Reapply -Scope Outside` / `-Verify -Scope Outside` | Every other step (anchor service, records, PE cache, denies outside the install folder, firewall rule, host env). The firewall rule names the worker exe by a path a swap does not change, and its query costs 3–5 s; the swap checks in its preflight that the candidate carries the exe. The scan walks the denied folders and skips the install folder. `InstallRoot` + `Outside` = the full plan. |
Both refuse to run without a full record. A node isolated by an installer older than #901 recorded only `DenyPaths`:
run the install once with its full arguments (the per-box deploy record lists them) to write the record.
`scripts/install/Invoke-OtOpcUaInstallSwap.ps1` is the swap with the gate built in: preflight (candidate carries
`script-worker\`, `PreviousInstall` is fresh and on the same volume, isolation record present when isolation is
installed, record `InstallRoot` equals `-InstallRoot`) → stop → move live → previous, candidate → live → copy `-Carry`
items back (default `appsettings*.json`, `pki`, `logs`; QA: `'appsettings*.json','logs'`; rollback: `-NoCarry`) →
`-Reapply` → `-Verify` → start (`-NoStart` for a pair window). A failed `-Verify` leaves the service **stopped**. Backups,
installed, record `InstallRoot` equals `-InstallRoot`) → `-Reapply -Scope Outside` + `-Verify -Scope Outside` **while
the old build still serves** (a failure stops here: nothing stopped or moved) → stop → move live → previous, candidate →
live → copy `-Carry` items back (default `appsettings*.json`, `pki`, `logs`; QA: `'appsettings*.json','logs'`; rollback:
`-NoCarry`) → `-Reapply -Scope InstallRoot` → `-Verify -Scope InstallRoot` → start (`-NoStart` for a pair window). The
two scoped gates together check every install step, so the new build never starts unverified; a failed post-swap
`-Verify` leaves the service **stopped**. The script prints the stopped window and its parts
(`stopped window (stop issued -> start issued): 6.0 s [stop …, move 0.0 s, carry 0.1 s, isolation 5.8 s]`). Backups,
`data\` restore, migrations and health checks stay in the deploy's own steps. Every deploy pack that swaps the folder runs
it (or `-Reapply` + `-Verify` right after its own swap), and `-Verify` is a post-deploy gate.
`Test-ScriptWorkerIsolation.Windows.ps1` exercises the whole sequence (swap → `-Verify` fails → `-Reapply` → `-Verify`
passes → `-Reapply` unchanged → the swap script) on a scratch tree.
passes → `-Reapply` unchanged → the swap script → the scoped modes, the pre-stop gate, `-NoStart`, a `-NoCarry` rollback)
on a scratch tree; `-TimingFiles N` times the swap on a vd03-like tree (`-ScriptsDir` for a before/after).
**Expected stopped window (lmxopcua#905).** Before #905 the swap ran the full `-Reapply` and `-Verify` while the service
was stopped, and each of them walked every denied folder for the protected-DACL coverage scan with `Get-Acl` per item
(about 3.4 ms an item on Windows PowerShell 5.1). vd03 denies all of `E:\ApiInstall`, so that was most of its 101 s outage
on 2026-11-13 (QA: 50–60 s per node). The scan now reads only the DACL through .NET (about 20× faster per item), walks
each denied folder once (a folder nested in another denied folder is no longer walked again), and inside the stopped
window walks only the swapped-in install folder. Setting the ACEs themselves was never the cost: an unchanged deny is
read, not rewritten, and the re-applied install-folder deny propagates over about 1,400 files in about 0.5 s. The firewall
query (3–5 s a run) moved to the Outside scope. Measured on windev with a vd03-like tree (17,851 items under the denied
folder, 48 protected files): the swap's stopped window went from **153.1 s to 6.0 s** on Windows PowerShell 5.1 and from
**101.6 s to 6.2 s** on PowerShell 7 (`Test-ScriptWorkerIsolation.Windows.ps1 -TimingFiles 17000`). Of the 6 s, about
1.5 s per installer run is process start + script parse + module import. The Outside part (full scan, firewall,
anchor-service query) takes about 23 s **before** the stop, with the old build serving. Add the stop, the deploy's own
steps inside the window (a `data\` backup) and the host start-up to `/health/ready`.
## Install / Uninstall
- `scripts/install/Install-Services.ps1 -Roles admin,driver` — installs `OtOpcUaHost`.
- `scripts/install/Install-ScriptWorkerIsolation.ps1` — the ScriptWorker identity, ACLs and firewall rule (above); only for `Scripting:Runner` ≠ `InProcess`.
- `scripts/install/Invoke-OtOpcUaInstallSwap.ps1` — upgrade/rollback folder swap with the isolation `-Reapply` + `-Verify` gate built in (lmxopcua#901, above).
- `scripts/install/Invoke-OtOpcUaInstallSwap.ps1` — upgrade/rollback folder swap with the isolation `-Reapply` + `-Verify` gate built in (lmxopcua#901), scoped so only the swapped-in folder is handled while the service is stopped (lmxopcua#905, above).
- `scripts/install/Uninstall-Services.ps1` — stops + removes the host service. (The historian backend is the external HistorianGateway — not installed/removed by these scripts.)
## Logging