Install swap: -Reapply/-Verify folder walk runs while the service is stopped (vd03 downtime ~101 s vs 20–40 s estimate) #905

New Issue

2026-11-13T03:01:45-05:00

dohertj2 commented

2026-11-13 03:01:45 -05:00

Found in the 2026-11-13 c9640f5c deploy (record docs/deploy/2026-11-13-vd03-qa-to-c9640f5c.md). Invoke-OtOpcUaInstallSwap.ps1 (#901) runs -Reapply + -Verify between stop and start. On vd03 the walk covers all of E:\ApiInstall, so downtime was about 101 s (the pack estimated 20–40 s; earlier swaps took 10–15 s). The QA pair outage was about 91 s against an estimate of 1–1.5 min.

Suggested fix: do as much of the ACL work as possible before stopping. The folder-level ACLs on everything except the swapped-in install folder can be re-applied or verified while the old build still serves. Only the new install folder (and anything inside it) needs re-applying in the stopped window, followed by a scoped -Verify of those entries. A full -Verify can run after start as a gate (stop the service again if it fails), or before stop for the untouched entries. Keep the guarantee that the new build never runs unverified. Measure before and after on windev with the Windows test harness.

Found in the 2026-11-13 `c9640f5c` deploy (record `docs/deploy/2026-11-13-vd03-qa-to-c9640f5c.md`). `Invoke-OtOpcUaInstallSwap.ps1` (#901) runs `-Reapply` + `-Verify` between stop and start. On vd03 the walk covers all of `E:\ApiInstall`, so downtime was about 101 s (the pack estimated 20–40 s; earlier swaps took 10–15 s). The QA pair outage was about 91 s against an estimate of 1–1.5 min. Suggested fix: do as much of the ACL work as possible before stopping. The folder-level ACLs on everything except the swapped-in install folder can be re-applied or verified while the old build still serves. Only the new install folder (and anything inside it) needs re-applying in the stopped window, followed by a scoped `-Verify` of those entries. A full `-Verify` can run after start as a gate (stop the service again if it fails), or before stop for the untouched entries. Keep the guarantee that the new build never runs unverified. Measure before and after on windev with the Windows test harness.

dohertj2 added the type:bug priority:p3 labels 2026-11-13 03:01:45 -05:00

dohertj2 referenced this issue from a commit

2026-11-13 03:51:54 -05:00

fix(install): cut the swap's stopped window to the swapped-in folder (#905)

dohertj2 commented

2026-11-13 03:52:17 -05:00

Fixed in 0c19f26c on master.

Where the time went. I measured on windev with a vd03-like tree. The cost was the protected-DACL coverage scan, not icacls propagation and not the ACL writes. Both -Reapply and -Verify walked every denied folder while the service was stopped. On vd03 that means all of E:\ApiInstall, and the nested roots (install folder, pki\) were walked again. Each walk used Get-ChildItem -Recurse, then Get-Acl per item, then a pipeline per item for -Skip.

17,584 items (PS 5.1 / PS 7):

Piece	5.1	7
Enumerate only	2.1 s	1.3 s
`Get-Acl` per item	36.2 s	27.8 s
`Find-OtOpcUaDenyCoverageGap` (old)	59.1 s	34.5 s
`GetAccessControl(Access)` per item (new)	2.6 s	2.5 s
`Set-Acl` deny on the whole tree, first time (propagation)	6.7 s	6.7 s
Same deny, already present	0.06 s	0.04 s
Install-folder deny re-applied after a swap (~1,400 files)	0.57 s	0.55 s

A second cost showed up once the walk was gone. Get-NetFirewallRule + Get-NetFirewallApplicationFilter take 3–5 s per run on windev.

Change

The coverage scan now enumerates with .NET, prunes the -Skip folders and reads only the DACL. It walks each denied folder once (outermost roots only) and does not follow reparse points.
Install-ScriptWorkerIsolation.ps1 -Scope All|InstallRoot|Outside (works with -Reapply, -Verify and -PlanOnly):
- InstallRoot is the folder ACEs a swap drops: WorkerFolderAcl, InstallRootDeny, and a denied folder inside the install folder such as vd03's pki\.
- Outside is everything else, including the firewall rule. Its exe path does not change in a swap, and the swap preflight now checks that the candidate carries the worker exe.
- Together the two scopes cover the plan exactly once each.
Invoke-OtOpcUaInstallSwap.ps1:
- Runs -Reapply/-Verify -Scope Outside before the stop, with the full scan. If that fails, nothing is stopped or moved.
- Inside the window it runs only -Reapply/-Verify -Scope InstallRoot, so the new build is still never started unverified.
- Prints stopped window (…): N s [stop, move, carry, isolation].
- -NoStart, -NoCarry and the pre-#901 refusal behave as before.

Before/after on windev (Test-ScriptWorkerIsolation.Windows.ps1 -TimingOnly -TimingFiles 17000 [-ScriptsDir <old>]; 17,851 items under the denied folder, 48 protected files, temporary local host account):

Host	Stopped window before	Stopped window after	Pre-stop (service still up)
Windows PowerShell 5.1	153.1 s	6.0 s (move 0.0, carry 0.1, isolation 5.8)	23.1 s
PowerShell 7.6	101.6 s	6.2 s (move 0.0, carry 0.2, isolation 5.9)	23.2 s

Of the remaining ~6 s, about 1.5 s per installer run is process start, script parse and module import.

Tests

Dry-run harness (macOS): 101 → 123 passed. New coverage: scope partition (vd03 and QA layouts), coverage roots, -Scope argument rules, and the swap order checked from the parsed script. The new assertions were mutation-checked.
Windows test: 72 passed, 0 failed on both PS 5.1 and PS 7 (-LocalHostAccount). New cases: scoped verify/reapply after a raw swap, each scope's negative control, scan scoping and dedup, the candidate-without-exe refusal, the pre-stop gate (nothing moves), -NoStart with service control, and a -NoCarry rollback.
windev was cleaned afterwards: scratch trees, the scratch services, the local account and the firewall rule are all gone. MxAccessGw was untouched and is Running.

Docs: docs/ServiceHosting.md, plus downtime notes in the c9640f5c pack (§7) and record. The next vd03 swap should be about 20–30 s from stop to ready, plus the data\ backup.

Fixed in `0c19f26c` on master. **Where the time went.** I measured on windev with a vd03-like tree. The cost was the protected-DACL coverage scan, not icacls propagation and not the ACL writes. Both `-Reapply` and `-Verify` walked every denied folder while the service was stopped. On vd03 that means all of `E:\ApiInstall`, and the nested roots (install folder, `pki\`) were walked again. Each walk used `Get-ChildItem -Recurse`, then `Get-Acl` per item, then a pipeline per item for `-Skip`. 17,584 items (PS 5.1 / PS 7): | Piece | 5.1 | 7 | |---|---|---| | Enumerate only | 2.1 s | 1.3 s | | `Get-Acl` per item | 36.2 s | 27.8 s | | `Find-OtOpcUaDenyCoverageGap` (old) | 59.1 s | 34.5 s | | `GetAccessControl(Access)` per item (new) | 2.6 s | 2.5 s | | `Set-Acl` deny on the whole tree, first time (propagation) | 6.7 s | 6.7 s | | Same deny, already present | 0.06 s | 0.04 s | | Install-folder deny re-applied after a swap (~1,400 files) | 0.57 s | 0.55 s | A second cost showed up once the walk was gone. `Get-NetFirewallRule` + `Get-NetFirewallApplicationFilter` take 3–5 s per run on windev. **Change** - The coverage scan now enumerates with .NET, prunes the `-Skip` folders and reads only the DACL. It walks each denied folder once (outermost roots only) and does not follow reparse points. - `Install-ScriptWorkerIsolation.ps1 -Scope All|InstallRoot|Outside` (works with `-Reapply`, `-Verify` and `-PlanOnly`): - `InstallRoot` is the folder ACEs a swap drops: `WorkerFolderAcl`, `InstallRootDeny`, and a denied folder inside the install folder such as vd03's `pki\`. - `Outside` is everything else, including the firewall rule. Its exe path does not change in a swap, and the swap preflight now checks that the candidate carries the worker exe. - Together the two scopes cover the plan exactly once each. - `Invoke-OtOpcUaInstallSwap.ps1`: - Runs `-Reapply`/`-Verify -Scope Outside` **before the stop**, with the full scan. If that fails, nothing is stopped or moved. - Inside the window it runs only `-Reapply`/`-Verify -Scope InstallRoot`, so the new build is still never started unverified. - Prints `stopped window (…): N s [stop, move, carry, isolation]`. - `-NoStart`, `-NoCarry` and the pre-#901 refusal behave as before. **Before/after on windev** (`Test-ScriptWorkerIsolation.Windows.ps1 -TimingOnly -TimingFiles 17000 [-ScriptsDir <old>]`; 17,851 items under the denied folder, 48 protected files, temporary local host account): | Host | Stopped window before | Stopped window after | Pre-stop (service still up) | |---|---|---|---| | Windows PowerShell 5.1 | 153.1 s | **6.0 s** (move 0.0, carry 0.1, isolation 5.8) | 23.1 s | | PowerShell 7.6 | 101.6 s | **6.2 s** (move 0.0, carry 0.2, isolation 5.9) | 23.2 s | Of the remaining ~6 s, about 1.5 s per installer run is process start, script parse and module import. **Tests** - Dry-run harness (macOS): 101 → **123 passed**. New coverage: scope partition (vd03 and QA layouts), coverage roots, `-Scope` argument rules, and the swap order checked from the parsed script. The new assertions were mutation-checked. - Windows test: **72 passed, 0 failed on both PS 5.1 and PS 7** (`-LocalHostAccount`). New cases: scoped verify/reapply after a raw swap, each scope's negative control, scan scoping and dedup, the candidate-without-exe refusal, the pre-stop gate (nothing moves), `-NoStart` with service control, and a `-NoCarry` rollback. - windev was cleaned afterwards: scratch trees, the scratch services, the local account and the firewall rule are all gone. MxAccessGw was untouched and is Running. Docs: `docs/ServiceHosting.md`, plus downtime notes in the c9640f5c pack (§7) and record. The next vd03 swap should be about 20–30 s from stop to ready, plus the `data\` backup.

dohertj2 closed this issue

2026-11-13 03:52:18 -05:00

dohertj2 referenced this issue from a commit

2026-11-13 07:02:24 -05:00

docs(deploy): next vd03 + QA deploy pack c9640f5c -> 47f0b98c (#905 #906 #717 LocalDb 0.3.8), read-only preflight

dohertj2 referenced this issue from a commit

2026-11-13 07:51:31 -05:00

docs(deploy): vd03 + QA deployed c9640f5c -> 76e73e7d (#905 #906 #900 #717, LocalDb 0.3.8)

dohertj2 commented

2026-11-13 07:53:49 -05:00

Deployed to DEV vd03 and the QA pair vq03/vq04 on 2026-11-13 (12:27–12:48Z), master 76e73e7d. Record: docs/deploy/2026-11-13-vd03-qa-to-76e73e7d.md (5baf05f6).

The swap ran with the service up and -NoStart, so -Scope Outside ran with the old build serving.

vd03

Outside: 11/11 unchanged, VERIFY OK 11/11, 28.0 s before the stop.
Window line: stop -> swapped and verified (left stopped): 5.7 s [stop 1.5 s, move 0.0 s, carry 0.8 s, isolation 3.4 s].
Downtime ~14 s stop → ready (c9640f5c: ~101 s). Full -Verify after start: OK 14/14.

QA

Outside: 19/19 unchanged + OK on each node (~18 s, before the stop).
InstallRoot window: 4.8 s (vq04) and 4.6 s (vq03). Full -Verify after every start: OK 21/21.
Pair outage 36.5 s (c9640f5c: ~91 s).

One new defect, filed as #908. On both QA nodes the first Move-Item app → app.bak… failed "file in use" right after the stop. The host was probably still exiting after SCM reported Stopped. Nothing had moved, so a retry with the service stopped went through. On vq03 the retry re-ran the Outside scope inside the outage (~20 s of the 36.5 s).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

**Deployed** to DEV vd03 and the QA pair vq03/vq04 on 2026-11-13 (12:27–12:48Z), master `76e73e7d`. Record: `docs/deploy/2026-11-13-vd03-qa-to-76e73e7d.md` (`5baf05f6`). The swap ran with the service up and `-NoStart`, so `-Scope Outside` ran with the old build serving. **vd03** - Outside: 11/11 unchanged, VERIFY OK 11/11, 28.0 s before the stop. - Window line: `stop -> swapped and verified (left stopped): 5.7 s [stop 1.5 s, move 0.0 s, carry 0.8 s, isolation 3.4 s]`. - **Downtime ~14 s** stop → ready (c9640f5c: ~101 s). Full `-Verify` after start: OK 14/14. **QA** - Outside: 19/19 unchanged + OK on each node (~18 s, before the stop). - InstallRoot window: `4.8 s` (vq04) and `4.6 s` (vq03). Full `-Verify` after every start: OK 21/21. - **Pair outage 36.5 s** (c9640f5c: ~91 s). **One new defect, filed as #908.** On both QA nodes the first `Move-Item app → app.bak…` failed "file in use" right after the stop. The host was probably still exiting after SCM reported Stopped. Nothing had moved, so a retry with the service stopped went through. On vq03 the retry re-ran the Outside scope inside the outage (~20 s of the 36.5 s). Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#905