fix(install): cut the swap's stopped window to the swapped-in folder (#905)
v2-ci / unit-tests (push) Has been cancelled

Root cause: the protected-DACL coverage scan, not the ACL writes. Both -Reapply and
-Verify walked every denied folder (all of E:\ApiInstall on vd03, nested roots again)
with Get-ChildItem -Recurse + Get-Acl per item + a pipeline per item for -Skip, about
3.4 ms an item on PS 5.1, all while the service was stopped. An unchanged deny is read,
not rewritten; the re-applied install-folder deny propagates over ~1,400 files in 0.5 s.

- Find-OtOpcUaDenyCoverageGap: .NET enumeration, -Skip folders pruned, DACL-only
  GetAccessControl(Access) (about 0.15 ms an item); reparse points not followed.
- Get-OtOpcUaCoverageWalkRoots: each denied folder walked once (outermost only).
- Install-ScriptWorkerIsolation.ps1 -Scope All|InstallRoot|Outside (with -Reapply,
  -Verify, -PlanOnly). InstallRoot = the folder ACEs a swap drops; Outside = the rest,
  including the firewall rule (its query costs 3-5 s; its exe path does not change).
  The two partition the plan. Prints the scan time.
- Invoke-OtOpcUaInstallSwap.ps1: -Reapply/-Verify -Scope Outside before the stop (a
  failure stops before anything moves), -Scope InstallRoot inside the window; preflight
  checks the candidate carries the worker exe; prints the stopped window and its parts.
  -NoStart, -NoCarry and the pre-#901 refusal unchanged.

windev, vd03-like tree (17,851 items): stopped window 153.1 s -> 6.0 s (PS 5.1),
101.6 s -> 6.2 s (PS 7); the Outside part takes ~23 s with the old build serving.
Dry-run harness 101 -> 123 passed; Windows test 72 passed on PS 5.1 and PS 7.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 03:51:41 -05:00
parent f817c77e02
commit 0c19f26c81
8 changed files with 475 additions and 58 deletions
+69 -21
View File
@@ -11,13 +11,27 @@
0. Preflight (nothing changed yet): the candidate exists and carries script-worker\; PreviousInstall does not exist
and is on the same volume; when isolation is installed, its settings are recorded (an install by an installer
older than #901 recorded only DenyPaths; then run the full-argument install once first, or pass -SkipIsolation).
1. Stop the service (unless -NoServiceControl).
2. Move InstallRoot -> PreviousInstall, then Candidate -> InstallRoot.
3. Copy each -Carry item (relative to InstallRoot; wildcards allowed) from PreviousInstall into the new InstallRoot.
Folders are copied with robocopy /E /COPY:DAT (data, attributes, timestamps; not ACLs: step 4 sets those).
4. Isolation, when recorded: Install-ScriptWorkerIsolation.ps1 -Reapply, then -Verify. If -Verify fails, the script
stops here with the service STOPPED (fail closed): fix it, re-run -Reapply/-Verify, then start the service.
5. Start the service (unless -NoStart, e.g. a Primary/Secondary pair window where both nodes start together).
1. Isolation, when recorded, with the old build still serving (lmxopcua#905): Install-ScriptWorkerIsolation.ps1
-Reapply -Scope Outside, then -Verify -Scope Outside: every step the swap cannot undo (anchor service, records,
PE cache, denies outside InstallRoot, firewall rule, host env) and the coverage scan of the denied folders
outside InstallRoot.
If either fails, the script stops here: nothing is stopped or moved.
2. Stop the service (unless -NoServiceControl). The stopped window starts here.
3. Move InstallRoot -> PreviousInstall, then Candidate -> InstallRoot.
4. Copy each -Carry item (relative to InstallRoot; wildcards allowed) from PreviousInstall into the new InstallRoot.
Folders are copied with robocopy /E /COPY:DAT (data, attributes, timestamps; not ACLs: step 5 sets those).
5. Isolation, when recorded: -Reapply -Scope InstallRoot, then -Verify -Scope InstallRoot: only the entries a swap
drops (the ACEs on InstallRoot, script-worker\ and a denied folder inside InstallRoot such as pki\) and the
coverage scan of InstallRoot. Steps 1 and 5 together check every install step (the firewall rule names the
worker exe by its unchanged path; preflight checks the candidate carries that exe), so the new build is never
started unverified. If -Verify fails, the script stops here with the service STOPPED
(fail closed): fix it, run -Reapply/-Verify (no -Scope), then start the service.
6. Start the service (unless -NoStart, e.g. a Primary/Secondary pair window where both nodes start together).
Prints the stopped window (stop issued -> start issued) and the time of each step in it.
lmxopcua#905: the coverage scan used to run over every denied folder in both -Reapply and -Verify while the service
was stopped, with Get-Acl per item; on vd03 (E:\ApiInstall denied) that was most of a 101 s outage. It is now about
20x faster per item and, inside the stopped window, walks only the new InstallRoot.
Not covered (layout-specific; keep them in the deploy's own scripts): backups of data\ / the registry / the ConfigDb,
restoring data\ on a rollback, migrations, health checks.
@@ -91,12 +105,16 @@ Import-Module (Join-Path $PSScriptRoot 'ScriptWorkerIsolation.psm1') -Force
$installer = Join-Path $PSScriptRoot 'Install-ScriptWorkerIsolation.ps1'
function Step([string]$Message) { Write-Host "==> $Message" }
function Invoke-Isolation([string]$Mode) {
function Invoke-Isolation([string]$Mode, [string]$Scope) {
# Same PowerShell host as this script; the installer's exit code is the gate.
$exe = if ($PSVersionTable.PSEdition -eq 'Core') { (Get-Process -Id $PID).Path } else { 'powershell.exe' }
& $exe -NoProfile -ExecutionPolicy Bypass -File $installer "-$Mode" -WorkerServiceName $WorkerServiceName | ForEach-Object { Write-Host " $_" }
& $exe -NoProfile -ExecutionPolicy Bypass -File $installer "-$Mode" -Scope $Scope -WorkerServiceName $WorkerServiceName | ForEach-Object { Write-Host " $_" }
$LASTEXITCODE
}
# lmxopcua#905: the stopped window and what it is spent on.
$window = [System.Diagnostics.Stopwatch]::new()
$phases = [System.Collections.Generic.List[string]]::new()
function Mark([string]$Name, [System.Diagnostics.Stopwatch]$Sw) { $phases.Add(('{0} {1:N1} s' -f $Name, $Sw.Elapsed.TotalSeconds)) }
$InstallRoot = $InstallRoot.TrimEnd('\')
$Candidate = $Candidate.TrimEnd('\')
@@ -109,6 +127,8 @@ $principal = [Security.Principal.WindowsPrincipal][Security.Principal.WindowsIde
if (-not $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) { throw 'run elevated' }
if (-not (Test-Path -LiteralPath $InstallRoot -PathType Container)) { throw "InstallRoot $InstallRoot not found" }
if (-not (Test-Path -LiteralPath (Join-Path $Candidate 'script-worker') -PathType Container)) { throw "candidate $Candidate has no script-worker\ (not a Host publish?)" }
# lmxopcua#905: the firewall rule (verified before the stop) blocks this exe by path; the swapped-in folder must carry it.
$candidateWorkerExe = Join-Path $Candidate 'script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe'
if (Test-Path -LiteralPath $PreviousInstall) { throw "PreviousInstall $PreviousInstall already exists; pick a fresh name" }
if ([IO.Path]::GetPathRoot($InstallRoot) -ne [IO.Path]::GetPathRoot($PreviousInstall) -or [IO.Path]::GetPathRoot($InstallRoot) -ne [IO.Path]::GetPathRoot($Candidate)) {
throw 'InstallRoot, Candidate and PreviousInstall must be on the same volume (Move-Item renames)'
@@ -126,14 +146,30 @@ if (-not $SkipIsolation) {
if ($isolation -and $isolation.InstallRoot.TrimEnd('\') -ine $InstallRoot) {
throw "the isolation record is for InstallRoot '$($isolation.InstallRoot)', not '$InstallRoot'; fix -InstallRoot or re-run the isolation install"
}
if ($isolation) { Write-Host "Isolation installed (record: InstallRoot $($isolation.InstallRoot)); it is re-applied and verified after the swap." }
if ($isolation -and -not (Test-Path -LiteralPath $candidateWorkerExe -PathType Leaf)) {
throw "candidate has no $candidateWorkerExe; the worker firewall rule blocks that path (a framework-dependent publish is not covered by it)"
}
if ($isolation) { Write-Host "Isolation installed (record: InstallRoot $($isolation.InstallRoot)); re-applied and verified outside InstallRoot before the stop, inside it after the swap." }
else { Write-Host 'Isolation not installed on this node; nothing to re-apply.' }
}
else { Write-Warning '-SkipIsolation: the worker-isolation ACLs are NOT re-applied; a swapped folder has lost them (lmxopcua#901).' }
if (-not $PSCmdlet.ShouldProcess($InstallRoot, "swap in $Candidate (previous install -> $PreviousInstall)")) { return }
# ---- 1. Stop ---------------------------------------------------------------------------------------------------------
# ---- 1. Isolation outside InstallRoot, old build still serving (lmxopcua#905) ---------------------------------------
if ($isolation) {
$sw = [System.Diagnostics.Stopwatch]::StartNew()
Step 'worker isolation outside the install folder: -Reapply -Scope Outside (service still running)'
$code = Invoke-Isolation 'Reapply' 'Outside'
if ($code -ne 0) { throw "Install-ScriptWorkerIsolation.ps1 -Reapply -Scope Outside failed ($code); nothing stopped or swapped" }
Step 'worker isolation outside the install folder: -Verify -Scope Outside (pre-stop gate)'
$code = Invoke-Isolation 'Verify' 'Outside'
if ($code -ne 0) { throw "Install-ScriptWorkerIsolation.ps1 -Verify -Scope Outside failed ($code); nothing stopped or swapped. Fix the [MISSING] steps (-Reapply), then re-run the swap." }
Write-Host (' (pre-stop isolation {0:N1} s, service still running)' -f $sw.Elapsed.TotalSeconds)
}
# ---- 2. Stop ---------------------------------------------------------------------------------------------------------
$window.Start()
if (-not $NoServiceControl) {
Step "stop $ServiceName"
$svc = Get-Service -Name $ServiceName
@@ -141,9 +177,11 @@ if (-not $NoServiceControl) {
Stop-Service -Name $ServiceName -Force
$svc.WaitForStatus('Stopped', [TimeSpan]::FromSeconds($StopTimeoutSeconds))
}
Mark 'stop' $window
}
# ---- 2. Swap ---------------------------------------------------------------------------------------------------------
# ---- 3. Swap ---------------------------------------------------------------------------------------------------------
$sw = [System.Diagnostics.Stopwatch]::StartNew()
Step "move $InstallRoot -> $PreviousInstall"
Move-Item -LiteralPath $InstallRoot -Destination $PreviousInstall
Step "move $Candidate -> $InstallRoot"
@@ -154,7 +192,10 @@ catch {
throw
}
# ---- 3. Carry --------------------------------------------------------------------------------------------------------
Mark 'move' $sw
# ---- 4. Carry --------------------------------------------------------------------------------------------------------
$sw = [System.Diagnostics.Stopwatch]::StartNew()
foreach ($item in $Carry) {
# Get-Item, not Get-ChildItem: a folder name must give the folder itself, not its children (wildcards still expand).
$sources = @(Get-Item -Path (Join-Path $PreviousInstall $item) -Force -ErrorAction SilentlyContinue)
@@ -170,17 +211,24 @@ foreach ($item in $Carry) {
}
}
# ---- 4. Isolation re-apply + verify (fail closed) --------------------------------------------------------------------
Mark 'carry' $sw
# ---- 5. Isolation inside InstallRoot: re-apply + verify (fail closed) ------------------------------------------------
if ($isolation) {
Step 'worker isolation: -Reapply'
$code = Invoke-Isolation 'Reapply'
if ($code -ne 0) { throw "Install-ScriptWorkerIsolation.ps1 -Reapply failed ($code); $ServiceName left stopped" }
Step 'worker isolation: -Verify (post-swap gate)'
$code = Invoke-Isolation 'Verify'
if ($code -ne 0) { throw "Install-ScriptWorkerIsolation.ps1 -Verify failed ($code); $ServiceName left stopped. Fix the [MISSING] steps, re-run -Reapply/-Verify, then start it." }
$sw = [System.Diagnostics.Stopwatch]::StartNew()
Step 'worker isolation in the swapped-in folder: -Reapply -Scope InstallRoot'
$code = Invoke-Isolation 'Reapply' 'InstallRoot'
if ($code -ne 0) { throw "Install-ScriptWorkerIsolation.ps1 -Reapply -Scope InstallRoot failed ($code); $ServiceName left stopped" }
Step 'worker isolation in the swapped-in folder: -Verify -Scope InstallRoot (post-swap gate)'
$code = Invoke-Isolation 'Verify' 'InstallRoot'
if ($code -ne 0) { throw "Install-ScriptWorkerIsolation.ps1 -Verify -Scope InstallRoot failed ($code); $ServiceName left stopped. Fix the [MISSING] steps, re-run -Reapply/-Verify, then start it." }
Mark 'isolation' $sw
}
# ---- 5. Start --------------------------------------------------------------------------------------------------------
# ---- 6. Start --------------------------------------------------------------------------------------------------------
$window.Stop()
$windowName = if ($NoServiceControl) { 'swap window (service control is the caller''s)' } elseif ($NoStart) { 'stop -> swapped and verified (left stopped)' } else { 'stopped window (stop issued -> start issued)' }
Write-Host ('{0}: {1:N1} s [{2}]' -f $windowName, $window.Elapsed.TotalSeconds, ($phases -join ', '))
if (-not $NoServiceControl -and -not $NoStart) {
Step "start $ServiceName"
Start-Service -Name $ServiceName