diff --git a/docs/ServiceHosting.md b/docs/ServiceHosting.md index db455b68d..9c743f65a 100644 --- a/docs/ServiceHosting.md +++ b/docs/ServiceHosting.md @@ -147,16 +147,24 @@ run the install once with its full arguments (the per-box deploy record lists th `scripts/install/Invoke-OtOpcUaInstallSwap.ps1` is the swap with the gate built in: preflight (candidate carries `script-worker\`, `PreviousInstall` is fresh and on the same volume, isolation record present when isolation is installed, record `InstallRoot` equals `-InstallRoot`) → `-Reapply -Scope Outside` + `-Verify -Scope Outside` **while -the old build still serves** (a failure stops here: nothing stopped or moved) → stop → move live → previous, candidate → +the old build still serves** (a failure stops here: nothing stopped or moved) → stop → **wait until the host process +(PID captured before the stop) and every process whose image is under `InstallRoot` have exited** (lmxopcua#908: SCM can +report `Stopped` while the host is still exiting, holding e.g. a log file open, and the folder move then fails "in use"; +bounded by `-ProcessExitTimeoutSeconds`, default 60 s, after which the script throws naming the processes, with nothing +moved) → move live → previous (one `[IO.Directory]::Move` rename that fails as a whole — PowerShell 7's `Move-Item` falls +back to a partial item-by-item move when the rename fails; retried for at most `-MoveRetrySeconds`, default 10 s, while it +fails because something under `InstallRoot` is open: a backstop for a handle held by a process outside it), candidate → live → copy `-Carry` items back (default `appsettings*.json`, `pki`, `logs`; QA: `'appsettings*.json','logs'`; rollback: `-NoCarry`) → `-Reapply -Scope InstallRoot` → `-Verify -Scope InstallRoot` → start (`-NoStart` for a pair window). The two scoped gates together check every install step, so the new build never starts unverified; a failed post-swap `-Verify` leaves the service **stopped**. The script prints the stopped window and its parts -(`stopped window (stop issued -> start issued): 6.0 s [stop …, move 0.0 s, carry 0.1 s, isolation 5.8 s]`). Backups, +(`stopped window (stop issued -> start issued): 6.0 s [stop …, exit …, move 0.0 s, carry 0.1 s, isolation 5.8 s]`; `exit` is the #908 wait). Backups, `data\` restore, migrations and health checks stay in the deploy's own steps. Every deploy pack that swaps the folder runs it (or `-Reapply` + `-Verify` right after its own swap), and `-Verify` is a post-deploy gate. `Test-ScriptWorkerIsolation.Windows.ps1` exercises the whole sequence (swap → `-Verify` fails → `-Reapply` → `-Verify` -passes → `-Reapply` unchanged → the swap script → the scoped modes, the pre-stop gate, `-NoStart`, a `-NoCarry` rollback) +passes → `-Reapply` unchanged → the swap script → the scoped modes, the pre-stop gate, `-NoStart`, a `-NoCarry` rollback, +and the #908 cases: a process running from `InstallRoot` past the budget, one exiting ~2 s after the wait starts, and an in-use +handle from outside `InstallRoot` that the first-move retry absorbs) on a scratch tree; `-TimingFiles N` times the swap on a vd03-like tree (`-ScriptsDir` for a before/after). **Expected stopped window (lmxopcua#905).** Before #905 the swap ran the full `-Reapply` and `-Verify` while the service diff --git a/docs/deploy/2026-11-13-vd03-qa-to-76e73e7d.md b/docs/deploy/2026-11-13-vd03-qa-to-76e73e7d.md index fb490c47b..b1ae3c222 100644 --- a/docs/deploy/2026-11-13-vd03-qa-to-76e73e7d.md +++ b/docs/deploy/2026-11-13-vd03-qa-to-76e73e7d.md @@ -184,6 +184,9 @@ Do not run a mixed pair: `RedundancyStateChanged` changed shape in this range. A process to exit; `Invoke-OtOpcUaInstallSwap.ps1` does not. - It cost about 20 s of the vq03 pair outage (the retry re-runs `-Scope Outside`). - Filed as **lmxopcua#908**. vd03 did not hit it. + - **Fixed (lmxopcua#908, closed):** `Invoke-OtOpcUaInstallSwap.ps1` now waits, after `Stopped`, for the host process + and every process imaged under `InstallRoot` to exit (`-ProcessExitTimeoutSeconds`, default 60) and retries the first + move on a sharing violation (`-MoveRetrySeconds`, default 10). The next deploy needs no hand retry. 3. `Stop-Service` on the new build took 5.7–6.1 s instead of the 1–3 s target, because of the ~5 s OPC UA session close after the cluster leave (see "Observation"). The hand-over, the actual #906 criterion, was 0.36 s and 0.47 s. 4. Logs: the QA host log for the window rolled to `otopcua-20261113_001.log` on vq04 (size limit). Read it from there. diff --git a/scripts/install/Invoke-OtOpcUaInstallSwap.ps1 b/scripts/install/Invoke-OtOpcUaInstallSwap.ps1 index f850cfbdc..ff6131601 100644 --- a/scripts/install/Invoke-OtOpcUaInstallSwap.ps1 +++ b/scripts/install/Invoke-OtOpcUaInstallSwap.ps1 @@ -16,8 +16,14 @@ PE cache, denies outside InstallRoot, firewall rule, host env) and the coverage scan of the denied folders outside InstallRoot. If either fails, the script stops here: nothing is stopped or moved. - 2. Stop the service (unless -NoServiceControl). The stopped window starts here. - 3. Move InstallRoot -> PreviousInstall, then Candidate -> InstallRoot. + 2. Stop the service (unless -NoServiceControl). The stopped window starts here. SCM can report Stopped while the + host process is still exiting (lmxopcua#908), so the script then waits, at most -ProcessExitTimeoutSeconds, until + the service's host process (PID captured before the stop) and every process whose image is under InstallRoot + (OtOpcUa.Host, the ScriptWorker) have exited. If one is still there, it throws naming them: nothing has moved. + 3. Move InstallRoot -> PreviousInstall, then Candidate -> InstallRoot, each one rename ([IO.Directory]::Move; not + Move-Item, which in PowerShell 7 falls back to a partial item-by-item move when the rename fails). The first + move is retried for at most -MoveRetrySeconds while it fails because something under InstallRoot is open + (nothing has moved yet; a backstop for a handle the process wait cannot see). 4. Copy each -Carry item (relative to InstallRoot; wildcards allowed) from PreviousInstall into the new InstallRoot. Folders are copied with robocopy /E /COPY:DAT (data, attributes, timestamps; not ACLs: step 5 sets those). 5. Isolation, when recorded: -Reapply -Scope InstallRoot, then -Verify -Scope InstallRoot: only the entries a swap @@ -66,6 +72,13 @@ .PARAMETER SkipIsolation Do not re-apply/verify the isolation even when it is installed. WARNs. Only for a box where it is being removed. +.PARAMETER ProcessExitTimeoutSeconds + After the stop, how long to wait for the host process and any process running from InstallRoot to exit before + failing with nothing moved (default 60; lmxopcua#908). + +.PARAMETER MoveRetrySeconds + How long to keep retrying the first folder move while it fails with a sharing violation (default 10; 0 = no retry). + .PARAMETER WorkerServiceName Passed to the isolation installer (default OtOpcUaScriptWorker). @@ -96,7 +109,9 @@ param( [switch]$NoStart, [switch]$SkipIsolation, [ValidatePattern('^[A-Za-z0-9_.\-]{1,80}$')] [string]$WorkerServiceName = 'OtOpcUaScriptWorker', - [int]$StopTimeoutSeconds = 120 + [int]$StopTimeoutSeconds = 120, + [ValidateRange(0, 3600)] [int]$ProcessExitTimeoutSeconds = 60, + [ValidateRange(0, 600)] [int]$MoveRetrySeconds = 10 ) Set-StrictMode -Version Latest @@ -131,7 +146,7 @@ if (-not (Test-Path -LiteralPath (Join-Path $Candidate 'script-worker') -PathTyp $candidateWorkerExe = Join-Path $Candidate 'script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe' if (Test-Path -LiteralPath $PreviousInstall) { throw "PreviousInstall $PreviousInstall already exists; pick a fresh name" } if ([IO.Path]::GetPathRoot($InstallRoot) -ne [IO.Path]::GetPathRoot($PreviousInstall) -or [IO.Path]::GetPathRoot($InstallRoot) -ne [IO.Path]::GetPathRoot($Candidate)) { - throw 'InstallRoot, Candidate and PreviousInstall must be on the same volume (Move-Item renames)' + throw 'InstallRoot, Candidate and PreviousInstall must be on the same volume (the swap renames folders)' } $isolation = $null if (-not $SkipIsolation) { @@ -169,26 +184,43 @@ if ($isolation) { } # ---- 2. Stop --------------------------------------------------------------------------------------------------------- +$hostPid = 0; $hostStart = $null $window.Start() if (-not $NoServiceControl) { Step "stop $ServiceName" $svc = Get-Service -Name $ServiceName if ($svc.Status -ne 'Stopped') { + # lmxopcua#908: remember the host process (PID + start time, so a reused PID does not count) before stopping it. + $hostPid = [int](Get-CimInstance -ClassName Win32_Service -Filter "Name='$ServiceName'").ProcessId + if ($hostPid -gt 0) { + $hostProc = Get-Process -Id $hostPid -ErrorAction SilentlyContinue + if ($hostProc) { try { $hostStart = $hostProc.StartTime.ToUniversalTime() } catch { } } + } Stop-Service -Name $ServiceName -Force $svc.WaitForStatus('Stopped', [TimeSpan]::FromSeconds($StopTimeoutSeconds)) } Mark 'stop' $window } +# lmxopcua#908: SCM reports Stopped before the host process is gone; the folder move fails "in use" until it is. +$sw = [System.Diagnostics.Stopwatch]::StartNew() +Step "wait for the host process (PID $(if ($hostPid) { $hostPid } else { 'n/a' })) and any process under $InstallRoot to exit (at most $ProcessExitTimeoutSeconds s)" +$waited = Wait-OtOpcUaInstallReleased -InstallRoot $InstallRoot -ProcessId $hostPid -ProcessStartTime $hostStart -TimeoutSeconds $ProcessExitTimeoutSeconds +Write-Host (' released after {0:N1} s' -f $waited) +Mark 'exit' $sw # ---- 3. Swap --------------------------------------------------------------------------------------------------------- $sw = [System.Diagnostics.Stopwatch]::StartNew() Step "move $InstallRoot -> $PreviousInstall" -Move-Item -LiteralPath $InstallRoot -Destination $PreviousInstall +# lmxopcua#908: [IO.Directory]::Move, not Move-Item. It is one rename (MoveFileEx, same volume: checked in preflight) that +# fails as a whole. PowerShell 7's Move-Item, when the rename fails because a file under the folder is open, falls back +# to moving item by item and leaves a partial PreviousInstall tree behind (windev), so a retry would then nest into it. +$attempts = Invoke-OtOpcUaMoveWithRetry -RetrySeconds $MoveRetrySeconds -Move { [System.IO.Directory]::Move($InstallRoot, $PreviousInstall) } +if ($attempts -gt 1) { Write-Host " moved on attempt $attempts" } Step "move $Candidate -> $InstallRoot" -try { Move-Item -LiteralPath $Candidate -Destination $InstallRoot } +try { [System.IO.Directory]::Move($Candidate, $InstallRoot) } catch { Write-Warning "candidate move failed; putting the previous install back: $($_.Exception.Message)" - Move-Item -LiteralPath $PreviousInstall -Destination $InstallRoot + [System.IO.Directory]::Move($PreviousInstall, $InstallRoot) throw } diff --git a/scripts/install/ScriptWorkerIsolation.psm1 b/scripts/install/ScriptWorkerIsolation.psm1 index 396f6b73a..5596c519e 100644 --- a/scripts/install/ScriptWorkerIsolation.psm1 +++ b/scripts/install/ScriptWorkerIsolation.psm1 @@ -692,8 +692,125 @@ function Get-OtOpcUaCoverageWalkRoots { [pscustomobject]@{ Roots = $outer.ToArray(); Skip = $(if ($Scope -eq 'Outside') { @($root) } else { @() }) } } +function Get-OtOpcUaProcessSnapshot { + <# + .SYNOPSIS + Every process as Id, Name, Path (image path; '' when unreadable), StartTime (UTC or $null). Windows: Win32_Process + (sees service processes in session 0 when elevated); elsewhere Get-Process (the dry-run harness injects its own). + #> + if ($PSVersionTable.PSEdition -eq 'Desktop' -or $IsWindows) { + foreach ($p in @(Get-CimInstance -ClassName Win32_Process -ErrorAction Stop)) { + [pscustomobject]@{ + Id = [int]$p.ProcessId; Name = [string]$p.Name; Path = [string]$p.ExecutablePath + StartTime = $(if ($p.CreationDate) { ([datetime]$p.CreationDate).ToUniversalTime() } else { $null }) + } + } + } + else { + foreach ($p in @(Get-Process)) { + $path = ''; $start = $null + try { $path = [string]$p.Path } catch { } + try { $start = $p.StartTime.ToUniversalTime() } catch { } + [pscustomobject]@{ Id = $p.Id; Name = $p.ProcessName; Path = $path; StartTime = $start } + } + } +} + +function Find-OtOpcUaInstallHolder { + <# + .SYNOPSIS + The processes that still pin InstallRoot (lmxopcua#908): the service's host process (by the PID captured before the + stop; when its start time was captured too, a different process reusing the PID does not count) and any process + whose image is under InstallRoot (OtOpcUa.Host, ZB.MOM.WW.OtOpcUa.ScriptWorker in script-worker\). Pure: the + snapshot is passed in. + #> + param( + [Parameter(Mandatory)] [AllowEmptyCollection()] [object[]]$Snapshot, + [Parameter(Mandatory)] [string]$InstallRoot, + [int]$ProcessId = 0, + [Nullable[datetime]]$ProcessStartTime = $null + ) + @($Snapshot | Where-Object { + $isHost = $ProcessId -gt 0 -and $_.Id -eq $ProcessId -and + ($null -eq $ProcessStartTime -or $null -eq $_.StartTime -or [math]::Abs(([datetime]$_.StartTime - [datetime]$ProcessStartTime).TotalSeconds) -lt 1) + $isUnder = $_.Path -and (Test-OtOpcUaPathWithin -Path ([string]$_.Path) -Ancestor $InstallRoot) + $isHost -or $isUnder + }) +} + +function Wait-OtOpcUaInstallReleased { + <# + .SYNOPSIS + After the service reports Stopped, waits until no process pins InstallRoot (Find-OtOpcUaInstallHolder), polling + every -PollMilliseconds for at most -TimeoutSeconds; throws with the holders' names, PIDs and paths when they are + still there (lmxopcua#908: SCM reports Stopped while the host is still exiting, and the folder move fails "in use"). + Returns the seconds waited. -GetSnapshot replaces the process list (tests). + #> + param( + [Parameter(Mandatory)] [string]$InstallRoot, + [int]$ProcessId = 0, + [Nullable[datetime]]$ProcessStartTime = $null, + [int]$TimeoutSeconds = 60, + [int]$PollMilliseconds = 250, + [scriptblock]$GetSnapshot = { Get-OtOpcUaProcessSnapshot } + ) + $sw = [System.Diagnostics.Stopwatch]::StartNew() + while ($true) { + $holders = @(Find-OtOpcUaInstallHolder -Snapshot @(& $GetSnapshot) -InstallRoot $InstallRoot -ProcessId $ProcessId -ProcessStartTime $ProcessStartTime) + if ($holders.Count -eq 0) { return $sw.Elapsed.TotalSeconds } + if ($sw.Elapsed.TotalSeconds -ge $TimeoutSeconds) { + $list = ($holders | ForEach-Object { '{0} (PID {1}, {2})' -f $_.Name, $_.Id, $(if ($_.Path) { $_.Path } else { 'path unreadable' }) }) -join '; ' + throw ("$InstallRoot is still in use $TimeoutSeconds s after the service stopped (lmxopcua#908): $list. Nothing was moved. " + + 'Wait for or end those processes, then re-run the swap (or raise -ProcessExitTimeoutSeconds).') + } + Start-Sleep -Milliseconds $PollMilliseconds + } +} + +function Test-OtOpcUaSharingViolation { + # True when a folder rename failed because something under it is open (lmxopcua#908, all seen on windev): + # [IO.Directory]::Move reports an IOException 0x80070005 "Access to the path '...' is denied." on both PowerShell hosts; + # Move-Item reports "The process cannot access the file because it is being used by another process" (IOException, + # ERROR_SHARING_VIOLATION 32 / ERROR_LOCK_VIOLATION 33) on PowerShell 7 and IOException COR_E_IO "Access to the path + # ... is denied." on 5.1. An ACL denial is an UnauthorizedAccessException and is not matched; an IOException of the + # access-denied shape that is a real denial only costs the bounded retry budget. Checks the exception and every inner one. + param([Exception]$Exception) + for ($e = $Exception; $e; $e = $e.InnerException) { + if ($e -is [System.IO.IOException] -and (($e.HResult -band 0xFFFF) -in 32, 33)) { return $true } + if ($e.Message -like '*being used by another process*') { return $true } + if ($e -is [System.IO.IOException] -and (($e.HResult -band 0xFFFF) -eq 5 -or $e.Message -like 'Access to the path*is denied*')) { return $true } + } + $false +} + +function Invoke-OtOpcUaMoveWithRetry { + <# + .SYNOPSIS + Runs -Move (the first folder rename of the swap, before anything has moved) and retries it while it fails with a + sharing violation, for at most -RetrySeconds (lmxopcua#908 backstop; the process wait is the fix). Any other error, + or a sharing violation past the budget, is rethrown. Returns the number of attempts. + #> + param( + [Parameter(Mandatory)] [scriptblock]$Move, + [int]$RetrySeconds = 10, + [int]$DelayMilliseconds = 500 + ) + $sw = [System.Diagnostics.Stopwatch]::StartNew() + $attempt = 0 + while ($true) { + $attempt++ + try { & $Move; return $attempt } + catch { + if (-not (Test-OtOpcUaSharingViolation -Exception $_.Exception) -or $sw.Elapsed.TotalSeconds -ge $RetrySeconds) { throw } + Write-Warning ("move attempt {0} failed (in use), retrying in {1} ms: {2}" -f $attempt, $DelayMilliseconds, $_.Exception.Message) + Start-Sleep -Milliseconds $DelayMilliseconds + } + } +} + Export-ModuleMember -Function Get-OtOpcUaServiceSid, Get-OtOpcUaPeCacheSddl, Get-OtOpcUaPeEntriesAce, Get-OtOpcUaScriptWorkerFolderAce, Get-OtOpcUaInstallRootDenyAce, Resolve-OtOpcUaDenyPaths, Get-ScriptWorkerIsolationPlan, Invoke-ScriptWorkerIsolationStep, Get-OtOpcUaRecordedDenyPaths, Find-OtOpcUaDenyCoverageGap, Get-OtOpcUaIsolationSettings, Test-OtOpcUaPathWithin, Test-ScriptWorkerIsolationStep, Test-OtOpcUaRecordKeyAcl, Test-OtOpcUaStepInInstallRoot, Select-ScriptWorkerIsolationPlanScope, - Get-OtOpcUaCoverageWalkRoots + Get-OtOpcUaCoverageWalkRoots, Get-OtOpcUaProcessSnapshot, Find-OtOpcUaInstallHolder, Wait-OtOpcUaInstallReleased, + Test-OtOpcUaSharingViolation, Invoke-OtOpcUaMoveWithRetry diff --git a/scripts/install/tests/Test-ScriptWorkerIsolation.Windows.ps1 b/scripts/install/tests/Test-ScriptWorkerIsolation.Windows.ps1 index b8862a1ac..61050f430 100644 --- a/scripts/install/tests/Test-ScriptWorkerIsolation.Windows.ps1 +++ b/scripts/install/tests/Test-ScriptWorkerIsolation.Windows.ps1 @@ -3,7 +3,8 @@ Windows-executed test of Install-ScriptWorkerIsolation.ps1 (lmxopcua#883, #890, #901): apply -> re-apply -> verify -> shorter -DenyPaths -> -Verify -> folder swap -> -Verify fails -> -Reapply -> -Verify passes -> -Reapply unchanged -> Invoke-OtOpcUaInstallSwap.ps1 (re-apply + verify built in; refuses a pre-#901 record) -> -Scope InstallRoot/Outside - after a swap, the swap's pre-stop gate, -NoStart, a -NoCarry rollback (lmxopcua#905) -> uninstall (from the record, + after a swap, the swap's pre-stop gate, -NoStart, a -NoCarry rollback (lmxopcua#905) -> a process still + running from InstallRoot after the stop: waited for, timed out clearly, and an unseen handle retried (lmxopcua#908) -> uninstall (from the record, without -DenyPaths) -> uninstall. ACL level only; the restricted-token proof that the worker cannot read the denied folders is InstallerLayoutIsolationTests (ScriptWorker.Tests, OTOPCUA_ISOLATION_INSTALLER_E2E=1). The dry-run harness (Test-ScriptWorkerIsolation.ps1) only asserts plan data; it cannot see host-specific failures such as @@ -323,6 +324,90 @@ try { $vEnd = Invoke-Mode 'Verify' Assert ($vEnd.Exit -eq 0) 'full verify passes after the rollback' + # lmxopcua#908: on QA the host was still exiting when SCM said Stopped, and the first folder move failed "in use". + # Stand-in for that host: a copy of powershell.exe under InstallRoot (so its image is under InstallRoot) holding + # logs\host908.log open (FileShare.ReadWrite, no Delete: a folder rename fails while it is open; a running image alone + # does not block it, checked on windev) until a release file appears. The swap runs as a child whose output is + # watched, so the holder is released a fixed time after the swap reaches a given step (its pre-stop isolation gate + # takes several seconds, longer than any fixed holder lifetime would be safe against). + $winPs = Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\powershell.exe' + $holderLog = Join-Path $root 'logs\host908.log' + $release = Join-Path $scratch 'release908' + function Start-InstallHolder([string]$Image) { + Remove-Item -LiteralPath $release -Force -ErrorAction SilentlyContinue + New-Item -ItemType Directory -Path (Join-Path $root 'logs') -Force | Out-Null + if ($Image -like "$root\*") { New-Item -ItemType Directory -Path (Split-Path -Parent $Image) -Force | Out-Null; Copy-Item -LiteralPath $winPs -Destination $Image -Force } + $cmd = "`$f = [IO.File]::Open('$holderLog', 'OpenOrCreate', 'Write', 'ReadWrite'); `$t = [Diagnostics.Stopwatch]::StartNew(); " + + "while (-not (Test-Path -LiteralPath '$release') -and `$t.Elapsed.TotalSeconds -lt 120) { Start-Sleep -Milliseconds 50 }; `$f.Dispose()" + $enc = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($cmd)) + $p = Start-Process -FilePath $Image -ArgumentList '-NoProfile', '-NonInteractive', '-EncodedCommand', $enc -WindowStyle Hidden -PassThru + $deadline = [DateTime]::UtcNow.AddSeconds(15) + while ([DateTime]::UtcNow -lt $deadline -and -not $p.HasExited -and -not (Test-Path -LiteralPath $holderLog)) { Start-Sleep -Milliseconds 100 } + Start-Sleep -Milliseconds 300 + if ($p.HasExited) { throw "holder exited early ($($p.ExitCode))" } + $p + } + function Invoke-SwapReleasing([string[]]$SwapArgs, [string]$Marker, [int]$ReleaseAfterMs, $Holder) { + # Runs the swap; ReleaseAfterMs after $Marker appears in its output, releases the holder. Returns output + exit code. + $out = Join-Path $scratch 'swap908.out'; $err = Join-Path $scratch 'swap908.err' + $argList = @('-NoProfile', '-ExecutionPolicy', 'Bypass', '-File', "`"$swapScript`"") + ($SwapArgs | ForEach-Object { if ($_ -match '\s') { "`"$_`"" } else { $_ } }) + $sp = Start-Process -FilePath $exe -ArgumentList $argList -RedirectStandardOutput $out -RedirectStandardError $err -WindowStyle Hidden -PassThru + $null = $sp.Handle # Windows PowerShell 5.1: without an open handle ExitCode is $null after the exit + $seenAt = $null + while (-not $sp.HasExited) { + if (-not $seenAt -and (Test-Path $out) -and ((Get-Content -LiteralPath $out -Raw -ErrorAction SilentlyContinue) -like "*$Marker*")) { $seenAt = [DateTime]::UtcNow } + if ($seenAt -and ([DateTime]::UtcNow - $seenAt).TotalMilliseconds -ge $ReleaseAfterMs -and -not (Test-Path $release)) { New-Item -ItemType File -Path $release | Out-Null } + Start-Sleep -Milliseconds 50 + } + $sp.WaitForExit() + if (-not (Test-Path $release)) { New-Item -ItemType File -Path $release | Out-Null } + $Holder.WaitForExit(30000) | Out-Null + $text = ((Get-Content -LiteralPath $out -Raw) + (Get-Content -LiteralPath $err -Raw)) -replace "\x1b\[[0-9;]*m", '' # no ANSI colour codes + [pscustomobject]@{ Exit = $sp.ExitCode; Output = $text } + } + function New-StubCandidate([string]$Dir, [string]$Stub) { + New-Item -ItemType Directory -Path (Join-Path $Dir 'script-worker') -Force | Out-Null + Set-Content -LiteralPath (Join-Path $Dir 'script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe') -Value $Stub + } + $holderExe = Join-Path $root 'holder908\OtOpcUa.Host908.exe' + $common = @('-Carry', 'appsettings*.json,pki', '-NoServiceControl', '-WorkerServiceName', $WorkerServiceName) + + Write-Host 'lmxopcua#908: a process running from InstallRoot that outlives the budget: fails clearly, nothing moved' + $cand8a = Join-Path $scratch 'install.new8a'; $prev8a = Join-Path $scratch 'install.bak.8a' + New-StubCandidate $cand8a 'stub v8a' + $hold = Start-InstallHolder $holderExe + $r8a = Invoke-SwapReleasing (@('-InstallRoot', $root, '-Candidate', $cand8a, '-PreviousInstall', $prev8a, '-ProcessExitTimeoutSeconds', '1') + $common) 'Swap complete' 0 $hold + $sq8a = & $squash $r8a.Output + Write-Host (" {0,-34} exit {1}; {2}" -f 'swap, holder kept, budget 1 s', $r8a.Exit, $(if ($sq8a -match 'stillinuse1safter') { 'refused: still in use' } else { 'NOT refused' })) + Assert ($r8a.Exit -ne 0 -and $sq8a -match 'stillinuse1safter.*OtOpcUa\.Host908\.exe\(PID' -and $sq8a -match 'Nothingwasmoved' -and + (Test-Path $cand8a) -and -not (Test-Path $prev8a) -and (Test-Path $holderExe)) "times out naming the holder, nothing moved ($($r8a.Output))" + Remove-Item -LiteralPath $holderLog -Force -ErrorAction SilentlyContinue + + Write-Host 'lmxopcua#908: a process running from InstallRoot until ~2 s after the wait starts: the swap waits for it, then moves' + $hold = Start-InstallHolder $holderExe + $r8b = Invoke-SwapReleasing (@('-InstallRoot', $root, '-Candidate', $cand8a, '-PreviousInstall', $prev8a, '-MoveRetrySeconds', '0') + $common) '==> wait for the host process' 2000 $hold + $released = [regex]::Match($r8b.Output, 'released after ([\d.,]+) s').Groups[1].Value + Write-Host (" {0,-34} exit {1}; released after {2} s" -f 'swap, holder +2 s (no move retry)', $r8b.Exit, $released) + Assert ($r8b.Exit -eq 0 -and $r8b.Output -match 'Swap complete' -and $released -and [double]($released -replace ',', '.') -ge 1.5 -and + (Get-Content -LiteralPath $workerExe) -eq 'stub v8a' -and $r8b.Output -match 'VERIFY OK \(scope InstallRoot\)') "waited for the holder, then swapped and verified ($($r8b.Output))" + + Write-Host 'lmxopcua#908: a handle the process wait cannot see (holder image outside InstallRoot): the first move is retried' + $cand8c = Join-Path $scratch 'install.new8c'; $prev8c = Join-Path $scratch 'install.bak.8c' + New-StubCandidate $cand8c 'stub v8c' + $hold = Start-InstallHolder $winPs + $r8c = Invoke-SwapReleasing (@('-InstallRoot', $root, '-Candidate', $cand8c, '-PreviousInstall', $prev8c, '-MoveRetrySeconds', '20') + $common) "==> move $root ->" 2000 $hold + $attempt = [regex]::Match($r8c.Output, 'moved on attempt (\d+)').Groups[1].Value + Write-Host (" {0,-34} exit {1}; moved on attempt {2}" -f 'swap, outside holder +2 s', $r8c.Exit, $attempt) + Assert ($r8c.Exit -eq 0 -and $attempt -and [int]$attempt -gt 1 -and (Get-Content -LiteralPath $workerExe) -eq 'stub v8c' -and + $r8c.Output -match 'VERIFY OK \(scope InstallRoot\)') "the in-use first move was retried and then went through ($($r8c.Output))" + $prevItems = @(Get-ChildItem -LiteralPath $prev8c -Recurse -Force | ForEach-Object { $_.FullName.Substring($prev8c.Length + 1) }) + Assert (-not ($prevItems -contains 'install') -and ($prevItems -contains 'logs\host908.log') -and + (Get-Content -LiteralPath (Join-Path $prev8c 'script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe')) -eq 'stub v8a') ` + "the previous install moved whole: no partial tree from a failed attempt, nothing nested into it ($($prevItems -join ', '))" + Remove-Item -LiteralPath $release -Force -ErrorAction SilentlyContinue + $v8 = Invoke-Mode 'Verify' + Assert ($v8.Exit -eq 0) 'full verify passes after the #908 swaps' + Write-Host 'Uninstall WITHOUT -DenyPaths (the record drives it), twice' $u1 = Invoke-Installer @('-Uninstall') Assert ($u1.Exit -eq 0) "uninstall exits 0 ($($u1.Output))" diff --git a/scripts/install/tests/Test-ScriptWorkerIsolation.ps1 b/scripts/install/tests/Test-ScriptWorkerIsolation.ps1 index 96b92f3f3..c0547ed17 100644 --- a/scripts/install/tests/Test-ScriptWorkerIsolation.ps1 +++ b/scripts/install/tests/Test-ScriptWorkerIsolation.ps1 @@ -257,15 +257,89 @@ function First-Offset([string]$Name, [string]$Contains = '') { $hit = @($cmds | Where-Object { $_.GetCommandName() -eq $Name -and $_.Extent.Text -like "*$Contains*" } | Sort-Object { $_.Extent.StartOffset }) if ($hit.Count) { $hit[0].Extent.StartOffset } else { -1 } } +$swapText = $swapAst.Extent.Text +function Text-Offset([string]$Text) { $swapText.IndexOf($Text, [StringComparison]::Ordinal) } $iso = @($cmds | Where-Object { $_.GetCommandName() -eq 'Invoke-Isolation' }) Assert ($iso.Count -eq 4 -and @($iso | Where-Object { $_.CommandElements.Count -ne 3 }).Count -eq 0) "every Invoke-Isolation call names a scope ($(@($iso | ForEach-Object { $_.Extent.Text }) -join ' | '))" $order = @( (First-Offset 'Invoke-Isolation' "'Reapply' 'Outside'"), (First-Offset 'Invoke-Isolation' "'Verify' 'Outside'"), (First-Offset 'Stop-Service'), - (First-Offset 'Move-Item' '$Candidate'), (First-Offset 'Invoke-Isolation' "'Reapply' 'InstallRoot'"), + (Text-Offset '[System.IO.Directory]::Move($Candidate, $InstallRoot)'), (First-Offset 'Invoke-Isolation' "'Reapply' 'InstallRoot'"), (First-Offset 'Invoke-Isolation' "'Verify' 'InstallRoot'"), (First-Offset 'Start-Service')) Assert (@($order | Where-Object { $_ -lt 0 }).Count -eq 0 -and (($order -join ',') -eq (@($order | Sort-Object) -join ','))) ` "Reapply/Verify Outside -> stop -> swap -> Reapply/Verify InstallRoot -> start ($($order -join ','))" +$swapExit = First-Offset 'Wait-OtOpcUaInstallReleased' +$swapRetry = First-Offset 'Invoke-OtOpcUaMoveWithRetry' '::Move($InstallRoot, $PreviousInstall)' +$swapPid = First-Offset 'Get-CimInstance' 'Win32_Service' +Assert ($swapPid -ge 0 -and $swapPid -lt (First-Offset 'Stop-Service') -and (First-Offset 'Stop-Service') -lt $swapExit -and $swapExit -lt $swapRetry -and + $swapRetry -lt (Text-Offset '[System.IO.Directory]::Move($Candidate')) ` + "lmxopcua#908: host PID captured -> stop -> wait for the processes to exit -> first move (with retry) -> candidate move ($swapPid,$(First-Offset 'Stop-Service'),$swapExit,$swapRetry,$(Text-Offset '[System.IO.Directory]::Move($Candidate'))" +function Test-InsideCommand($Ast, [string]$Name) { + for ($p = $Ast.Parent; $p; $p = $p.Parent) { if ($p -is [System.Management.Automation.Language.CommandAst] -and $p.GetCommandName() -eq $Name) { return $true } } + $false +} +$firstMoves = @($swapAst.FindAll({ param($n) $n -is [System.Management.Automation.Language.InvokeMemberExpressionAst] -and + $n.Extent.Text -like '*::Move($InstallRoot, $PreviousInstall)' }, $true)) +Assert ($firstMoves.Count -eq 1 -and @($firstMoves | Where-Object { -not (Test-InsideCommand $_ 'Invoke-OtOpcUaMoveWithRetry') }).Count -eq 0) ` + "the first move (InstallRoot -> PreviousInstall) runs only inside Invoke-OtOpcUaMoveWithRetry ($($firstMoves.Count))" +$moveItems = @($cmds | Where-Object { $_.GetCommandName() -eq 'Move-Item' }) +Assert ($moveItems.Count -eq 0) "no Move-Item: PowerShell 7's falls back to a partial item-by-item move when the rename fails (lmxopcua#908) ($(@($moveItems | ForEach-Object { $_.Extent.Text }) -join ' | '))" + +Write-Host 'Host process exit wait (lmxopcua#908; fake process lists)' +$qaApp = 'E:\ApiInstall\OtOpcUa\app' +$t0 = [datetime]::new(2026, 11, 13, 12, 0, 0, [DateTimeKind]::Utc) +$hostP = [pscustomobject]@{ Id = 4100; Name = 'OtOpcUa.Host.exe'; Path = "$qaApp\OtOpcUa.Host.exe"; StartTime = $t0 } +$workerP = [pscustomobject]@{ Id = 4200; Name = 'ZB.MOM.WW.OtOpcUa.ScriptWorker.exe'; Path = "$qaApp\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe"; StartTime = $t0 } +$hostNoPath = [pscustomobject]@{ Id = 4100; Name = 'OtOpcUa.Host.exe'; Path = ''; StartTime = $t0 } +$others = @( + [pscustomobject]@{ Id = 10; Name = 'svchost.exe'; Path = 'C:\Windows\System32\svchost.exe'; StartTime = $t0 }, + [pscustomobject]@{ Id = 11; Name = 'OtOpcUa.Host.exe'; Path = "$qaApp.new-76e73e7d\OtOpcUa.Host.exe"; StartTime = $t0 }, # the candidate: a prefix sibling + [pscustomobject]@{ Id = 12; Name = 'pwsh.exe'; Path = $null; StartTime = $null }) +$h = @(Find-OtOpcUaInstallHolder -Snapshot (@($hostP, $workerP) + $others) -InstallRoot "$qaApp\" -ProcessId 4100 -ProcessStartTime $t0) +Assert ((@($h | ForEach-Object { $_.Id }) -join ',') -eq '4100,4200') "host + worker under app\ hold it; the candidate folder and others do not ($(@($h | ForEach-Object { $_.Id }) -join ','))" +$h = @(Find-OtOpcUaInstallHolder -Snapshot (@($hostNoPath) + $others) -InstallRoot $qaApp -ProcessId 4100 -ProcessStartTime $t0) +Assert ((@($h | ForEach-Object { $_.Id }) -join ',') -eq '4100') "the host counts by PID even when its image path is unreadable ($(@($h | ForEach-Object { $_.Id }) -join ','))" +$reused = [pscustomobject]@{ Id = 4100; Name = 'notepad.exe'; Path = 'C:\Windows\notepad.exe'; StartTime = $t0.AddSeconds(30) } +$h = @(Find-OtOpcUaInstallHolder -Snapshot (@($reused) + $others) -InstallRoot $qaApp -ProcessId 4100 -ProcessStartTime $t0) +Assert ($h.Count -eq 0) 'a different process that reused the PID does not count' +$h = @(Find-OtOpcUaInstallHolder -Snapshot $others -InstallRoot $qaApp) +Assert ($h.Count -eq 0) 'nothing under app\ and no PID: released' + +# The QA case: SCM said Stopped, the host + worker are gone ~1 s later. The wait must see them, then return. +$script:calls = 0 +$fake = { $script:calls++; if ($script:calls -le 3) { @($hostP, $workerP) + $others } else { $others } } +$waited = Wait-OtOpcUaInstallReleased -InstallRoot $qaApp -ProcessId 4100 -ProcessStartTime $t0 -TimeoutSeconds 10 -PollMilliseconds 20 -GetSnapshot $fake +Assert ($script:calls -eq 4) "waits while the host and worker are still exiting, returns once they are gone (snapshots taken: $($script:calls), $([math]::Round($waited, 2)) s)" +$script:calls = 0 +$null = Wait-OtOpcUaInstallReleased -InstallRoot $qaApp -ProcessId 4100 -TimeoutSeconds 10 -PollMilliseconds 20 -GetSnapshot { $script:calls++; $others } +Assert ($script:calls -eq 1) "already released: one snapshot, no wait ($($script:calls))" +$sw908 = [System.Diagnostics.Stopwatch]::StartNew() +Assert-Throws { Wait-OtOpcUaInstallReleased -InstallRoot $qaApp -ProcessId 4100 -TimeoutSeconds 1 -PollMilliseconds 50 -GetSnapshot { @($workerP) + $others } } ` + '*still in use 1 s after*ZB.MOM.WW.OtOpcUa.ScriptWorker.exe (PID 4200*Nothing was moved*' 'a process that never exits fails clearly, naming it' +Assert ($sw908.Elapsed.TotalSeconds -lt 5) "the wait is bounded ($([math]::Round($sw908.Elapsed.TotalSeconds, 1)) s for a 1 s budget)" + +Write-Host 'First-move retry on a sharing violation (lmxopcua#908)' +$inUse = [System.IO.IOException]::new('The process cannot access the file because it is being used by another process.', -2147024864) # 0x80070020 +Assert (Test-OtOpcUaSharingViolation -Exception $inUse) 'ERROR_SHARING_VIOLATION is recognised' +Assert (Test-OtOpcUaSharingViolation -Exception ([System.Management.Automation.RuntimeException]::new('wrapped', $inUse))) 'also when wrapped' +Assert (-not (Test-OtOpcUaSharingViolation -Exception ([System.IO.IOException]::new('Cannot create a file when that file already exists.', -2147024713)))) 'ERROR_ALREADY_EXISTS is not' +Assert (Test-OtOpcUaSharingViolation -Exception ([System.Management.Automation.MethodInvocationException]::new('Exception calling "Move" with "2" argument(s)', + [System.IO.IOException]::new("Access to the path 'E:\ApiInstall\OtOpcUa\app' is denied.", -2147024891)))) ` + 'the [IO.Directory]::Move shape (MethodInvocationException over IOException 0x80070005, both hosts, windev)' +Assert (Test-OtOpcUaSharingViolation -Exception ([System.IO.IOException]::new("Access to the path 'E:\ApiInstall\OtOpcUa\app' is denied.", -2146232800))) ` + 'the Windows PowerShell 5.1 shape of the same failure (IOException COR_E_IO "Access to the path ... is denied", windev)' +Assert (-not (Test-OtOpcUaSharingViolation -Exception ([System.UnauthorizedAccessException]::new("Access to the path 'x' is denied.")))) 'an UnauthorizedAccessException is not' +$script:calls = 0 +$n = try { Invoke-OtOpcUaMoveWithRetry -RetrySeconds 5 -DelayMilliseconds 10 -Move { $script:calls++; if ($script:calls -le 2) { throw $inUse } } 3>$null } catch { "threw: $($_.Exception.Message)" } +Assert ($n -eq 3 -and $script:calls -eq 3) "in use twice, then moved on attempt 3 ($n)" +$script:calls = 0 +Assert-Throws { Invoke-OtOpcUaMoveWithRetry -RetrySeconds 5 -DelayMilliseconds 10 -Move { $script:calls++; throw [System.UnauthorizedAccessException]::new('Access to the path is denied.') } 3>$null } ` + '*denied*' 'any other error is not retried' +Assert ($script:calls -eq 1) " (one attempt: $($script:calls))" +$script:calls = 0 +Assert-Throws { Invoke-OtOpcUaMoveWithRetry -RetrySeconds 0 -DelayMilliseconds 10 -Move { $script:calls++; throw $inUse } 3>$null } '*being used by another process*' '-MoveRetrySeconds 0: no retry' +Assert ($script:calls -eq 1) " (one attempt: $($script:calls))" + if ($IsWindows) { Write-Host 'SDDL parses on Windows' $ds = [System.Security.AccessControl.DirectorySecurity]::new()