fix(install): swap waits for the host process to exit before moving the folder (#908)

On QA (76e73e7d deploy) Invoke-OtOpcUaInstallSwap.ps1's first Move-Item of
app\ failed "file in use" right after the service reported Stopped: SCM
reports Stopped while the host is still exiting (e.g. a log file under
InstallRoot still open).

- After the stop, wait (bounded, -ProcessExitTimeoutSeconds, default 60)
  until the service's host process (PID + start time captured before the
  stop) and every process imaged under InstallRoot have exited; otherwise
  throw naming them, with nothing moved.
- The renames use [IO.Directory]::Move instead of Move-Item. On windev,
  PowerShell 7's Move-Item falls back to an item-by-item move when the
  rename fails because a file under the folder is open, and leaves a partial
  PreviousInstall tree behind (a retry then nests into it).
- The first rename is retried for at most -MoveRetrySeconds (default 10)
  while it fails because something under InstallRoot is open (a backstop
  for a holder whose image is outside InstallRoot).
- -Verify/-Reapply/-Scope semantics unchanged.

Tests: dry-run harness gains fake-process-list tests for the wait and the
retry plus AST order checks (145 passed); the Windows harness gains three
real cases (holder past the budget, holder released 2 s into the wait,
holder outside InstallRoot) - 77/0 on pwsh 7 and Windows PowerShell 5.1 on
windev; with the wait removed the first two fail.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

scripts/install/tests/Test-ScriptWorkerIsolation.Windows.ps1

@@ -3,7 +3,8 @@
     Windows-executed test of Install-ScriptWorkerIsolation.ps1 (lmxopcua#883, #890, #901): apply -> re-apply -> verify -> shorter
     -DenyPaths -> -Verify -> folder swap -> -Verify fails -> -Reapply -> -Verify passes -> -Reapply unchanged ->
     Invoke-OtOpcUaInstallSwap.ps1 (re-apply + verify built in; refuses a pre-#901 record) -> -Scope InstallRoot/Outside
-    after a swap, the swap's pre-stop gate, -NoStart, a -NoCarry rollback (lmxopcua#905) -> uninstall (from the record,
+    after a swap, the swap's pre-stop gate, -NoStart, a -NoCarry rollback (lmxopcua#905) -> a process still
+    running from InstallRoot after the stop: waited for, timed out clearly, and an unseen handle retried (lmxopcua#908) -> uninstall (from the record,
     without -DenyPaths) -> uninstall. ACL level only; the restricted-token proof that
     the worker cannot read the denied folders is InstallerLayoutIsolationTests (ScriptWorker.Tests, OTOPCUA_ISOLATION_INSTALLER_E2E=1).
     The dry-run harness (Test-ScriptWorkerIsolation.ps1) only asserts plan data; it cannot see host-specific failures such as
@@ -323,6 +324,90 @@ try {
     $vEnd = Invoke-Mode 'Verify'
     Assert ($vEnd.Exit -eq 0) 'full verify passes after the rollback'
 
+    # lmxopcua#908: on QA the host was still exiting when SCM said Stopped, and the first folder move failed "in use".
+    # Stand-in for that host: a copy of powershell.exe under InstallRoot (so its image is under InstallRoot) holding
+    # logs\host908.log open (FileShare.ReadWrite, no Delete: a folder rename fails while it is open; a running image alone
+    # does not block it, checked on windev) until a release file appears. The swap runs as a child whose output is
+    # watched, so the holder is released a fixed time after the swap reaches a given step (its pre-stop isolation gate
+    # takes several seconds, longer than any fixed holder lifetime would be safe against).
+    $winPs = Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\powershell.exe'
+    $holderLog = Join-Path $root 'logs\host908.log'
+    $release = Join-Path $scratch 'release908'
+    function Start-InstallHolder([string]$Image) {
+        Remove-Item -LiteralPath $release -Force -ErrorAction SilentlyContinue
+        New-Item -ItemType Directory -Path (Join-Path $root 'logs') -Force | Out-Null
+        if ($Image -like "$root\*") { New-Item -ItemType Directory -Path (Split-Path -Parent $Image) -Force | Out-Null; Copy-Item -LiteralPath $winPs -Destination $Image -Force }
+        $cmd = "`$f = [IO.File]::Open('$holderLog', 'OpenOrCreate', 'Write', 'ReadWrite'); `$t = [Diagnostics.Stopwatch]::StartNew(); " +
+            "while (-not (Test-Path -LiteralPath '$release') -and `$t.Elapsed.TotalSeconds -lt 120) { Start-Sleep -Milliseconds 50 }; `$f.Dispose()"
+        $enc = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($cmd))
+        $p = Start-Process -FilePath $Image -ArgumentList '-NoProfile', '-NonInteractive', '-EncodedCommand', $enc -WindowStyle Hidden -PassThru
+        $deadline = [DateTime]::UtcNow.AddSeconds(15)
+        while ([DateTime]::UtcNow -lt $deadline -and -not $p.HasExited -and -not (Test-Path -LiteralPath $holderLog)) { Start-Sleep -Milliseconds 100 }
+        Start-Sleep -Milliseconds 300
+        if ($p.HasExited) { throw "holder exited early ($($p.ExitCode))" }
+        $p
+    }
+    function Invoke-SwapReleasing([string[]]$SwapArgs, [string]$Marker, [int]$ReleaseAfterMs, $Holder) {
+        # Runs the swap; ReleaseAfterMs after $Marker appears in its output, releases the holder. Returns output + exit code.
+        $out = Join-Path $scratch 'swap908.out'; $err = Join-Path $scratch 'swap908.err'
+        $argList = @('-NoProfile', '-ExecutionPolicy', 'Bypass', '-File', "`"$swapScript`"") + ($SwapArgs | ForEach-Object { if ($_ -match '\s') { "`"$_`"" } else { $_ } })
+        $sp = Start-Process -FilePath $exe -ArgumentList $argList -RedirectStandardOutput $out -RedirectStandardError $err -WindowStyle Hidden -PassThru
+        $null = $sp.Handle   # Windows PowerShell 5.1: without an open handle ExitCode is $null after the exit
+        $seenAt = $null
+        while (-not $sp.HasExited) {
+            if (-not $seenAt -and (Test-Path $out) -and ((Get-Content -LiteralPath $out -Raw -ErrorAction SilentlyContinue) -like "*$Marker*")) { $seenAt = [DateTime]::UtcNow }
+            if ($seenAt -and ([DateTime]::UtcNow - $seenAt).TotalMilliseconds -ge $ReleaseAfterMs -and -not (Test-Path $release)) { New-Item -ItemType File -Path $release | Out-Null }
+            Start-Sleep -Milliseconds 50
+        }
+        $sp.WaitForExit()
+        if (-not (Test-Path $release)) { New-Item -ItemType File -Path $release | Out-Null }
+        $Holder.WaitForExit(30000) | Out-Null
+        $text = ((Get-Content -LiteralPath $out -Raw) + (Get-Content -LiteralPath $err -Raw)) -replace "\x1b\[[0-9;]*m", ''   # no ANSI colour codes
+        [pscustomobject]@{ Exit = $sp.ExitCode; Output = $text }
+    }
+    function New-StubCandidate([string]$Dir, [string]$Stub) {
+        New-Item -ItemType Directory -Path (Join-Path $Dir 'script-worker') -Force | Out-Null
+        Set-Content -LiteralPath (Join-Path $Dir 'script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe') -Value $Stub
+    }
+    $holderExe = Join-Path $root 'holder908\OtOpcUa.Host908.exe'
+    $common = @('-Carry', 'appsettings*.json,pki', '-NoServiceControl', '-WorkerServiceName', $WorkerServiceName)
+
+    Write-Host 'lmxopcua#908: a process running from InstallRoot that outlives the budget: fails clearly, nothing moved'
+    $cand8a = Join-Path $scratch 'install.new8a'; $prev8a = Join-Path $scratch 'install.bak.8a'
+    New-StubCandidate $cand8a 'stub v8a'
+    $hold = Start-InstallHolder $holderExe
+    $r8a = Invoke-SwapReleasing (@('-InstallRoot', $root, '-Candidate', $cand8a, '-PreviousInstall', $prev8a, '-ProcessExitTimeoutSeconds', '1') + $common) 'Swap complete' 0 $hold
+    $sq8a = & $squash $r8a.Output
+    Write-Host ("    {0,-34} exit {1}; {2}" -f 'swap, holder kept, budget 1 s', $r8a.Exit, $(if ($sq8a -match 'stillinuse1safter') { 'refused: still in use' } else { 'NOT refused' }))
+    Assert ($r8a.Exit -ne 0 -and $sq8a -match 'stillinuse1safter.*OtOpcUa\.Host908\.exe\(PID' -and $sq8a -match 'Nothingwasmoved' -and
+        (Test-Path $cand8a) -and -not (Test-Path $prev8a) -and (Test-Path $holderExe)) "times out naming the holder, nothing moved ($($r8a.Output))"
+    Remove-Item -LiteralPath $holderLog -Force -ErrorAction SilentlyContinue
+
+    Write-Host 'lmxopcua#908: a process running from InstallRoot until ~2 s after the wait starts: the swap waits for it, then moves'
+    $hold = Start-InstallHolder $holderExe
+    $r8b = Invoke-SwapReleasing (@('-InstallRoot', $root, '-Candidate', $cand8a, '-PreviousInstall', $prev8a, '-MoveRetrySeconds', '0') + $common) '==> wait for the host process' 2000 $hold
+    $released = [regex]::Match($r8b.Output, 'released after ([\d.,]+) s').Groups[1].Value
+    Write-Host ("    {0,-34} exit {1}; released after {2} s" -f 'swap, holder +2 s (no move retry)', $r8b.Exit, $released)
+    Assert ($r8b.Exit -eq 0 -and $r8b.Output -match 'Swap complete' -and $released -and [double]($released -replace ',', '.') -ge 1.5 -and
+        (Get-Content -LiteralPath $workerExe) -eq 'stub v8a' -and $r8b.Output -match 'VERIFY OK \(scope InstallRoot\)') "waited for the holder, then swapped and verified ($($r8b.Output))"
+
+    Write-Host 'lmxopcua#908: a handle the process wait cannot see (holder image outside InstallRoot): the first move is retried'
+    $cand8c = Join-Path $scratch 'install.new8c'; $prev8c = Join-Path $scratch 'install.bak.8c'
+    New-StubCandidate $cand8c 'stub v8c'
+    $hold = Start-InstallHolder $winPs
+    $r8c = Invoke-SwapReleasing (@('-InstallRoot', $root, '-Candidate', $cand8c, '-PreviousInstall', $prev8c, '-MoveRetrySeconds', '20') + $common) "==> move $root ->" 2000 $hold
+    $attempt = [regex]::Match($r8c.Output, 'moved on attempt (\d+)').Groups[1].Value
+    Write-Host ("    {0,-34} exit {1}; moved on attempt {2}" -f 'swap, outside holder +2 s', $r8c.Exit, $attempt)
+    Assert ($r8c.Exit -eq 0 -and $attempt -and [int]$attempt -gt 1 -and (Get-Content -LiteralPath $workerExe) -eq 'stub v8c' -and
+        $r8c.Output -match 'VERIFY OK \(scope InstallRoot\)') "the in-use first move was retried and then went through ($($r8c.Output))"
+    $prevItems = @(Get-ChildItem -LiteralPath $prev8c -Recurse -Force | ForEach-Object { $_.FullName.Substring($prev8c.Length + 1) })
+    Assert (-not ($prevItems -contains 'install') -and ($prevItems -contains 'logs\host908.log') -and
+        (Get-Content -LiteralPath (Join-Path $prev8c 'script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe')) -eq 'stub v8a') `
+        "the previous install moved whole: no partial tree from a failed attempt, nothing nested into it ($($prevItems -join ', '))"
+    Remove-Item -LiteralPath $release -Force -ErrorAction SilentlyContinue
+    $v8 = Invoke-Mode 'Verify'
+    Assert ($v8.Exit -eq 0) 'full verify passes after the #908 swaps'
+
     Write-Host 'Uninstall WITHOUT -DenyPaths (the record drives it), twice'
     $u1 = Invoke-Installer @('-Uninstall')
     Assert ($u1.Exit -eq 0) "uninstall exits 0 ($($u1.Output))"