fix(install): swap waits for the host process to exit before moving the folder (#908)

On QA (76e73e7d deploy) Invoke-OtOpcUaInstallSwap.ps1's first Move-Item of
app\ failed "file in use" right after the service reported Stopped: SCM
reports Stopped while the host is still exiting (e.g. a log file under
InstallRoot still open).

- After the stop, wait (bounded, -ProcessExitTimeoutSeconds, default 60)
  until the service's host process (PID + start time captured before the
  stop) and every process imaged under InstallRoot have exited; otherwise
  throw naming them, with nothing moved.
- The renames use [IO.Directory]::Move instead of Move-Item. On windev,
  PowerShell 7's Move-Item falls back to an item-by-item move when the
  rename fails because a file under the folder is open, and leaves a partial
  PreviousInstall tree behind (a retry then nests into it).
- The first rename is retried for at most -MoveRetrySeconds (default 10)
  while it fails because something under InstallRoot is open (a backstop
  for a holder whose image is outside InstallRoot).
- -Verify/-Reapply/-Scope semantics unchanged.

Tests: dry-run harness gains fake-process-list tests for the wait and the
retry plus AST order checks (145 passed); the Windows harness gains three
real cases (holder past the budget, holder released 2 s into the wait,
holder outside InstallRoot) - 77/0 on pwsh 7 and Windows PowerShell 5.1 on
windev; with the wait removed the first two fail.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

docs/ServiceHosting.md

@@ -147,16 +147,24 @@ run the install once with its full arguments (the per-box deploy record lists th
 `scripts/install/Invoke-OtOpcUaInstallSwap.ps1` is the swap with the gate built in: preflight (candidate carries
 `script-worker\`, `PreviousInstall` is fresh and on the same volume, isolation record present when isolation is
 installed, record `InstallRoot` equals `-InstallRoot`) → `-Reapply -Scope Outside` + `-Verify -Scope Outside` **while
-the old build still serves** (a failure stops here: nothing stopped or moved) → stop → move live → previous, candidate →
+the old build still serves** (a failure stops here: nothing stopped or moved) → stop → **wait until the host process
+(PID captured before the stop) and every process whose image is under `InstallRoot` have exited** (lmxopcua#908: SCM can
+report `Stopped` while the host is still exiting, holding e.g. a log file open, and the folder move then fails "in use";
+bounded by `-ProcessExitTimeoutSeconds`, default 60 s, after which the script throws naming the processes, with nothing
+moved) → move live → previous (one `[IO.Directory]::Move` rename that fails as a whole — PowerShell 7's `Move-Item` falls
+back to a partial item-by-item move when the rename fails; retried for at most `-MoveRetrySeconds`, default 10 s, while it
+fails because something under `InstallRoot` is open: a backstop for a handle held by a process outside it), candidate →
 live → copy `-Carry` items back (default `appsettings*.json`, `pki`, `logs`; QA: `'appsettings*.json','logs'`; rollback:
 `-NoCarry`) → `-Reapply -Scope InstallRoot` → `-Verify -Scope InstallRoot` → start (`-NoStart` for a pair window). The
 two scoped gates together check every install step, so the new build never starts unverified; a failed post-swap
 `-Verify` leaves the service **stopped**. The script prints the stopped window and its parts
-(`stopped window (stop issued -> start issued): 6.0 s [stop …, move 0.0 s, carry 0.1 s, isolation 5.8 s]`). Backups,
+(`stopped window (stop issued -> start issued): 6.0 s [stop …, exit …, move 0.0 s, carry 0.1 s, isolation 5.8 s]`; `exit` is the #908 wait). Backups,
 `data\` restore, migrations and health checks stay in the deploy's own steps. Every deploy pack that swaps the folder runs
 it (or `-Reapply` + `-Verify` right after its own swap), and `-Verify` is a post-deploy gate.
 `Test-ScriptWorkerIsolation.Windows.ps1` exercises the whole sequence (swap → `-Verify` fails → `-Reapply` → `-Verify`
-passes → `-Reapply` unchanged → the swap script → the scoped modes, the pre-stop gate, `-NoStart`, a `-NoCarry` rollback)
+passes → `-Reapply` unchanged → the swap script → the scoped modes, the pre-stop gate, `-NoStart`, a `-NoCarry` rollback,
+and the #908 cases: a process running from `InstallRoot` past the budget, one exiting ~2 s after the wait starts, and an in-use
+handle from outside `InstallRoot` that the first-move retry absorbs)
 on a scratch tree; `-TimingFiles N` times the swap on a vd03-like tree (`-ScriptsDir` for a before/after).
 
 **Expected stopped window (lmxopcua#905).** Before #905 the swap ran the full `-Reapply` and `-Verify` while the service

docs/deploy/2026-11-13-vd03-qa-to-76e73e7d.md

@@ -184,6 +184,9 @@ Do not run a mixed pair: `RedundancyStateChanged` changed shape in this range. A
      process to exit; `Invoke-OtOpcUaInstallSwap.ps1` does not.
    - It cost about 20 s of the vq03 pair outage (the retry re-runs `-Scope Outside`).
    - Filed as **lmxopcua#908**. vd03 did not hit it.
+   - **Fixed (lmxopcua#908, closed):** `Invoke-OtOpcUaInstallSwap.ps1` now waits, after `Stopped`, for the host process
+     and every process imaged under `InstallRoot` to exit (`-ProcessExitTimeoutSeconds`, default 60) and retries the first
+     move on a sharing violation (`-MoveRetrySeconds`, default 10). The next deploy needs no hand retry.
 3. `Stop-Service` on the new build took 5.7–6.1 s instead of the 1–3 s target, because of the ~5 s OPC UA session close
    after the cluster leave (see "Observation"). The hand-over, the actual #906 criterion, was 0.36 s and 0.47 s.
 4. Logs: the QA host log for the window rolled to `otopcua-20261113_001.log` on vq04 (size limit). Read it from there.

scripts/install/Invoke-OtOpcUaInstallSwap.ps1

@@ -16,8 +16,14 @@
          PE cache, denies outside InstallRoot, firewall rule, host env) and the coverage scan of the denied folders
          outside InstallRoot.
          If either fails, the script stops here: nothing is stopped or moved.
-      2. Stop the service (unless -NoServiceControl). The stopped window starts here.
-      3. Move InstallRoot -> PreviousInstall, then Candidate -> InstallRoot.
+      2. Stop the service (unless -NoServiceControl). The stopped window starts here. SCM can report Stopped while the
+         host process is still exiting (lmxopcua#908), so the script then waits, at most -ProcessExitTimeoutSeconds, until
+         the service's host process (PID captured before the stop) and every process whose image is under InstallRoot
+         (OtOpcUa.Host, the ScriptWorker) have exited. If one is still there, it throws naming them: nothing has moved.
+      3. Move InstallRoot -> PreviousInstall, then Candidate -> InstallRoot, each one rename ([IO.Directory]::Move; not
+         Move-Item, which in PowerShell 7 falls back to a partial item-by-item move when the rename fails). The first
+         move is retried for at most -MoveRetrySeconds while it fails because something under InstallRoot is open
+         (nothing has moved yet; a backstop for a handle the process wait cannot see).
       4. Copy each -Carry item (relative to InstallRoot; wildcards allowed) from PreviousInstall into the new InstallRoot.
          Folders are copied with robocopy /E /COPY:DAT (data, attributes, timestamps; not ACLs: step 5 sets those).
       5. Isolation, when recorded: -Reapply -Scope InstallRoot, then -Verify -Scope InstallRoot: only the entries a swap
@@ -66,6 +72,13 @@
 .PARAMETER SkipIsolation
     Do not re-apply/verify the isolation even when it is installed. WARNs. Only for a box where it is being removed.
 
+.PARAMETER ProcessExitTimeoutSeconds
+    After the stop, how long to wait for the host process and any process running from InstallRoot to exit before
+    failing with nothing moved (default 60; lmxopcua#908).
+
+.PARAMETER MoveRetrySeconds
+    How long to keep retrying the first folder move while it fails with a sharing violation (default 10; 0 = no retry).
+
 .PARAMETER WorkerServiceName
     Passed to the isolation installer (default OtOpcUaScriptWorker).
 
@@ -96,7 +109,9 @@ param(
     [switch]$NoStart,
     [switch]$SkipIsolation,
     [ValidatePattern('^[A-Za-z0-9_.\-]{1,80}$')] [string]$WorkerServiceName = 'OtOpcUaScriptWorker',
-    [int]$StopTimeoutSeconds = 120
+    [int]$StopTimeoutSeconds = 120,
+    [ValidateRange(0, 3600)] [int]$ProcessExitTimeoutSeconds = 60,
+    [ValidateRange(0, 600)] [int]$MoveRetrySeconds = 10
 )
 
 Set-StrictMode -Version Latest
@@ -131,7 +146,7 @@ if (-not (Test-Path -LiteralPath (Join-Path $Candidate 'script-worker') -PathTyp
 $candidateWorkerExe = Join-Path $Candidate 'script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe'
 if (Test-Path -LiteralPath $PreviousInstall) { throw "PreviousInstall $PreviousInstall already exists; pick a fresh name" }
 if ([IO.Path]::GetPathRoot($InstallRoot) -ne [IO.Path]::GetPathRoot($PreviousInstall) -or [IO.Path]::GetPathRoot($InstallRoot) -ne [IO.Path]::GetPathRoot($Candidate)) {
-    throw 'InstallRoot, Candidate and PreviousInstall must be on the same volume (Move-Item renames)'
+    throw 'InstallRoot, Candidate and PreviousInstall must be on the same volume (the swap renames folders)'
 }
 $isolation = $null
 if (-not $SkipIsolation) {
@@ -169,26 +184,43 @@ if ($isolation) {
 }
 
 # ---- 2. Stop ---------------------------------------------------------------------------------------------------------
+$hostPid = 0; $hostStart = $null
 $window.Start()
 if (-not $NoServiceControl) {
     Step "stop $ServiceName"
     $svc = Get-Service -Name $ServiceName
     if ($svc.Status -ne 'Stopped') {
+        # lmxopcua#908: remember the host process (PID + start time, so a reused PID does not count) before stopping it.
+        $hostPid = [int](Get-CimInstance -ClassName Win32_Service -Filter "Name='$ServiceName'").ProcessId
+        if ($hostPid -gt 0) {
+            $hostProc = Get-Process -Id $hostPid -ErrorAction SilentlyContinue
+            if ($hostProc) { try { $hostStart = $hostProc.StartTime.ToUniversalTime() } catch { } }
+        }
         Stop-Service -Name $ServiceName -Force
         $svc.WaitForStatus('Stopped', [TimeSpan]::FromSeconds($StopTimeoutSeconds))
     }
     Mark 'stop' $window
 }
+# lmxopcua#908: SCM reports Stopped before the host process is gone; the folder move fails "in use" until it is.
+$sw = [System.Diagnostics.Stopwatch]::StartNew()
+Step "wait for the host process (PID $(if ($hostPid) { $hostPid } else { 'n/a' })) and any process under $InstallRoot to exit (at most $ProcessExitTimeoutSeconds s)"
+$waited = Wait-OtOpcUaInstallReleased -InstallRoot $InstallRoot -ProcessId $hostPid -ProcessStartTime $hostStart -TimeoutSeconds $ProcessExitTimeoutSeconds
+Write-Host ('    released after {0:N1} s' -f $waited)
+Mark 'exit' $sw
 
 # ---- 3. Swap ---------------------------------------------------------------------------------------------------------
 $sw = [System.Diagnostics.Stopwatch]::StartNew()
 Step "move $InstallRoot -> $PreviousInstall"
-Move-Item -LiteralPath $InstallRoot -Destination $PreviousInstall
+# lmxopcua#908: [IO.Directory]::Move, not Move-Item. It is one rename (MoveFileEx, same volume: checked in preflight) that
+# fails as a whole. PowerShell 7's Move-Item, when the rename fails because a file under the folder is open, falls back
+# to moving item by item and leaves a partial PreviousInstall tree behind (windev), so a retry would then nest into it.
+$attempts = Invoke-OtOpcUaMoveWithRetry -RetrySeconds $MoveRetrySeconds -Move { [System.IO.Directory]::Move($InstallRoot, $PreviousInstall) }
+if ($attempts -gt 1) { Write-Host "    moved on attempt $attempts" }
 Step "move $Candidate -> $InstallRoot"
-try { Move-Item -LiteralPath $Candidate -Destination $InstallRoot }
+try { [System.IO.Directory]::Move($Candidate, $InstallRoot) }
 catch {
     Write-Warning "candidate move failed; putting the previous install back: $($_.Exception.Message)"
-    Move-Item -LiteralPath $PreviousInstall -Destination $InstallRoot
+    [System.IO.Directory]::Move($PreviousInstall, $InstallRoot)
     throw
 }
 

scripts/install/ScriptWorkerIsolation.psm1

@@ -692,8 +692,125 @@ function Get-OtOpcUaCoverageWalkRoots {
     [pscustomobject]@{ Roots = $outer.ToArray(); Skip = $(if ($Scope -eq 'Outside') { @($root) } else { @() }) }
 }
 
+function Get-OtOpcUaProcessSnapshot {
+    <#
+    .SYNOPSIS
+        Every process as Id, Name, Path (image path; '' when unreadable), StartTime (UTC or $null). Windows: Win32_Process
+        (sees service processes in session 0 when elevated); elsewhere Get-Process (the dry-run harness injects its own).
+    #>
+    if ($PSVersionTable.PSEdition -eq 'Desktop' -or $IsWindows) {
+        foreach ($p in @(Get-CimInstance -ClassName Win32_Process -ErrorAction Stop)) {
+            [pscustomobject]@{
+                Id = [int]$p.ProcessId; Name = [string]$p.Name; Path = [string]$p.ExecutablePath
+                StartTime = $(if ($p.CreationDate) { ([datetime]$p.CreationDate).ToUniversalTime() } else { $null })
+            }
+        }
+    }
+    else {
+        foreach ($p in @(Get-Process)) {
+            $path = ''; $start = $null
+            try { $path = [string]$p.Path } catch { }
+            try { $start = $p.StartTime.ToUniversalTime() } catch { }
+            [pscustomobject]@{ Id = $p.Id; Name = $p.ProcessName; Path = $path; StartTime = $start }
+        }
+    }
+}
+
+function Find-OtOpcUaInstallHolder {
+    <#
+    .SYNOPSIS
+        The processes that still pin InstallRoot (lmxopcua#908): the service's host process (by the PID captured before the
+        stop; when its start time was captured too, a different process reusing the PID does not count) and any process
+        whose image is under InstallRoot (OtOpcUa.Host, ZB.MOM.WW.OtOpcUa.ScriptWorker in script-worker\). Pure: the
+        snapshot is passed in.
+    #>
+    param(
+        [Parameter(Mandatory)] [AllowEmptyCollection()] [object[]]$Snapshot,
+        [Parameter(Mandatory)] [string]$InstallRoot,
+        [int]$ProcessId = 0,
+        [Nullable[datetime]]$ProcessStartTime = $null
+    )
+    @($Snapshot | Where-Object {
+        $isHost = $ProcessId -gt 0 -and $_.Id -eq $ProcessId -and
+            ($null -eq $ProcessStartTime -or $null -eq $_.StartTime -or [math]::Abs(([datetime]$_.StartTime - [datetime]$ProcessStartTime).TotalSeconds) -lt 1)
+        $isUnder = $_.Path -and (Test-OtOpcUaPathWithin -Path ([string]$_.Path) -Ancestor $InstallRoot)
+        $isHost -or $isUnder
+    })
+}
+
+function Wait-OtOpcUaInstallReleased {
+    <#
+    .SYNOPSIS
+        After the service reports Stopped, waits until no process pins InstallRoot (Find-OtOpcUaInstallHolder), polling
+        every -PollMilliseconds for at most -TimeoutSeconds; throws with the holders' names, PIDs and paths when they are
+        still there (lmxopcua#908: SCM reports Stopped while the host is still exiting, and the folder move fails "in use").
+        Returns the seconds waited. -GetSnapshot replaces the process list (tests).
+    #>
+    param(
+        [Parameter(Mandatory)] [string]$InstallRoot,
+        [int]$ProcessId = 0,
+        [Nullable[datetime]]$ProcessStartTime = $null,
+        [int]$TimeoutSeconds = 60,
+        [int]$PollMilliseconds = 250,
+        [scriptblock]$GetSnapshot = { Get-OtOpcUaProcessSnapshot }
+    )
+    $sw = [System.Diagnostics.Stopwatch]::StartNew()
+    while ($true) {
+        $holders = @(Find-OtOpcUaInstallHolder -Snapshot @(& $GetSnapshot) -InstallRoot $InstallRoot -ProcessId $ProcessId -ProcessStartTime $ProcessStartTime)
+        if ($holders.Count -eq 0) { return $sw.Elapsed.TotalSeconds }
+        if ($sw.Elapsed.TotalSeconds -ge $TimeoutSeconds) {
+            $list = ($holders | ForEach-Object { '{0} (PID {1}, {2})' -f $_.Name, $_.Id, $(if ($_.Path) { $_.Path } else { 'path unreadable' }) }) -join '; '
+            throw ("$InstallRoot is still in use $TimeoutSeconds s after the service stopped (lmxopcua#908): $list. Nothing was moved. " +
+                'Wait for or end those processes, then re-run the swap (or raise -ProcessExitTimeoutSeconds).')
+        }
+        Start-Sleep -Milliseconds $PollMilliseconds
+    }
+}
+
+function Test-OtOpcUaSharingViolation {
+    # True when a folder rename failed because something under it is open (lmxopcua#908, all seen on windev):
+    # [IO.Directory]::Move reports an IOException 0x80070005 "Access to the path '...' is denied." on both PowerShell hosts;
+    # Move-Item reports "The process cannot access the file because it is being used by another process" (IOException,
+    # ERROR_SHARING_VIOLATION 32 / ERROR_LOCK_VIOLATION 33) on PowerShell 7 and IOException COR_E_IO "Access to the path
+    # ... is denied." on 5.1. An ACL denial is an UnauthorizedAccessException and is not matched; an IOException of the
+    # access-denied shape that is a real denial only costs the bounded retry budget. Checks the exception and every inner one.
+    param([Exception]$Exception)
+    for ($e = $Exception; $e; $e = $e.InnerException) {
+        if ($e -is [System.IO.IOException] -and (($e.HResult -band 0xFFFF) -in 32, 33)) { return $true }
+        if ($e.Message -like '*being used by another process*') { return $true }
+        if ($e -is [System.IO.IOException] -and (($e.HResult -band 0xFFFF) -eq 5 -or $e.Message -like 'Access to the path*is denied*')) { return $true }
+    }
+    $false
+}
+
+function Invoke-OtOpcUaMoveWithRetry {
+    <#
+    .SYNOPSIS
+        Runs -Move (the first folder rename of the swap, before anything has moved) and retries it while it fails with a
+        sharing violation, for at most -RetrySeconds (lmxopcua#908 backstop; the process wait is the fix). Any other error,
+        or a sharing violation past the budget, is rethrown. Returns the number of attempts.
+    #>
+    param(
+        [Parameter(Mandatory)] [scriptblock]$Move,
+        [int]$RetrySeconds = 10,
+        [int]$DelayMilliseconds = 500
+    )
+    $sw = [System.Diagnostics.Stopwatch]::StartNew()
+    $attempt = 0
+    while ($true) {
+        $attempt++
+        try { & $Move; return $attempt }
+        catch {
+            if (-not (Test-OtOpcUaSharingViolation -Exception $_.Exception) -or $sw.Elapsed.TotalSeconds -ge $RetrySeconds) { throw }
+            Write-Warning ("move attempt {0} failed (in use), retrying in {1} ms: {2}" -f $attempt, $DelayMilliseconds, $_.Exception.Message)
+            Start-Sleep -Milliseconds $DelayMilliseconds
+        }
+    }
+}
+
 Export-ModuleMember -Function Get-OtOpcUaServiceSid, Get-OtOpcUaPeCacheSddl, Get-OtOpcUaPeEntriesAce,
     Get-OtOpcUaScriptWorkerFolderAce, Get-OtOpcUaInstallRootDenyAce, Resolve-OtOpcUaDenyPaths, Get-ScriptWorkerIsolationPlan,
     Invoke-ScriptWorkerIsolationStep, Get-OtOpcUaRecordedDenyPaths, Find-OtOpcUaDenyCoverageGap, Get-OtOpcUaIsolationSettings, Test-OtOpcUaPathWithin,
     Test-ScriptWorkerIsolationStep, Test-OtOpcUaRecordKeyAcl, Test-OtOpcUaStepInInstallRoot, Select-ScriptWorkerIsolationPlanScope,
-    Get-OtOpcUaCoverageWalkRoots
+    Get-OtOpcUaCoverageWalkRoots, Get-OtOpcUaProcessSnapshot, Find-OtOpcUaInstallHolder, Wait-OtOpcUaInstallReleased,
+    Test-OtOpcUaSharingViolation, Invoke-OtOpcUaMoveWithRetry

scripts/install/tests/Test-ScriptWorkerIsolation.Windows.ps1

@@ -3,7 +3,8 @@
     Windows-executed test of Install-ScriptWorkerIsolation.ps1 (lmxopcua#883, #890, #901): apply -> re-apply -> verify -> shorter
     -DenyPaths -> -Verify -> folder swap -> -Verify fails -> -Reapply -> -Verify passes -> -Reapply unchanged ->
     Invoke-OtOpcUaInstallSwap.ps1 (re-apply + verify built in; refuses a pre-#901 record) -> -Scope InstallRoot/Outside
-    after a swap, the swap's pre-stop gate, -NoStart, a -NoCarry rollback (lmxopcua#905) -> uninstall (from the record,
+    after a swap, the swap's pre-stop gate, -NoStart, a -NoCarry rollback (lmxopcua#905) -> a process still
+    running from InstallRoot after the stop: waited for, timed out clearly, and an unseen handle retried (lmxopcua#908) -> uninstall (from the record,
     without -DenyPaths) -> uninstall. ACL level only; the restricted-token proof that
     the worker cannot read the denied folders is InstallerLayoutIsolationTests (ScriptWorker.Tests, OTOPCUA_ISOLATION_INSTALLER_E2E=1).
     The dry-run harness (Test-ScriptWorkerIsolation.ps1) only asserts plan data; it cannot see host-specific failures such as
@@ -323,6 +324,90 @@ try {
     $vEnd = Invoke-Mode 'Verify'
     Assert ($vEnd.Exit -eq 0) 'full verify passes after the rollback'
 
+    # lmxopcua#908: on QA the host was still exiting when SCM said Stopped, and the first folder move failed "in use".
+    # Stand-in for that host: a copy of powershell.exe under InstallRoot (so its image is under InstallRoot) holding
+    # logs\host908.log open (FileShare.ReadWrite, no Delete: a folder rename fails while it is open; a running image alone
+    # does not block it, checked on windev) until a release file appears. The swap runs as a child whose output is
+    # watched, so the holder is released a fixed time after the swap reaches a given step (its pre-stop isolation gate
+    # takes several seconds, longer than any fixed holder lifetime would be safe against).
+    $winPs = Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\powershell.exe'
+    $holderLog = Join-Path $root 'logs\host908.log'
+    $release = Join-Path $scratch 'release908'
+    function Start-InstallHolder([string]$Image) {
+        Remove-Item -LiteralPath $release -Force -ErrorAction SilentlyContinue
+        New-Item -ItemType Directory -Path (Join-Path $root 'logs') -Force | Out-Null
+        if ($Image -like "$root\*") { New-Item -ItemType Directory -Path (Split-Path -Parent $Image) -Force | Out-Null; Copy-Item -LiteralPath $winPs -Destination $Image -Force }
+        $cmd = "`$f = [IO.File]::Open('$holderLog', 'OpenOrCreate', 'Write', 'ReadWrite'); `$t = [Diagnostics.Stopwatch]::StartNew(); " +
+            "while (-not (Test-Path -LiteralPath '$release') -and `$t.Elapsed.TotalSeconds -lt 120) { Start-Sleep -Milliseconds 50 }; `$f.Dispose()"
+        $enc = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($cmd))
+        $p = Start-Process -FilePath $Image -ArgumentList '-NoProfile', '-NonInteractive', '-EncodedCommand', $enc -WindowStyle Hidden -PassThru
+        $deadline = [DateTime]::UtcNow.AddSeconds(15)
+        while ([DateTime]::UtcNow -lt $deadline -and -not $p.HasExited -and -not (Test-Path -LiteralPath $holderLog)) { Start-Sleep -Milliseconds 100 }
+        Start-Sleep -Milliseconds 300
+        if ($p.HasExited) { throw "holder exited early ($($p.ExitCode))" }
+        $p
+    }
+    function Invoke-SwapReleasing([string[]]$SwapArgs, [string]$Marker, [int]$ReleaseAfterMs, $Holder) {
+        # Runs the swap; ReleaseAfterMs after $Marker appears in its output, releases the holder. Returns output + exit code.
+        $out = Join-Path $scratch 'swap908.out'; $err = Join-Path $scratch 'swap908.err'
+        $argList = @('-NoProfile', '-ExecutionPolicy', 'Bypass', '-File', "`"$swapScript`"") + ($SwapArgs | ForEach-Object { if ($_ -match '\s') { "`"$_`"" } else { $_ } })
+        $sp = Start-Process -FilePath $exe -ArgumentList $argList -RedirectStandardOutput $out -RedirectStandardError $err -WindowStyle Hidden -PassThru
+        $null = $sp.Handle   # Windows PowerShell 5.1: without an open handle ExitCode is $null after the exit
+        $seenAt = $null
+        while (-not $sp.HasExited) {
+            if (-not $seenAt -and (Test-Path $out) -and ((Get-Content -LiteralPath $out -Raw -ErrorAction SilentlyContinue) -like "*$Marker*")) { $seenAt = [DateTime]::UtcNow }
+            if ($seenAt -and ([DateTime]::UtcNow - $seenAt).TotalMilliseconds -ge $ReleaseAfterMs -and -not (Test-Path $release)) { New-Item -ItemType File -Path $release | Out-Null }
+            Start-Sleep -Milliseconds 50
+        }
+        $sp.WaitForExit()
+        if (-not (Test-Path $release)) { New-Item -ItemType File -Path $release | Out-Null }
+        $Holder.WaitForExit(30000) | Out-Null
+        $text = ((Get-Content -LiteralPath $out -Raw) + (Get-Content -LiteralPath $err -Raw)) -replace "\x1b\[[0-9;]*m", ''   # no ANSI colour codes
+        [pscustomobject]@{ Exit = $sp.ExitCode; Output = $text }
+    }
+    function New-StubCandidate([string]$Dir, [string]$Stub) {
+        New-Item -ItemType Directory -Path (Join-Path $Dir 'script-worker') -Force | Out-Null
+        Set-Content -LiteralPath (Join-Path $Dir 'script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe') -Value $Stub
+    }
+    $holderExe = Join-Path $root 'holder908\OtOpcUa.Host908.exe'
+    $common = @('-Carry', 'appsettings*.json,pki', '-NoServiceControl', '-WorkerServiceName', $WorkerServiceName)
+
+    Write-Host 'lmxopcua#908: a process running from InstallRoot that outlives the budget: fails clearly, nothing moved'
+    $cand8a = Join-Path $scratch 'install.new8a'; $prev8a = Join-Path $scratch 'install.bak.8a'
+    New-StubCandidate $cand8a 'stub v8a'
+    $hold = Start-InstallHolder $holderExe
+    $r8a = Invoke-SwapReleasing (@('-InstallRoot', $root, '-Candidate', $cand8a, '-PreviousInstall', $prev8a, '-ProcessExitTimeoutSeconds', '1') + $common) 'Swap complete' 0 $hold
+    $sq8a = & $squash $r8a.Output
+    Write-Host ("    {0,-34} exit {1}; {2}" -f 'swap, holder kept, budget 1 s', $r8a.Exit, $(if ($sq8a -match 'stillinuse1safter') { 'refused: still in use' } else { 'NOT refused' }))
+    Assert ($r8a.Exit -ne 0 -and $sq8a -match 'stillinuse1safter.*OtOpcUa\.Host908\.exe\(PID' -and $sq8a -match 'Nothingwasmoved' -and
+        (Test-Path $cand8a) -and -not (Test-Path $prev8a) -and (Test-Path $holderExe)) "times out naming the holder, nothing moved ($($r8a.Output))"
+    Remove-Item -LiteralPath $holderLog -Force -ErrorAction SilentlyContinue
+
+    Write-Host 'lmxopcua#908: a process running from InstallRoot until ~2 s after the wait starts: the swap waits for it, then moves'
+    $hold = Start-InstallHolder $holderExe
+    $r8b = Invoke-SwapReleasing (@('-InstallRoot', $root, '-Candidate', $cand8a, '-PreviousInstall', $prev8a, '-MoveRetrySeconds', '0') + $common) '==> wait for the host process' 2000 $hold
+    $released = [regex]::Match($r8b.Output, 'released after ([\d.,]+) s').Groups[1].Value
+    Write-Host ("    {0,-34} exit {1}; released after {2} s" -f 'swap, holder +2 s (no move retry)', $r8b.Exit, $released)
+    Assert ($r8b.Exit -eq 0 -and $r8b.Output -match 'Swap complete' -and $released -and [double]($released -replace ',', '.') -ge 1.5 -and
+        (Get-Content -LiteralPath $workerExe) -eq 'stub v8a' -and $r8b.Output -match 'VERIFY OK \(scope InstallRoot\)') "waited for the holder, then swapped and verified ($($r8b.Output))"
+
+    Write-Host 'lmxopcua#908: a handle the process wait cannot see (holder image outside InstallRoot): the first move is retried'
+    $cand8c = Join-Path $scratch 'install.new8c'; $prev8c = Join-Path $scratch 'install.bak.8c'
+    New-StubCandidate $cand8c 'stub v8c'
+    $hold = Start-InstallHolder $winPs
+    $r8c = Invoke-SwapReleasing (@('-InstallRoot', $root, '-Candidate', $cand8c, '-PreviousInstall', $prev8c, '-MoveRetrySeconds', '20') + $common) "==> move $root ->" 2000 $hold
+    $attempt = [regex]::Match($r8c.Output, 'moved on attempt (\d+)').Groups[1].Value
+    Write-Host ("    {0,-34} exit {1}; moved on attempt {2}" -f 'swap, outside holder +2 s', $r8c.Exit, $attempt)
+    Assert ($r8c.Exit -eq 0 -and $attempt -and [int]$attempt -gt 1 -and (Get-Content -LiteralPath $workerExe) -eq 'stub v8c' -and
+        $r8c.Output -match 'VERIFY OK \(scope InstallRoot\)') "the in-use first move was retried and then went through ($($r8c.Output))"
+    $prevItems = @(Get-ChildItem -LiteralPath $prev8c -Recurse -Force | ForEach-Object { $_.FullName.Substring($prev8c.Length + 1) })
+    Assert (-not ($prevItems -contains 'install') -and ($prevItems -contains 'logs\host908.log') -and
+        (Get-Content -LiteralPath (Join-Path $prev8c 'script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe')) -eq 'stub v8a') `
+        "the previous install moved whole: no partial tree from a failed attempt, nothing nested into it ($($prevItems -join ', '))"
+    Remove-Item -LiteralPath $release -Force -ErrorAction SilentlyContinue
+    $v8 = Invoke-Mode 'Verify'
+    Assert ($v8.Exit -eq 0) 'full verify passes after the #908 swaps'
+
     Write-Host 'Uninstall WITHOUT -DenyPaths (the record drives it), twice'
     $u1 = Invoke-Installer @('-Uninstall')
     Assert ($u1.Exit -eq 0) "uninstall exits 0 ($($u1.Output))"

scripts/install/tests/Test-ScriptWorkerIsolation.ps1

@@ -257,15 +257,89 @@ function First-Offset([string]$Name, [string]$Contains = '') {
     $hit = @($cmds | Where-Object { $_.GetCommandName() -eq $Name -and $_.Extent.Text -like "*$Contains*" } | Sort-Object { $_.Extent.StartOffset })
     if ($hit.Count) { $hit[0].Extent.StartOffset } else { -1 }
 }
+$swapText = $swapAst.Extent.Text
+function Text-Offset([string]$Text) { $swapText.IndexOf($Text, [StringComparison]::Ordinal) }
 $iso = @($cmds | Where-Object { $_.GetCommandName() -eq 'Invoke-Isolation' })
 Assert ($iso.Count -eq 4 -and @($iso | Where-Object { $_.CommandElements.Count -ne 3 }).Count -eq 0) "every Invoke-Isolation call names a scope ($(@($iso | ForEach-Object { $_.Extent.Text }) -join ' | '))"
 $order = @(
     (First-Offset 'Invoke-Isolation' "'Reapply' 'Outside'"), (First-Offset 'Invoke-Isolation' "'Verify' 'Outside'"), (First-Offset 'Stop-Service'),
-    (First-Offset 'Move-Item' '$Candidate'), (First-Offset 'Invoke-Isolation' "'Reapply' 'InstallRoot'"),
+    (Text-Offset '[System.IO.Directory]::Move($Candidate, $InstallRoot)'), (First-Offset 'Invoke-Isolation' "'Reapply' 'InstallRoot'"),
     (First-Offset 'Invoke-Isolation' "'Verify' 'InstallRoot'"), (First-Offset 'Start-Service'))
 Assert (@($order | Where-Object { $_ -lt 0 }).Count -eq 0 -and (($order -join ',') -eq (@($order | Sort-Object) -join ','))) `
     "Reapply/Verify Outside -> stop -> swap -> Reapply/Verify InstallRoot -> start ($($order -join ','))"
 
+$swapExit = First-Offset 'Wait-OtOpcUaInstallReleased'
+$swapRetry = First-Offset 'Invoke-OtOpcUaMoveWithRetry' '::Move($InstallRoot, $PreviousInstall)'
+$swapPid = First-Offset 'Get-CimInstance' 'Win32_Service'
+Assert ($swapPid -ge 0 -and $swapPid -lt (First-Offset 'Stop-Service') -and (First-Offset 'Stop-Service') -lt $swapExit -and $swapExit -lt $swapRetry -and
+    $swapRetry -lt (Text-Offset '[System.IO.Directory]::Move($Candidate')) `
+    "lmxopcua#908: host PID captured -> stop -> wait for the processes to exit -> first move (with retry) -> candidate move ($swapPid,$(First-Offset 'Stop-Service'),$swapExit,$swapRetry,$(Text-Offset '[System.IO.Directory]::Move($Candidate'))"
+function Test-InsideCommand($Ast, [string]$Name) {
+    for ($p = $Ast.Parent; $p; $p = $p.Parent) { if ($p -is [System.Management.Automation.Language.CommandAst] -and $p.GetCommandName() -eq $Name) { return $true } }
+    $false
+}
+$firstMoves = @($swapAst.FindAll({ param($n) $n -is [System.Management.Automation.Language.InvokeMemberExpressionAst] -and
+    $n.Extent.Text -like '*::Move($InstallRoot, $PreviousInstall)' }, $true))
+Assert ($firstMoves.Count -eq 1 -and @($firstMoves | Where-Object { -not (Test-InsideCommand $_ 'Invoke-OtOpcUaMoveWithRetry') }).Count -eq 0) `
+    "the first move (InstallRoot -> PreviousInstall) runs only inside Invoke-OtOpcUaMoveWithRetry ($($firstMoves.Count))"
+$moveItems = @($cmds | Where-Object { $_.GetCommandName() -eq 'Move-Item' })
+Assert ($moveItems.Count -eq 0) "no Move-Item: PowerShell 7's falls back to a partial item-by-item move when the rename fails (lmxopcua#908) ($(@($moveItems | ForEach-Object { $_.Extent.Text }) -join ' | '))"
+
+Write-Host 'Host process exit wait (lmxopcua#908; fake process lists)'
+$qaApp = 'E:\ApiInstall\OtOpcUa\app'
+$t0 = [datetime]::new(2026, 11, 13, 12, 0, 0, [DateTimeKind]::Utc)
+$hostP = [pscustomobject]@{ Id = 4100; Name = 'OtOpcUa.Host.exe'; Path = "$qaApp\OtOpcUa.Host.exe"; StartTime = $t0 }
+$workerP = [pscustomobject]@{ Id = 4200; Name = 'ZB.MOM.WW.OtOpcUa.ScriptWorker.exe'; Path = "$qaApp\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe"; StartTime = $t0 }
+$hostNoPath = [pscustomobject]@{ Id = 4100; Name = 'OtOpcUa.Host.exe'; Path = ''; StartTime = $t0 }
+$others = @(
+    [pscustomobject]@{ Id = 10; Name = 'svchost.exe'; Path = 'C:\Windows\System32\svchost.exe'; StartTime = $t0 },
+    [pscustomobject]@{ Id = 11; Name = 'OtOpcUa.Host.exe'; Path = "$qaApp.new-76e73e7d\OtOpcUa.Host.exe"; StartTime = $t0 },   # the candidate: a prefix sibling
+    [pscustomobject]@{ Id = 12; Name = 'pwsh.exe'; Path = $null; StartTime = $null })
+$h = @(Find-OtOpcUaInstallHolder -Snapshot (@($hostP, $workerP) + $others) -InstallRoot "$qaApp\" -ProcessId 4100 -ProcessStartTime $t0)
+Assert ((@($h | ForEach-Object { $_.Id }) -join ',') -eq '4100,4200') "host + worker under app\ hold it; the candidate folder and others do not ($(@($h | ForEach-Object { $_.Id }) -join ','))"
+$h = @(Find-OtOpcUaInstallHolder -Snapshot (@($hostNoPath) + $others) -InstallRoot $qaApp -ProcessId 4100 -ProcessStartTime $t0)
+Assert ((@($h | ForEach-Object { $_.Id }) -join ',') -eq '4100') "the host counts by PID even when its image path is unreadable ($(@($h | ForEach-Object { $_.Id }) -join ','))"
+$reused = [pscustomobject]@{ Id = 4100; Name = 'notepad.exe'; Path = 'C:\Windows\notepad.exe'; StartTime = $t0.AddSeconds(30) }
+$h = @(Find-OtOpcUaInstallHolder -Snapshot (@($reused) + $others) -InstallRoot $qaApp -ProcessId 4100 -ProcessStartTime $t0)
+Assert ($h.Count -eq 0) 'a different process that reused the PID does not count'
+$h = @(Find-OtOpcUaInstallHolder -Snapshot $others -InstallRoot $qaApp)
+Assert ($h.Count -eq 0) 'nothing under app\ and no PID: released'
+
+# The QA case: SCM said Stopped, the host + worker are gone ~1 s later. The wait must see them, then return.
+$script:calls = 0
+$fake = { $script:calls++; if ($script:calls -le 3) { @($hostP, $workerP) + $others } else { $others } }
+$waited = Wait-OtOpcUaInstallReleased -InstallRoot $qaApp -ProcessId 4100 -ProcessStartTime $t0 -TimeoutSeconds 10 -PollMilliseconds 20 -GetSnapshot $fake
+Assert ($script:calls -eq 4) "waits while the host and worker are still exiting, returns once they are gone (snapshots taken: $($script:calls), $([math]::Round($waited, 2)) s)"
+$script:calls = 0
+$null = Wait-OtOpcUaInstallReleased -InstallRoot $qaApp -ProcessId 4100 -TimeoutSeconds 10 -PollMilliseconds 20 -GetSnapshot { $script:calls++; $others }
+Assert ($script:calls -eq 1) "already released: one snapshot, no wait ($($script:calls))"
+$sw908 = [System.Diagnostics.Stopwatch]::StartNew()
+Assert-Throws { Wait-OtOpcUaInstallReleased -InstallRoot $qaApp -ProcessId 4100 -TimeoutSeconds 1 -PollMilliseconds 50 -GetSnapshot { @($workerP) + $others } } `
+    '*still in use 1 s after*ZB.MOM.WW.OtOpcUa.ScriptWorker.exe (PID 4200*Nothing was moved*' 'a process that never exits fails clearly, naming it'
+Assert ($sw908.Elapsed.TotalSeconds -lt 5) "the wait is bounded ($([math]::Round($sw908.Elapsed.TotalSeconds, 1)) s for a 1 s budget)"
+
+Write-Host 'First-move retry on a sharing violation (lmxopcua#908)'
+$inUse = [System.IO.IOException]::new('The process cannot access the file because it is being used by another process.', -2147024864)   # 0x80070020
+Assert (Test-OtOpcUaSharingViolation -Exception $inUse) 'ERROR_SHARING_VIOLATION is recognised'
+Assert (Test-OtOpcUaSharingViolation -Exception ([System.Management.Automation.RuntimeException]::new('wrapped', $inUse))) 'also when wrapped'
+Assert (-not (Test-OtOpcUaSharingViolation -Exception ([System.IO.IOException]::new('Cannot create a file when that file already exists.', -2147024713)))) 'ERROR_ALREADY_EXISTS is not'
+Assert (Test-OtOpcUaSharingViolation -Exception ([System.Management.Automation.MethodInvocationException]::new('Exception calling "Move" with "2" argument(s)',
+    [System.IO.IOException]::new("Access to the path 'E:\ApiInstall\OtOpcUa\app' is denied.", -2147024891)))) `
+    'the [IO.Directory]::Move shape (MethodInvocationException over IOException 0x80070005, both hosts, windev)'
+Assert (Test-OtOpcUaSharingViolation -Exception ([System.IO.IOException]::new("Access to the path 'E:\ApiInstall\OtOpcUa\app' is denied.", -2146232800))) `
+    'the Windows PowerShell 5.1 shape of the same failure (IOException COR_E_IO "Access to the path ... is denied", windev)'
+Assert (-not (Test-OtOpcUaSharingViolation -Exception ([System.UnauthorizedAccessException]::new("Access to the path 'x' is denied.")))) 'an UnauthorizedAccessException is not'
+$script:calls = 0
+$n = try { Invoke-OtOpcUaMoveWithRetry -RetrySeconds 5 -DelayMilliseconds 10 -Move { $script:calls++; if ($script:calls -le 2) { throw $inUse } } 3>$null } catch { "threw: $($_.Exception.Message)" }
+Assert ($n -eq 3 -and $script:calls -eq 3) "in use twice, then moved on attempt 3 ($n)"
+$script:calls = 0
+Assert-Throws { Invoke-OtOpcUaMoveWithRetry -RetrySeconds 5 -DelayMilliseconds 10 -Move { $script:calls++; throw [System.UnauthorizedAccessException]::new('Access to the path is denied.') } 3>$null } `
+    '*denied*' 'any other error is not retried'
+Assert ($script:calls -eq 1) "  (one attempt: $($script:calls))"
+$script:calls = 0
+Assert-Throws { Invoke-OtOpcUaMoveWithRetry -RetrySeconds 0 -DelayMilliseconds 10 -Move { $script:calls++; throw $inUse } 3>$null } '*being used by another process*' '-MoveRetrySeconds 0: no retry'
+Assert ($script:calls -eq 1) "  (one attempt: $($script:calls))"
+
 if ($IsWindows) {
     Write-Host 'SDDL parses on Windows'
     $ds = [System.Security.AccessControl.DirectorySecurity]::new()