lmxopcua/scripts/install/Invoke-OtOpcUaInstallSwap.ps1

<#
.SYNOPSIS
    Swaps an OtOpcUa install folder for a candidate build and restores the ScriptWorker isolation before the host starts
    again (lmxopcua#901). Also the rollback: swap the previous install back in the same way.

.DESCRIPTION
    A folder swap drops the explicit worker-isolation ACEs on script-worker\, on the install folder and on any denied
    folder that is recreated or copied in (pki\ on vd03), because a moved-in folder carries its own DACL. This script makes
    the re-apply part of the swap, in this order:

      0. Preflight (nothing changed yet): the candidate exists and carries script-worker\; PreviousInstall does not exist
         and is on the same volume; when isolation is installed, its settings are recorded (an install by an installer
         older than #901 recorded only DenyPaths; then run the full-argument install once first, or pass -SkipIsolation).
      1. Isolation, when recorded, with the old build still serving (lmxopcua#905): Install-ScriptWorkerIsolation.ps1
         -Reapply -Scope Outside, then -Verify -Scope Outside: every step the swap cannot undo (anchor service, records,
         PE cache, denies outside InstallRoot, firewall rule, host env) and the coverage scan of the denied folders
         outside InstallRoot.
         If either fails, the script stops here: nothing is stopped or moved.
      2. Stop the service (unless -NoServiceControl). The stopped window starts here. SCM can report Stopped while the
         host process is still exiting (lmxopcua#908), so the script then waits, at most -ProcessExitTimeoutSeconds, until
         the service's host process (PID captured before the stop) and every process whose image is under InstallRoot
         (OtOpcUa.Host, the ScriptWorker) have exited. If one is still there, it throws naming them: nothing has moved.
      3. Move InstallRoot -> PreviousInstall, then Candidate -> InstallRoot, each one rename ([IO.Directory]::Move; not
         Move-Item, which in PowerShell 7 falls back to a partial item-by-item move when the rename fails). The first
         move is retried for at most -MoveRetrySeconds while it fails because something under InstallRoot is open
         (nothing has moved yet; a backstop for a handle the process wait cannot see).
      4. Copy each -Carry item (relative to InstallRoot; wildcards allowed) from PreviousInstall into the new InstallRoot.
         Folders are copied with robocopy /E /COPY:DAT (data, attributes, timestamps; not ACLs: step 5 sets those).
      5. Isolation, when recorded: -Reapply -Scope InstallRoot, then -Verify -Scope InstallRoot: only the entries a swap
         drops (the ACEs on InstallRoot, script-worker\ and a denied folder inside InstallRoot such as pki\) and the
         coverage scan of InstallRoot. Steps 1 and 5 together check every install step (the firewall rule names the
         worker exe by its unchanged path; preflight checks the candidate carries that exe), so the new build is never
         started unverified. If -Verify fails, the script stops here with the service STOPPED
         (fail closed): fix it, run -Reapply/-Verify (no -Scope), then start the service.
      6. Start the service (unless -NoStart, e.g. a Primary/Secondary pair window where both nodes start together).
         Prints the stopped window (stop issued -> start issued) and the time of each step in it.

    lmxopcua#905: the coverage scan used to run over every denied folder in both -Reapply and -Verify while the service
    was stopped, with Get-Acl per item; on vd03 (E:\ApiInstall denied) that was most of a 101 s outage. It is now about
    20x faster per item and, inside the stopped window, walks only the new InstallRoot.

    Not covered (layout-specific; keep them in the deploy's own scripts): backups of data\ / the registry / the ConfigDb,
    restoring data\ on a rollback, migrations, health checks.

.PARAMETER InstallRoot
    The live install folder (OtOpcUa.Host.exe + script-worker\). vd03: E:\ApiInstall\OtOpcUa. QA: E:\ApiInstall\OtOpcUa\app.

.PARAMETER Candidate
    The expanded new build (for a rollback: the previous install).

.PARAMETER PreviousInstall
    Where the live folder is moved to. Must not exist. E.g. E:\ApiInstall\OtOpcUa.bak.20261113-pre-<sha>; for a rollback
    E:\ApiInstall\OtOpcUa.failed-<sha>.

.PARAMETER Carry
    Items copied from the previous install into the new one, relative to InstallRoot. Default: appsettings*.json, pki, logs
    (the vd03 layout, where pki\ and logs\ live inside the install folder). QA keeps pki\ and data\ beside app\: pass
    -Carry 'appsettings*.json','logs'. A rollback swaps back a complete folder: pass -NoCarry.

.PARAMETER NoCarry
    Copy nothing from the previous install (a rollback).

.PARAMETER ServiceName
    The host service (default OtOpcUaHost).

.PARAMETER NoServiceControl
    Neither stop nor start the service (the caller owns it; used by the scratch test).

.PARAMETER NoStart
    Leave the service stopped after a successful swap and verify.

.PARAMETER SkipIsolation
    Do not re-apply/verify the isolation even when it is installed. WARNs. Only for a box where it is being removed.

.PARAMETER ProcessExitTimeoutSeconds
    After the stop, how long to wait for the host process and any process running from InstallRoot to exit before
    failing with nothing moved (default 60; lmxopcua#908).

.PARAMETER MoveRetrySeconds
    How long to keep retrying the first folder move while it fails with a sharing violation (default 10; 0 = no retry).

.PARAMETER WorkerServiceName
    Passed to the isolation installer (default OtOpcUaScriptWorker).

.EXAMPLE
    # vd03 (pki\ and logs\ inside the install folder)
    .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.new-6c4d2990 `
        -PreviousInstall E:\ApiInstall\OtOpcUa.bak.20261113-pre-6c4d2990

.EXAMPLE
    # QA, one node of the stop-both/start-both window (start both afterwards)
    .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate E:\ApiInstall\OtOpcUa\app.new-<sha> `
        -PreviousInstall E:\ApiInstall\OtOpcUa\app.bak.<date>-pre-<sha> -Carry 'appsettings*.json','logs' -NoStart

.EXAMPLE
    # Rollback on vd03: the failed build out, the previous one back (restore data\ separately)
    .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa -Candidate E:\ApiInstall\OtOpcUa.bak.20261113-pre-6c4d2990 `
        -PreviousInstall E:\ApiInstall\OtOpcUa.failed-6c4d2990 -NoCarry
#>
[CmdletBinding(SupportsShouldProcess, ConfirmImpact = 'Medium')]
param(
    [Parameter(Mandatory)] [string]$InstallRoot,
    [Parameter(Mandatory)] [string]$Candidate,
    [Parameter(Mandatory)] [string]$PreviousInstall,
    [string[]]$Carry = @('appsettings*.json', 'pki', 'logs'),
    [switch]$NoCarry,
    [string]$ServiceName = 'OtOpcUaHost',
    [switch]$NoServiceControl,
    [switch]$NoStart,
    [switch]$SkipIsolation,
    [ValidatePattern('^[A-Za-z0-9_.\-]{1,80}$')] [string]$WorkerServiceName = 'OtOpcUaScriptWorker',
    [int]$StopTimeoutSeconds = 120,
    [ValidateRange(0, 3600)] [int]$ProcessExitTimeoutSeconds = 60,
    [ValidateRange(0, 600)] [int]$MoveRetrySeconds = 10
)

Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
Import-Module (Join-Path $PSScriptRoot 'ScriptWorkerIsolation.psm1') -Force
$installer = Join-Path $PSScriptRoot 'Install-ScriptWorkerIsolation.ps1'

function Step([string]$Message) { Write-Host "==> $Message" }
function Invoke-Isolation([string]$Mode, [string]$Scope) {
    # Same PowerShell host as this script; the installer's exit code is the gate.
    $exe = if ($PSVersionTable.PSEdition -eq 'Core') { (Get-Process -Id $PID).Path } else { 'powershell.exe' }
    & $exe -NoProfile -ExecutionPolicy Bypass -File $installer "-$Mode" -Scope $Scope -WorkerServiceName $WorkerServiceName | ForEach-Object { Write-Host "    $_" }
    $LASTEXITCODE
}
# lmxopcua#905: the stopped window and what it is spent on.
$window = [System.Diagnostics.Stopwatch]::new()
$phases = [System.Collections.Generic.List[string]]::new()
function Mark([string]$Name, [System.Diagnostics.Stopwatch]$Sw) { $phases.Add(('{0} {1:N1} s' -f $Name, $Sw.Elapsed.TotalSeconds)) }

$InstallRoot = $InstallRoot.TrimEnd('\')
$Candidate = $Candidate.TrimEnd('\')
$PreviousInstall = $PreviousInstall.TrimEnd('\')
if ($NoCarry) { $Carry = @() }
$Carry = @($Carry | ForEach-Object { $_ -split '[,;]' } | ForEach-Object { $_.Trim() } | Where-Object { $_ })

# ---- 0. Preflight: nothing changes until all of it passes -----------------------------------------------------------
$principal = [Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()
if (-not $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) { throw 'run elevated' }
if (-not (Test-Path -LiteralPath $InstallRoot -PathType Container)) { throw "InstallRoot $InstallRoot not found" }
if (-not (Test-Path -LiteralPath (Join-Path $Candidate 'script-worker') -PathType Container)) { throw "candidate $Candidate has no script-worker\ (not a Host publish?)" }
# lmxopcua#905: the firewall rule (verified before the stop) blocks this exe by path; the swapped-in folder must carry it.
$candidateWorkerExe = Join-Path $Candidate 'script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe'
if (Test-Path -LiteralPath $PreviousInstall) { throw "PreviousInstall $PreviousInstall already exists; pick a fresh name" }
if ([IO.Path]::GetPathRoot($InstallRoot) -ne [IO.Path]::GetPathRoot($PreviousInstall) -or [IO.Path]::GetPathRoot($InstallRoot) -ne [IO.Path]::GetPathRoot($Candidate)) {
    throw 'InstallRoot, Candidate and PreviousInstall must be on the same volume (the swap renames folders)'
}
$isolation = $null
if (-not $SkipIsolation) {
    $isolation = Get-OtOpcUaIsolationSettings -WorkerServiceName $WorkerServiceName
    $legacy = @(Get-OtOpcUaRecordedDenyPaths -WorkerServiceName $WorkerServiceName)
    $anchor = Get-Service -Name $WorkerServiceName -ErrorAction SilentlyContinue
    if (-not $isolation -and ($anchor -or $legacy.Count)) {
        throw ("ScriptWorker isolation is installed but its settings are not recorded (installed by an installer older than lmxopcua#901). " +
            'Before swapping, run Install-ScriptWorkerIsolation.ps1 once with its full arguments (the deploy pack lists them); that writes the record. ' +
            'Then re-run this swap.')
    }
    if ($isolation -and $isolation.InstallRoot.TrimEnd('\') -ine $InstallRoot) {
        throw "the isolation record is for InstallRoot '$($isolation.InstallRoot)', not '$InstallRoot'; fix -InstallRoot or re-run the isolation install"
    }
    if ($isolation -and -not (Test-Path -LiteralPath $candidateWorkerExe -PathType Leaf)) {
        throw "candidate has no $candidateWorkerExe; the worker firewall rule blocks that path (a framework-dependent publish is not covered by it)"
    }
    if ($isolation) { Write-Host "Isolation installed (record: InstallRoot $($isolation.InstallRoot)); re-applied and verified outside InstallRoot before the stop, inside it after the swap." }
    else { Write-Host 'Isolation not installed on this node; nothing to re-apply.' }
}
else { Write-Warning '-SkipIsolation: the worker-isolation ACLs are NOT re-applied; a swapped folder has lost them (lmxopcua#901).' }

if (-not $PSCmdlet.ShouldProcess($InstallRoot, "swap in $Candidate (previous install -> $PreviousInstall)")) { return }

# ---- 1. Isolation outside InstallRoot, old build still serving (lmxopcua#905) ---------------------------------------
if ($isolation) {
    $sw = [System.Diagnostics.Stopwatch]::StartNew()
    Step 'worker isolation outside the install folder: -Reapply -Scope Outside (service still running)'
    $code = Invoke-Isolation 'Reapply' 'Outside'
    if ($code -ne 0) { throw "Install-ScriptWorkerIsolation.ps1 -Reapply -Scope Outside failed ($code); nothing stopped or swapped" }
    Step 'worker isolation outside the install folder: -Verify -Scope Outside (pre-stop gate)'
    $code = Invoke-Isolation 'Verify' 'Outside'
    if ($code -ne 0) { throw "Install-ScriptWorkerIsolation.ps1 -Verify -Scope Outside failed ($code); nothing stopped or swapped. Fix the [MISSING] steps (-Reapply), then re-run the swap." }
    Write-Host ('    (pre-stop isolation {0:N1} s, service still running)' -f $sw.Elapsed.TotalSeconds)
}

# ---- 2. Stop ---------------------------------------------------------------------------------------------------------
$hostPid = 0; $hostStart = $null
$window.Start()
if (-not $NoServiceControl) {
    Step "stop $ServiceName"
    $svc = Get-Service -Name $ServiceName
    if ($svc.Status -ne 'Stopped') {
        # lmxopcua#908: remember the host process (PID + start time, so a reused PID does not count) before stopping it.
        $hostPid = [int](Get-CimInstance -ClassName Win32_Service -Filter "Name='$ServiceName'").ProcessId
        if ($hostPid -gt 0) {
            $hostProc = Get-Process -Id $hostPid -ErrorAction SilentlyContinue
            if ($hostProc) { try { $hostStart = $hostProc.StartTime.ToUniversalTime() } catch { } }
        }
        Stop-Service -Name $ServiceName -Force
        $svc.WaitForStatus('Stopped', [TimeSpan]::FromSeconds($StopTimeoutSeconds))
    }
    Mark 'stop' $window
}
# lmxopcua#908: SCM reports Stopped before the host process is gone; the folder move fails "in use" until it is.
$sw = [System.Diagnostics.Stopwatch]::StartNew()
Step "wait for the host process (PID $(if ($hostPid) { $hostPid } else { 'n/a' })) and any process under $InstallRoot to exit (at most $ProcessExitTimeoutSeconds s)"
$waited = Wait-OtOpcUaInstallReleased -InstallRoot $InstallRoot -ProcessId $hostPid -ProcessStartTime $hostStart -TimeoutSeconds $ProcessExitTimeoutSeconds
Write-Host ('    released after {0:N1} s' -f $waited)
Mark 'exit' $sw

# ---- 3. Swap ---------------------------------------------------------------------------------------------------------
$sw = [System.Diagnostics.Stopwatch]::StartNew()
Step "move $InstallRoot -> $PreviousInstall"
# lmxopcua#908: [IO.Directory]::Move, not Move-Item. It is one rename (MoveFileEx, same volume: checked in preflight) that
# fails as a whole. PowerShell 7's Move-Item, when the rename fails because a file under the folder is open, falls back
# to moving item by item and leaves a partial PreviousInstall tree behind (windev), so a retry would then nest into it.
$attempts = Invoke-OtOpcUaMoveWithRetry -RetrySeconds $MoveRetrySeconds -Move { [System.IO.Directory]::Move($InstallRoot, $PreviousInstall) }
if ($attempts -gt 1) { Write-Host "    moved on attempt $attempts" }
Step "move $Candidate -> $InstallRoot"
try { [System.IO.Directory]::Move($Candidate, $InstallRoot) }
catch {
    Write-Warning "candidate move failed; putting the previous install back: $($_.Exception.Message)"
    [System.IO.Directory]::Move($PreviousInstall, $InstallRoot)
    throw
}

Mark 'move' $sw

# ---- 4. Carry --------------------------------------------------------------------------------------------------------
$sw = [System.Diagnostics.Stopwatch]::StartNew()
foreach ($item in $Carry) {
    # Get-Item, not Get-ChildItem: a folder name must give the folder itself, not its children (wildcards still expand).
    $sources = @(Get-Item -Path (Join-Path $PreviousInstall $item) -Force -ErrorAction SilentlyContinue)
    if ($sources.Count -eq 0) { Write-Host "    (carry: nothing matches $item)"; continue }
    foreach ($src in $sources) {
        $dst = Join-Path $InstallRoot $src.Name
        if ($src.PSIsContainer) {
            & robocopy $src.FullName $dst /E /COPY:DAT /DCOPY:DAT /R:1 /W:1 /NFL /NDL /NJH /NJS /NP | Out-Null
            if ($LASTEXITCODE -ge 8) { throw "robocopy $($src.FullName) -> $dst failed ($LASTEXITCODE)" }
        }
        else { Copy-Item -LiteralPath $src.FullName -Destination $dst -Force }
        Write-Host "    carried $($src.Name)"
    }
}

Mark 'carry' $sw

# ---- 5. Isolation inside InstallRoot: re-apply + verify (fail closed) ------------------------------------------------
if ($isolation) {
    $sw = [System.Diagnostics.Stopwatch]::StartNew()
    Step 'worker isolation in the swapped-in folder: -Reapply -Scope InstallRoot'
    $code = Invoke-Isolation 'Reapply' 'InstallRoot'
    if ($code -ne 0) { throw "Install-ScriptWorkerIsolation.ps1 -Reapply -Scope InstallRoot failed ($code); $ServiceName left stopped" }
    Step 'worker isolation in the swapped-in folder: -Verify -Scope InstallRoot (post-swap gate)'
    $code = Invoke-Isolation 'Verify' 'InstallRoot'
    if ($code -ne 0) { throw "Install-ScriptWorkerIsolation.ps1 -Verify -Scope InstallRoot failed ($code); $ServiceName left stopped. Fix the [MISSING] steps, re-run -Reapply/-Verify, then start it." }
    Mark 'isolation' $sw
}

# ---- 6. Start --------------------------------------------------------------------------------------------------------
$window.Stop()
$windowName = if ($NoServiceControl) { 'swap window (service control is the caller''s)' } elseif ($NoStart) { 'stop -> swapped and verified (left stopped)' } else { 'stopped window (stop issued -> start issued)' }
Write-Host ('{0}: {1:N1} s [{2}]' -f $windowName, $window.Elapsed.TotalSeconds, ($phases -join ', '))
if (-not $NoServiceControl -and -not $NoStart) {
    Step "start $ServiceName"
    Start-Service -Name $ServiceName
    (Get-Service -Name $ServiceName).WaitForStatus('Running', [TimeSpan]::FromSeconds(60))
}
Write-Host "Swap complete: $InstallRoot is the candidate; previous install at $PreviousInstall."