Files
mxaccessgw/scripts/deploy/Upgrade-MxGatewayService.ps1
T
Joseph Doherty 4890a3d16e
ci / windows-x86 (push) Successful in 2m41s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Failing after 12m34s
ci / java (push) Failing after 14m50s
fix(deploy): Upgrade-MxGatewayService.ps1 safe on the wonder layout, guards fail closed (#230)
The process guards matched only ZB.MOM.WW.MxGateway.*; the wonder boxes run
MxGateway.Server.exe / MxGateway.Worker.exe, so the one-gateway guard, the
pre-start OrphanWorkerTerminator guard and the STOP_PENDING kill passed vacuously.

- Image names come from the installed exes (NSSM Application, Worker env path) plus
  both known schemes; processes are also matched by exact path (Win32_Process).
- One-gateway guard fails closed: a RUNNING service whose gateway is not visible, at
  the Application path, as the service host's child, is an abort.
- STOP_PENDING kill is scoped to this install's processes.
- Staged checks use the installed exe names; staged Worker must be x86; a staged
  self-contained runtime older than the live one is refused (-AllowRuntimeDowngrade).
- Carry-over: appsettings*.json and certificates/keys (dashboard.pfx) always keep the
  live copy; other live-only non-binary top-level files are carried; hash-verified.
- Env block read from the registry and fingerprinted (names + SHA-256 prefixes, never
  values) after stop, swap, start and the stability window; any change aborts.
  Sensitive values are redacted from all output and notes.
- File-based master key (vd03) accepted and backed up; a missing env master key
  (windev) is a warning. Backup dir is ACL-restricted before anything is written.
- Rollback accepts .bak-<stamp> and .bak.<yyyyMMddTHHmmss> sets and refuses a
  mixed Server/Worker stamp; auto-rollback no longer fails on the old SHA.
- -WhatIf now works on Windows PowerShell 5.1 (Get-FileHash returned nothing under
  WhatIfPreference); an empty Win32_Service result no longer reads as "unavailable".
- Install-MxGatewayService.ps1 process guard matches both schemes.
- Dry-run harness scripts/deploy/tests (fake box in both layouts, 140 assertions,
  passes on pwsh/macOS and Windows PowerShell 5.1) plus a mutation runner (12/12 killed).
- Runbook updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 22:23:19 -05:00

868 lines
52 KiB
PowerShell

#requires -Version 5.1
<#
.SYNOPSIS
On-box upgrade / rollback of the NSSM-wrapped MxAccess Gateway (Server + x86 Worker).
Implements docs/runbooks/DeployUpgradeNssmService.md sections 3, 5, 6, 7 and 8 (mxaccessgw#220, #230).
.DESCRIPTION
RUN THIS ON THE WINDOWS BOX ITSELF (Windows PowerShell 5.1 is enough). The new Server publish and the
new x86 Worker build must already be staged on the box (UNC/SMB copy; build both from the same SHA).
Upgrade flow: preflight -> stop (STOP_PENDING handled) -> backup -> swap Server + Worker -> start ->
verify (/health/ready, VersionInfo SHA, service stable, env block unchanged) -> automatic rollback if
start/verify fails. Rollback flow (-Rollback): stop -> swap the .bak directories back -> optional store
restore -> start -> verify.
Safety properties:
* Exactly one gateway per box. The gateway and Worker image names are taken from the INSTALLED exes
(NSSM Application and env MxGateway__Worker__ExecutablePath) plus both known naming schemes
(ZB.MOM.WW.MxGateway.* on windev, MxGateway.* on the wonder boxes), and processes are also matched by
exact path. The guards FAIL CLOSED: a RUNNING service whose gateway process cannot be found is an
abort, not a pass (mxaccessgw#230: the guards used to pass vacuously on the wonder boxes).
* Never writes the NSSM env block (AppEnvironmentExtra), Application or AppDirectory: the swap is
rename-in-place. The env block is fingerprinted (entry names + SHA-256 prefix of each value; values are
never printed) at preflight, after the stop, after the swap, after the start and after the stability
window. Any added, removed or changed entry aborts (and triggers the automatic rollback).
* Live-only top-level files in Server\ and Worker\ are carried into the new directories (appsettings*.json,
dashboard.pfx and other certificates/keys always keep the LIVE copy), hash-verified.
* API-key DB, Secrets store and a file-based master key must live outside the swapped directories; they are
only COPIED (db + -wal + -shm) into *.bak-<stamp> siblings while the service is stopped.
* A self-contained Server is refused if its .NET runtime is older than the live one (no silent downgrade).
-DryRun (or -WhatIf) prints every mutating action and performs none. Read-only probing (service state,
registry env block, process list, disk space, file hashes) still runs, so the guards are exercised for real.
.PARAMETER ServerSource Staged Server publish output (must contain the same exe name as NSSM Application).
.PARAMETER WorkerSource Staged x86 Worker output (must contain the exe name of MxGateway__Worker__ExecutablePath).
.PARAMETER ServiceName NSSM service name. Default MxAccessGw.
.PARAMETER NssmPath nssm executable. Default 'nssm' (on PATH).
.PARAMETER ServerDir Live Server directory. Default: NSSM AppDirectory.
.PARAMETER WorkerDir Live Worker directory. Default: parent of env MxGateway__Worker__ExecutablePath.
.PARAMETER ExpectedSha Short/long git SHA. Staged Server+Worker (preflight) and the deployed Server (verify)
must carry it in ProductVersion ('0.1.2+<sha>'). Builds from 2026-07-09..08-10 have a bogus stamp.
.PARAMETER Label Free-text suffix for backup names (e.g. 'planwrites').
.PARAMETER HealthBaseUrl Default http://localhost:5130 (dashboard port; /health/ready, /healthz anonymous).
.PARAMETER Rollback Roll back instead of upgrading. Uses the newest Server.bak-* / Server.bak.* unless -BackupStamp.
.PARAMETER BackupStamp Stamp of the backup set to roll back to (yyyyMMdd-HHmmss, or yyyyMMddTHHmmss for .bak.<stamp> sets).
.PARAMETER RestoreStores With -Rollback: also restore gateway-auth.db / secrets db from *.bak-<stamp> (schema-bump case).
.PARAMETER NoAutoRollback Do not roll back automatically when start/verification fails.
.PARAMETER TakeNewAppSettings Do not carry the live appsettings*.json over into the new Server dir.
.PARAMETER AllowRuntimeDowngrade Accept a staged self-contained runtime older than the live one.
.PARAMETER SkipFreeSpaceCheck / MinFreeGB / StopTimeoutSeconds / StableSeconds / ReadyTimeoutSeconds tunables.
.PARAMETER DryRun Print, do not act. -WhatIf is equivalent.
#>
[CmdletBinding(SupportsShouldProcess = $true)]
param(
[string]$ServerSource,
[string]$WorkerSource,
[string]$ServiceName = 'MxAccessGw',
[string]$NssmPath = 'nssm',
[string]$ServerDir,
[string]$WorkerDir,
[string]$ExpectedSha,
[string]$Label,
[string]$HealthBaseUrl = 'http://localhost:5130',
[switch]$Rollback,
[string]$BackupStamp,
[switch]$RestoreStores,
[switch]$NoAutoRollback,
[switch]$TakeNewAppSettings,
[switch]$AllowRuntimeDowngrade,
[switch]$SkipFreeSpaceCheck,
[double]$MinFreeGB = 2,
[int]$StopTimeoutSeconds = 30,
[int]$StableSeconds = 60,
[int]$ReadyTimeoutSeconds = 120,
[switch]$DryRun
)
$ErrorActionPreference = 'Stop'
$script:Dry = $DryRun.IsPresent -or $WhatIfPreference
# -WhatIf is handled by $script:Dry (Step). Leaving the preference on makes Windows PowerShell 5.1 cmdlets used for READ-ONLY
# probing misbehave (Get-FileHash returns nothing under -WhatIf), so the plan would abort; found on windev, #230.
$WhatIfPreference = $false
$script:Stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
# Both naming schemes seen in the field: windev runs ZB.MOM.WW.MxGateway.*.exe, the wonder boxes MxGateway.*.exe.
# The installed exe names (NSSM Application, MxGateway__Worker__ExecutablePath) are added to these in Get-Context.
$script:KnownServerImages = @('ZB.MOM.WW.MxGateway.Server', 'MxGateway.Server')
$script:KnownWorkerImages = @('ZB.MOM.WW.MxGateway.Worker', 'MxGateway.Worker')
$script:ServerImages = $script:KnownServerImages
$script:WorkerImages = $script:KnownWorkerImages
$script:EnvWorkerExe = 'MxGateway__Worker__ExecutablePath'
# Live copy always wins for these (host config, certificates, keys), even when the staged build ships one.
$script:PreservePatterns = @('appsettings*.json', '*.pfx', '*.p12', '*.pem', '*.key', '*.crt', '*.cer')
# Never carried from the old build (they are build output; a live-only binary is a leftover of an older build).
$script:BinaryExtensions = @('.dll', '.exe', '.pdb', '.so', '.dylib', '.winmd')
$script:Notes = New-Object System.Collections.Generic.List[string]
$script:Redact = New-Object System.Collections.Generic.List[string]
$script:LockStream = $null
$script:BackupDir = $null
$script:EnvWarnOnly = $false
# --- output (secret values are redacted from every line) ----------------------------------------------------
function Hide([string]$m) {
if ($null -eq $m) { return '' }
foreach ($s in $script:Redact) { if ($s) { $m = $m.Replace($s, '<redacted>') } }
$m
}
function Say([string]$m) { $m = Hide $m; Write-Host $m; $script:Notes.Add($m) }
function Info([string]$m) { Say "[info] $m" }
function Warn([string]$m) { $m = Hide $m; Write-Warning $m; $script:Notes.Add("[WARN] $m") }
function Fail([string]$m) { throw "ABORT: $(Hide $m)" }
# Mutating step: print in dry-run, run otherwise.
function Step([string]$Description, [scriptblock]$Action) {
if ($script:Dry) { Say "[DRYRUN] $Description"; return }
Say "[ACTION] $Description"
& $Action
}
# --- environment seams (the dry-run harness in scripts/deploy/tests replaces these) ---------------------------
function Test-IsWindowsHost { $env:OS -eq 'Windows_NT' }
function Invoke-Nssm {
param([string[]]$NssmArgs, [switch]$AllowFail)
$raw = & $NssmPath @NssmArgs 2>&1
$code = $LASTEXITCODE
$text = (($raw | ForEach-Object { "$_" }) -join "`n") -replace "`0", ''
if ($code -ne 0 -and -not $AllowFail) { Fail "nssm $($NssmArgs -join ' ') failed (exit $code): $text" }
[pscustomobject]@{ Code = $code; Text = $text.Trim() }
}
function Get-NssmValue([string]$Name) { (Invoke-Nssm -NssmArgs @('get', $ServiceName, $Name)).Text }
function Get-ServiceState {
$r = Invoke-Nssm -NssmArgs @('status', $ServiceName) -AllowFail
if ($r.Code -ne 0) { return 'MISSING' }
($r.Text -split "`n")[0].Trim()
}
# The env block exactly as NSSM stores it (REG_MULTI_SZ), one 'NAME=value' per element. Falls back to nssm get.
function Read-EnvBlockLines {
$key = "HKLM:\SYSTEM\CurrentControlSet\Services\$ServiceName\Parameters"
if ((Test-IsWindowsHost) -and (Test-Path -LiteralPath $key)) {
$p = Get-ItemProperty -LiteralPath $key -Name AppEnvironmentExtra -ErrorAction SilentlyContinue
if ($p) { return @($p.AppEnvironmentExtra | Where-Object { $_ }) }
return @()
}
@((Get-NssmValue 'AppEnvironmentExtra') -split "[`r`n]+" | Where-Object { $_.Trim() })
}
# Id, ParentId, Name (image name without .exe), Path (may be $null when unreadable).
function Get-ProcessTable {
if (Get-Command Get-CimInstance -ErrorAction SilentlyContinue) {
return @(Get-CimInstance Win32_Process | ForEach-Object {
[pscustomobject]@{ Id = [int]$_.ProcessId; ParentId = [int]$_.ParentProcessId; Name = [IO.Path]::GetFileNameWithoutExtension([string]$_.Name); Path = $_.ExecutablePath }
})
}
@(Get-Process -ErrorAction SilentlyContinue | ForEach-Object {
$p = $null; try { $p = $_.Path } catch { }
[pscustomobject]@{ Id = $_.Id; ParentId = 0; Name = $_.ProcessName; Path = $p }
})
}
# PID of the NSSM service host (the gateway Server is its child). $null when it cannot be read.
function Get-ServiceHostPid {
if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null }
$s = Get-CimInstance Win32_Service -Filter "Name='$ServiceName'" -ErrorAction SilentlyContinue
if ($s -and $s.ProcessId) { return [int]$s.ProcessId }
$null
}
# Other services whose command line launches the gateway exe. $null when the check is unavailable.
function Get-OtherGatewayServices([string]$ExeName) {
if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null }
# The leading comma keeps an EMPTY result an array: a bare @() reaches the caller as $null ('check unavailable').
, @(Get-CimInstance Win32_Service | Where-Object { $_.Name -ne $ServiceName -and $_.PathName -and $_.PathName -like "*$ExeName*" } | ForEach-Object { $_.Name })
}
function Stop-ProcessById([int]$Id) { Stop-Process -Id $Id -Force -ErrorAction SilentlyContinue }
function Get-HttpStatus([string]$Url) {
try { return [int](Invoke-WebRequest -Uri $Url -UseBasicParsing -TimeoutSec 10).StatusCode }
catch { if ($_.Exception.Response) { return [int]$_.Exception.Response.StatusCode }; return 0 }
}
function Get-ProductSha([string]$Exe) {
try {
$pv = [System.Diagnostics.FileVersionInfo]::GetVersionInfo($Exe).ProductVersion
if ($pv -match '\+([0-9a-fA-F]{7,40})(\b|$)') { return $Matches[1].ToLowerInvariant() }
} catch { }
$null
}
# 'X86', 'Amd64', 'MSIL', ... or $null when it cannot be read.
function Get-ExeArchitecture([string]$Exe) {
try { return "$([Reflection.AssemblyName]::GetAssemblyName($Exe).ProcessorArchitecture)" } catch { return $null }
}
# Restrict a directory to SYSTEM + Administrators, inheritance off. Throws if that fails.
function Protect-Directory([string]$Path) {
if (-not (Test-IsWindowsHost)) { Warn "not Windows: $Path is not ACL-restricted"; return }
$null = & icacls.exe $Path /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' 2>&1
if ($LASTEXITCODE -ne 0) { Fail "icacls could not restrict $Path (exit $LASTEXITCODE); refusing to write the env block there" }
}
function Export-ServiceKey([string]$Destination) {
if (-not (Test-IsWindowsHost)) { Info 'not Windows: service registry export skipped'; return }
$null = & reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$ServiceName" $Destination /y 2>&1
if ($LASTEXITCODE -ne 0) { Fail "reg export of the service key failed (exit $LASTEXITCODE)" }
}
function Get-DirSddl([string]$Path) {
if (-not (Test-IsWindowsHost)) { return $null }
try { (Get-Acl -LiteralPath $Path).Sddl } catch { $null }
}
# Give a carried file the old file's ACL when the old one had explicit (non-inherited) entries.
function Copy-ExplicitAcl([string]$From, [string]$To) {
if (-not (Test-IsWindowsHost)) { return }
try {
$a = Get-Acl -LiteralPath $From
if (@($a.Access | Where-Object { -not $_.IsInherited }).Count -gt 0 -or $a.AreAccessRulesProtected) {
Set-Acl -LiteralPath $To -AclObject $a
Info " ACL of $(Split-Path -Leaf $From) copied (it had explicit entries)"
}
} catch { Warn "could not copy the ACL onto ${To}: $($_.Exception.Message)" }
}
# --- pure helpers ---------------------------------------------------------------------------------------------
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
function Test-Inside([string]$Path, [string]$Root) {
if (-not $Path -or -not $Root) { return $false }
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
$a -eq $b -or $a.StartsWith("$b/")
}
function Test-SamePath([string]$A, [string]$B) { $A -and $B -and ((Norm $A).ToLowerInvariant() -eq (Norm $B).ToLowerInvariant()) }
function Get-Leaf([string]$p) { if (-not $p) { return $null }; ((Norm $p) -split '/')[-1] }
function Get-Stem([string]$p) { $l = Get-Leaf $p; if (-not $l) { return $null }; [IO.Path]::GetFileNameWithoutExtension($l) }
function Test-ShaMatch([string]$Actual, [string]$Expected) {
if (-not $Actual -or -not $Expected) { return $false }
$e = $Expected.ToLowerInvariant()
$Actual.StartsWith($e) -or $e.StartsWith($Actual)
}
function Get-TextHashPrefix([string]$Text) {
$sha = [Security.Cryptography.SHA256]::Create()
try { (($sha.ComputeHash([Text.Encoding]::UTF8.GetBytes($Text)) | ForEach-Object { $_.ToString('x2') }) -join '').Substring(0, 12) }
finally { $sha.Dispose() }
}
function Get-FileSha([string]$Path) {
$sha = [Security.Cryptography.SHA256]::Create(); $fs = [IO.File]::OpenRead($Path)
try { (($sha.ComputeHash($fs) | ForEach-Object { $_.ToString('X2') }) -join '') } finally { $fs.Dispose(); $sha.Dispose() }
}
function Test-SensitiveName([string]$Name) { ($Name -match 'KEY|PEPPER|PASS|SECRET|TOKEN|PWD|CREDENTIAL') -and ($Name -notmatch '(Path|Source|EnvVarName|Name)$') }
# name -> value. Every sensitive value is registered for redaction.
function ConvertTo-EnvMap([string[]]$Lines) {
$map = [ordered]@{}
foreach ($line in $Lines) {
$i = $line.IndexOf('=')
if ($i -le 0) { continue }
$k = $line.Substring(0, $i).Trim(); $v = $line.Substring($i + 1)
$map[$k] = $v
if ((Test-SensitiveName $k) -and $v.Length -ge 4 -and -not $script:Redact.Contains($v)) { $script:Redact.Add($v) }
}
$map
}
# Names + per-value SHA-256 prefixes. Never carries a value.
function Get-EnvFingerprint([string[]]$Lines) {
$map = ConvertTo-EnvMap $Lines
$entries = [ordered]@{}
foreach ($k in $map.Keys) { $entries[$k] = Get-TextHashPrefix ([string]$map[$k]) }
[pscustomobject]@{ Count = @($Lines).Count; Names = @($map.Keys); Entries = $entries; BlockHash = (Get-TextHashPrefix (@($Lines) -join "`n")) }
}
# List of differences (names only). Empty when identical.
function Compare-EnvFingerprint($Before, $After) {
$d = New-Object System.Collections.Generic.List[string]
foreach ($k in $Before.Entries.Keys) {
if (-not $After.Entries.Contains($k)) { $d.Add("removed $k") }
elseif ($After.Entries[$k] -ne $Before.Entries[$k]) { $d.Add("changed $k (sha256 $($Before.Entries[$k]) -> $($After.Entries[$k]))") }
}
foreach ($k in $After.Entries.Keys) { if (-not $Before.Entries.Contains($k)) { $d.Add("added $k") } }
if ($d.Count -eq 0 -and $Before.BlockHash -ne $After.BlockHash) { $d.Add("block hash $($Before.BlockHash) -> $($After.BlockHash) (order, duplicates or blank entries changed)") }
, $d.ToArray()
}
function Format-EnvFingerprint($Fp) { "$($Fp.Count) entries, block sha256 $($Fp.BlockHash): " + (($Fp.Entries.Keys | ForEach-Object { "$_=$($Fp.Entries[$_])" }) -join ', ') }
# Re-read the env block and abort on any difference from the preflight fingerprint.
function Assert-EnvUnchanged($ctx, [string]$When) {
$now = Get-EnvFingerprint (Read-EnvBlockLines)
$diff = Compare-EnvFingerprint $ctx.EnvFp $now
if ($diff.Count -gt 0) {
# During an automatic rollback the difference is already the reported failure; restoring the old build still goes ahead.
if ($script:EnvWarnOnly) { Warn "NSSM env block changed ${When}: $($diff -join '; ') (not written by this script; restore it from the backup dir's AppEnvironmentExtra.txt if unintended)"; return }
Fail "NSSM env block changed ${When}: $($diff -join '; ')"
}
Info "env block unchanged $When ($($now.Count) entries, block sha256 $($now.BlockHash))"
}
# Directory listings via .NET (FileInfo/DirectoryInfo, hidden files included). Get-ChildItem costs ~250 ms per call
# under pwsh on macOS (per-file owner lookups), which made the dry-run harness 20x slower; on Windows both are fine.
function Get-Files([string]$Dir, [string]$Filter = '*', [switch]$Recurse) {
if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return }
$o = if ($Recurse) { [IO.SearchOption]::AllDirectories } else { [IO.SearchOption]::TopDirectoryOnly }
try { (New-Object IO.DirectoryInfo $Dir).GetFiles($Filter, $o) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" }
}
function Get-Dirs([string]$Dir, [string]$Filter = '*') {
if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return }
try { (New-Object IO.DirectoryInfo $Dir).GetDirectories($Filter) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" }
}
function Get-StoreFiles([string]$Db) { @($Db, "$Db-wal", "$Db-shm") | Where-Object { $_ -and (Test-Path -LiteralPath $_) } }
function Get-DirSize([string]$p) {
if (-not (Test-Path -LiteralPath $p)) { return 0 }
$s = (@(Get-Files $p -Recurse) | Measure-Object Length -Sum).Sum
if ($s) { [double]$s } else { 0 }
}
# Version of a self-contained .NET runtime in a directory (from mscordaccore_<arch>_<arch>_<ver>.dll), or $null (framework-dependent).
function Get-BundledRuntimeVersion([string]$Dir) {
$f = @(Get-Files $Dir 'mscordaccore_*.dll')
foreach ($x in $f) { if ($x.Name -match '_(\d+\.\d+\.\d+(\.\d+)?)\.dll$') { return [version]$Matches[1] } }
$null
}
# --- processes --------------------------------------------------------------------------------------------------
function Test-GatewayServer($p, $ctx) { ($script:ServerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.Application) }
function Test-GatewayWorker($p, $ctx) { ($script:WorkerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.WorkerExe) }
function Get-GatewayProcesses($ctx) {
@(Get-ProcessTable | Where-Object { (Test-GatewayServer $_ $ctx) -or (Test-GatewayWorker $_ $ctx) } | ForEach-Object {
$role = if (Test-GatewayServer $_ $ctx) { 'Server' } else { 'Worker' }
$_ | Add-Member -NotePropertyName Role -NotePropertyValue $role -PassThru -Force
})
}
function Format-Procs($procs) { (@($procs) | ForEach-Object { "$($_.Name)#$($_.Id)<-$($_.ParentId) [$(if ($_.Path) { $_.Path } else { 'path unreadable' })]" }) -join ', ' }
function Assert-NoGatewayProcess($ctx, [string]$When) {
$left = @(Get-GatewayProcesses $ctx)
if ($left.Count -gt 0) { Fail "${When}: gateway process(es) still alive: $(Format-Procs $left)" }
}
# The one-gateway guard. Fails closed: a RUNNING service whose gateway cannot be seen is an abort.
function Assert-SingleGateway($ctx, [string]$State) {
$procs = @(Get-GatewayProcesses $ctx)
$servers = @($procs | Where-Object { $_.Role -eq 'Server' })
$workers = @($procs | Where-Object { $_.Role -eq 'Worker' })
Info "gateway processes on the box (images $($script:ServerImages -join '|') / $($script:WorkerImages -join '|')): $(if ($procs.Count) { Format-Procs $procs } else { 'none' })"
if ($servers.Count -gt 1) { Fail "$($servers.Count) gateway Server processes are running ($(Format-Procs $servers)). A second gateway is the OrphanWorkerTerminator trap - resolve manually." }
switch ($State) {
'SERVICE_STOPPED' {
if ($procs.Count -gt 0) { Fail "service is STOPPED but gateway process(es) exist ($(Format-Procs $procs)): a rogue second gateway or an unclean stop. Starting would terminate those workers." }
}
'SERVICE_RUNNING' {
if ($servers.Count -eq 0) {
Fail "service is RUNNING but no gateway Server process was found (looked for image names $($script:ServerImages -join ', ') and path $($ctx.Application)). The process guards cannot see this gateway, so they would pass vacuously; refusing."
}
$s = $servers[0]
if ($s.Path -and -not (Test-SamePath $s.Path $ctx.Application)) { Fail "the running gateway is $($s.Path), not the service's Application $($ctx.Application): a second gateway is running and the service's own is not" }
if (-not $s.Path) { Warn "gateway Server #$($s.Id) path is unreadable; matched by image name only" }
$hostPid = Get-ServiceHostPid
if ($hostPid) {
if ($s.ParentId -ne $hostPid) { Fail "gateway Server #$($s.Id) is not a child of the $ServiceName service host #${hostPid} (parent #$($s.ParentId)): it is another gateway" }
Info "gateway Server #$($s.Id) is the service's own (child of service host #$hostPid)"
} else { Warn 'service host PID unavailable; parent check skipped' }
$foreign = @($workers | Where-Object { $_.ParentId -and $_.ParentId -ne $s.Id })
if ($foreign.Count -gt 0) { Warn "Worker process(es) not owned by the service's gateway: $(Format-Procs $foreign). They are killed as orphans on the next start." }
$wrongPath = @($workers | Where-Object { $_.Path -and $ctx.WorkerExe -and -not (Test-SamePath $_.Path $ctx.WorkerExe) })
if ($wrongPath.Count -gt 0) { Warn "Worker process(es) not running the configured $($ctx.WorkerExe): $(Format-Procs $wrongPath)" }
}
'SERVICE_STOP_PENDING' { Warn 'service is STOP_PENDING; the stop step will finish it' }
default {
if ($servers.Count -gt 0) { Fail "a gateway Server process exists while the service is $State" }
}
}
}
# --- context discovery (read only) -------------------------------------------------------------------------
function Get-Context {
$state = Get-ServiceState
if ($state -eq 'MISSING') { Fail "service '$ServiceName' does not exist (use Install-MxGatewayService.ps1 for a new box)" }
$ctx = [ordered]@{}
$ctx.State = $state
$ctx.Application = Get-NssmValue 'Application'
$ctx.AppDirectory = Get-NssmValue 'AppDirectory'
$lines = @(Read-EnvBlockLines)
$ctx.Env = ConvertTo-EnvMap $lines
$ctx.EnvFp = Get-EnvFingerprint $lines
$ctx.EnvCount = $ctx.EnvFp.Count
if ($ServerDir) { $ctx.ServerDir = $ServerDir } else { $ctx.ServerDir = $ctx.AppDirectory }
$wexe = $ctx.Env[$script:EnvWorkerExe]
$ctx.WorkerExe = $wexe
if ($WorkerDir) { $ctx.WorkerDir = $WorkerDir }
elseif ($wexe) { $ctx.WorkerDir = Split-Path -Parent $wexe }
else { Fail "cannot determine the Worker directory: env $($script:EnvWorkerExe) is not in the NSSM env block; pass -WorkerDir" }
$pd = 'C:\ProgramData\MxGateway'
$auth = $ctx.Env['MxGateway__Authentication__SqlitePath']; if (-not $auth) { $auth = "$pd\gateway-auth.db" }
$sec = $ctx.Env['Secrets__SqlitePath']; if (-not $sec) { $sec = "$pd\mxgateway-secrets.db" }
$ctx.AuthDb = $auth; $ctx.SecretsDb = $sec
$ctx.KekFile = $ctx.Env['Secrets__MasterKey__FilePath']
$ctx.ServerExeName = Get-Leaf $ctx.Application
$ctx.WorkerExeName = Get-Leaf $wexe
if (-not $ctx.WorkerExeName) { $ctx.WorkerExeName = 'ZB.MOM.WW.MxGateway.Worker.exe' }
# Process image names: the installed ones first, then both known schemes.
$script:ServerImages = @(@(Get-Stem $ctx.Application) + $script:KnownServerImages | Where-Object { $_ } | Select-Object -Unique)
$script:WorkerImages = @(@(Get-Stem $wexe) + $script:KnownWorkerImages | Where-Object { $_ } | Select-Object -Unique)
[pscustomobject]$ctx
}
# --- preflight -----------------------------------------------------------------------------------------------
function Test-MasterKey($ctx) {
$src = $ctx.Env['Secrets__MasterKey__Source']
$envVar = $ctx.Env['Secrets__MasterKey__EnvVarName']; if (-not $envVar) { $envVar = 'ZB_SECRETS_MASTER_KEY' }
if ($ctx.KekFile -or $src -eq 'File') {
if (-not $ctx.KekFile) { Fail 'Secrets__MasterKey__Source=File but no Secrets__MasterKey__FilePath in the env block' }
if (-not (Test-Rooted $ctx.KekFile)) { Fail "master key file '$($ctx.KekFile)' is not absolute" }
foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) { if (Test-Inside $ctx.KekFile $root) { Fail "master key file '$($ctx.KekFile)' is INSIDE '$root' - the swap would lose it" } }
if (Test-Path -LiteralPath $ctx.KekFile) { Info "master key: file $($ctx.KekFile) (present, outside the swap)" } else { Fail "master key file '$($ctx.KekFile)' does not exist (the Secrets store would be unreadable)" }
} elseif ($ctx.Env.Contains($envVar)) {
Info "master key: env block entry $envVar"
} else {
Warn "the env block carries no master key ($envVar or Secrets__MasterKey__FilePath); it comes from elsewhere (appsettings or the account environment). The upgrade does not change that, but confirm the Secrets store opens after the start."
}
}
function Invoke-Preflight($ctx) {
Say '== Preflight =='
Info "service '$ServiceName' state: $($ctx.State)"
Info "Application : $($ctx.Application)"
Info "AppDirectory: $($ctx.AppDirectory)"
Info "Server dir : $($ctx.ServerDir)"
Info "Worker dir : $($ctx.WorkerDir) (exe env: $($ctx.WorkerExe))"
Info "API-key DB : $($ctx.AuthDb)"
Info "Secrets DB : $($ctx.SecretsDb)"
Info "NSSM env block fingerprint (names + value sha256 prefixes, no values): $(Format-EnvFingerprint $ctx.EnvFp)"
if ($ctx.EnvCount -eq 0) { Fail 'NSSM env block is empty - refusing: it holds Kestrel/store paths, the master key and the pepper' }
if (-not ($ctx.Env.Keys | Where-Object { $_ -match 'Pepper' })) {
Warn 'no *Pepper* entry in the env block; confirm the API-key pepper is supplied another way or issued tokens will not verify'
}
if ($ctx.Env['ASPNETCORE_ENVIRONMENT'] -ne 'Production' -and $ctx.Env['DOTNET_ENVIRONMENT'] -ne 'Production') {
Warn 'neither ASPNETCORE_ENVIRONMENT nor DOTNET_ENVIRONMENT is exactly Production (runbook trap 6) - do not change it during an upgrade'
}
if (-not (Test-SamePath (Split-Path -Parent $ctx.Application) $ctx.ServerDir)) { Fail "NSSM Application '$($ctx.Application)' is not inside the Server dir '$($ctx.ServerDir)'" }
if ($ctx.WorkerExe -and -not (Test-SamePath (Split-Path -Parent $ctx.WorkerExe) $ctx.WorkerDir)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' is not inside the Worker dir '$($ctx.WorkerDir)'" }
# State that must survive: rooted and outside the swapped directories.
foreach ($pair in @(@('API-key DB', $ctx.AuthDb), @('Secrets DB', $ctx.SecretsDb))) {
if (-not (Test-Rooted $pair[1])) { Fail "$($pair[0]) path '$($pair[1])' is not absolute" }
foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) {
if (Test-Inside $pair[1] $root) { Fail "$($pair[0]) '$($pair[1])' is INSIDE '$root' - the swap would lose it. Move it out first (runbook section 1)." }
}
if (Test-Path -LiteralPath $pair[1]) { Info "$($pair[0]) present ($((Get-StoreFiles $pair[1]).Count) file(s) incl. wal/shm)" }
else { Warn "$($pair[0]) '$($pair[1])' does not exist yet (fresh store will be created on start)" }
}
Test-MasterKey $ctx
# Directories
foreach ($d in @($ctx.ServerDir, $ctx.WorkerDir)) {
if (-not (Test-Path -LiteralPath $d -PathType Container)) { Fail "live directory '$d' does not exist" }
}
if ((Test-Inside $ctx.WorkerDir $ctx.ServerDir) -or (Test-Inside $ctx.ServerDir $ctx.WorkerDir)) {
Fail 'Server and Worker directories are nested; the rename-in-place swap needs sibling directories'
}
if (-not (Test-Path -LiteralPath $ctx.Application)) { Fail "NSSM Application '$($ctx.Application)' does not exist" }
if ($ctx.WorkerExe -and -not (Test-Path -LiteralPath $ctx.WorkerExe)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' does not exist" }
$dbs = @(@(Get-Files $ctx.ServerDir -Recurse) + @(Get-Files $ctx.WorkerDir -Recurse) | Where-Object { $_.Name -match '\.db(-wal|-shm)?$' })
if ($dbs.Count -gt 0) { Fail "SQLite file(s) inside the swapped directories would be orphaned by the rename: $(($dbs | ForEach-Object { $_.FullName }) -join ', ')" }
# One-gateway guard (fail closed)
Assert-SingleGateway $ctx $ctx.State
$others = Get-OtherGatewayServices $ctx.ServerExeName
if ($null -eq $others) { Info 'Win32_Service cross-check unavailable on this host (skipped)' }
elseif (@($others).Count -gt 0) { Fail "another service ($(@($others) -join ', ')) also launches $($ctx.ServerExeName); two gateways on one box is forbidden" }
else { Info "no other service launches $($ctx.ServerExeName)" }
if ($Rollback) { Say '== Preflight OK =='; return }
# Sources
if (-not $ServerSource -or -not $WorkerSource) { Fail '-ServerSource and -WorkerSource are required for an upgrade' }
$stagedServerExe = Join-Path $ServerSource $ctx.ServerExeName
$stagedWorkerExe = Join-Path $WorkerSource $ctx.WorkerExeName
foreach ($s in @(@($ServerSource, $stagedServerExe), @($WorkerSource, $stagedWorkerExe))) {
if (-not (Test-Path -LiteralPath $s[1])) { Fail "staged source '$($s[0])' does not contain $(Get-Leaf $s[1]) (the installed exe name; NSSM and $($script:EnvWorkerExe) point at it)" }
}
if ((Test-Inside $ServerSource $ctx.ServerDir) -or (Test-Inside $WorkerSource $ctx.WorkerDir) -or (Test-Inside $ServerSource $ctx.WorkerDir) -or (Test-Inside $WorkerSource $ctx.ServerDir)) { Fail 'staging directory is inside a live directory' }
$arch = Get-ExeArchitecture $stagedWorkerExe
if ($arch -ne 'X86') { Fail "staged Worker $stagedWorkerExe architecture is '$arch', not X86 (trap 5)" }
Info "staged Worker architecture: X86"
$sSha = Get-ProductSha $stagedServerExe
$wSha = Get-ProductSha $stagedWorkerExe
Info "staged Server SHA: $sSha ; staged Worker SHA: $wSha ; live Server SHA: $(Get-ProductSha $ctx.Application)"
if ($sSha -and $wSha -and -not (Test-ShaMatch $sSha $wSha)) { Fail "Server ($sSha) and Worker ($wSha) were built from different commits (wire protocol risk)" }
if ($ExpectedSha) {
if (-not (Test-ShaMatch $sSha $ExpectedSha)) {
if ($script:Dry) { Warn "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha' (dry-run: continuing)" }
else { Fail "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha'" }
}
} else { Warn 'no -ExpectedSha given: the post-deploy SHA check will only record the SHA' }
$leafSha = if ((Get-Leaf $ctx.ServerDir) -match '([0-9a-f]{7,40})$') { $Matches[1] } else { $null }
if ($leafSha -and $sSha -and -not (Test-ShaMatch $leafSha $sSha)) { Warn "the Server dir name '$(Get-Leaf $ctx.ServerDir)' carries SHA $leafSha but will hold $sSha after the rename-in-place swap" }
# Runtime: never silently downgrade a self-contained Server.
$liveRt = Get-BundledRuntimeVersion $ctx.ServerDir; $newRt = Get-BundledRuntimeVersion $ServerSource
Info "bundled .NET runtime: live $(if ($liveRt) { $liveRt } else { 'none (framework-dependent)' }), staged $(if ($newRt) { $newRt } else { 'none (framework-dependent)' })"
if ($liveRt -and -not $newRt) { Warn 'live Server is self-contained, staged is framework-dependent: the box must have a matching .NET 10 runtime installed' }
if ($liveRt -and $newRt -and $newRt -lt $liveRt) {
if ($AllowRuntimeDowngrade) { Warn "staged runtime $newRt is OLDER than live $liveRt (-AllowRuntimeDowngrade)" }
else { Fail "staged self-contained runtime $newRt is OLDER than the live $liveRt; rebuild with a newer SDK or pass -AllowRuntimeDowngrade" }
}
# What the swap will carry over / drop (top-level files).
foreach ($pair in @(@('Server', $ctx.ServerDir, $ServerSource), @('Worker', $ctx.WorkerDir, $WorkerSource))) {
$plan = Get-CarryPlan $pair[1] $pair[2] ($pair[0] -eq 'Server' -and -not $TakeNewAppSettings)
Info "$($pair[0]): carry from live -> $(if ($plan.Carry.Count) { $plan.Carry -join ', ' } else { 'nothing' }); live-only binaries dropped -> $(if ($plan.Dropped.Count) { $plan.Dropped -join ', ' } else { 'none' })"
if ($plan.Dirs.Count) { Warn "$($pair[0]): live-only directories NOT carried (they stay in the .bak): $($plan.Dirs -join ', ')" }
}
$lw = @(Get-Files $ctx.WorkerDir | ForEach-Object { $_.Name }); $nw = @(Get-Files $WorkerSource | ForEach-Object { $_.Name })
$onlyNew = @($nw | Where-Object { $lw -notcontains $_ })
if ($onlyNew.Count) { Info "Worker: files only in the staged build: $($onlyNew -join ', ')" }
# Disk space: 2x (server + worker + stores + payload), at least MinFreeGB
if (-not $SkipFreeSpaceCheck) {
$need = (Get-DirSize $ctx.ServerDir) + (Get-DirSize $ctx.WorkerDir) + (Get-DirSize $ServerSource) + (Get-DirSize $WorkerSource)
foreach ($f in (Get-StoreFiles $ctx.AuthDb) + (Get-StoreFiles $ctx.SecretsDb)) { $need += (Get-Item -LiteralPath $f).Length }
$need = [math]::Max($need * 2, $MinFreeGB * 1GB)
$root = [IO.Path]::GetPathRoot((Resolve-Path -LiteralPath $ctx.ServerDir).Path)
$free = (New-Object IO.DriveInfo $root).AvailableFreeSpace
Info ("free on {0}: {1:N2} GB, required {2:N2} GB" -f $root, ($free / 1GB), ($need / 1GB))
if ($free -lt $need) { Fail 'not enough free disk space for the backups' }
}
Say '== Preflight OK =='
}
# --- stop / start ---------------------------------------------------------------------------------------------
function Wait-State([string]$Want, [int]$TimeoutSec) {
$deadline = (Get-Date).AddSeconds($TimeoutSec)
do { $s = Get-ServiceState; if ($s -eq $Want) { return $true }; Start-Sleep -Seconds 1 } while ((Get-Date) -lt $deadline)
$false
}
function Wait-NoGatewayProcess($ctx, [int]$TimeoutSec) {
$deadline = (Get-Date).AddSeconds($TimeoutSec)
do { if (@(Get-GatewayProcesses $ctx).Count -eq 0) { return $true }; Start-Sleep -Seconds 1 } while ((Get-Date) -lt $deadline)
$false
}
# Kill only this install's gateway processes: path under the Server/Worker dir, or (path unreadable) the installed image name.
function Stop-InstallProcesses($ctx, [string]$Why) {
$installed = @((Get-Stem $ctx.Application), (Get-Stem $ctx.WorkerExe)) | Where-Object { $_ }
$mine = @(Get-GatewayProcesses $ctx | Where-Object {
if ($_.Path) { (Test-Inside $_.Path $ctx.ServerDir) -or (Test-Inside $_.Path $ctx.WorkerDir) } else { $installed -contains $_.Name }
} | Sort-Object { if ($_.Role -eq 'Server') { 0 } else { 1 } })
foreach ($p in $mine) { Say "[ACTION] Stop-Process -Id $($p.Id) -Force ($($p.Name), $Why)"; Stop-ProcessById $p.Id }
$mine.Count
}
function Stop-Gateway($ctx) {
Say '-- Stop --'
$state = Get-ServiceState
if ($state -eq 'SERVICE_STOPPED') { Info 'already stopped' }
else {
Step "nssm stop $ServiceName (wait up to $StopTimeoutSeconds s for SERVICE_STOPPED, then for every gateway/Worker process to exit)" {
[void](Invoke-Nssm -NssmArgs @('stop', $ServiceName) -AllowFail)
if (-not (Wait-State 'SERVICE_STOPPED' $StopTimeoutSeconds)) {
# STOP_PENDING trap: workers did not exit in time. Kill this install's Server, then its Workers.
Warn "service still $(Get-ServiceState) after $StopTimeoutSeconds s (STOP_PENDING trap) - killing this install's gateway processes"
$n = Stop-InstallProcesses $ctx 'STOP_PENDING'
if ($n -eq 0) { Warn 'no process of this install was found to kill' }
if (-not (Wait-State 'SERVICE_STOPPED' 20)) {
Warn 'still not STOPPED; killing the NSSM service host process'
$svcPid = Get-ServiceHostPid
if ($svcPid) { Say "[ACTION] Stop-Process -Id $svcPid -Force (service host)"; Stop-ProcessById $svcPid }
if (-not (Wait-State 'SERVICE_STOPPED' 20)) { Fail "service '$ServiceName' will not stop; investigate manually (state $(Get-ServiceState))" }
}
}
if (-not (Wait-NoGatewayProcess $ctx 20)) {
Warn "gateway/Worker process(es) outlived the service stop (unclean stop): $(Format-Procs (Get-GatewayProcesses $ctx))"
[void](Stop-InstallProcesses $ctx 'left over after the stop')
[void](Wait-NoGatewayProcess $ctx 10)
}
}
}
if (-not $script:Dry) {
Assert-NoGatewayProcess $ctx 'after stop'
Info 'stopped; no gateway/Worker process on the box'
Assert-EnvUnchanged $ctx 'after the stop'
}
}
function Start-Gateway($ctx) {
Say '-- Start --'
if (-not $script:Dry) {
Assert-NoGatewayProcess $ctx 'before start (OrphanWorkerTerminator guard)'
Assert-EnvUnchanged $ctx 'before the start'
} else { Say "[DRYRUN] assert no process named $($script:ServerImages -join '|') / $($script:WorkerImages -join '|') exists, and the env block fingerprint is unchanged (OrphanWorkerTerminator guard)" }
Step "nssm start $ServiceName (once)" {
[void](Invoke-Nssm -NssmArgs @('start', $ServiceName))
}
Step "wait for SERVICE_RUNNING and compare the env block fingerprint" {
if (-not (Wait-State 'SERVICE_RUNNING' 60)) { Fail "service did not reach RUNNING (state $(Get-ServiceState))" }
Assert-EnvUnchanged $ctx 'after the start'
}
}
# --- backup / swap -----------------------------------------------------------------------------------------------
function Get-Suffix { if ($Label) { "$($script:Stamp)-$Label" } else { $script:Stamp } }
function Backup-State($ctx) {
Say '-- Backup (service stopped) --'
$suffix = Get-Suffix
$root = Split-Path -Parent $ctx.ServerDir
$dir = Join-Path $root "_deploy-backup-$suffix"
$script:BackupDir = $dir
Step "create $dir, restrict it to SYSTEM + Administrators, then save the env block (secrets), service registry key and env fingerprint" {
New-Item -ItemType Directory -Path $dir -Force | Out-Null
Protect-Directory $dir
$body = @("Application=$($ctx.Application)", "AppDirectory=$($ctx.AppDirectory)", '--- AppEnvironmentExtra ---') + @(Read-EnvBlockLines)
Set-Content -LiteralPath (Join-Path $dir 'AppEnvironmentExtra.txt') -Value $body -Encoding UTF8
Export-ServiceKey (Join-Path $dir "$ServiceName-servicekey.reg")
$fp = @("BlockHash=$($ctx.EnvFp.BlockHash)") + @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" })
Set-Content -LiteralPath (Join-Path $dir 'env-fingerprint.txt') -Value $fp -Encoding ASCII
}
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
foreach ($f in (Get-StoreFiles $db)) {
$dst = "$f.bak-$suffix"
Step "copy store file $f -> $dst (live file is never moved)" {
Copy-Item -LiteralPath $f -Destination $dst -Force
if ((Get-FileSha $f) -ne (Get-FileSha $dst)) { Fail "store backup hash mismatch for $f" }
}
}
if (-not (Test-Path -LiteralPath $db)) { Info "no store at $db to back up" }
}
if ($ctx.KekFile -and (Test-Path -LiteralPath $ctx.KekFile)) {
Step "copy master key file $($ctx.KekFile) into $dir (live file is never moved)" { Copy-Item -LiteralPath $ctx.KekFile -Destination (Join-Path $dir (Get-Leaf $ctx.KekFile)) -Force }
}
}
function Test-Preserved([string]$Name) { foreach ($p in $script:PreservePatterns) { if ($Name -like $p) { return $true } }; $false }
# Which top-level files of the live dir go into the new one. Carry = names copied from live (live wins).
function Get-CarryPlan([string]$Live, [string]$Source, [bool]$KeepSettings) {
$liveFiles = @(Get-Files $Live)
$srcNames = @(Get-Files $Source | ForEach-Object { $_.Name })
$srcDirs = @(Get-Dirs $Source | ForEach-Object { $_.Name })
$carry = New-Object System.Collections.Generic.List[string]; $dropped = New-Object System.Collections.Generic.List[string]
foreach ($f in $liveFiles) {
$inSource = $srcNames -contains $f.Name
if ($f.Name -like 'appsettings*.json') { if ($KeepSettings) { $carry.Add($f.Name) }; continue }
if (Test-Preserved $f.Name) { $carry.Add($f.Name); continue }
if ($inSource) { continue }
if ($script:BinaryExtensions -contains $f.Extension.ToLowerInvariant()) { $dropped.Add($f.Name) } else { $carry.Add($f.Name) }
}
$dirs = @(Get-Dirs $Live | Where-Object { $srcDirs -notcontains $_.Name } | ForEach-Object { $_.Name })
[pscustomobject]@{ Carry = $carry.ToArray(); Dropped = $dropped.ToArray(); Dirs = $dirs }
}
function Swap-Directory([string]$Live, [string]$Source, [string]$Name, [bool]$KeepSettings) {
$bak = "$Live.bak-$(Get-Suffix)"
$plan = Get-CarryPlan $Live $Source $KeepSettings
$hashes = @{}
foreach ($f in $plan.Carry) { $hashes[$f] = Get-FileSha (Join-Path $Live $f) }
$sddlOld = Get-DirSddl $Live
Step "rename $Name dir: $Live -> $bak" { Rename-Item -LiteralPath $Live -NewName (Split-Path -Leaf $bak) }
Step "copy new ${Name}: $Source -> $Live" { Copy-Item -LiteralPath $Source -Destination $Live -Recurse -Force }
foreach ($f in $plan.Carry) {
Step "carry live $Name\$f forward from $bak (sha256 $($hashes[$f].Substring(0, 12)))" {
Copy-Item -LiteralPath (Join-Path $bak $f) -Destination (Join-Path $Live $f) -Force
if ((Get-FileSha (Join-Path $Live $f)) -ne $hashes[$f]) { Fail "carried $Name\$f does not match the live copy" }
Copy-ExplicitAcl (Join-Path $bak $f) (Join-Path $Live $f)
}
}
foreach ($f in $plan.Dropped) { Info "$Name\$f exists only in the old build (binary) and is not carried; it stays in $bak" }
if (-not $script:Dry) {
$sddlNew = Get-DirSddl $Live
if ($sddlOld -and $sddlNew -and $sddlOld -ne $sddlNew) { Warn "$Name dir ACL differs from the old one (new inherits from the parent). old: $sddlOld new: $sddlNew" }
}
$bak
}
# --- verification ----------------------------------------------------------------------------------------------------
function Invoke-Verify($ctx, [bool]$AfterRollback = $false) {
Say '== Verify =='
if ($script:Dry) {
Say "[DRYRUN] GET $HealthBaseUrl/health/ready until 200 (<= $ReadyTimeoutSeconds s)"
Say "[DRYRUN] GET $HealthBaseUrl/healthz ; read ProductVersion SHA of $($ctx.Application)$(if ($ExpectedSha -and -not $Rollback) { " and require it to match $ExpectedSha" })"
Say "[DRYRUN] exactly one gateway Server ($($ctx.Application)) on the box, same PID, service SERVICE_RUNNING for $StableSeconds s, env fingerprint unchanged"
return
}
$ready = $false; $deadline = (Get-Date).AddSeconds($ReadyTimeoutSeconds)
while (-not $ready -and (Get-Date) -lt $deadline) {
$ready = ((Get-HttpStatus "$HealthBaseUrl/health/ready") -eq 200)
if (-not $ready) { Start-Sleep -Seconds 3 }
}
if (-not $ready) { Fail "$HealthBaseUrl/health/ready did not return 200 within $ReadyTimeoutSeconds s" }
Info '/health/ready = 200'
$hz = Get-HttpStatus "$HealthBaseUrl/healthz"
if ($hz -ne 200) { Fail "/healthz = $hz" }
Info '/healthz = 200'
$sha = Get-ProductSha $ctx.Application
Info "deployed Server SHA (ProductVersion): $sha"
if ($ExpectedSha -and -not $Rollback -and -not $AfterRollback -and -not (Test-ShaMatch $sha $ExpectedSha)) { Fail "deployed SHA '$sha' != expected '$ExpectedSha'" }
Assert-SingleGateway $ctx 'SERVICE_RUNNING'
$pid0 = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' })[0].Id
Info "watching the service for $StableSeconds s (gateway #$pid0)"
$until = (Get-Date).AddSeconds($StableSeconds)
do {
if ((Get-ServiceState) -ne 'SERVICE_RUNNING') { Fail 'service left RUNNING during the stability window (restart loop?)' }
Start-Sleep -Seconds 5
} while ((Get-Date) -lt $until)
$servers = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' })
if ($servers.Count -ne 1 -or $servers[0].Id -ne $pid0) { Fail "gateway process changed during the stability window: #$pid0 -> $(Format-Procs $servers) (restart loop or second gateway)" }
Assert-EnvUnchanged $ctx 'after the stability window'
Warn 'health is green with zero sessions; still prove the Worker path: open a session with a PRE-EXISTING API key and read a known tag (runbook 7.3)'
}
# --- rollback ------------------------------------------------------------------------------------------------------------
# Backup dirs come in two spellings: <leaf>.bak-yyyyMMdd-HHmmss[-label] (this script) and <leaf>.bak.yyyyMMddTHHmmss (manual / adapted scripts).
function Get-BackupKey([string]$Name) {
if ($Name -match '\.bak[-.](\d{8})[-T]?(\d{6})') { return "$($Matches[1])$($Matches[2])" }
$null
}
function Find-Backup([string]$Live, [string]$Stamp) {
$leaf = Split-Path -Leaf $Live; $parent = Split-Path -Parent $Live
$cands = @(Get-Dirs $parent | Where-Object { ($_.Name -like "$leaf.bak-*" -or $_.Name -like "$leaf.bak.*") -and (Get-BackupKey $_.Name) })
if ($Stamp) {
$key = ($Stamp -replace '[^0-9]', '')
$cands = @($cands | Where-Object { (Get-BackupKey $_.Name) -eq $key -or $_.Name -like "$leaf.bak-$Stamp*" -or $_.Name -like "$leaf.bak.$Stamp*" })
}
$cands | Sort-Object { Get-BackupKey $_.Name } -Descending | Select-Object -First 1
}
function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores, [bool]$AllowPartial) {
Say '== Rollback =='
$sb = Find-Backup $ctx.ServerDir $Stamp
if (-not $sb -and -not $AllowPartial) { Fail "no Server backup directory found (stamp '$Stamp')" }
$key = if ($sb) { Get-BackupKey $sb.Name } else { ($Stamp -replace '[^0-9]', '') }
$wb = Find-Backup $ctx.WorkerDir $key
if ($wb -and (Get-BackupKey $wb.Name) -ne $key) { $wb = $null }
if (-not $wb -and -not $AllowPartial) { Fail "no Worker backup matching the Server backup $($sb.Name) (stamp key $key); refusing a mixed rollback" }
if (-not $sb -and -not $wb) { Fail "nothing to roll back to (stamp '$Stamp')" }
Info "rolling back to Server=$(if ($sb) { $sb.Name } else { '(unchanged)' }) Worker=$(if ($wb) { $wb.Name } else { '(unchanged)' })"
$fpFile = @(Get-Dirs (Split-Path -Parent $ctx.ServerDir) '_deploy-backup-*' | Where-Object { ($_.Name -replace '[^0-9]', '').StartsWith($key) } | ForEach-Object { Join-Path $_.FullName 'env-fingerprint.txt' } | Where-Object { Test-Path -LiteralPath $_ })
if ($fpFile.Count) {
$saved = Get-Content -LiteralPath $fpFile[0] | Where-Object { $_ -notlike 'BlockHash=*' }
$now = @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" })
if (($saved -join '|') -ne ($now -join '|')) { Warn "the env block differs from the one saved before the upgrade ($($fpFile[0])); restore it from AppEnvironmentExtra.txt there if that was not intended (runbook section 8 step 4)" }
else { Info "env block matches the pre-upgrade fingerprint in $($fpFile[0])" }
}
Stop-Gateway $ctx
foreach ($pair in @(@($ctx.ServerDir, $sb), @($ctx.WorkerDir, $wb))) {
if (-not $pair[1]) { Info "$($pair[0]) has no backup for this stamp; left unchanged"; continue }
$failed = "$($pair[0]).failed-$($script:Stamp)"
if (Test-Path -LiteralPath $pair[0]) { Step "rename $($pair[0]) -> $failed" { Rename-Item -LiteralPath $pair[0] -NewName (Split-Path -Leaf $failed) } }
Step "rename $($pair[1].FullName) -> $($pair[0])" { Rename-Item -LiteralPath $pair[1].FullName -NewName (Split-Path -Leaf $pair[0]) }
}
if ($Stores) {
$bstamp = ($sb.Name -replace '^.*\.bak[-.]', '')
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
foreach ($ext in '', '-wal', '-shm') {
$live = "$db$ext"; $bak = "$db$ext.bak-$bstamp"
if (Test-Path -LiteralPath $bak) { Step "restore store file $bak -> $live" { Copy-Item -LiteralPath $bak -Destination $live -Force } }
elseif ($ext -and (Test-Path -LiteralPath $live) -and (Test-Path -LiteralPath "$db.bak-$bstamp")) {
# a stale wal/shm must not pair with the restored db: move it aside, never delete
Step "move stale $live aside -> $live.stale-$($script:Stamp)" { Rename-Item -LiteralPath $live -NewName ("$(Split-Path -Leaf $live).stale-$($script:Stamp)") }
}
elseif (-not $ext) { Warn "no backup $bak for $db; live store left untouched" }
}
}
} else { Info 'stores left untouched (they live outside the swap); pass -RestoreStores after a schema bump' }
Start-Gateway $ctx
}
# --- main -------------------------------------------------------------------------------------------------------------------
function Invoke-Main {
$exit = 0
$ctx = $null
try {
Say "Upgrade-MxGatewayService stamp=$($script:Stamp) mode=$(if ($Rollback) {'ROLLBACK'} else {'UPGRADE'}) dry=$($script:Dry) host=$env:COMPUTERNAME"
$ctx = Get-Context
Invoke-Preflight $ctx
if (-not $script:Dry) {
$lockPath = Join-Path (Split-Path -Parent $ctx.ServerDir) '.mxgw-deploy.lock'
try { $script:LockStream = [IO.File]::Open($lockPath, 'OpenOrCreate', 'ReadWrite', 'None') }
catch { Fail "another deploy run holds $lockPath" }
}
if ($Rollback) {
Invoke-RollbackSwap $ctx $BackupStamp $RestoreStores.IsPresent $false
Invoke-Verify $ctx
} else {
Stop-Gateway $ctx
try { Backup-State $ctx }
catch {
$err = $_
if (-not $script:Dry) {
Warn 'backup failed before anything was swapped - starting the unchanged build again'
try { Start-Gateway $ctx } catch { Warn "restart of the unchanged build FAILED: $($_.Exception.Message)" }
}
throw $err
}
$touched = $false
try {
$touched = $true
$null = Swap-Directory $ctx.ServerDir $ServerSource 'Server' (-not $TakeNewAppSettings)
$null = Swap-Directory $ctx.WorkerDir $WorkerSource 'Worker' $false
if (-not $script:Dry) { Assert-EnvUnchanged $ctx 'after the swap' }
Start-Gateway $ctx
Invoke-Verify $ctx
} catch {
$err = $_
Say "[FAIL] $(Hide $err.Exception.Message)"
if ($script:Dry -or $NoAutoRollback -or -not $touched) { throw }
Warn 'upgrade failed - rolling back automatically'
$script:EnvWarnOnly = $true
try {
Invoke-RollbackSwap $ctx $script:Stamp $false $true
Invoke-Verify $ctx $true
Warn 'ROLLED BACK to the previous build; original failure follows'
} catch { Warn "AUTOMATIC ROLLBACK FAILED: $($_.Exception.Message). Manual intervention required (backups: *.bak-$(Get-Suffix))." }
throw $err
}
}
Say '== DONE =='
} catch {
$exit = 1
$m = Hide $_.Exception.Message
$script:Notes.Add("ERROR: $m")
Write-Host "ERROR: $m" -ForegroundColor Red
} finally {
if ($script:LockStream) { $script:LockStream.Dispose(); $script:LockStream = $null }
try {
if (-not $script:Dry -and $ctx) {
$notes = Join-Path (Split-Path -Parent $ctx.ServerDir) "_DEPLOY-NOTES-$($script:Stamp).txt"
Set-Content -LiteralPath $notes -Value $script:Notes -Encoding UTF8
Write-Host "deploy notes: $notes"
}
} catch { }
$script:Redact.Clear()
}
$exit
}
# Dot-sourcing (the dry-run harness) only defines the functions.
if ($MyInvocation.InvocationName -ne '.') { $rc = Invoke-Main | Select-Object -Last 1; exit $rc }