fix(deploy): Upgrade-MxGatewayService.ps1 safe on the wonder layout, guards fail closed (#230)
ci / windows-x86 (push) Successful in 2m41s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Failing after 12m34s
ci / java (push) Failing after 14m50s

The process guards matched only ZB.MOM.WW.MxGateway.*; the wonder boxes run
MxGateway.Server.exe / MxGateway.Worker.exe, so the one-gateway guard, the
pre-start OrphanWorkerTerminator guard and the STOP_PENDING kill passed vacuously.

- Image names come from the installed exes (NSSM Application, Worker env path) plus
  both known schemes; processes are also matched by exact path (Win32_Process).
- One-gateway guard fails closed: a RUNNING service whose gateway is not visible, at
  the Application path, as the service host's child, is an abort.
- STOP_PENDING kill is scoped to this install's processes.
- Staged checks use the installed exe names; staged Worker must be x86; a staged
  self-contained runtime older than the live one is refused (-AllowRuntimeDowngrade).
- Carry-over: appsettings*.json and certificates/keys (dashboard.pfx) always keep the
  live copy; other live-only non-binary top-level files are carried; hash-verified.
- Env block read from the registry and fingerprinted (names + SHA-256 prefixes, never
  values) after stop, swap, start and the stability window; any change aborts.
  Sensitive values are redacted from all output and notes.
- File-based master key (vd03) accepted and backed up; a missing env master key
  (windev) is a warning. Backup dir is ACL-restricted before anything is written.
- Rollback accepts .bak-<stamp> and .bak.<yyyyMMddTHHmmss> sets and refuses a
  mixed Server/Worker stamp; auto-rollback no longer fails on the old SHA.
- -WhatIf now works on Windows PowerShell 5.1 (Get-FileHash returned nothing under
  WhatIfPreference); an empty Win32_Service result no longer reads as "unavailable".
- Install-MxGatewayService.ps1 process guard matches both schemes.
- Dry-run harness scripts/deploy/tests (fake box in both layouts, 140 assertions,
  passes on pwsh/macOS and Windows PowerShell 5.1) plus a mutation runner (12/12 killed).
- Runbook updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 22:23:14 -05:00
parent 453fe01525
commit 4890a3d16e
7 changed files with 1192 additions and 218 deletions
+562 -185
View File
@@ -2,32 +2,38 @@
<#
.SYNOPSIS
On-box upgrade / rollback of the NSSM-wrapped MxAccess Gateway (Server + x86 Worker).
Implements docs/runbooks/DeployUpgradeNssmService.md sections 3, 5, 6, 7 and 8 (mxaccessgw#220).
Implements docs/runbooks/DeployUpgradeNssmService.md sections 3, 5, 6, 7 and 8 (mxaccessgw#220, #230).
.DESCRIPTION
RUN THIS ON THE WINDOWS BOX ITSELF (Windows PowerShell 5.1 is enough). The new Server publish and the
new x86 Worker build must already be staged on the box (UNC/SMB copy; build both from the same SHA).
Upgrade flow: preflight -> stop (STOP_PENDING handled) -> backup -> swap Server + Worker -> start ->
verify (/health/ready, VersionInfo SHA, service stable) -> automatic rollback if start/verify fails.
Rollback flow (-Rollback): stop -> swap the .bak directories back -> optional store restore -> start -> verify.
verify (/health/ready, VersionInfo SHA, service stable, env block unchanged) -> automatic rollback if
start/verify fails. Rollback flow (-Rollback): stop -> swap the .bak directories back -> optional store
restore -> start -> verify.
Safety properties:
* Exactly one gateway per box. Refuses to run if a second gateway process / service points at the same
exe, if the service is stopped but a gateway process exists, or if a Server/Worker process is still
alive when a start is about to happen (OrphanWorkerTerminator trap, runbook section 2.1).
* Never calls `nssm set ... AppEnvironmentExtra` (that replaces the whole block). The env block is only
READ (saved to the backup dir, entry count compared before/after). The Application/AppDirectory
are not changed either: the swap is rename-in-place, so the NSSM config is untouched.
* API-key DB, Secrets store and master key live outside the content root and are never moved or
deleted. They are only COPIED (db + -wal + -shm) into *.bak-<stamp> siblings while the service is stopped.
The script refuses to run if a store path is inside the content root (the 2026-08-09 incident).
* Exactly one gateway per box. The gateway and Worker image names are taken from the INSTALLED exes
(NSSM Application and env MxGateway__Worker__ExecutablePath) plus both known naming schemes
(ZB.MOM.WW.MxGateway.* on windev, MxGateway.* on the wonder boxes), and processes are also matched by
exact path. The guards FAIL CLOSED: a RUNNING service whose gateway process cannot be found is an
abort, not a pass (mxaccessgw#230: the guards used to pass vacuously on the wonder boxes).
* Never writes the NSSM env block (AppEnvironmentExtra), Application or AppDirectory: the swap is
rename-in-place. The env block is fingerprinted (entry names + SHA-256 prefix of each value; values are
never printed) at preflight, after the stop, after the swap, after the start and after the stability
window. Any added, removed or changed entry aborts (and triggers the automatic rollback).
* Live-only top-level files in Server\ and Worker\ are carried into the new directories (appsettings*.json,
dashboard.pfx and other certificates/keys always keep the LIVE copy), hash-verified.
* API-key DB, Secrets store and a file-based master key must live outside the swapped directories; they are
only COPIED (db + -wal + -shm) into *.bak-<stamp> siblings while the service is stopped.
* A self-contained Server is refused if its .NET runtime is older than the live one (no silent downgrade).
-DryRun (or -WhatIf) prints every mutating action and performs none. Read-only probing (nssm get/status,
process list, disk space, file existence) still runs so the decision logic is exercised.
-DryRun (or -WhatIf) prints every mutating action and performs none. Read-only probing (service state,
registry env block, process list, disk space, file hashes) still runs, so the guards are exercised for real.
.PARAMETER ServerSource Staged Server publish output (contains ZB.MOM.WW.MxGateway.Server.exe).
.PARAMETER WorkerSource Staged x86 Worker output (contains ZB.MOM.WW.MxGateway.Worker.exe).
.PARAMETER ServerSource Staged Server publish output (must contain the same exe name as NSSM Application).
.PARAMETER WorkerSource Staged x86 Worker output (must contain the exe name of MxGateway__Worker__ExecutablePath).
.PARAMETER ServiceName NSSM service name. Default MxAccessGw.
.PARAMETER NssmPath nssm executable. Default 'nssm' (on PATH).
.PARAMETER ServerDir Live Server directory. Default: NSSM AppDirectory.
@@ -36,11 +42,12 @@
must carry it in ProductVersion ('0.1.2+<sha>'). Builds from 2026-07-09..08-10 have a bogus stamp.
.PARAMETER Label Free-text suffix for backup names (e.g. 'planwrites').
.PARAMETER HealthBaseUrl Default http://localhost:5130 (dashboard port; /health/ready, /healthz anonymous).
.PARAMETER Rollback Roll back instead of upgrading. Uses newest Server.bak-* unless -BackupStamp is given.
.PARAMETER BackupStamp Stamp (yyyyMMdd-HHmmss) of the backup set to roll back to.
.PARAMETER Rollback Roll back instead of upgrading. Uses the newest Server.bak-* / Server.bak.* unless -BackupStamp.
.PARAMETER BackupStamp Stamp of the backup set to roll back to (yyyyMMdd-HHmmss, or yyyyMMddTHHmmss for .bak.<stamp> sets).
.PARAMETER RestoreStores With -Rollback: also restore gateway-auth.db / secrets db from *.bak-<stamp> (schema-bump case).
.PARAMETER NoAutoRollback Do not roll back automatically when start/verification fails.
.PARAMETER TakeNewAppSettings Do not carry the live appsettings*.json over into the new Server dir.
.PARAMETER AllowRuntimeDowngrade Accept a staged self-contained runtime older than the live one.
.PARAMETER SkipFreeSpaceCheck / MinFreeGB / StopTimeoutSeconds / StableSeconds / ReadyTimeoutSeconds tunables.
.PARAMETER DryRun Print, do not act. -WhatIf is equivalent.
#>
@@ -60,6 +67,7 @@ param(
[switch]$RestoreStores,
[switch]$NoAutoRollback,
[switch]$TakeNewAppSettings,
[switch]$AllowRuntimeDowngrade,
[switch]$SkipFreeSpaceCheck,
[double]$MinFreeGB = 2,
[int]$StopTimeoutSeconds = 30,
@@ -70,16 +78,38 @@ param(
$ErrorActionPreference = 'Stop'
$script:Dry = $DryRun.IsPresent -or $WhatIfPreference
# -WhatIf is handled by $script:Dry (Step). Leaving the preference on makes Windows PowerShell 5.1 cmdlets used for READ-ONLY
# probing misbehave (Get-FileHash returns nothing under -WhatIf), so the plan would abort; found on windev, #230.
$WhatIfPreference = $false
$script:Stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
$script:ServerProc = 'ZB.MOM.WW.MxGateway.Server'
$script:WorkerProc = 'ZB.MOM.WW.MxGateway.Worker'
# Both naming schemes seen in the field: windev runs ZB.MOM.WW.MxGateway.*.exe, the wonder boxes MxGateway.*.exe.
# The installed exe names (NSSM Application, MxGateway__Worker__ExecutablePath) are added to these in Get-Context.
$script:KnownServerImages = @('ZB.MOM.WW.MxGateway.Server', 'MxGateway.Server')
$script:KnownWorkerImages = @('ZB.MOM.WW.MxGateway.Worker', 'MxGateway.Worker')
$script:ServerImages = $script:KnownServerImages
$script:WorkerImages = $script:KnownWorkerImages
$script:EnvWorkerExe = 'MxGateway__Worker__ExecutablePath'
# Live copy always wins for these (host config, certificates, keys), even when the staged build ships one.
$script:PreservePatterns = @('appsettings*.json', '*.pfx', '*.p12', '*.pem', '*.key', '*.crt', '*.cer')
# Never carried from the old build (they are build output; a live-only binary is a leftover of an older build).
$script:BinaryExtensions = @('.dll', '.exe', '.pdb', '.so', '.dylib', '.winmd')
$script:Notes = New-Object System.Collections.Generic.List[string]
$script:Redact = New-Object System.Collections.Generic.List[string]
$script:LockStream = $null
$script:BackupDir = $null
$script:EnvWarnOnly = $false
function Say([string]$m) { Write-Host $m; $script:Notes.Add($m) }
# --- output (secret values are redacted from every line) ----------------------------------------------------
function Hide([string]$m) {
if ($null -eq $m) { return '' }
foreach ($s in $script:Redact) { if ($s) { $m = $m.Replace($s, '<redacted>') } }
$m
}
function Say([string]$m) { $m = Hide $m; Write-Host $m; $script:Notes.Add($m) }
function Info([string]$m) { Say "[info] $m" }
function Warn([string]$m) { Write-Warning $m; $script:Notes.Add("[WARN] $m") }
function Fail([string]$m) { throw "ABORT: $m" }
function Warn([string]$m) { $m = Hide $m; Write-Warning $m; $script:Notes.Add("[WARN] $m") }
function Fail([string]$m) { throw "ABORT: $(Hide $m)" }
# Mutating step: print in dry-run, run otherwise.
function Step([string]$Description, [scriptblock]$Action) {
@@ -88,7 +118,9 @@ function Step([string]$Description, [scriptblock]$Action) {
& $Action
}
# --- helpers --------------------------------------------------------------------------------------------
# --- environment seams (the dry-run harness in scripts/deploy/tests replaces these) ---------------------------
function Test-IsWindowsHost { $env:OS -eq 'Windows_NT' }
function Invoke-Nssm {
param([string[]]$NssmArgs, [switch]$AllowFail)
@@ -107,20 +139,50 @@ function Get-ServiceState {
($r.Text -split "`n")[0].Trim()
}
function Parse-EnvBlock([string]$Text) {
$map = [ordered]@{}
foreach ($line in ($Text -split "[`r`n]+")) {
$t = $line.Trim()
if ($t -match '^([^=\s]+)=(.*)$') { $map[$Matches[1]] = $Matches[2] }
# The env block exactly as NSSM stores it (REG_MULTI_SZ), one 'NAME=value' per element. Falls back to nssm get.
function Read-EnvBlockLines {
$key = "HKLM:\SYSTEM\CurrentControlSet\Services\$ServiceName\Parameters"
if ((Test-IsWindowsHost) -and (Test-Path -LiteralPath $key)) {
$p = Get-ItemProperty -LiteralPath $key -Name AppEnvironmentExtra -ErrorAction SilentlyContinue
if ($p) { return @($p.AppEnvironmentExtra | Where-Object { $_ }) }
return @()
}
$map
@((Get-NssmValue 'AppEnvironmentExtra') -split "[`r`n]+" | Where-Object { $_.Trim() })
}
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
function Test-Inside([string]$Path, [string]$Root) {
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
$a -eq $b -or $a.StartsWith("$b/")
# Id, ParentId, Name (image name without .exe), Path (may be $null when unreadable).
function Get-ProcessTable {
if (Get-Command Get-CimInstance -ErrorAction SilentlyContinue) {
return @(Get-CimInstance Win32_Process | ForEach-Object {
[pscustomobject]@{ Id = [int]$_.ProcessId; ParentId = [int]$_.ParentProcessId; Name = [IO.Path]::GetFileNameWithoutExtension([string]$_.Name); Path = $_.ExecutablePath }
})
}
@(Get-Process -ErrorAction SilentlyContinue | ForEach-Object {
$p = $null; try { $p = $_.Path } catch { }
[pscustomobject]@{ Id = $_.Id; ParentId = 0; Name = $_.ProcessName; Path = $p }
})
}
# PID of the NSSM service host (the gateway Server is its child). $null when it cannot be read.
function Get-ServiceHostPid {
if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null }
$s = Get-CimInstance Win32_Service -Filter "Name='$ServiceName'" -ErrorAction SilentlyContinue
if ($s -and $s.ProcessId) { return [int]$s.ProcessId }
$null
}
# Other services whose command line launches the gateway exe. $null when the check is unavailable.
function Get-OtherGatewayServices([string]$ExeName) {
if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null }
# The leading comma keeps an EMPTY result an array: a bare @() reaches the caller as $null ('check unavailable').
, @(Get-CimInstance Win32_Service | Where-Object { $_.Name -ne $ServiceName -and $_.PathName -and $_.PathName -like "*$ExeName*" } | ForEach-Object { $_.Name })
}
function Stop-ProcessById([int]$Id) { Stop-Process -Id $Id -Force -ErrorAction SilentlyContinue }
function Get-HttpStatus([string]$Url) {
try { return [int](Invoke-WebRequest -Uri $Url -UseBasicParsing -TimeoutSec 10).StatusCode }
catch { if ($_.Exception.Response) { return [int]$_.Exception.Response.StatusCode }; return 0 }
}
function Get-ProductSha([string]$Exe) {
@@ -130,25 +192,200 @@ function Get-ProductSha([string]$Exe) {
} catch { }
$null
}
# 'X86', 'Amd64', 'MSIL', ... or $null when it cannot be read.
function Get-ExeArchitecture([string]$Exe) {
try { return "$([Reflection.AssemblyName]::GetAssemblyName($Exe).ProcessorArchitecture)" } catch { return $null }
}
# Restrict a directory to SYSTEM + Administrators, inheritance off. Throws if that fails.
function Protect-Directory([string]$Path) {
if (-not (Test-IsWindowsHost)) { Warn "not Windows: $Path is not ACL-restricted"; return }
$null = & icacls.exe $Path /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' 2>&1
if ($LASTEXITCODE -ne 0) { Fail "icacls could not restrict $Path (exit $LASTEXITCODE); refusing to write the env block there" }
}
function Export-ServiceKey([string]$Destination) {
if (-not (Test-IsWindowsHost)) { Info 'not Windows: service registry export skipped'; return }
$null = & reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$ServiceName" $Destination /y 2>&1
if ($LASTEXITCODE -ne 0) { Fail "reg export of the service key failed (exit $LASTEXITCODE)" }
}
function Get-DirSddl([string]$Path) {
if (-not (Test-IsWindowsHost)) { return $null }
try { (Get-Acl -LiteralPath $Path).Sddl } catch { $null }
}
# Give a carried file the old file's ACL when the old one had explicit (non-inherited) entries.
function Copy-ExplicitAcl([string]$From, [string]$To) {
if (-not (Test-IsWindowsHost)) { return }
try {
$a = Get-Acl -LiteralPath $From
if (@($a.Access | Where-Object { -not $_.IsInherited }).Count -gt 0 -or $a.AreAccessRulesProtected) {
Set-Acl -LiteralPath $To -AclObject $a
Info " ACL of $(Split-Path -Leaf $From) copied (it had explicit entries)"
}
} catch { Warn "could not copy the ACL onto ${To}: $($_.Exception.Message)" }
}
# --- pure helpers ---------------------------------------------------------------------------------------------
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
function Test-Inside([string]$Path, [string]$Root) {
if (-not $Path -or -not $Root) { return $false }
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
$a -eq $b -or $a.StartsWith("$b/")
}
function Test-SamePath([string]$A, [string]$B) { $A -and $B -and ((Norm $A).ToLowerInvariant() -eq (Norm $B).ToLowerInvariant()) }
function Get-Leaf([string]$p) { if (-not $p) { return $null }; ((Norm $p) -split '/')[-1] }
function Get-Stem([string]$p) { $l = Get-Leaf $p; if (-not $l) { return $null }; [IO.Path]::GetFileNameWithoutExtension($l) }
function Test-ShaMatch([string]$Actual, [string]$Expected) {
if (-not $Actual -or -not $Expected) { return $false }
$e = $Expected.ToLowerInvariant()
$Actual.StartsWith($e) -or $e.StartsWith($Actual)
}
function Get-GatewayProcesses {
@(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in @($script:ServerProc, $script:WorkerProc) })
function Get-TextHashPrefix([string]$Text) {
$sha = [Security.Cryptography.SHA256]::Create()
try { (($sha.ComputeHash([Text.Encoding]::UTF8.GetBytes($Text)) | ForEach-Object { $_.ToString('x2') }) -join '').Substring(0, 12) }
finally { $sha.Dispose() }
}
function Get-FileSha([string]$Path) {
$sha = [Security.Cryptography.SHA256]::Create(); $fs = [IO.File]::OpenRead($Path)
try { (($sha.ComputeHash($fs) | ForEach-Object { $_.ToString('X2') }) -join '') } finally { $fs.Dispose(); $sha.Dispose() }
}
# Store file set: db + -wal + -shm
function Get-StoreFiles([string]$Db) { @($Db, "$Db-wal", "$Db-shm") | Where-Object { Test-Path -LiteralPath $_ } }
function Test-SensitiveName([string]$Name) { ($Name -match 'KEY|PEPPER|PASS|SECRET|TOKEN|PWD|CREDENTIAL') -and ($Name -notmatch '(Path|Source|EnvVarName|Name)$') }
# name -> value. Every sensitive value is registered for redaction.
function ConvertTo-EnvMap([string[]]$Lines) {
$map = [ordered]@{}
foreach ($line in $Lines) {
$i = $line.IndexOf('=')
if ($i -le 0) { continue }
$k = $line.Substring(0, $i).Trim(); $v = $line.Substring($i + 1)
$map[$k] = $v
if ((Test-SensitiveName $k) -and $v.Length -ge 4 -and -not $script:Redact.Contains($v)) { $script:Redact.Add($v) }
}
$map
}
# Names + per-value SHA-256 prefixes. Never carries a value.
function Get-EnvFingerprint([string[]]$Lines) {
$map = ConvertTo-EnvMap $Lines
$entries = [ordered]@{}
foreach ($k in $map.Keys) { $entries[$k] = Get-TextHashPrefix ([string]$map[$k]) }
[pscustomobject]@{ Count = @($Lines).Count; Names = @($map.Keys); Entries = $entries; BlockHash = (Get-TextHashPrefix (@($Lines) -join "`n")) }
}
# List of differences (names only). Empty when identical.
function Compare-EnvFingerprint($Before, $After) {
$d = New-Object System.Collections.Generic.List[string]
foreach ($k in $Before.Entries.Keys) {
if (-not $After.Entries.Contains($k)) { $d.Add("removed $k") }
elseif ($After.Entries[$k] -ne $Before.Entries[$k]) { $d.Add("changed $k (sha256 $($Before.Entries[$k]) -> $($After.Entries[$k]))") }
}
foreach ($k in $After.Entries.Keys) { if (-not $Before.Entries.Contains($k)) { $d.Add("added $k") } }
if ($d.Count -eq 0 -and $Before.BlockHash -ne $After.BlockHash) { $d.Add("block hash $($Before.BlockHash) -> $($After.BlockHash) (order, duplicates or blank entries changed)") }
, $d.ToArray()
}
function Format-EnvFingerprint($Fp) { "$($Fp.Count) entries, block sha256 $($Fp.BlockHash): " + (($Fp.Entries.Keys | ForEach-Object { "$_=$($Fp.Entries[$_])" }) -join ', ') }
# Re-read the env block and abort on any difference from the preflight fingerprint.
function Assert-EnvUnchanged($ctx, [string]$When) {
$now = Get-EnvFingerprint (Read-EnvBlockLines)
$diff = Compare-EnvFingerprint $ctx.EnvFp $now
if ($diff.Count -gt 0) {
# During an automatic rollback the difference is already the reported failure; restoring the old build still goes ahead.
if ($script:EnvWarnOnly) { Warn "NSSM env block changed ${When}: $($diff -join '; ') (not written by this script; restore it from the backup dir's AppEnvironmentExtra.txt if unintended)"; return }
Fail "NSSM env block changed ${When}: $($diff -join '; ')"
}
Info "env block unchanged $When ($($now.Count) entries, block sha256 $($now.BlockHash))"
}
# Directory listings via .NET (FileInfo/DirectoryInfo, hidden files included). Get-ChildItem costs ~250 ms per call
# under pwsh on macOS (per-file owner lookups), which made the dry-run harness 20x slower; on Windows both are fine.
function Get-Files([string]$Dir, [string]$Filter = '*', [switch]$Recurse) {
if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return }
$o = if ($Recurse) { [IO.SearchOption]::AllDirectories } else { [IO.SearchOption]::TopDirectoryOnly }
try { (New-Object IO.DirectoryInfo $Dir).GetFiles($Filter, $o) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" }
}
function Get-Dirs([string]$Dir, [string]$Filter = '*') {
if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return }
try { (New-Object IO.DirectoryInfo $Dir).GetDirectories($Filter) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" }
}
function Get-StoreFiles([string]$Db) { @($Db, "$Db-wal", "$Db-shm") | Where-Object { $_ -and (Test-Path -LiteralPath $_) } }
function Get-DirSize([string]$p) {
if (-not (Test-Path -LiteralPath $p)) { return 0 }
$s = (Get-ChildItem -LiteralPath $p -Recurse -File -Force -ErrorAction SilentlyContinue | Measure-Object Length -Sum).Sum
$s = (@(Get-Files $p -Recurse) | Measure-Object Length -Sum).Sum
if ($s) { [double]$s } else { 0 }
}
# Version of a self-contained .NET runtime in a directory (from mscordaccore_<arch>_<arch>_<ver>.dll), or $null (framework-dependent).
function Get-BundledRuntimeVersion([string]$Dir) {
$f = @(Get-Files $Dir 'mscordaccore_*.dll')
foreach ($x in $f) { if ($x.Name -match '_(\d+\.\d+\.\d+(\.\d+)?)\.dll$') { return [version]$Matches[1] } }
$null
}
# --- processes --------------------------------------------------------------------------------------------------
function Test-GatewayServer($p, $ctx) { ($script:ServerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.Application) }
function Test-GatewayWorker($p, $ctx) { ($script:WorkerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.WorkerExe) }
function Get-GatewayProcesses($ctx) {
@(Get-ProcessTable | Where-Object { (Test-GatewayServer $_ $ctx) -or (Test-GatewayWorker $_ $ctx) } | ForEach-Object {
$role = if (Test-GatewayServer $_ $ctx) { 'Server' } else { 'Worker' }
$_ | Add-Member -NotePropertyName Role -NotePropertyValue $role -PassThru -Force
})
}
function Format-Procs($procs) { (@($procs) | ForEach-Object { "$($_.Name)#$($_.Id)<-$($_.ParentId) [$(if ($_.Path) { $_.Path } else { 'path unreadable' })]" }) -join ', ' }
function Assert-NoGatewayProcess($ctx, [string]$When) {
$left = @(Get-GatewayProcesses $ctx)
if ($left.Count -gt 0) { Fail "${When}: gateway process(es) still alive: $(Format-Procs $left)" }
}
# The one-gateway guard. Fails closed: a RUNNING service whose gateway cannot be seen is an abort.
function Assert-SingleGateway($ctx, [string]$State) {
$procs = @(Get-GatewayProcesses $ctx)
$servers = @($procs | Where-Object { $_.Role -eq 'Server' })
$workers = @($procs | Where-Object { $_.Role -eq 'Worker' })
Info "gateway processes on the box (images $($script:ServerImages -join '|') / $($script:WorkerImages -join '|')): $(if ($procs.Count) { Format-Procs $procs } else { 'none' })"
if ($servers.Count -gt 1) { Fail "$($servers.Count) gateway Server processes are running ($(Format-Procs $servers)). A second gateway is the OrphanWorkerTerminator trap - resolve manually." }
switch ($State) {
'SERVICE_STOPPED' {
if ($procs.Count -gt 0) { Fail "service is STOPPED but gateway process(es) exist ($(Format-Procs $procs)): a rogue second gateway or an unclean stop. Starting would terminate those workers." }
}
'SERVICE_RUNNING' {
if ($servers.Count -eq 0) {
Fail "service is RUNNING but no gateway Server process was found (looked for image names $($script:ServerImages -join ', ') and path $($ctx.Application)). The process guards cannot see this gateway, so they would pass vacuously; refusing."
}
$s = $servers[0]
if ($s.Path -and -not (Test-SamePath $s.Path $ctx.Application)) { Fail "the running gateway is $($s.Path), not the service's Application $($ctx.Application): a second gateway is running and the service's own is not" }
if (-not $s.Path) { Warn "gateway Server #$($s.Id) path is unreadable; matched by image name only" }
$hostPid = Get-ServiceHostPid
if ($hostPid) {
if ($s.ParentId -ne $hostPid) { Fail "gateway Server #$($s.Id) is not a child of the $ServiceName service host #${hostPid} (parent #$($s.ParentId)): it is another gateway" }
Info "gateway Server #$($s.Id) is the service's own (child of service host #$hostPid)"
} else { Warn 'service host PID unavailable; parent check skipped' }
$foreign = @($workers | Where-Object { $_.ParentId -and $_.ParentId -ne $s.Id })
if ($foreign.Count -gt 0) { Warn "Worker process(es) not owned by the service's gateway: $(Format-Procs $foreign). They are killed as orphans on the next start." }
$wrongPath = @($workers | Where-Object { $_.Path -and $ctx.WorkerExe -and -not (Test-SamePath $_.Path $ctx.WorkerExe) })
if ($wrongPath.Count -gt 0) { Warn "Worker process(es) not running the configured $($ctx.WorkerExe): $(Format-Procs $wrongPath)" }
}
'SERVICE_STOP_PENDING' { Warn 'service is STOP_PENDING; the stop step will finish it' }
default {
if ($servers.Count -gt 0) { Fail "a gateway Server process exists while the service is $State" }
}
}
}
# --- context discovery (read only) -------------------------------------------------------------------------
function Get-Context {
@@ -158,25 +395,47 @@ function Get-Context {
$ctx.State = $state
$ctx.Application = Get-NssmValue 'Application'
$ctx.AppDirectory = Get-NssmValue 'AppDirectory'
$ctx.EnvText = Get-NssmValue 'AppEnvironmentExtra'
$ctx.Env = Parse-EnvBlock $ctx.EnvText
$ctx.EnvCount = $ctx.Env.Count
if (-not $ServerDir) { $ctx.ServerDir = $ctx.AppDirectory } else { $ctx.ServerDir = $ServerDir }
$wexe = $ctx.Env['MxGateway__Worker__ExecutablePath']
$lines = @(Read-EnvBlockLines)
$ctx.Env = ConvertTo-EnvMap $lines
$ctx.EnvFp = Get-EnvFingerprint $lines
$ctx.EnvCount = $ctx.EnvFp.Count
if ($ServerDir) { $ctx.ServerDir = $ServerDir } else { $ctx.ServerDir = $ctx.AppDirectory }
$wexe = $ctx.Env[$script:EnvWorkerExe]
$ctx.WorkerExe = $wexe
if ($WorkerDir) { $ctx.WorkerDir = $WorkerDir }
elseif ($wexe) { $ctx.WorkerDir = Split-Path -Parent $wexe }
else { Fail 'cannot determine the Worker directory: env MxGateway__Worker__ExecutablePath is not in the NSSM env block; pass -WorkerDir' }
else { Fail "cannot determine the Worker directory: env $($script:EnvWorkerExe) is not in the NSSM env block; pass -WorkerDir" }
$pd = 'C:\ProgramData\MxGateway'
$auth = $ctx.Env['MxGateway__Authentication__SqlitePath']; if (-not $auth) { $auth = "$pd\gateway-auth.db" }
$sec = $ctx.Env['Secrets__SqlitePath']; if (-not $sec) { $sec = "$pd\mxgateway-secrets.db" }
$ctx.AuthDb = $auth; $ctx.SecretsDb = $sec
$ctx.ServerExeName = Split-Path -Leaf $ctx.Application
$ctx.KekFile = $ctx.Env['Secrets__MasterKey__FilePath']
$ctx.ServerExeName = Get-Leaf $ctx.Application
$ctx.WorkerExeName = Get-Leaf $wexe
if (-not $ctx.WorkerExeName) { $ctx.WorkerExeName = 'ZB.MOM.WW.MxGateway.Worker.exe' }
# Process image names: the installed ones first, then both known schemes.
$script:ServerImages = @(@(Get-Stem $ctx.Application) + $script:KnownServerImages | Where-Object { $_ } | Select-Object -Unique)
$script:WorkerImages = @(@(Get-Stem $wexe) + $script:KnownWorkerImages | Where-Object { $_ } | Select-Object -Unique)
[pscustomobject]$ctx
}
# --- preflight -----------------------------------------------------------------------------------------------
function Test-MasterKey($ctx) {
$src = $ctx.Env['Secrets__MasterKey__Source']
$envVar = $ctx.Env['Secrets__MasterKey__EnvVarName']; if (-not $envVar) { $envVar = 'ZB_SECRETS_MASTER_KEY' }
if ($ctx.KekFile -or $src -eq 'File') {
if (-not $ctx.KekFile) { Fail 'Secrets__MasterKey__Source=File but no Secrets__MasterKey__FilePath in the env block' }
if (-not (Test-Rooted $ctx.KekFile)) { Fail "master key file '$($ctx.KekFile)' is not absolute" }
foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) { if (Test-Inside $ctx.KekFile $root) { Fail "master key file '$($ctx.KekFile)' is INSIDE '$root' - the swap would lose it" } }
if (Test-Path -LiteralPath $ctx.KekFile) { Info "master key: file $($ctx.KekFile) (present, outside the swap)" } else { Fail "master key file '$($ctx.KekFile)' does not exist (the Secrets store would be unreadable)" }
} elseif ($ctx.Env.Contains($envVar)) {
Info "master key: env block entry $envVar"
} else {
Warn "the env block carries no master key ($envVar or Secrets__MasterKey__FilePath); it comes from elsewhere (appsettings or the account environment). The upgrade does not change that, but confirm the Secrets store opens after the start."
}
}
function Invoke-Preflight($ctx) {
Say '== Preflight =='
Info "service '$ServiceName' state: $($ctx.State)"
@@ -186,20 +445,19 @@ function Invoke-Preflight($ctx) {
Info "Worker dir : $($ctx.WorkerDir) (exe env: $($ctx.WorkerExe))"
Info "API-key DB : $($ctx.AuthDb)"
Info "Secrets DB : $($ctx.SecretsDb)"
Info "NSSM env block: $($ctx.EnvCount) entries (names only): $(($ctx.Env.Keys) -join ', ')"
Info "NSSM env block fingerprint (names + value sha256 prefixes, no values): $(Format-EnvFingerprint $ctx.EnvFp)"
if ($ctx.EnvCount -eq 0) { Fail 'NSSM env block is empty - refusing: it holds Kestrel/store paths, the KEK and the pepper' }
foreach ($need in 'ZB_SECRETS_MASTER_KEY') {
if (-not $ctx.Env.Contains($need)) { Fail "env block has no $need (the Secrets store would be unreadable after the restart)" }
}
if ($ctx.EnvCount -eq 0) { Fail 'NSSM env block is empty - refusing: it holds Kestrel/store paths, the master key and the pepper' }
if (-not ($ctx.Env.Keys | Where-Object { $_ -match 'Pepper' })) {
Warn 'no *Pepper* entry in the env block; confirm the API-key pepper is supplied another way or issued tokens will not verify'
}
if ($ctx.Env['ASPNETCORE_ENVIRONMENT'] -ne 'Production' -and $ctx.Env['DOTNET_ENVIRONMENT'] -ne 'Production') {
Warn 'neither ASPNETCORE_ENVIRONMENT nor DOTNET_ENVIRONMENT is exactly Production (runbook trap 6) - do not change it during an upgrade'
}
if (-not (Test-SamePath (Split-Path -Parent $ctx.Application) $ctx.ServerDir)) { Fail "NSSM Application '$($ctx.Application)' is not inside the Server dir '$($ctx.ServerDir)'" }
if ($ctx.WorkerExe -and -not (Test-SamePath (Split-Path -Parent $ctx.WorkerExe) $ctx.WorkerDir)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' is not inside the Worker dir '$($ctx.WorkerDir)'" }
# State that must survive: rooted and outside the content root.
# State that must survive: rooted and outside the swapped directories.
foreach ($pair in @(@('API-key DB', $ctx.AuthDb), @('Secrets DB', $ctx.SecretsDb))) {
if (-not (Test-Rooted $pair[1])) { Fail "$($pair[0]) path '$($pair[1])' is not absolute" }
foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) {
@@ -208,6 +466,7 @@ function Invoke-Preflight($ctx) {
if (Test-Path -LiteralPath $pair[1]) { Info "$($pair[0]) present ($((Get-StoreFiles $pair[1]).Count) file(s) incl. wal/shm)" }
else { Warn "$($pair[0]) '$($pair[1])' does not exist yet (fresh store will be created on start)" }
}
Test-MasterKey $ctx
# Directories
foreach ($d in @($ctx.ServerDir, $ctx.WorkerDir)) {
@@ -216,33 +475,34 @@ function Invoke-Preflight($ctx) {
if ((Test-Inside $ctx.WorkerDir $ctx.ServerDir) -or (Test-Inside $ctx.ServerDir $ctx.WorkerDir)) {
Fail 'Server and Worker directories are nested; the rename-in-place swap needs sibling directories'
}
if (-not (Test-Path -LiteralPath $ctx.Application)) { Fail "NSSM Application '$($ctx.Application)' does not exist" }
if ($ctx.WorkerExe -and -not (Test-Path -LiteralPath $ctx.WorkerExe)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' does not exist" }
$dbs = @(@(Get-Files $ctx.ServerDir -Recurse) + @(Get-Files $ctx.WorkerDir -Recurse) | Where-Object { $_.Name -match '\.db(-wal|-shm)?$' })
if ($dbs.Count -gt 0) { Fail "SQLite file(s) inside the swapped directories would be orphaned by the rename: $(($dbs | ForEach-Object { $_.FullName }) -join ', ')" }
# One-gateway guard
$procs = Get-GatewayProcesses
$servers = @($procs | Where-Object { $_.ProcessName -eq $script:ServerProc })
if ($servers.Count -gt 1) { Fail "$($servers.Count) gateway Server processes are running (PIDs $(($servers.Id) -join ',')). A second gateway is the OrphanWorkerTerminator trap - resolve manually." }
if ($ctx.State -eq 'SERVICE_STOPPED' -and $procs.Count -gt 0) {
Fail "service is STOPPED but gateway process(es) exist (PIDs $(($procs.Id) -join ',')): a rogue second gateway. Starting would terminate its live workers."
}
if ($servers.Count -eq 1 -and $ctx.State -ne 'SERVICE_RUNNING' -and $ctx.State -ne 'SERVICE_STOP_PENDING') {
Fail "a Server process exists while the service is $($ctx.State)"
}
if (Get-Command Get-CimInstance -ErrorAction SilentlyContinue) {
$svcs = @(Get-CimInstance Win32_Service | Where-Object { $_.Name -ne $ServiceName -and $_.PathName -and $_.PathName -like "*$($ctx.ServerExeName)*" })
if ($svcs.Count -gt 0) { Fail "another service ($(($svcs.Name) -join ', ')) also launches $($ctx.ServerExeName); two gateways on one box is forbidden" }
} else { Info 'Win32_Service cross-check unavailable on this host (skipped)' }
# One-gateway guard (fail closed)
Assert-SingleGateway $ctx $ctx.State
$others = Get-OtherGatewayServices $ctx.ServerExeName
if ($null -eq $others) { Info 'Win32_Service cross-check unavailable on this host (skipped)' }
elseif (@($others).Count -gt 0) { Fail "another service ($(@($others) -join ', ')) also launches $($ctx.ServerExeName); two gateways on one box is forbidden" }
else { Info "no other service launches $($ctx.ServerExeName)" }
if ($Rollback) { return }
if ($Rollback) { Say '== Preflight OK =='; return }
# Sources
if (-not $ServerSource -or -not $WorkerSource) { Fail '-ServerSource and -WorkerSource are required for an upgrade' }
foreach ($s in @(@($ServerSource, $ctx.ServerExeName), @($WorkerSource, "$($script:WorkerProc).exe"))) {
if (-not (Test-Path -LiteralPath (Join-Path $s[0] $s[1]))) { Fail "staged source '$($s[0])' does not contain $($s[1])" }
$stagedServerExe = Join-Path $ServerSource $ctx.ServerExeName
$stagedWorkerExe = Join-Path $WorkerSource $ctx.WorkerExeName
foreach ($s in @(@($ServerSource, $stagedServerExe), @($WorkerSource, $stagedWorkerExe))) {
if (-not (Test-Path -LiteralPath $s[1])) { Fail "staged source '$($s[0])' does not contain $(Get-Leaf $s[1]) (the installed exe name; NSSM and $($script:EnvWorkerExe) point at it)" }
}
if ((Test-Inside $ServerSource $ctx.ServerDir) -or (Test-Inside $WorkerSource $ctx.WorkerDir)) { Fail 'staging directory is inside a live directory' }
$sSha = Get-ProductSha (Join-Path $ServerSource $ctx.ServerExeName)
$wSha = Get-ProductSha (Join-Path $WorkerSource "$($script:WorkerProc).exe")
Info "staged Server SHA: $sSha ; staged Worker SHA: $wSha"
if ((Test-Inside $ServerSource $ctx.ServerDir) -or (Test-Inside $WorkerSource $ctx.WorkerDir) -or (Test-Inside $ServerSource $ctx.WorkerDir) -or (Test-Inside $WorkerSource $ctx.ServerDir)) { Fail 'staging directory is inside a live directory' }
$arch = Get-ExeArchitecture $stagedWorkerExe
if ($arch -ne 'X86') { Fail "staged Worker $stagedWorkerExe architecture is '$arch', not X86 (trap 5)" }
Info "staged Worker architecture: X86"
$sSha = Get-ProductSha $stagedServerExe
$wSha = Get-ProductSha $stagedWorkerExe
Info "staged Server SHA: $sSha ; staged Worker SHA: $wSha ; live Server SHA: $(Get-ProductSha $ctx.Application)"
if ($sSha -and $wSha -and -not (Test-ShaMatch $sSha $wSha)) { Fail "Server ($sSha) and Worker ($wSha) were built from different commits (wire protocol risk)" }
if ($ExpectedSha) {
if (-not (Test-ShaMatch $sSha $ExpectedSha)) {
@@ -250,6 +510,27 @@ function Invoke-Preflight($ctx) {
else { Fail "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha'" }
}
} else { Warn 'no -ExpectedSha given: the post-deploy SHA check will only record the SHA' }
$leafSha = if ((Get-Leaf $ctx.ServerDir) -match '([0-9a-f]{7,40})$') { $Matches[1] } else { $null }
if ($leafSha -and $sSha -and -not (Test-ShaMatch $leafSha $sSha)) { Warn "the Server dir name '$(Get-Leaf $ctx.ServerDir)' carries SHA $leafSha but will hold $sSha after the rename-in-place swap" }
# Runtime: never silently downgrade a self-contained Server.
$liveRt = Get-BundledRuntimeVersion $ctx.ServerDir; $newRt = Get-BundledRuntimeVersion $ServerSource
Info "bundled .NET runtime: live $(if ($liveRt) { $liveRt } else { 'none (framework-dependent)' }), staged $(if ($newRt) { $newRt } else { 'none (framework-dependent)' })"
if ($liveRt -and -not $newRt) { Warn 'live Server is self-contained, staged is framework-dependent: the box must have a matching .NET 10 runtime installed' }
if ($liveRt -and $newRt -and $newRt -lt $liveRt) {
if ($AllowRuntimeDowngrade) { Warn "staged runtime $newRt is OLDER than live $liveRt (-AllowRuntimeDowngrade)" }
else { Fail "staged self-contained runtime $newRt is OLDER than the live $liveRt; rebuild with a newer SDK or pass -AllowRuntimeDowngrade" }
}
# What the swap will carry over / drop (top-level files).
foreach ($pair in @(@('Server', $ctx.ServerDir, $ServerSource), @('Worker', $ctx.WorkerDir, $WorkerSource))) {
$plan = Get-CarryPlan $pair[1] $pair[2] ($pair[0] -eq 'Server' -and -not $TakeNewAppSettings)
Info "$($pair[0]): carry from live -> $(if ($plan.Carry.Count) { $plan.Carry -join ', ' } else { 'nothing' }); live-only binaries dropped -> $(if ($plan.Dropped.Count) { $plan.Dropped -join ', ' } else { 'none' })"
if ($plan.Dirs.Count) { Warn "$($pair[0]): live-only directories NOT carried (they stay in the .bak): $($plan.Dirs -join ', ')" }
}
$lw = @(Get-Files $ctx.WorkerDir | ForEach-Object { $_.Name }); $nw = @(Get-Files $WorkerSource | ForEach-Object { $_.Name })
$onlyNew = @($nw | Where-Object { $lw -notcontains $_ })
if ($onlyNew.Count) { Info "Worker: files only in the staged build: $($onlyNew -join ', ')" }
# Disk space: 2x (server + worker + stores + payload), at least MinFreeGB
if (-not $SkipFreeSpaceCheck) {
@@ -272,52 +553,67 @@ function Wait-State([string]$Want, [int]$TimeoutSec) {
$false
}
function Assert-NoGatewayProcess([string]$When) {
$left = Get-GatewayProcesses
if ($left.Count -gt 0) { Fail "${When}: gateway process(es) still alive: $(($left | ForEach-Object { "$($_.ProcessName)#$($_.Id)" }) -join ', ')" }
function Wait-NoGatewayProcess($ctx, [int]$TimeoutSec) {
$deadline = (Get-Date).AddSeconds($TimeoutSec)
do { if (@(Get-GatewayProcesses $ctx).Count -eq 0) { return $true }; Start-Sleep -Seconds 1 } while ((Get-Date) -lt $deadline)
$false
}
function Stop-Gateway {
# Kill only this install's gateway processes: path under the Server/Worker dir, or (path unreadable) the installed image name.
function Stop-InstallProcesses($ctx, [string]$Why) {
$installed = @((Get-Stem $ctx.Application), (Get-Stem $ctx.WorkerExe)) | Where-Object { $_ }
$mine = @(Get-GatewayProcesses $ctx | Where-Object {
if ($_.Path) { (Test-Inside $_.Path $ctx.ServerDir) -or (Test-Inside $_.Path $ctx.WorkerDir) } else { $installed -contains $_.Name }
} | Sort-Object { if ($_.Role -eq 'Server') { 0 } else { 1 } })
foreach ($p in $mine) { Say "[ACTION] Stop-Process -Id $($p.Id) -Force ($($p.Name), $Why)"; Stop-ProcessById $p.Id }
$mine.Count
}
function Stop-Gateway($ctx) {
Say '-- Stop --'
$state = Get-ServiceState
if ($state -eq 'SERVICE_STOPPED') { Info 'already stopped' }
else {
Step "nssm stop $ServiceName (wait up to $StopTimeoutSeconds s for SERVICE_STOPPED)" {
Step "nssm stop $ServiceName (wait up to $StopTimeoutSeconds s for SERVICE_STOPPED, then for every gateway/Worker process to exit)" {
[void](Invoke-Nssm -NssmArgs @('stop', $ServiceName) -AllowFail)
if (-not (Wait-State 'SERVICE_STOPPED' $StopTimeoutSeconds)) {
# STOP_PENDING trap: workers did not exit in time. Kill the Server (then Workers) by image name -
# safe because preflight proved there is exactly one gateway on the box.
Warn "service still $(Get-ServiceState) after $StopTimeoutSeconds s (STOP_PENDING trap) - killing gateway processes"
foreach ($n in @($script:ServerProc, $script:WorkerProc)) {
Get-Process -Name $n -ErrorAction SilentlyContinue | ForEach-Object { Say "[ACTION] Stop-Process -Id $($_.Id) -Force ($n)"; Stop-Process -Id $_.Id -Force }
}
# STOP_PENDING trap: workers did not exit in time. Kill this install's Server, then its Workers.
Warn "service still $(Get-ServiceState) after $StopTimeoutSeconds s (STOP_PENDING trap) - killing this install's gateway processes"
$n = Stop-InstallProcesses $ctx 'STOP_PENDING'
if ($n -eq 0) { Warn 'no process of this install was found to kill' }
if (-not (Wait-State 'SERVICE_STOPPED' 20)) {
Warn 'still not STOPPED; killing the NSSM service host process'
try {
$svcPid = (Get-CimInstance Win32_Service -Filter "Name='$ServiceName'").ProcessId
if ($svcPid) { Stop-Process -Id $svcPid -Force }
} catch { Warn "could not kill the service host: $_" }
$svcPid = Get-ServiceHostPid
if ($svcPid) { Say "[ACTION] Stop-Process -Id $svcPid -Force (service host)"; Stop-ProcessById $svcPid }
if (-not (Wait-State 'SERVICE_STOPPED' 20)) { Fail "service '$ServiceName' will not stop; investigate manually (state $(Get-ServiceState))" }
}
}
Start-Sleep -Seconds 2
Assert-NoGatewayProcess 'after stop'
if (-not (Wait-NoGatewayProcess $ctx 20)) {
Warn "gateway/Worker process(es) outlived the service stop (unclean stop): $(Format-Procs (Get-GatewayProcesses $ctx))"
[void](Stop-InstallProcesses $ctx 'left over after the stop')
[void](Wait-NoGatewayProcess $ctx 10)
}
}
}
if (-not $script:Dry) { Assert-NoGatewayProcess 'after stop' }
if (-not $script:Dry) {
Assert-NoGatewayProcess $ctx 'after stop'
Info 'stopped; no gateway/Worker process on the box'
Assert-EnvUnchanged $ctx 'after the stop'
}
}
function Start-Gateway($ctx) {
Say '-- Start --'
if (-not $script:Dry) { Assert-NoGatewayProcess 'before start (OrphanWorkerTerminator guard)' }
else { Say '[DRYRUN] assert no Server/Worker process exists (OrphanWorkerTerminator guard)' }
if (-not $script:Dry) {
Assert-NoGatewayProcess $ctx 'before start (OrphanWorkerTerminator guard)'
Assert-EnvUnchanged $ctx 'before the start'
} else { Say "[DRYRUN] assert no process named $($script:ServerImages -join '|') / $($script:WorkerImages -join '|') exists, and the env block fingerprint is unchanged (OrphanWorkerTerminator guard)" }
Step "nssm start $ServiceName (once)" {
[void](Invoke-Nssm -NssmArgs @('start', $ServiceName))
}
Step "wait for SERVICE_RUNNING and compare env entry count (expect $($ctx.EnvCount))" {
Step "wait for SERVICE_RUNNING and compare the env block fingerprint" {
if (-not (Wait-State 'SERVICE_RUNNING' 60)) { Fail "service did not reach RUNNING (state $(Get-ServiceState))" }
$now = (Parse-EnvBlock (Get-NssmValue 'AppEnvironmentExtra')).Count
if ($now -ne $ctx.EnvCount) { Fail "NSSM env block entry count changed: $($ctx.EnvCount) -> $now" }
Assert-EnvUnchanged $ctx 'after the start'
}
}
@@ -329,94 +625,157 @@ function Backup-State($ctx) {
Say '-- Backup (service stopped) --'
$suffix = Get-Suffix
$root = Split-Path -Parent $ctx.ServerDir
$envFile = Join-Path $root "_env-AppEnvironmentExtra-$($script:Stamp).txt"
Step "save the NSSM env block + Application/AppDirectory to $envFile (contains secrets: keep it ACL-restricted)" {
$body = @("Application=$($ctx.Application)", "AppDirectory=$($ctx.AppDirectory)", '--- AppEnvironmentExtra ---', $ctx.EnvText)
Set-Content -LiteralPath $envFile -Value $body -Encoding UTF8
try { & icacls.exe $envFile /inheritance:r /grant:r 'SYSTEM:(F)' 'BUILTIN\Administrators:(F)' | Out-Null } catch { Warn "could not restrict ACL on ${envFile}: $_" }
$dir = Join-Path $root "_deploy-backup-$suffix"
$script:BackupDir = $dir
Step "create $dir, restrict it to SYSTEM + Administrators, then save the env block (secrets), service registry key and env fingerprint" {
New-Item -ItemType Directory -Path $dir -Force | Out-Null
Protect-Directory $dir
$body = @("Application=$($ctx.Application)", "AppDirectory=$($ctx.AppDirectory)", '--- AppEnvironmentExtra ---') + @(Read-EnvBlockLines)
Set-Content -LiteralPath (Join-Path $dir 'AppEnvironmentExtra.txt') -Value $body -Encoding UTF8
Export-ServiceKey (Join-Path $dir "$ServiceName-servicekey.reg")
$fp = @("BlockHash=$($ctx.EnvFp.BlockHash)") + @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" })
Set-Content -LiteralPath (Join-Path $dir 'env-fingerprint.txt') -Value $fp -Encoding ASCII
}
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
foreach ($f in (Get-StoreFiles $db)) {
$dst = "$f.bak-$suffix"
Step "copy store file $f -> $dst (live file is never moved)" { Copy-Item -LiteralPath $f -Destination $dst -Force }
Step "copy store file $f -> $dst (live file is never moved)" {
Copy-Item -LiteralPath $f -Destination $dst -Force
if ((Get-FileSha $f) -ne (Get-FileSha $dst)) { Fail "store backup hash mismatch for $f" }
}
}
if (-not (Test-Path -LiteralPath $db)) { Info "no store at $db to back up" }
}
if ($ctx.KekFile -and (Test-Path -LiteralPath $ctx.KekFile)) {
Step "copy master key file $($ctx.KekFile) into $dir (live file is never moved)" { Copy-Item -LiteralPath $ctx.KekFile -Destination (Join-Path $dir (Get-Leaf $ctx.KekFile)) -Force }
}
}
function Test-Preserved([string]$Name) { foreach ($p in $script:PreservePatterns) { if ($Name -like $p) { return $true } }; $false }
# Which top-level files of the live dir go into the new one. Carry = names copied from live (live wins).
function Get-CarryPlan([string]$Live, [string]$Source, [bool]$KeepSettings) {
$liveFiles = @(Get-Files $Live)
$srcNames = @(Get-Files $Source | ForEach-Object { $_.Name })
$srcDirs = @(Get-Dirs $Source | ForEach-Object { $_.Name })
$carry = New-Object System.Collections.Generic.List[string]; $dropped = New-Object System.Collections.Generic.List[string]
foreach ($f in $liveFiles) {
$inSource = $srcNames -contains $f.Name
if ($f.Name -like 'appsettings*.json') { if ($KeepSettings) { $carry.Add($f.Name) }; continue }
if (Test-Preserved $f.Name) { $carry.Add($f.Name); continue }
if ($inSource) { continue }
if ($script:BinaryExtensions -contains $f.Extension.ToLowerInvariant()) { $dropped.Add($f.Name) } else { $carry.Add($f.Name) }
}
$dirs = @(Get-Dirs $Live | Where-Object { $srcDirs -notcontains $_.Name } | ForEach-Object { $_.Name })
[pscustomobject]@{ Carry = $carry.ToArray(); Dropped = $dropped.ToArray(); Dirs = $dirs }
}
function Swap-Directory([string]$Live, [string]$Source, [string]$Name, [bool]$KeepSettings) {
$bak = "$Live.bak-$(Get-Suffix)"
$plan = Get-CarryPlan $Live $Source $KeepSettings
$hashes = @{}
foreach ($f in $plan.Carry) { $hashes[$f] = Get-FileSha (Join-Path $Live $f) }
$sddlOld = Get-DirSddl $Live
Step "rename $Name dir: $Live -> $bak" { Rename-Item -LiteralPath $Live -NewName (Split-Path -Leaf $bak) }
Step "copy new ${Name}: $Source -> $Live" { Copy-Item -LiteralPath $Source -Destination $Live -Recurse -Force }
if ($KeepSettings) {
$from = if ($script:Dry) { $Live } else { $bak } # in a dry run nothing was renamed yet
foreach ($f in 'appsettings.json', 'appsettings.Development.json') {
if (Test-Path -LiteralPath (Join-Path $from $f)) {
Step "restore live $f byte-identical from $bak" { Copy-Item -LiteralPath (Join-Path $bak $f) -Destination (Join-Path $Live $f) -Force }
} else { Info "$f not present in the old $Name dir (nothing to carry over)" }
foreach ($f in $plan.Carry) {
Step "carry live $Name\$f forward from $bak (sha256 $($hashes[$f].Substring(0, 12)))" {
Copy-Item -LiteralPath (Join-Path $bak $f) -Destination (Join-Path $Live $f) -Force
if ((Get-FileSha (Join-Path $Live $f)) -ne $hashes[$f]) { Fail "carried $Name\$f does not match the live copy" }
Copy-ExplicitAcl (Join-Path $bak $f) (Join-Path $Live $f)
}
}
foreach ($f in $plan.Dropped) { Info "$Name\$f exists only in the old build (binary) and is not carried; it stays in $bak" }
if (-not $script:Dry) {
$sddlNew = Get-DirSddl $Live
if ($sddlOld -and $sddlNew -and $sddlOld -ne $sddlNew) { Warn "$Name dir ACL differs from the old one (new inherits from the parent). old: $sddlOld new: $sddlNew" }
}
$bak
}
# --- verification ----------------------------------------------------------------------------------------------------
function Invoke-Verify($ctx) {
function Invoke-Verify($ctx, [bool]$AfterRollback = $false) {
Say '== Verify =='
if ($script:Dry) {
Say "[DRYRUN] GET $HealthBaseUrl/health/ready until 200 (<= $ReadyTimeoutSeconds s)"
Say "[DRYRUN] GET $HealthBaseUrl/healthz ; read ProductVersion SHA of $(Join-Path $ctx.ServerDir $ctx.ServerExeName)$(if ($ExpectedSha) { " and require it to match $ExpectedSha" })"
Say "[DRYRUN] service must stay SERVICE_RUNNING for $StableSeconds s with env entry count $($ctx.EnvCount)"
Say "[DRYRUN] GET $HealthBaseUrl/healthz ; read ProductVersion SHA of $($ctx.Application)$(if ($ExpectedSha -and -not $Rollback) { " and require it to match $ExpectedSha" })"
Say "[DRYRUN] exactly one gateway Server ($($ctx.Application)) on the box, same PID, service SERVICE_RUNNING for $StableSeconds s, env fingerprint unchanged"
return
}
$ready = $false; $deadline = (Get-Date).AddSeconds($ReadyTimeoutSeconds)
while (-not $ready -and (Get-Date) -lt $deadline) {
try { $r = Invoke-WebRequest -Uri "$HealthBaseUrl/health/ready" -UseBasicParsing -TimeoutSec 10; $ready = ($r.StatusCode -eq 200) }
catch { Start-Sleep -Seconds 3 }
$ready = ((Get-HttpStatus "$HealthBaseUrl/health/ready") -eq 200)
if (-not $ready) { Start-Sleep -Seconds 3 }
}
if (-not $ready) { Fail "$HealthBaseUrl/health/ready did not return 200 within $ReadyTimeoutSeconds s" }
Info '/health/ready = 200'
try { Info "/healthz = $((Invoke-WebRequest -Uri "$HealthBaseUrl/healthz" -UseBasicParsing -TimeoutSec 10).StatusCode)" } catch { Fail "/healthz failed: $_" }
$hz = Get-HttpStatus "$HealthBaseUrl/healthz"
if ($hz -ne 200) { Fail "/healthz = $hz" }
Info '/healthz = 200'
$sha = Get-ProductSha (Join-Path $ctx.ServerDir $ctx.ServerExeName)
$sha = Get-ProductSha $ctx.Application
Info "deployed Server SHA (ProductVersion): $sha"
if ($ExpectedSha -and -not $Rollback -and -not (Test-ShaMatch $sha $ExpectedSha)) { Fail "deployed SHA '$sha' != expected '$ExpectedSha'" }
if ($ExpectedSha -and -not $Rollback -and -not $AfterRollback -and -not (Test-ShaMatch $sha $ExpectedSha)) { Fail "deployed SHA '$sha' != expected '$ExpectedSha'" }
Info "watching the service for $StableSeconds s"
Assert-SingleGateway $ctx 'SERVICE_RUNNING'
$pid0 = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' })[0].Id
Info "watching the service for $StableSeconds s (gateway #$pid0)"
$until = (Get-Date).AddSeconds($StableSeconds)
while ((Get-Date) -lt $until) {
do {
if ((Get-ServiceState) -ne 'SERVICE_RUNNING') { Fail 'service left RUNNING during the stability window (restart loop?)' }
Start-Sleep -Seconds 5
}
$n = (Parse-EnvBlock (Get-NssmValue 'AppEnvironmentExtra')).Count
if ($n -ne $ctx.EnvCount) { Fail "env block entry count $n != preflight $($ctx.EnvCount)" }
Info "env block still $n entries"
} while ((Get-Date) -lt $until)
$servers = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' })
if ($servers.Count -ne 1 -or $servers[0].Id -ne $pid0) { Fail "gateway process changed during the stability window: #$pid0 -> $(Format-Procs $servers) (restart loop or second gateway)" }
Assert-EnvUnchanged $ctx 'after the stability window'
Warn 'health is green with zero sessions; still prove the Worker path: open a session with a PRE-EXISTING API key and read a known tag (runbook 7.3)'
}
# --- rollback ------------------------------------------------------------------------------------------------------------
function Find-Backup([string]$Live, [string]$Stamp) {
$leaf = Split-Path -Leaf $Live; $parent = Split-Path -Parent $Live
$cands = @(Get-ChildItem -LiteralPath $parent -Directory -Filter "$leaf.bak-*" -ErrorAction SilentlyContinue)
if ($Stamp) { $cands = @($cands | Where-Object { $_.Name -like "$leaf.bak-$Stamp*" }) }
$cands | Sort-Object Name -Descending | Select-Object -First 1
# Backup dirs come in two spellings: <leaf>.bak-yyyyMMdd-HHmmss[-label] (this script) and <leaf>.bak.yyyyMMddTHHmmss (manual / adapted scripts).
function Get-BackupKey([string]$Name) {
if ($Name -match '\.bak[-.](\d{8})[-T]?(\d{6})') { return "$($Matches[1])$($Matches[2])" }
$null
}
function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores) {
function Find-Backup([string]$Live, [string]$Stamp) {
$leaf = Split-Path -Leaf $Live; $parent = Split-Path -Parent $Live
$cands = @(Get-Dirs $parent | Where-Object { ($_.Name -like "$leaf.bak-*" -or $_.Name -like "$leaf.bak.*") -and (Get-BackupKey $_.Name) })
if ($Stamp) {
$key = ($Stamp -replace '[^0-9]', '')
$cands = @($cands | Where-Object { (Get-BackupKey $_.Name) -eq $key -or $_.Name -like "$leaf.bak-$Stamp*" -or $_.Name -like "$leaf.bak.$Stamp*" })
}
$cands | Sort-Object { Get-BackupKey $_.Name } -Descending | Select-Object -First 1
}
function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores, [bool]$AllowPartial) {
Say '== Rollback =='
$sb = Find-Backup $ctx.ServerDir $Stamp
$wb = Find-Backup $ctx.WorkerDir $Stamp
if (-not $sb -or -not $wb) { Fail "no matching Server/Worker backup directories found (stamp '$Stamp')" }
Info "rolling back to Server=$($sb.Name) Worker=$($wb.Name)"
Stop-Gateway
if (-not $sb -and -not $AllowPartial) { Fail "no Server backup directory found (stamp '$Stamp')" }
$key = if ($sb) { Get-BackupKey $sb.Name } else { ($Stamp -replace '[^0-9]', '') }
$wb = Find-Backup $ctx.WorkerDir $key
if ($wb -and (Get-BackupKey $wb.Name) -ne $key) { $wb = $null }
if (-not $wb -and -not $AllowPartial) { Fail "no Worker backup matching the Server backup $($sb.Name) (stamp key $key); refusing a mixed rollback" }
if (-not $sb -and -not $wb) { Fail "nothing to roll back to (stamp '$Stamp')" }
Info "rolling back to Server=$(if ($sb) { $sb.Name } else { '(unchanged)' }) Worker=$(if ($wb) { $wb.Name } else { '(unchanged)' })"
$fpFile = @(Get-Dirs (Split-Path -Parent $ctx.ServerDir) '_deploy-backup-*' | Where-Object { ($_.Name -replace '[^0-9]', '').StartsWith($key) } | ForEach-Object { Join-Path $_.FullName 'env-fingerprint.txt' } | Where-Object { Test-Path -LiteralPath $_ })
if ($fpFile.Count) {
$saved = Get-Content -LiteralPath $fpFile[0] | Where-Object { $_ -notlike 'BlockHash=*' }
$now = @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" })
if (($saved -join '|') -ne ($now -join '|')) { Warn "the env block differs from the one saved before the upgrade ($($fpFile[0])); restore it from AppEnvironmentExtra.txt there if that was not intended (runbook section 8 step 4)" }
else { Info "env block matches the pre-upgrade fingerprint in $($fpFile[0])" }
}
Stop-Gateway $ctx
foreach ($pair in @(@($ctx.ServerDir, $sb), @($ctx.WorkerDir, $wb))) {
if (-not $pair[1]) { Info "$($pair[0]) has no backup for this stamp; left unchanged"; continue }
$failed = "$($pair[0]).failed-$($script:Stamp)"
if (Test-Path -LiteralPath $pair[0]) { Step "rename $($pair[0]) -> $failed" { Rename-Item -LiteralPath $pair[0] -NewName (Split-Path -Leaf $failed) } }
Step "rename $($pair[1].FullName) -> $($pair[0])" { Rename-Item -LiteralPath $pair[1].FullName -NewName (Split-Path -Leaf $pair[0]) }
}
if ($Stores) {
$bstamp = ($sb.Name -replace '^.*\.bak-', '')
$bstamp = ($sb.Name -replace '^.*\.bak[-.]', '')
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
foreach ($ext in '', '-wal', '-shm') {
$live = "$db$ext"; $bak = "$db$ext.bak-$bstamp"
@@ -434,57 +793,75 @@ function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores) {
# --- main -------------------------------------------------------------------------------------------------------------------
$exit = 0
try {
Say "Upgrade-MxGatewayService stamp=$($script:Stamp) mode=$(if ($Rollback) {'ROLLBACK'} else {'UPGRADE'}) dry=$($script:Dry)"
$ctx = Get-Context
Invoke-Preflight $ctx
if (-not $script:Dry) {
$lockPath = Join-Path (Split-Path -Parent $ctx.ServerDir) '.mxgw-deploy.lock'
try { $script:LockStream = [IO.File]::Open($lockPath, 'OpenOrCreate', 'ReadWrite', 'None') }
catch { Fail "another deploy run holds $lockPath" }
}
if ($Rollback) {
Invoke-RollbackSwap $ctx $BackupStamp $RestoreStores.IsPresent
Invoke-Verify $ctx
} else {
Stop-Gateway
Backup-State $ctx
$swapped = $false
try {
$null = Swap-Directory $ctx.ServerDir $ServerSource 'Server' (-not $TakeNewAppSettings)
$null = Swap-Directory $ctx.WorkerDir $WorkerSource 'Worker' $false
$swapped = $true
Start-Gateway $ctx
Invoke-Verify $ctx
} catch {
$err = $_
Say "[FAIL] $($err.Exception.Message)"
if ($script:Dry -or $NoAutoRollback) { throw }
Warn 'upgrade failed - rolling back automatically'
try {
if (-not $swapped) { Warn 'swap did not complete; rollback swaps whatever .bak sets exist for this stamp' }
Invoke-RollbackSwap $ctx $script:Stamp $false
Invoke-Verify $ctx
Warn 'ROLLED BACK to the previous build; original failure follows'
} catch { Warn "AUTOMATIC ROLLBACK FAILED: $($_.Exception.Message). Manual intervention required (backups: *.bak-$(Get-Suffix))." }
throw $err
}
}
Say '== DONE =='
} catch {
$exit = 1
Write-Host "ERROR: $($_.Exception.Message)" -ForegroundColor Red
} finally {
if ($script:LockStream) { $script:LockStream.Dispose() }
function Invoke-Main {
$exit = 0
$ctx = $null
try {
if (-not $script:Dry -and $ctx) {
$notes = Join-Path (Split-Path -Parent $ctx.ServerDir) "_DEPLOY-NOTES-$($script:Stamp).txt"
Set-Content -LiteralPath $notes -Value $script:Notes -Encoding UTF8
Write-Host "deploy notes: $notes"
Say "Upgrade-MxGatewayService stamp=$($script:Stamp) mode=$(if ($Rollback) {'ROLLBACK'} else {'UPGRADE'}) dry=$($script:Dry) host=$env:COMPUTERNAME"
$ctx = Get-Context
Invoke-Preflight $ctx
if (-not $script:Dry) {
$lockPath = Join-Path (Split-Path -Parent $ctx.ServerDir) '.mxgw-deploy.lock'
try { $script:LockStream = [IO.File]::Open($lockPath, 'OpenOrCreate', 'ReadWrite', 'None') }
catch { Fail "another deploy run holds $lockPath" }
}
} catch { }
if ($Rollback) {
Invoke-RollbackSwap $ctx $BackupStamp $RestoreStores.IsPresent $false
Invoke-Verify $ctx
} else {
Stop-Gateway $ctx
try { Backup-State $ctx }
catch {
$err = $_
if (-not $script:Dry) {
Warn 'backup failed before anything was swapped - starting the unchanged build again'
try { Start-Gateway $ctx } catch { Warn "restart of the unchanged build FAILED: $($_.Exception.Message)" }
}
throw $err
}
$touched = $false
try {
$touched = $true
$null = Swap-Directory $ctx.ServerDir $ServerSource 'Server' (-not $TakeNewAppSettings)
$null = Swap-Directory $ctx.WorkerDir $WorkerSource 'Worker' $false
if (-not $script:Dry) { Assert-EnvUnchanged $ctx 'after the swap' }
Start-Gateway $ctx
Invoke-Verify $ctx
} catch {
$err = $_
Say "[FAIL] $(Hide $err.Exception.Message)"
if ($script:Dry -or $NoAutoRollback -or -not $touched) { throw }
Warn 'upgrade failed - rolling back automatically'
$script:EnvWarnOnly = $true
try {
Invoke-RollbackSwap $ctx $script:Stamp $false $true
Invoke-Verify $ctx $true
Warn 'ROLLED BACK to the previous build; original failure follows'
} catch { Warn "AUTOMATIC ROLLBACK FAILED: $($_.Exception.Message). Manual intervention required (backups: *.bak-$(Get-Suffix))." }
throw $err
}
}
Say '== DONE =='
} catch {
$exit = 1
$m = Hide $_.Exception.Message
$script:Notes.Add("ERROR: $m")
Write-Host "ERROR: $m" -ForegroundColor Red
} finally {
if ($script:LockStream) { $script:LockStream.Dispose(); $script:LockStream = $null }
try {
if (-not $script:Dry -and $ctx) {
$notes = Join-Path (Split-Path -Parent $ctx.ServerDir) "_DEPLOY-NOTES-$($script:Stamp).txt"
Set-Content -LiteralPath $notes -Value $script:Notes -Encoding UTF8
Write-Host "deploy notes: $notes"
}
} catch { }
$script:Redact.Clear()
}
$exit
}
exit $exit
# Dot-sourcing (the dry-run harness) only defines the functions.
if ($MyInvocation.InvocationName -ne '.') { $rc = Invoke-Main | Select-Object -Last 1; exit $rc }