fix(deploy): Upgrade-MxGatewayService.ps1 safe on the wonder layout, guards fail closed (#230)
The process guards matched only ZB.MOM.WW.MxGateway.*; the wonder boxes run MxGateway.Server.exe / MxGateway.Worker.exe, so the one-gateway guard, the pre-start OrphanWorkerTerminator guard and the STOP_PENDING kill passed vacuously. - Image names come from the installed exes (NSSM Application, Worker env path) plus both known schemes; processes are also matched by exact path (Win32_Process). - One-gateway guard fails closed: a RUNNING service whose gateway is not visible, at the Application path, as the service host's child, is an abort. - STOP_PENDING kill is scoped to this install's processes. - Staged checks use the installed exe names; staged Worker must be x86; a staged self-contained runtime older than the live one is refused (-AllowRuntimeDowngrade). - Carry-over: appsettings*.json and certificates/keys (dashboard.pfx) always keep the live copy; other live-only non-binary top-level files are carried; hash-verified. - Env block read from the registry and fingerprinted (names + SHA-256 prefixes, never values) after stop, swap, start and the stability window; any change aborts. Sensitive values are redacted from all output and notes. - File-based master key (vd03) accepted and backed up; a missing env master key (windev) is a warning. Backup dir is ACL-restricted before anything is written. - Rollback accepts .bak-<stamp> and .bak.<yyyyMMddTHHmmss> sets and refuses a mixed Server/Worker stamp; auto-rollback no longer fails on the old SHA. - -WhatIf now works on Windows PowerShell 5.1 (Get-FileHash returned nothing under WhatIfPreference); an empty Win32_Service result no longer reads as "unavailable". - Install-MxGatewayService.ps1 process guard matches both schemes. - Dry-run harness scripts/deploy/tests (fake box in both layouts, 140 assertions, passes on pwsh/macOS and Windows PowerShell 5.1) plus a mutation runner (12/12 killed). - Runbook updated. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
@@ -2,32 +2,38 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
On-box upgrade / rollback of the NSSM-wrapped MxAccess Gateway (Server + x86 Worker).
|
||||
Implements docs/runbooks/DeployUpgradeNssmService.md sections 3, 5, 6, 7 and 8 (mxaccessgw#220).
|
||||
Implements docs/runbooks/DeployUpgradeNssmService.md sections 3, 5, 6, 7 and 8 (mxaccessgw#220, #230).
|
||||
|
||||
.DESCRIPTION
|
||||
RUN THIS ON THE WINDOWS BOX ITSELF (Windows PowerShell 5.1 is enough). The new Server publish and the
|
||||
new x86 Worker build must already be staged on the box (UNC/SMB copy; build both from the same SHA).
|
||||
|
||||
Upgrade flow: preflight -> stop (STOP_PENDING handled) -> backup -> swap Server + Worker -> start ->
|
||||
verify (/health/ready, VersionInfo SHA, service stable) -> automatic rollback if start/verify fails.
|
||||
Rollback flow (-Rollback): stop -> swap the .bak directories back -> optional store restore -> start -> verify.
|
||||
verify (/health/ready, VersionInfo SHA, service stable, env block unchanged) -> automatic rollback if
|
||||
start/verify fails. Rollback flow (-Rollback): stop -> swap the .bak directories back -> optional store
|
||||
restore -> start -> verify.
|
||||
|
||||
Safety properties:
|
||||
* Exactly one gateway per box. Refuses to run if a second gateway process / service points at the same
|
||||
exe, if the service is stopped but a gateway process exists, or if a Server/Worker process is still
|
||||
alive when a start is about to happen (OrphanWorkerTerminator trap, runbook section 2.1).
|
||||
* Never calls `nssm set ... AppEnvironmentExtra` (that replaces the whole block). The env block is only
|
||||
READ (saved to the backup dir, entry count compared before/after). The Application/AppDirectory
|
||||
are not changed either: the swap is rename-in-place, so the NSSM config is untouched.
|
||||
* API-key DB, Secrets store and master key live outside the content root and are never moved or
|
||||
deleted. They are only COPIED (db + -wal + -shm) into *.bak-<stamp> siblings while the service is stopped.
|
||||
The script refuses to run if a store path is inside the content root (the 2026-08-09 incident).
|
||||
* Exactly one gateway per box. The gateway and Worker image names are taken from the INSTALLED exes
|
||||
(NSSM Application and env MxGateway__Worker__ExecutablePath) plus both known naming schemes
|
||||
(ZB.MOM.WW.MxGateway.* on windev, MxGateway.* on the wonder boxes), and processes are also matched by
|
||||
exact path. The guards FAIL CLOSED: a RUNNING service whose gateway process cannot be found is an
|
||||
abort, not a pass (mxaccessgw#230: the guards used to pass vacuously on the wonder boxes).
|
||||
* Never writes the NSSM env block (AppEnvironmentExtra), Application or AppDirectory: the swap is
|
||||
rename-in-place. The env block is fingerprinted (entry names + SHA-256 prefix of each value; values are
|
||||
never printed) at preflight, after the stop, after the swap, after the start and after the stability
|
||||
window. Any added, removed or changed entry aborts (and triggers the automatic rollback).
|
||||
* Live-only top-level files in Server\ and Worker\ are carried into the new directories (appsettings*.json,
|
||||
dashboard.pfx and other certificates/keys always keep the LIVE copy), hash-verified.
|
||||
* API-key DB, Secrets store and a file-based master key must live outside the swapped directories; they are
|
||||
only COPIED (db + -wal + -shm) into *.bak-<stamp> siblings while the service is stopped.
|
||||
* A self-contained Server is refused if its .NET runtime is older than the live one (no silent downgrade).
|
||||
|
||||
-DryRun (or -WhatIf) prints every mutating action and performs none. Read-only probing (nssm get/status,
|
||||
process list, disk space, file existence) still runs so the decision logic is exercised.
|
||||
-DryRun (or -WhatIf) prints every mutating action and performs none. Read-only probing (service state,
|
||||
registry env block, process list, disk space, file hashes) still runs, so the guards are exercised for real.
|
||||
|
||||
.PARAMETER ServerSource Staged Server publish output (contains ZB.MOM.WW.MxGateway.Server.exe).
|
||||
.PARAMETER WorkerSource Staged x86 Worker output (contains ZB.MOM.WW.MxGateway.Worker.exe).
|
||||
.PARAMETER ServerSource Staged Server publish output (must contain the same exe name as NSSM Application).
|
||||
.PARAMETER WorkerSource Staged x86 Worker output (must contain the exe name of MxGateway__Worker__ExecutablePath).
|
||||
.PARAMETER ServiceName NSSM service name. Default MxAccessGw.
|
||||
.PARAMETER NssmPath nssm executable. Default 'nssm' (on PATH).
|
||||
.PARAMETER ServerDir Live Server directory. Default: NSSM AppDirectory.
|
||||
@@ -36,11 +42,12 @@
|
||||
must carry it in ProductVersion ('0.1.2+<sha>'). Builds from 2026-07-09..08-10 have a bogus stamp.
|
||||
.PARAMETER Label Free-text suffix for backup names (e.g. 'planwrites').
|
||||
.PARAMETER HealthBaseUrl Default http://localhost:5130 (dashboard port; /health/ready, /healthz anonymous).
|
||||
.PARAMETER Rollback Roll back instead of upgrading. Uses newest Server.bak-* unless -BackupStamp is given.
|
||||
.PARAMETER BackupStamp Stamp (yyyyMMdd-HHmmss) of the backup set to roll back to.
|
||||
.PARAMETER Rollback Roll back instead of upgrading. Uses the newest Server.bak-* / Server.bak.* unless -BackupStamp.
|
||||
.PARAMETER BackupStamp Stamp of the backup set to roll back to (yyyyMMdd-HHmmss, or yyyyMMddTHHmmss for .bak.<stamp> sets).
|
||||
.PARAMETER RestoreStores With -Rollback: also restore gateway-auth.db / secrets db from *.bak-<stamp> (schema-bump case).
|
||||
.PARAMETER NoAutoRollback Do not roll back automatically when start/verification fails.
|
||||
.PARAMETER TakeNewAppSettings Do not carry the live appsettings*.json over into the new Server dir.
|
||||
.PARAMETER AllowRuntimeDowngrade Accept a staged self-contained runtime older than the live one.
|
||||
.PARAMETER SkipFreeSpaceCheck / MinFreeGB / StopTimeoutSeconds / StableSeconds / ReadyTimeoutSeconds tunables.
|
||||
.PARAMETER DryRun Print, do not act. -WhatIf is equivalent.
|
||||
#>
|
||||
@@ -60,6 +67,7 @@ param(
|
||||
[switch]$RestoreStores,
|
||||
[switch]$NoAutoRollback,
|
||||
[switch]$TakeNewAppSettings,
|
||||
[switch]$AllowRuntimeDowngrade,
|
||||
[switch]$SkipFreeSpaceCheck,
|
||||
[double]$MinFreeGB = 2,
|
||||
[int]$StopTimeoutSeconds = 30,
|
||||
@@ -70,16 +78,38 @@ param(
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$script:Dry = $DryRun.IsPresent -or $WhatIfPreference
|
||||
# -WhatIf is handled by $script:Dry (Step). Leaving the preference on makes Windows PowerShell 5.1 cmdlets used for READ-ONLY
|
||||
# probing misbehave (Get-FileHash returns nothing under -WhatIf), so the plan would abort; found on windev, #230.
|
||||
$WhatIfPreference = $false
|
||||
$script:Stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
|
||||
$script:ServerProc = 'ZB.MOM.WW.MxGateway.Server'
|
||||
$script:WorkerProc = 'ZB.MOM.WW.MxGateway.Worker'
|
||||
# Both naming schemes seen in the field: windev runs ZB.MOM.WW.MxGateway.*.exe, the wonder boxes MxGateway.*.exe.
|
||||
# The installed exe names (NSSM Application, MxGateway__Worker__ExecutablePath) are added to these in Get-Context.
|
||||
$script:KnownServerImages = @('ZB.MOM.WW.MxGateway.Server', 'MxGateway.Server')
|
||||
$script:KnownWorkerImages = @('ZB.MOM.WW.MxGateway.Worker', 'MxGateway.Worker')
|
||||
$script:ServerImages = $script:KnownServerImages
|
||||
$script:WorkerImages = $script:KnownWorkerImages
|
||||
$script:EnvWorkerExe = 'MxGateway__Worker__ExecutablePath'
|
||||
# Live copy always wins for these (host config, certificates, keys), even when the staged build ships one.
|
||||
$script:PreservePatterns = @('appsettings*.json', '*.pfx', '*.p12', '*.pem', '*.key', '*.crt', '*.cer')
|
||||
# Never carried from the old build (they are build output; a live-only binary is a leftover of an older build).
|
||||
$script:BinaryExtensions = @('.dll', '.exe', '.pdb', '.so', '.dylib', '.winmd')
|
||||
$script:Notes = New-Object System.Collections.Generic.List[string]
|
||||
$script:Redact = New-Object System.Collections.Generic.List[string]
|
||||
$script:LockStream = $null
|
||||
$script:BackupDir = $null
|
||||
$script:EnvWarnOnly = $false
|
||||
|
||||
function Say([string]$m) { Write-Host $m; $script:Notes.Add($m) }
|
||||
# --- output (secret values are redacted from every line) ----------------------------------------------------
|
||||
|
||||
function Hide([string]$m) {
|
||||
if ($null -eq $m) { return '' }
|
||||
foreach ($s in $script:Redact) { if ($s) { $m = $m.Replace($s, '<redacted>') } }
|
||||
$m
|
||||
}
|
||||
function Say([string]$m) { $m = Hide $m; Write-Host $m; $script:Notes.Add($m) }
|
||||
function Info([string]$m) { Say "[info] $m" }
|
||||
function Warn([string]$m) { Write-Warning $m; $script:Notes.Add("[WARN] $m") }
|
||||
function Fail([string]$m) { throw "ABORT: $m" }
|
||||
function Warn([string]$m) { $m = Hide $m; Write-Warning $m; $script:Notes.Add("[WARN] $m") }
|
||||
function Fail([string]$m) { throw "ABORT: $(Hide $m)" }
|
||||
|
||||
# Mutating step: print in dry-run, run otherwise.
|
||||
function Step([string]$Description, [scriptblock]$Action) {
|
||||
@@ -88,7 +118,9 @@ function Step([string]$Description, [scriptblock]$Action) {
|
||||
& $Action
|
||||
}
|
||||
|
||||
# --- helpers --------------------------------------------------------------------------------------------
|
||||
# --- environment seams (the dry-run harness in scripts/deploy/tests replaces these) ---------------------------
|
||||
|
||||
function Test-IsWindowsHost { $env:OS -eq 'Windows_NT' }
|
||||
|
||||
function Invoke-Nssm {
|
||||
param([string[]]$NssmArgs, [switch]$AllowFail)
|
||||
@@ -107,20 +139,50 @@ function Get-ServiceState {
|
||||
($r.Text -split "`n")[0].Trim()
|
||||
}
|
||||
|
||||
function Parse-EnvBlock([string]$Text) {
|
||||
$map = [ordered]@{}
|
||||
foreach ($line in ($Text -split "[`r`n]+")) {
|
||||
$t = $line.Trim()
|
||||
if ($t -match '^([^=\s]+)=(.*)$') { $map[$Matches[1]] = $Matches[2] }
|
||||
# The env block exactly as NSSM stores it (REG_MULTI_SZ), one 'NAME=value' per element. Falls back to nssm get.
|
||||
function Read-EnvBlockLines {
|
||||
$key = "HKLM:\SYSTEM\CurrentControlSet\Services\$ServiceName\Parameters"
|
||||
if ((Test-IsWindowsHost) -and (Test-Path -LiteralPath $key)) {
|
||||
$p = Get-ItemProperty -LiteralPath $key -Name AppEnvironmentExtra -ErrorAction SilentlyContinue
|
||||
if ($p) { return @($p.AppEnvironmentExtra | Where-Object { $_ }) }
|
||||
return @()
|
||||
}
|
||||
$map
|
||||
@((Get-NssmValue 'AppEnvironmentExtra') -split "[`r`n]+" | Where-Object { $_.Trim() })
|
||||
}
|
||||
|
||||
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
|
||||
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
|
||||
function Test-Inside([string]$Path, [string]$Root) {
|
||||
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
|
||||
$a -eq $b -or $a.StartsWith("$b/")
|
||||
# Id, ParentId, Name (image name without .exe), Path (may be $null when unreadable).
|
||||
function Get-ProcessTable {
|
||||
if (Get-Command Get-CimInstance -ErrorAction SilentlyContinue) {
|
||||
return @(Get-CimInstance Win32_Process | ForEach-Object {
|
||||
[pscustomobject]@{ Id = [int]$_.ProcessId; ParentId = [int]$_.ParentProcessId; Name = [IO.Path]::GetFileNameWithoutExtension([string]$_.Name); Path = $_.ExecutablePath }
|
||||
})
|
||||
}
|
||||
@(Get-Process -ErrorAction SilentlyContinue | ForEach-Object {
|
||||
$p = $null; try { $p = $_.Path } catch { }
|
||||
[pscustomobject]@{ Id = $_.Id; ParentId = 0; Name = $_.ProcessName; Path = $p }
|
||||
})
|
||||
}
|
||||
|
||||
# PID of the NSSM service host (the gateway Server is its child). $null when it cannot be read.
|
||||
function Get-ServiceHostPid {
|
||||
if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null }
|
||||
$s = Get-CimInstance Win32_Service -Filter "Name='$ServiceName'" -ErrorAction SilentlyContinue
|
||||
if ($s -and $s.ProcessId) { return [int]$s.ProcessId }
|
||||
$null
|
||||
}
|
||||
|
||||
# Other services whose command line launches the gateway exe. $null when the check is unavailable.
|
||||
function Get-OtherGatewayServices([string]$ExeName) {
|
||||
if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null }
|
||||
# The leading comma keeps an EMPTY result an array: a bare @() reaches the caller as $null ('check unavailable').
|
||||
, @(Get-CimInstance Win32_Service | Where-Object { $_.Name -ne $ServiceName -and $_.PathName -and $_.PathName -like "*$ExeName*" } | ForEach-Object { $_.Name })
|
||||
}
|
||||
|
||||
function Stop-ProcessById([int]$Id) { Stop-Process -Id $Id -Force -ErrorAction SilentlyContinue }
|
||||
|
||||
function Get-HttpStatus([string]$Url) {
|
||||
try { return [int](Invoke-WebRequest -Uri $Url -UseBasicParsing -TimeoutSec 10).StatusCode }
|
||||
catch { if ($_.Exception.Response) { return [int]$_.Exception.Response.StatusCode }; return 0 }
|
||||
}
|
||||
|
||||
function Get-ProductSha([string]$Exe) {
|
||||
@@ -130,25 +192,200 @@ function Get-ProductSha([string]$Exe) {
|
||||
} catch { }
|
||||
$null
|
||||
}
|
||||
|
||||
# 'X86', 'Amd64', 'MSIL', ... or $null when it cannot be read.
|
||||
function Get-ExeArchitecture([string]$Exe) {
|
||||
try { return "$([Reflection.AssemblyName]::GetAssemblyName($Exe).ProcessorArchitecture)" } catch { return $null }
|
||||
}
|
||||
|
||||
# Restrict a directory to SYSTEM + Administrators, inheritance off. Throws if that fails.
|
||||
function Protect-Directory([string]$Path) {
|
||||
if (-not (Test-IsWindowsHost)) { Warn "not Windows: $Path is not ACL-restricted"; return }
|
||||
$null = & icacls.exe $Path /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' 2>&1
|
||||
if ($LASTEXITCODE -ne 0) { Fail "icacls could not restrict $Path (exit $LASTEXITCODE); refusing to write the env block there" }
|
||||
}
|
||||
|
||||
function Export-ServiceKey([string]$Destination) {
|
||||
if (-not (Test-IsWindowsHost)) { Info 'not Windows: service registry export skipped'; return }
|
||||
$null = & reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$ServiceName" $Destination /y 2>&1
|
||||
if ($LASTEXITCODE -ne 0) { Fail "reg export of the service key failed (exit $LASTEXITCODE)" }
|
||||
}
|
||||
|
||||
function Get-DirSddl([string]$Path) {
|
||||
if (-not (Test-IsWindowsHost)) { return $null }
|
||||
try { (Get-Acl -LiteralPath $Path).Sddl } catch { $null }
|
||||
}
|
||||
|
||||
# Give a carried file the old file's ACL when the old one had explicit (non-inherited) entries.
|
||||
function Copy-ExplicitAcl([string]$From, [string]$To) {
|
||||
if (-not (Test-IsWindowsHost)) { return }
|
||||
try {
|
||||
$a = Get-Acl -LiteralPath $From
|
||||
if (@($a.Access | Where-Object { -not $_.IsInherited }).Count -gt 0 -or $a.AreAccessRulesProtected) {
|
||||
Set-Acl -LiteralPath $To -AclObject $a
|
||||
Info " ACL of $(Split-Path -Leaf $From) copied (it had explicit entries)"
|
||||
}
|
||||
} catch { Warn "could not copy the ACL onto ${To}: $($_.Exception.Message)" }
|
||||
}
|
||||
|
||||
# --- pure helpers ---------------------------------------------------------------------------------------------
|
||||
|
||||
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
|
||||
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
|
||||
function Test-Inside([string]$Path, [string]$Root) {
|
||||
if (-not $Path -or -not $Root) { return $false }
|
||||
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
|
||||
$a -eq $b -or $a.StartsWith("$b/")
|
||||
}
|
||||
function Test-SamePath([string]$A, [string]$B) { $A -and $B -and ((Norm $A).ToLowerInvariant() -eq (Norm $B).ToLowerInvariant()) }
|
||||
function Get-Leaf([string]$p) { if (-not $p) { return $null }; ((Norm $p) -split '/')[-1] }
|
||||
function Get-Stem([string]$p) { $l = Get-Leaf $p; if (-not $l) { return $null }; [IO.Path]::GetFileNameWithoutExtension($l) }
|
||||
|
||||
function Test-ShaMatch([string]$Actual, [string]$Expected) {
|
||||
if (-not $Actual -or -not $Expected) { return $false }
|
||||
$e = $Expected.ToLowerInvariant()
|
||||
$Actual.StartsWith($e) -or $e.StartsWith($Actual)
|
||||
}
|
||||
|
||||
function Get-GatewayProcesses {
|
||||
@(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in @($script:ServerProc, $script:WorkerProc) })
|
||||
function Get-TextHashPrefix([string]$Text) {
|
||||
$sha = [Security.Cryptography.SHA256]::Create()
|
||||
try { (($sha.ComputeHash([Text.Encoding]::UTF8.GetBytes($Text)) | ForEach-Object { $_.ToString('x2') }) -join '').Substring(0, 12) }
|
||||
finally { $sha.Dispose() }
|
||||
}
|
||||
function Get-FileSha([string]$Path) {
|
||||
$sha = [Security.Cryptography.SHA256]::Create(); $fs = [IO.File]::OpenRead($Path)
|
||||
try { (($sha.ComputeHash($fs) | ForEach-Object { $_.ToString('X2') }) -join '') } finally { $fs.Dispose(); $sha.Dispose() }
|
||||
}
|
||||
|
||||
# Store file set: db + -wal + -shm
|
||||
function Get-StoreFiles([string]$Db) { @($Db, "$Db-wal", "$Db-shm") | Where-Object { Test-Path -LiteralPath $_ } }
|
||||
function Test-SensitiveName([string]$Name) { ($Name -match 'KEY|PEPPER|PASS|SECRET|TOKEN|PWD|CREDENTIAL') -and ($Name -notmatch '(Path|Source|EnvVarName|Name)$') }
|
||||
|
||||
# name -> value. Every sensitive value is registered for redaction.
|
||||
function ConvertTo-EnvMap([string[]]$Lines) {
|
||||
$map = [ordered]@{}
|
||||
foreach ($line in $Lines) {
|
||||
$i = $line.IndexOf('=')
|
||||
if ($i -le 0) { continue }
|
||||
$k = $line.Substring(0, $i).Trim(); $v = $line.Substring($i + 1)
|
||||
$map[$k] = $v
|
||||
if ((Test-SensitiveName $k) -and $v.Length -ge 4 -and -not $script:Redact.Contains($v)) { $script:Redact.Add($v) }
|
||||
}
|
||||
$map
|
||||
}
|
||||
|
||||
# Names + per-value SHA-256 prefixes. Never carries a value.
|
||||
function Get-EnvFingerprint([string[]]$Lines) {
|
||||
$map = ConvertTo-EnvMap $Lines
|
||||
$entries = [ordered]@{}
|
||||
foreach ($k in $map.Keys) { $entries[$k] = Get-TextHashPrefix ([string]$map[$k]) }
|
||||
[pscustomobject]@{ Count = @($Lines).Count; Names = @($map.Keys); Entries = $entries; BlockHash = (Get-TextHashPrefix (@($Lines) -join "`n")) }
|
||||
}
|
||||
|
||||
# List of differences (names only). Empty when identical.
|
||||
function Compare-EnvFingerprint($Before, $After) {
|
||||
$d = New-Object System.Collections.Generic.List[string]
|
||||
foreach ($k in $Before.Entries.Keys) {
|
||||
if (-not $After.Entries.Contains($k)) { $d.Add("removed $k") }
|
||||
elseif ($After.Entries[$k] -ne $Before.Entries[$k]) { $d.Add("changed $k (sha256 $($Before.Entries[$k]) -> $($After.Entries[$k]))") }
|
||||
}
|
||||
foreach ($k in $After.Entries.Keys) { if (-not $Before.Entries.Contains($k)) { $d.Add("added $k") } }
|
||||
if ($d.Count -eq 0 -and $Before.BlockHash -ne $After.BlockHash) { $d.Add("block hash $($Before.BlockHash) -> $($After.BlockHash) (order, duplicates or blank entries changed)") }
|
||||
, $d.ToArray()
|
||||
}
|
||||
|
||||
function Format-EnvFingerprint($Fp) { "$($Fp.Count) entries, block sha256 $($Fp.BlockHash): " + (($Fp.Entries.Keys | ForEach-Object { "$_=$($Fp.Entries[$_])" }) -join ', ') }
|
||||
|
||||
# Re-read the env block and abort on any difference from the preflight fingerprint.
|
||||
function Assert-EnvUnchanged($ctx, [string]$When) {
|
||||
$now = Get-EnvFingerprint (Read-EnvBlockLines)
|
||||
$diff = Compare-EnvFingerprint $ctx.EnvFp $now
|
||||
if ($diff.Count -gt 0) {
|
||||
# During an automatic rollback the difference is already the reported failure; restoring the old build still goes ahead.
|
||||
if ($script:EnvWarnOnly) { Warn "NSSM env block changed ${When}: $($diff -join '; ') (not written by this script; restore it from the backup dir's AppEnvironmentExtra.txt if unintended)"; return }
|
||||
Fail "NSSM env block changed ${When}: $($diff -join '; ')"
|
||||
}
|
||||
Info "env block unchanged $When ($($now.Count) entries, block sha256 $($now.BlockHash))"
|
||||
}
|
||||
|
||||
# Directory listings via .NET (FileInfo/DirectoryInfo, hidden files included). Get-ChildItem costs ~250 ms per call
|
||||
# under pwsh on macOS (per-file owner lookups), which made the dry-run harness 20x slower; on Windows both are fine.
|
||||
function Get-Files([string]$Dir, [string]$Filter = '*', [switch]$Recurse) {
|
||||
if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return }
|
||||
$o = if ($Recurse) { [IO.SearchOption]::AllDirectories } else { [IO.SearchOption]::TopDirectoryOnly }
|
||||
try { (New-Object IO.DirectoryInfo $Dir).GetFiles($Filter, $o) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" }
|
||||
}
|
||||
function Get-Dirs([string]$Dir, [string]$Filter = '*') {
|
||||
if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return }
|
||||
try { (New-Object IO.DirectoryInfo $Dir).GetDirectories($Filter) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" }
|
||||
}
|
||||
|
||||
function Get-StoreFiles([string]$Db) { @($Db, "$Db-wal", "$Db-shm") | Where-Object { $_ -and (Test-Path -LiteralPath $_) } }
|
||||
|
||||
function Get-DirSize([string]$p) {
|
||||
if (-not (Test-Path -LiteralPath $p)) { return 0 }
|
||||
$s = (Get-ChildItem -LiteralPath $p -Recurse -File -Force -ErrorAction SilentlyContinue | Measure-Object Length -Sum).Sum
|
||||
$s = (@(Get-Files $p -Recurse) | Measure-Object Length -Sum).Sum
|
||||
if ($s) { [double]$s } else { 0 }
|
||||
}
|
||||
|
||||
# Version of a self-contained .NET runtime in a directory (from mscordaccore_<arch>_<arch>_<ver>.dll), or $null (framework-dependent).
|
||||
function Get-BundledRuntimeVersion([string]$Dir) {
|
||||
$f = @(Get-Files $Dir 'mscordaccore_*.dll')
|
||||
foreach ($x in $f) { if ($x.Name -match '_(\d+\.\d+\.\d+(\.\d+)?)\.dll$') { return [version]$Matches[1] } }
|
||||
$null
|
||||
}
|
||||
|
||||
# --- processes --------------------------------------------------------------------------------------------------
|
||||
|
||||
function Test-GatewayServer($p, $ctx) { ($script:ServerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.Application) }
|
||||
function Test-GatewayWorker($p, $ctx) { ($script:WorkerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.WorkerExe) }
|
||||
|
||||
function Get-GatewayProcesses($ctx) {
|
||||
@(Get-ProcessTable | Where-Object { (Test-GatewayServer $_ $ctx) -or (Test-GatewayWorker $_ $ctx) } | ForEach-Object {
|
||||
$role = if (Test-GatewayServer $_ $ctx) { 'Server' } else { 'Worker' }
|
||||
$_ | Add-Member -NotePropertyName Role -NotePropertyValue $role -PassThru -Force
|
||||
})
|
||||
}
|
||||
function Format-Procs($procs) { (@($procs) | ForEach-Object { "$($_.Name)#$($_.Id)<-$($_.ParentId) [$(if ($_.Path) { $_.Path } else { 'path unreadable' })]" }) -join ', ' }
|
||||
|
||||
function Assert-NoGatewayProcess($ctx, [string]$When) {
|
||||
$left = @(Get-GatewayProcesses $ctx)
|
||||
if ($left.Count -gt 0) { Fail "${When}: gateway process(es) still alive: $(Format-Procs $left)" }
|
||||
}
|
||||
|
||||
# The one-gateway guard. Fails closed: a RUNNING service whose gateway cannot be seen is an abort.
|
||||
function Assert-SingleGateway($ctx, [string]$State) {
|
||||
$procs = @(Get-GatewayProcesses $ctx)
|
||||
$servers = @($procs | Where-Object { $_.Role -eq 'Server' })
|
||||
$workers = @($procs | Where-Object { $_.Role -eq 'Worker' })
|
||||
Info "gateway processes on the box (images $($script:ServerImages -join '|') / $($script:WorkerImages -join '|')): $(if ($procs.Count) { Format-Procs $procs } else { 'none' })"
|
||||
if ($servers.Count -gt 1) { Fail "$($servers.Count) gateway Server processes are running ($(Format-Procs $servers)). A second gateway is the OrphanWorkerTerminator trap - resolve manually." }
|
||||
switch ($State) {
|
||||
'SERVICE_STOPPED' {
|
||||
if ($procs.Count -gt 0) { Fail "service is STOPPED but gateway process(es) exist ($(Format-Procs $procs)): a rogue second gateway or an unclean stop. Starting would terminate those workers." }
|
||||
}
|
||||
'SERVICE_RUNNING' {
|
||||
if ($servers.Count -eq 0) {
|
||||
Fail "service is RUNNING but no gateway Server process was found (looked for image names $($script:ServerImages -join ', ') and path $($ctx.Application)). The process guards cannot see this gateway, so they would pass vacuously; refusing."
|
||||
}
|
||||
$s = $servers[0]
|
||||
if ($s.Path -and -not (Test-SamePath $s.Path $ctx.Application)) { Fail "the running gateway is $($s.Path), not the service's Application $($ctx.Application): a second gateway is running and the service's own is not" }
|
||||
if (-not $s.Path) { Warn "gateway Server #$($s.Id) path is unreadable; matched by image name only" }
|
||||
$hostPid = Get-ServiceHostPid
|
||||
if ($hostPid) {
|
||||
if ($s.ParentId -ne $hostPid) { Fail "gateway Server #$($s.Id) is not a child of the $ServiceName service host #${hostPid} (parent #$($s.ParentId)): it is another gateway" }
|
||||
Info "gateway Server #$($s.Id) is the service's own (child of service host #$hostPid)"
|
||||
} else { Warn 'service host PID unavailable; parent check skipped' }
|
||||
$foreign = @($workers | Where-Object { $_.ParentId -and $_.ParentId -ne $s.Id })
|
||||
if ($foreign.Count -gt 0) { Warn "Worker process(es) not owned by the service's gateway: $(Format-Procs $foreign). They are killed as orphans on the next start." }
|
||||
$wrongPath = @($workers | Where-Object { $_.Path -and $ctx.WorkerExe -and -not (Test-SamePath $_.Path $ctx.WorkerExe) })
|
||||
if ($wrongPath.Count -gt 0) { Warn "Worker process(es) not running the configured $($ctx.WorkerExe): $(Format-Procs $wrongPath)" }
|
||||
}
|
||||
'SERVICE_STOP_PENDING' { Warn 'service is STOP_PENDING; the stop step will finish it' }
|
||||
default {
|
||||
if ($servers.Count -gt 0) { Fail "a gateway Server process exists while the service is $State" }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# --- context discovery (read only) -------------------------------------------------------------------------
|
||||
|
||||
function Get-Context {
|
||||
@@ -158,25 +395,47 @@ function Get-Context {
|
||||
$ctx.State = $state
|
||||
$ctx.Application = Get-NssmValue 'Application'
|
||||
$ctx.AppDirectory = Get-NssmValue 'AppDirectory'
|
||||
$ctx.EnvText = Get-NssmValue 'AppEnvironmentExtra'
|
||||
$ctx.Env = Parse-EnvBlock $ctx.EnvText
|
||||
$ctx.EnvCount = $ctx.Env.Count
|
||||
if (-not $ServerDir) { $ctx.ServerDir = $ctx.AppDirectory } else { $ctx.ServerDir = $ServerDir }
|
||||
$wexe = $ctx.Env['MxGateway__Worker__ExecutablePath']
|
||||
$lines = @(Read-EnvBlockLines)
|
||||
$ctx.Env = ConvertTo-EnvMap $lines
|
||||
$ctx.EnvFp = Get-EnvFingerprint $lines
|
||||
$ctx.EnvCount = $ctx.EnvFp.Count
|
||||
if ($ServerDir) { $ctx.ServerDir = $ServerDir } else { $ctx.ServerDir = $ctx.AppDirectory }
|
||||
$wexe = $ctx.Env[$script:EnvWorkerExe]
|
||||
$ctx.WorkerExe = $wexe
|
||||
if ($WorkerDir) { $ctx.WorkerDir = $WorkerDir }
|
||||
elseif ($wexe) { $ctx.WorkerDir = Split-Path -Parent $wexe }
|
||||
else { Fail 'cannot determine the Worker directory: env MxGateway__Worker__ExecutablePath is not in the NSSM env block; pass -WorkerDir' }
|
||||
else { Fail "cannot determine the Worker directory: env $($script:EnvWorkerExe) is not in the NSSM env block; pass -WorkerDir" }
|
||||
$pd = 'C:\ProgramData\MxGateway'
|
||||
$auth = $ctx.Env['MxGateway__Authentication__SqlitePath']; if (-not $auth) { $auth = "$pd\gateway-auth.db" }
|
||||
$sec = $ctx.Env['Secrets__SqlitePath']; if (-not $sec) { $sec = "$pd\mxgateway-secrets.db" }
|
||||
$ctx.AuthDb = $auth; $ctx.SecretsDb = $sec
|
||||
$ctx.ServerExeName = Split-Path -Leaf $ctx.Application
|
||||
$ctx.KekFile = $ctx.Env['Secrets__MasterKey__FilePath']
|
||||
$ctx.ServerExeName = Get-Leaf $ctx.Application
|
||||
$ctx.WorkerExeName = Get-Leaf $wexe
|
||||
if (-not $ctx.WorkerExeName) { $ctx.WorkerExeName = 'ZB.MOM.WW.MxGateway.Worker.exe' }
|
||||
# Process image names: the installed ones first, then both known schemes.
|
||||
$script:ServerImages = @(@(Get-Stem $ctx.Application) + $script:KnownServerImages | Where-Object { $_ } | Select-Object -Unique)
|
||||
$script:WorkerImages = @(@(Get-Stem $wexe) + $script:KnownWorkerImages | Where-Object { $_ } | Select-Object -Unique)
|
||||
[pscustomobject]$ctx
|
||||
}
|
||||
|
||||
# --- preflight -----------------------------------------------------------------------------------------------
|
||||
|
||||
function Test-MasterKey($ctx) {
|
||||
$src = $ctx.Env['Secrets__MasterKey__Source']
|
||||
$envVar = $ctx.Env['Secrets__MasterKey__EnvVarName']; if (-not $envVar) { $envVar = 'ZB_SECRETS_MASTER_KEY' }
|
||||
if ($ctx.KekFile -or $src -eq 'File') {
|
||||
if (-not $ctx.KekFile) { Fail 'Secrets__MasterKey__Source=File but no Secrets__MasterKey__FilePath in the env block' }
|
||||
if (-not (Test-Rooted $ctx.KekFile)) { Fail "master key file '$($ctx.KekFile)' is not absolute" }
|
||||
foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) { if (Test-Inside $ctx.KekFile $root) { Fail "master key file '$($ctx.KekFile)' is INSIDE '$root' - the swap would lose it" } }
|
||||
if (Test-Path -LiteralPath $ctx.KekFile) { Info "master key: file $($ctx.KekFile) (present, outside the swap)" } else { Fail "master key file '$($ctx.KekFile)' does not exist (the Secrets store would be unreadable)" }
|
||||
} elseif ($ctx.Env.Contains($envVar)) {
|
||||
Info "master key: env block entry $envVar"
|
||||
} else {
|
||||
Warn "the env block carries no master key ($envVar or Secrets__MasterKey__FilePath); it comes from elsewhere (appsettings or the account environment). The upgrade does not change that, but confirm the Secrets store opens after the start."
|
||||
}
|
||||
}
|
||||
|
||||
function Invoke-Preflight($ctx) {
|
||||
Say '== Preflight =='
|
||||
Info "service '$ServiceName' state: $($ctx.State)"
|
||||
@@ -186,20 +445,19 @@ function Invoke-Preflight($ctx) {
|
||||
Info "Worker dir : $($ctx.WorkerDir) (exe env: $($ctx.WorkerExe))"
|
||||
Info "API-key DB : $($ctx.AuthDb)"
|
||||
Info "Secrets DB : $($ctx.SecretsDb)"
|
||||
Info "NSSM env block: $($ctx.EnvCount) entries (names only): $(($ctx.Env.Keys) -join ', ')"
|
||||
Info "NSSM env block fingerprint (names + value sha256 prefixes, no values): $(Format-EnvFingerprint $ctx.EnvFp)"
|
||||
|
||||
if ($ctx.EnvCount -eq 0) { Fail 'NSSM env block is empty - refusing: it holds Kestrel/store paths, the KEK and the pepper' }
|
||||
foreach ($need in 'ZB_SECRETS_MASTER_KEY') {
|
||||
if (-not $ctx.Env.Contains($need)) { Fail "env block has no $need (the Secrets store would be unreadable after the restart)" }
|
||||
}
|
||||
if ($ctx.EnvCount -eq 0) { Fail 'NSSM env block is empty - refusing: it holds Kestrel/store paths, the master key and the pepper' }
|
||||
if (-not ($ctx.Env.Keys | Where-Object { $_ -match 'Pepper' })) {
|
||||
Warn 'no *Pepper* entry in the env block; confirm the API-key pepper is supplied another way or issued tokens will not verify'
|
||||
}
|
||||
if ($ctx.Env['ASPNETCORE_ENVIRONMENT'] -ne 'Production' -and $ctx.Env['DOTNET_ENVIRONMENT'] -ne 'Production') {
|
||||
Warn 'neither ASPNETCORE_ENVIRONMENT nor DOTNET_ENVIRONMENT is exactly Production (runbook trap 6) - do not change it during an upgrade'
|
||||
}
|
||||
if (-not (Test-SamePath (Split-Path -Parent $ctx.Application) $ctx.ServerDir)) { Fail "NSSM Application '$($ctx.Application)' is not inside the Server dir '$($ctx.ServerDir)'" }
|
||||
if ($ctx.WorkerExe -and -not (Test-SamePath (Split-Path -Parent $ctx.WorkerExe) $ctx.WorkerDir)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' is not inside the Worker dir '$($ctx.WorkerDir)'" }
|
||||
|
||||
# State that must survive: rooted and outside the content root.
|
||||
# State that must survive: rooted and outside the swapped directories.
|
||||
foreach ($pair in @(@('API-key DB', $ctx.AuthDb), @('Secrets DB', $ctx.SecretsDb))) {
|
||||
if (-not (Test-Rooted $pair[1])) { Fail "$($pair[0]) path '$($pair[1])' is not absolute" }
|
||||
foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) {
|
||||
@@ -208,6 +466,7 @@ function Invoke-Preflight($ctx) {
|
||||
if (Test-Path -LiteralPath $pair[1]) { Info "$($pair[0]) present ($((Get-StoreFiles $pair[1]).Count) file(s) incl. wal/shm)" }
|
||||
else { Warn "$($pair[0]) '$($pair[1])' does not exist yet (fresh store will be created on start)" }
|
||||
}
|
||||
Test-MasterKey $ctx
|
||||
|
||||
# Directories
|
||||
foreach ($d in @($ctx.ServerDir, $ctx.WorkerDir)) {
|
||||
@@ -216,33 +475,34 @@ function Invoke-Preflight($ctx) {
|
||||
if ((Test-Inside $ctx.WorkerDir $ctx.ServerDir) -or (Test-Inside $ctx.ServerDir $ctx.WorkerDir)) {
|
||||
Fail 'Server and Worker directories are nested; the rename-in-place swap needs sibling directories'
|
||||
}
|
||||
if (-not (Test-Path -LiteralPath $ctx.Application)) { Fail "NSSM Application '$($ctx.Application)' does not exist" }
|
||||
if ($ctx.WorkerExe -and -not (Test-Path -LiteralPath $ctx.WorkerExe)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' does not exist" }
|
||||
$dbs = @(@(Get-Files $ctx.ServerDir -Recurse) + @(Get-Files $ctx.WorkerDir -Recurse) | Where-Object { $_.Name -match '\.db(-wal|-shm)?$' })
|
||||
if ($dbs.Count -gt 0) { Fail "SQLite file(s) inside the swapped directories would be orphaned by the rename: $(($dbs | ForEach-Object { $_.FullName }) -join ', ')" }
|
||||
|
||||
# One-gateway guard
|
||||
$procs = Get-GatewayProcesses
|
||||
$servers = @($procs | Where-Object { $_.ProcessName -eq $script:ServerProc })
|
||||
if ($servers.Count -gt 1) { Fail "$($servers.Count) gateway Server processes are running (PIDs $(($servers.Id) -join ',')). A second gateway is the OrphanWorkerTerminator trap - resolve manually." }
|
||||
if ($ctx.State -eq 'SERVICE_STOPPED' -and $procs.Count -gt 0) {
|
||||
Fail "service is STOPPED but gateway process(es) exist (PIDs $(($procs.Id) -join ',')): a rogue second gateway. Starting would terminate its live workers."
|
||||
}
|
||||
if ($servers.Count -eq 1 -and $ctx.State -ne 'SERVICE_RUNNING' -and $ctx.State -ne 'SERVICE_STOP_PENDING') {
|
||||
Fail "a Server process exists while the service is $($ctx.State)"
|
||||
}
|
||||
if (Get-Command Get-CimInstance -ErrorAction SilentlyContinue) {
|
||||
$svcs = @(Get-CimInstance Win32_Service | Where-Object { $_.Name -ne $ServiceName -and $_.PathName -and $_.PathName -like "*$($ctx.ServerExeName)*" })
|
||||
if ($svcs.Count -gt 0) { Fail "another service ($(($svcs.Name) -join ', ')) also launches $($ctx.ServerExeName); two gateways on one box is forbidden" }
|
||||
} else { Info 'Win32_Service cross-check unavailable on this host (skipped)' }
|
||||
# One-gateway guard (fail closed)
|
||||
Assert-SingleGateway $ctx $ctx.State
|
||||
$others = Get-OtherGatewayServices $ctx.ServerExeName
|
||||
if ($null -eq $others) { Info 'Win32_Service cross-check unavailable on this host (skipped)' }
|
||||
elseif (@($others).Count -gt 0) { Fail "another service ($(@($others) -join ', ')) also launches $($ctx.ServerExeName); two gateways on one box is forbidden" }
|
||||
else { Info "no other service launches $($ctx.ServerExeName)" }
|
||||
|
||||
if ($Rollback) { return }
|
||||
if ($Rollback) { Say '== Preflight OK =='; return }
|
||||
|
||||
# Sources
|
||||
if (-not $ServerSource -or -not $WorkerSource) { Fail '-ServerSource and -WorkerSource are required for an upgrade' }
|
||||
foreach ($s in @(@($ServerSource, $ctx.ServerExeName), @($WorkerSource, "$($script:WorkerProc).exe"))) {
|
||||
if (-not (Test-Path -LiteralPath (Join-Path $s[0] $s[1]))) { Fail "staged source '$($s[0])' does not contain $($s[1])" }
|
||||
$stagedServerExe = Join-Path $ServerSource $ctx.ServerExeName
|
||||
$stagedWorkerExe = Join-Path $WorkerSource $ctx.WorkerExeName
|
||||
foreach ($s in @(@($ServerSource, $stagedServerExe), @($WorkerSource, $stagedWorkerExe))) {
|
||||
if (-not (Test-Path -LiteralPath $s[1])) { Fail "staged source '$($s[0])' does not contain $(Get-Leaf $s[1]) (the installed exe name; NSSM and $($script:EnvWorkerExe) point at it)" }
|
||||
}
|
||||
if ((Test-Inside $ServerSource $ctx.ServerDir) -or (Test-Inside $WorkerSource $ctx.WorkerDir)) { Fail 'staging directory is inside a live directory' }
|
||||
$sSha = Get-ProductSha (Join-Path $ServerSource $ctx.ServerExeName)
|
||||
$wSha = Get-ProductSha (Join-Path $WorkerSource "$($script:WorkerProc).exe")
|
||||
Info "staged Server SHA: $sSha ; staged Worker SHA: $wSha"
|
||||
if ((Test-Inside $ServerSource $ctx.ServerDir) -or (Test-Inside $WorkerSource $ctx.WorkerDir) -or (Test-Inside $ServerSource $ctx.WorkerDir) -or (Test-Inside $WorkerSource $ctx.ServerDir)) { Fail 'staging directory is inside a live directory' }
|
||||
$arch = Get-ExeArchitecture $stagedWorkerExe
|
||||
if ($arch -ne 'X86') { Fail "staged Worker $stagedWorkerExe architecture is '$arch', not X86 (trap 5)" }
|
||||
Info "staged Worker architecture: X86"
|
||||
$sSha = Get-ProductSha $stagedServerExe
|
||||
$wSha = Get-ProductSha $stagedWorkerExe
|
||||
Info "staged Server SHA: $sSha ; staged Worker SHA: $wSha ; live Server SHA: $(Get-ProductSha $ctx.Application)"
|
||||
if ($sSha -and $wSha -and -not (Test-ShaMatch $sSha $wSha)) { Fail "Server ($sSha) and Worker ($wSha) were built from different commits (wire protocol risk)" }
|
||||
if ($ExpectedSha) {
|
||||
if (-not (Test-ShaMatch $sSha $ExpectedSha)) {
|
||||
@@ -250,6 +510,27 @@ function Invoke-Preflight($ctx) {
|
||||
else { Fail "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha'" }
|
||||
}
|
||||
} else { Warn 'no -ExpectedSha given: the post-deploy SHA check will only record the SHA' }
|
||||
$leafSha = if ((Get-Leaf $ctx.ServerDir) -match '([0-9a-f]{7,40})$') { $Matches[1] } else { $null }
|
||||
if ($leafSha -and $sSha -and -not (Test-ShaMatch $leafSha $sSha)) { Warn "the Server dir name '$(Get-Leaf $ctx.ServerDir)' carries SHA $leafSha but will hold $sSha after the rename-in-place swap" }
|
||||
|
||||
# Runtime: never silently downgrade a self-contained Server.
|
||||
$liveRt = Get-BundledRuntimeVersion $ctx.ServerDir; $newRt = Get-BundledRuntimeVersion $ServerSource
|
||||
Info "bundled .NET runtime: live $(if ($liveRt) { $liveRt } else { 'none (framework-dependent)' }), staged $(if ($newRt) { $newRt } else { 'none (framework-dependent)' })"
|
||||
if ($liveRt -and -not $newRt) { Warn 'live Server is self-contained, staged is framework-dependent: the box must have a matching .NET 10 runtime installed' }
|
||||
if ($liveRt -and $newRt -and $newRt -lt $liveRt) {
|
||||
if ($AllowRuntimeDowngrade) { Warn "staged runtime $newRt is OLDER than live $liveRt (-AllowRuntimeDowngrade)" }
|
||||
else { Fail "staged self-contained runtime $newRt is OLDER than the live $liveRt; rebuild with a newer SDK or pass -AllowRuntimeDowngrade" }
|
||||
}
|
||||
|
||||
# What the swap will carry over / drop (top-level files).
|
||||
foreach ($pair in @(@('Server', $ctx.ServerDir, $ServerSource), @('Worker', $ctx.WorkerDir, $WorkerSource))) {
|
||||
$plan = Get-CarryPlan $pair[1] $pair[2] ($pair[0] -eq 'Server' -and -not $TakeNewAppSettings)
|
||||
Info "$($pair[0]): carry from live -> $(if ($plan.Carry.Count) { $plan.Carry -join ', ' } else { 'nothing' }); live-only binaries dropped -> $(if ($plan.Dropped.Count) { $plan.Dropped -join ', ' } else { 'none' })"
|
||||
if ($plan.Dirs.Count) { Warn "$($pair[0]): live-only directories NOT carried (they stay in the .bak): $($plan.Dirs -join ', ')" }
|
||||
}
|
||||
$lw = @(Get-Files $ctx.WorkerDir | ForEach-Object { $_.Name }); $nw = @(Get-Files $WorkerSource | ForEach-Object { $_.Name })
|
||||
$onlyNew = @($nw | Where-Object { $lw -notcontains $_ })
|
||||
if ($onlyNew.Count) { Info "Worker: files only in the staged build: $($onlyNew -join ', ')" }
|
||||
|
||||
# Disk space: 2x (server + worker + stores + payload), at least MinFreeGB
|
||||
if (-not $SkipFreeSpaceCheck) {
|
||||
@@ -272,52 +553,67 @@ function Wait-State([string]$Want, [int]$TimeoutSec) {
|
||||
$false
|
||||
}
|
||||
|
||||
function Assert-NoGatewayProcess([string]$When) {
|
||||
$left = Get-GatewayProcesses
|
||||
if ($left.Count -gt 0) { Fail "${When}: gateway process(es) still alive: $(($left | ForEach-Object { "$($_.ProcessName)#$($_.Id)" }) -join ', ')" }
|
||||
function Wait-NoGatewayProcess($ctx, [int]$TimeoutSec) {
|
||||
$deadline = (Get-Date).AddSeconds($TimeoutSec)
|
||||
do { if (@(Get-GatewayProcesses $ctx).Count -eq 0) { return $true }; Start-Sleep -Seconds 1 } while ((Get-Date) -lt $deadline)
|
||||
$false
|
||||
}
|
||||
|
||||
function Stop-Gateway {
|
||||
# Kill only this install's gateway processes: path under the Server/Worker dir, or (path unreadable) the installed image name.
|
||||
function Stop-InstallProcesses($ctx, [string]$Why) {
|
||||
$installed = @((Get-Stem $ctx.Application), (Get-Stem $ctx.WorkerExe)) | Where-Object { $_ }
|
||||
$mine = @(Get-GatewayProcesses $ctx | Where-Object {
|
||||
if ($_.Path) { (Test-Inside $_.Path $ctx.ServerDir) -or (Test-Inside $_.Path $ctx.WorkerDir) } else { $installed -contains $_.Name }
|
||||
} | Sort-Object { if ($_.Role -eq 'Server') { 0 } else { 1 } })
|
||||
foreach ($p in $mine) { Say "[ACTION] Stop-Process -Id $($p.Id) -Force ($($p.Name), $Why)"; Stop-ProcessById $p.Id }
|
||||
$mine.Count
|
||||
}
|
||||
|
||||
function Stop-Gateway($ctx) {
|
||||
Say '-- Stop --'
|
||||
$state = Get-ServiceState
|
||||
if ($state -eq 'SERVICE_STOPPED') { Info 'already stopped' }
|
||||
else {
|
||||
Step "nssm stop $ServiceName (wait up to $StopTimeoutSeconds s for SERVICE_STOPPED)" {
|
||||
Step "nssm stop $ServiceName (wait up to $StopTimeoutSeconds s for SERVICE_STOPPED, then for every gateway/Worker process to exit)" {
|
||||
[void](Invoke-Nssm -NssmArgs @('stop', $ServiceName) -AllowFail)
|
||||
if (-not (Wait-State 'SERVICE_STOPPED' $StopTimeoutSeconds)) {
|
||||
# STOP_PENDING trap: workers did not exit in time. Kill the Server (then Workers) by image name -
|
||||
# safe because preflight proved there is exactly one gateway on the box.
|
||||
Warn "service still $(Get-ServiceState) after $StopTimeoutSeconds s (STOP_PENDING trap) - killing gateway processes"
|
||||
foreach ($n in @($script:ServerProc, $script:WorkerProc)) {
|
||||
Get-Process -Name $n -ErrorAction SilentlyContinue | ForEach-Object { Say "[ACTION] Stop-Process -Id $($_.Id) -Force ($n)"; Stop-Process -Id $_.Id -Force }
|
||||
}
|
||||
# STOP_PENDING trap: workers did not exit in time. Kill this install's Server, then its Workers.
|
||||
Warn "service still $(Get-ServiceState) after $StopTimeoutSeconds s (STOP_PENDING trap) - killing this install's gateway processes"
|
||||
$n = Stop-InstallProcesses $ctx 'STOP_PENDING'
|
||||
if ($n -eq 0) { Warn 'no process of this install was found to kill' }
|
||||
if (-not (Wait-State 'SERVICE_STOPPED' 20)) {
|
||||
Warn 'still not STOPPED; killing the NSSM service host process'
|
||||
try {
|
||||
$svcPid = (Get-CimInstance Win32_Service -Filter "Name='$ServiceName'").ProcessId
|
||||
if ($svcPid) { Stop-Process -Id $svcPid -Force }
|
||||
} catch { Warn "could not kill the service host: $_" }
|
||||
$svcPid = Get-ServiceHostPid
|
||||
if ($svcPid) { Say "[ACTION] Stop-Process -Id $svcPid -Force (service host)"; Stop-ProcessById $svcPid }
|
||||
if (-not (Wait-State 'SERVICE_STOPPED' 20)) { Fail "service '$ServiceName' will not stop; investigate manually (state $(Get-ServiceState))" }
|
||||
}
|
||||
}
|
||||
Start-Sleep -Seconds 2
|
||||
Assert-NoGatewayProcess 'after stop'
|
||||
if (-not (Wait-NoGatewayProcess $ctx 20)) {
|
||||
Warn "gateway/Worker process(es) outlived the service stop (unclean stop): $(Format-Procs (Get-GatewayProcesses $ctx))"
|
||||
[void](Stop-InstallProcesses $ctx 'left over after the stop')
|
||||
[void](Wait-NoGatewayProcess $ctx 10)
|
||||
}
|
||||
}
|
||||
}
|
||||
if (-not $script:Dry) { Assert-NoGatewayProcess 'after stop' }
|
||||
if (-not $script:Dry) {
|
||||
Assert-NoGatewayProcess $ctx 'after stop'
|
||||
Info 'stopped; no gateway/Worker process on the box'
|
||||
Assert-EnvUnchanged $ctx 'after the stop'
|
||||
}
|
||||
}
|
||||
|
||||
function Start-Gateway($ctx) {
|
||||
Say '-- Start --'
|
||||
if (-not $script:Dry) { Assert-NoGatewayProcess 'before start (OrphanWorkerTerminator guard)' }
|
||||
else { Say '[DRYRUN] assert no Server/Worker process exists (OrphanWorkerTerminator guard)' }
|
||||
if (-not $script:Dry) {
|
||||
Assert-NoGatewayProcess $ctx 'before start (OrphanWorkerTerminator guard)'
|
||||
Assert-EnvUnchanged $ctx 'before the start'
|
||||
} else { Say "[DRYRUN] assert no process named $($script:ServerImages -join '|') / $($script:WorkerImages -join '|') exists, and the env block fingerprint is unchanged (OrphanWorkerTerminator guard)" }
|
||||
Step "nssm start $ServiceName (once)" {
|
||||
[void](Invoke-Nssm -NssmArgs @('start', $ServiceName))
|
||||
}
|
||||
Step "wait for SERVICE_RUNNING and compare env entry count (expect $($ctx.EnvCount))" {
|
||||
Step "wait for SERVICE_RUNNING and compare the env block fingerprint" {
|
||||
if (-not (Wait-State 'SERVICE_RUNNING' 60)) { Fail "service did not reach RUNNING (state $(Get-ServiceState))" }
|
||||
$now = (Parse-EnvBlock (Get-NssmValue 'AppEnvironmentExtra')).Count
|
||||
if ($now -ne $ctx.EnvCount) { Fail "NSSM env block entry count changed: $($ctx.EnvCount) -> $now" }
|
||||
Assert-EnvUnchanged $ctx 'after the start'
|
||||
}
|
||||
}
|
||||
|
||||
@@ -329,94 +625,157 @@ function Backup-State($ctx) {
|
||||
Say '-- Backup (service stopped) --'
|
||||
$suffix = Get-Suffix
|
||||
$root = Split-Path -Parent $ctx.ServerDir
|
||||
$envFile = Join-Path $root "_env-AppEnvironmentExtra-$($script:Stamp).txt"
|
||||
Step "save the NSSM env block + Application/AppDirectory to $envFile (contains secrets: keep it ACL-restricted)" {
|
||||
$body = @("Application=$($ctx.Application)", "AppDirectory=$($ctx.AppDirectory)", '--- AppEnvironmentExtra ---', $ctx.EnvText)
|
||||
Set-Content -LiteralPath $envFile -Value $body -Encoding UTF8
|
||||
try { & icacls.exe $envFile /inheritance:r /grant:r 'SYSTEM:(F)' 'BUILTIN\Administrators:(F)' | Out-Null } catch { Warn "could not restrict ACL on ${envFile}: $_" }
|
||||
$dir = Join-Path $root "_deploy-backup-$suffix"
|
||||
$script:BackupDir = $dir
|
||||
Step "create $dir, restrict it to SYSTEM + Administrators, then save the env block (secrets), service registry key and env fingerprint" {
|
||||
New-Item -ItemType Directory -Path $dir -Force | Out-Null
|
||||
Protect-Directory $dir
|
||||
$body = @("Application=$($ctx.Application)", "AppDirectory=$($ctx.AppDirectory)", '--- AppEnvironmentExtra ---') + @(Read-EnvBlockLines)
|
||||
Set-Content -LiteralPath (Join-Path $dir 'AppEnvironmentExtra.txt') -Value $body -Encoding UTF8
|
||||
Export-ServiceKey (Join-Path $dir "$ServiceName-servicekey.reg")
|
||||
$fp = @("BlockHash=$($ctx.EnvFp.BlockHash)") + @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" })
|
||||
Set-Content -LiteralPath (Join-Path $dir 'env-fingerprint.txt') -Value $fp -Encoding ASCII
|
||||
}
|
||||
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
|
||||
foreach ($f in (Get-StoreFiles $db)) {
|
||||
$dst = "$f.bak-$suffix"
|
||||
Step "copy store file $f -> $dst (live file is never moved)" { Copy-Item -LiteralPath $f -Destination $dst -Force }
|
||||
Step "copy store file $f -> $dst (live file is never moved)" {
|
||||
Copy-Item -LiteralPath $f -Destination $dst -Force
|
||||
if ((Get-FileSha $f) -ne (Get-FileSha $dst)) { Fail "store backup hash mismatch for $f" }
|
||||
}
|
||||
}
|
||||
if (-not (Test-Path -LiteralPath $db)) { Info "no store at $db to back up" }
|
||||
}
|
||||
if ($ctx.KekFile -and (Test-Path -LiteralPath $ctx.KekFile)) {
|
||||
Step "copy master key file $($ctx.KekFile) into $dir (live file is never moved)" { Copy-Item -LiteralPath $ctx.KekFile -Destination (Join-Path $dir (Get-Leaf $ctx.KekFile)) -Force }
|
||||
}
|
||||
}
|
||||
|
||||
function Test-Preserved([string]$Name) { foreach ($p in $script:PreservePatterns) { if ($Name -like $p) { return $true } }; $false }
|
||||
|
||||
# Which top-level files of the live dir go into the new one. Carry = names copied from live (live wins).
|
||||
function Get-CarryPlan([string]$Live, [string]$Source, [bool]$KeepSettings) {
|
||||
$liveFiles = @(Get-Files $Live)
|
||||
$srcNames = @(Get-Files $Source | ForEach-Object { $_.Name })
|
||||
$srcDirs = @(Get-Dirs $Source | ForEach-Object { $_.Name })
|
||||
$carry = New-Object System.Collections.Generic.List[string]; $dropped = New-Object System.Collections.Generic.List[string]
|
||||
foreach ($f in $liveFiles) {
|
||||
$inSource = $srcNames -contains $f.Name
|
||||
if ($f.Name -like 'appsettings*.json') { if ($KeepSettings) { $carry.Add($f.Name) }; continue }
|
||||
if (Test-Preserved $f.Name) { $carry.Add($f.Name); continue }
|
||||
if ($inSource) { continue }
|
||||
if ($script:BinaryExtensions -contains $f.Extension.ToLowerInvariant()) { $dropped.Add($f.Name) } else { $carry.Add($f.Name) }
|
||||
}
|
||||
$dirs = @(Get-Dirs $Live | Where-Object { $srcDirs -notcontains $_.Name } | ForEach-Object { $_.Name })
|
||||
[pscustomobject]@{ Carry = $carry.ToArray(); Dropped = $dropped.ToArray(); Dirs = $dirs }
|
||||
}
|
||||
|
||||
function Swap-Directory([string]$Live, [string]$Source, [string]$Name, [bool]$KeepSettings) {
|
||||
$bak = "$Live.bak-$(Get-Suffix)"
|
||||
$plan = Get-CarryPlan $Live $Source $KeepSettings
|
||||
$hashes = @{}
|
||||
foreach ($f in $plan.Carry) { $hashes[$f] = Get-FileSha (Join-Path $Live $f) }
|
||||
$sddlOld = Get-DirSddl $Live
|
||||
Step "rename $Name dir: $Live -> $bak" { Rename-Item -LiteralPath $Live -NewName (Split-Path -Leaf $bak) }
|
||||
Step "copy new ${Name}: $Source -> $Live" { Copy-Item -LiteralPath $Source -Destination $Live -Recurse -Force }
|
||||
if ($KeepSettings) {
|
||||
$from = if ($script:Dry) { $Live } else { $bak } # in a dry run nothing was renamed yet
|
||||
foreach ($f in 'appsettings.json', 'appsettings.Development.json') {
|
||||
if (Test-Path -LiteralPath (Join-Path $from $f)) {
|
||||
Step "restore live $f byte-identical from $bak" { Copy-Item -LiteralPath (Join-Path $bak $f) -Destination (Join-Path $Live $f) -Force }
|
||||
} else { Info "$f not present in the old $Name dir (nothing to carry over)" }
|
||||
foreach ($f in $plan.Carry) {
|
||||
Step "carry live $Name\$f forward from $bak (sha256 $($hashes[$f].Substring(0, 12)))" {
|
||||
Copy-Item -LiteralPath (Join-Path $bak $f) -Destination (Join-Path $Live $f) -Force
|
||||
if ((Get-FileSha (Join-Path $Live $f)) -ne $hashes[$f]) { Fail "carried $Name\$f does not match the live copy" }
|
||||
Copy-ExplicitAcl (Join-Path $bak $f) (Join-Path $Live $f)
|
||||
}
|
||||
}
|
||||
foreach ($f in $plan.Dropped) { Info "$Name\$f exists only in the old build (binary) and is not carried; it stays in $bak" }
|
||||
if (-not $script:Dry) {
|
||||
$sddlNew = Get-DirSddl $Live
|
||||
if ($sddlOld -and $sddlNew -and $sddlOld -ne $sddlNew) { Warn "$Name dir ACL differs from the old one (new inherits from the parent). old: $sddlOld new: $sddlNew" }
|
||||
}
|
||||
$bak
|
||||
}
|
||||
|
||||
# --- verification ----------------------------------------------------------------------------------------------------
|
||||
|
||||
function Invoke-Verify($ctx) {
|
||||
function Invoke-Verify($ctx, [bool]$AfterRollback = $false) {
|
||||
Say '== Verify =='
|
||||
if ($script:Dry) {
|
||||
Say "[DRYRUN] GET $HealthBaseUrl/health/ready until 200 (<= $ReadyTimeoutSeconds s)"
|
||||
Say "[DRYRUN] GET $HealthBaseUrl/healthz ; read ProductVersion SHA of $(Join-Path $ctx.ServerDir $ctx.ServerExeName)$(if ($ExpectedSha) { " and require it to match $ExpectedSha" })"
|
||||
Say "[DRYRUN] service must stay SERVICE_RUNNING for $StableSeconds s with env entry count $($ctx.EnvCount)"
|
||||
Say "[DRYRUN] GET $HealthBaseUrl/healthz ; read ProductVersion SHA of $($ctx.Application)$(if ($ExpectedSha -and -not $Rollback) { " and require it to match $ExpectedSha" })"
|
||||
Say "[DRYRUN] exactly one gateway Server ($($ctx.Application)) on the box, same PID, service SERVICE_RUNNING for $StableSeconds s, env fingerprint unchanged"
|
||||
return
|
||||
}
|
||||
$ready = $false; $deadline = (Get-Date).AddSeconds($ReadyTimeoutSeconds)
|
||||
while (-not $ready -and (Get-Date) -lt $deadline) {
|
||||
try { $r = Invoke-WebRequest -Uri "$HealthBaseUrl/health/ready" -UseBasicParsing -TimeoutSec 10; $ready = ($r.StatusCode -eq 200) }
|
||||
catch { Start-Sleep -Seconds 3 }
|
||||
$ready = ((Get-HttpStatus "$HealthBaseUrl/health/ready") -eq 200)
|
||||
if (-not $ready) { Start-Sleep -Seconds 3 }
|
||||
}
|
||||
if (-not $ready) { Fail "$HealthBaseUrl/health/ready did not return 200 within $ReadyTimeoutSeconds s" }
|
||||
Info '/health/ready = 200'
|
||||
try { Info "/healthz = $((Invoke-WebRequest -Uri "$HealthBaseUrl/healthz" -UseBasicParsing -TimeoutSec 10).StatusCode)" } catch { Fail "/healthz failed: $_" }
|
||||
$hz = Get-HttpStatus "$HealthBaseUrl/healthz"
|
||||
if ($hz -ne 200) { Fail "/healthz = $hz" }
|
||||
Info '/healthz = 200'
|
||||
|
||||
$sha = Get-ProductSha (Join-Path $ctx.ServerDir $ctx.ServerExeName)
|
||||
$sha = Get-ProductSha $ctx.Application
|
||||
Info "deployed Server SHA (ProductVersion): $sha"
|
||||
if ($ExpectedSha -and -not $Rollback -and -not (Test-ShaMatch $sha $ExpectedSha)) { Fail "deployed SHA '$sha' != expected '$ExpectedSha'" }
|
||||
if ($ExpectedSha -and -not $Rollback -and -not $AfterRollback -and -not (Test-ShaMatch $sha $ExpectedSha)) { Fail "deployed SHA '$sha' != expected '$ExpectedSha'" }
|
||||
|
||||
Info "watching the service for $StableSeconds s"
|
||||
Assert-SingleGateway $ctx 'SERVICE_RUNNING'
|
||||
$pid0 = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' })[0].Id
|
||||
Info "watching the service for $StableSeconds s (gateway #$pid0)"
|
||||
$until = (Get-Date).AddSeconds($StableSeconds)
|
||||
while ((Get-Date) -lt $until) {
|
||||
do {
|
||||
if ((Get-ServiceState) -ne 'SERVICE_RUNNING') { Fail 'service left RUNNING during the stability window (restart loop?)' }
|
||||
Start-Sleep -Seconds 5
|
||||
}
|
||||
$n = (Parse-EnvBlock (Get-NssmValue 'AppEnvironmentExtra')).Count
|
||||
if ($n -ne $ctx.EnvCount) { Fail "env block entry count $n != preflight $($ctx.EnvCount)" }
|
||||
Info "env block still $n entries"
|
||||
} while ((Get-Date) -lt $until)
|
||||
$servers = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' })
|
||||
if ($servers.Count -ne 1 -or $servers[0].Id -ne $pid0) { Fail "gateway process changed during the stability window: #$pid0 -> $(Format-Procs $servers) (restart loop or second gateway)" }
|
||||
Assert-EnvUnchanged $ctx 'after the stability window'
|
||||
Warn 'health is green with zero sessions; still prove the Worker path: open a session with a PRE-EXISTING API key and read a known tag (runbook 7.3)'
|
||||
}
|
||||
|
||||
# --- rollback ------------------------------------------------------------------------------------------------------------
|
||||
|
||||
function Find-Backup([string]$Live, [string]$Stamp) {
|
||||
$leaf = Split-Path -Leaf $Live; $parent = Split-Path -Parent $Live
|
||||
$cands = @(Get-ChildItem -LiteralPath $parent -Directory -Filter "$leaf.bak-*" -ErrorAction SilentlyContinue)
|
||||
if ($Stamp) { $cands = @($cands | Where-Object { $_.Name -like "$leaf.bak-$Stamp*" }) }
|
||||
$cands | Sort-Object Name -Descending | Select-Object -First 1
|
||||
# Backup dirs come in two spellings: <leaf>.bak-yyyyMMdd-HHmmss[-label] (this script) and <leaf>.bak.yyyyMMddTHHmmss (manual / adapted scripts).
|
||||
function Get-BackupKey([string]$Name) {
|
||||
if ($Name -match '\.bak[-.](\d{8})[-T]?(\d{6})') { return "$($Matches[1])$($Matches[2])" }
|
||||
$null
|
||||
}
|
||||
|
||||
function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores) {
|
||||
function Find-Backup([string]$Live, [string]$Stamp) {
|
||||
$leaf = Split-Path -Leaf $Live; $parent = Split-Path -Parent $Live
|
||||
$cands = @(Get-Dirs $parent | Where-Object { ($_.Name -like "$leaf.bak-*" -or $_.Name -like "$leaf.bak.*") -and (Get-BackupKey $_.Name) })
|
||||
if ($Stamp) {
|
||||
$key = ($Stamp -replace '[^0-9]', '')
|
||||
$cands = @($cands | Where-Object { (Get-BackupKey $_.Name) -eq $key -or $_.Name -like "$leaf.bak-$Stamp*" -or $_.Name -like "$leaf.bak.$Stamp*" })
|
||||
}
|
||||
$cands | Sort-Object { Get-BackupKey $_.Name } -Descending | Select-Object -First 1
|
||||
}
|
||||
|
||||
function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores, [bool]$AllowPartial) {
|
||||
Say '== Rollback =='
|
||||
$sb = Find-Backup $ctx.ServerDir $Stamp
|
||||
$wb = Find-Backup $ctx.WorkerDir $Stamp
|
||||
if (-not $sb -or -not $wb) { Fail "no matching Server/Worker backup directories found (stamp '$Stamp')" }
|
||||
Info "rolling back to Server=$($sb.Name) Worker=$($wb.Name)"
|
||||
Stop-Gateway
|
||||
if (-not $sb -and -not $AllowPartial) { Fail "no Server backup directory found (stamp '$Stamp')" }
|
||||
$key = if ($sb) { Get-BackupKey $sb.Name } else { ($Stamp -replace '[^0-9]', '') }
|
||||
$wb = Find-Backup $ctx.WorkerDir $key
|
||||
if ($wb -and (Get-BackupKey $wb.Name) -ne $key) { $wb = $null }
|
||||
if (-not $wb -and -not $AllowPartial) { Fail "no Worker backup matching the Server backup $($sb.Name) (stamp key $key); refusing a mixed rollback" }
|
||||
if (-not $sb -and -not $wb) { Fail "nothing to roll back to (stamp '$Stamp')" }
|
||||
Info "rolling back to Server=$(if ($sb) { $sb.Name } else { '(unchanged)' }) Worker=$(if ($wb) { $wb.Name } else { '(unchanged)' })"
|
||||
$fpFile = @(Get-Dirs (Split-Path -Parent $ctx.ServerDir) '_deploy-backup-*' | Where-Object { ($_.Name -replace '[^0-9]', '').StartsWith($key) } | ForEach-Object { Join-Path $_.FullName 'env-fingerprint.txt' } | Where-Object { Test-Path -LiteralPath $_ })
|
||||
if ($fpFile.Count) {
|
||||
$saved = Get-Content -LiteralPath $fpFile[0] | Where-Object { $_ -notlike 'BlockHash=*' }
|
||||
$now = @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" })
|
||||
if (($saved -join '|') -ne ($now -join '|')) { Warn "the env block differs from the one saved before the upgrade ($($fpFile[0])); restore it from AppEnvironmentExtra.txt there if that was not intended (runbook section 8 step 4)" }
|
||||
else { Info "env block matches the pre-upgrade fingerprint in $($fpFile[0])" }
|
||||
}
|
||||
Stop-Gateway $ctx
|
||||
foreach ($pair in @(@($ctx.ServerDir, $sb), @($ctx.WorkerDir, $wb))) {
|
||||
if (-not $pair[1]) { Info "$($pair[0]) has no backup for this stamp; left unchanged"; continue }
|
||||
$failed = "$($pair[0]).failed-$($script:Stamp)"
|
||||
if (Test-Path -LiteralPath $pair[0]) { Step "rename $($pair[0]) -> $failed" { Rename-Item -LiteralPath $pair[0] -NewName (Split-Path -Leaf $failed) } }
|
||||
Step "rename $($pair[1].FullName) -> $($pair[0])" { Rename-Item -LiteralPath $pair[1].FullName -NewName (Split-Path -Leaf $pair[0]) }
|
||||
}
|
||||
if ($Stores) {
|
||||
$bstamp = ($sb.Name -replace '^.*\.bak-', '')
|
||||
$bstamp = ($sb.Name -replace '^.*\.bak[-.]', '')
|
||||
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
|
||||
foreach ($ext in '', '-wal', '-shm') {
|
||||
$live = "$db$ext"; $bak = "$db$ext.bak-$bstamp"
|
||||
@@ -434,57 +793,75 @@ function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores) {
|
||||
|
||||
# --- main -------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
$exit = 0
|
||||
try {
|
||||
Say "Upgrade-MxGatewayService stamp=$($script:Stamp) mode=$(if ($Rollback) {'ROLLBACK'} else {'UPGRADE'}) dry=$($script:Dry)"
|
||||
$ctx = Get-Context
|
||||
Invoke-Preflight $ctx
|
||||
|
||||
if (-not $script:Dry) {
|
||||
$lockPath = Join-Path (Split-Path -Parent $ctx.ServerDir) '.mxgw-deploy.lock'
|
||||
try { $script:LockStream = [IO.File]::Open($lockPath, 'OpenOrCreate', 'ReadWrite', 'None') }
|
||||
catch { Fail "another deploy run holds $lockPath" }
|
||||
}
|
||||
|
||||
if ($Rollback) {
|
||||
Invoke-RollbackSwap $ctx $BackupStamp $RestoreStores.IsPresent
|
||||
Invoke-Verify $ctx
|
||||
} else {
|
||||
Stop-Gateway
|
||||
Backup-State $ctx
|
||||
$swapped = $false
|
||||
try {
|
||||
$null = Swap-Directory $ctx.ServerDir $ServerSource 'Server' (-not $TakeNewAppSettings)
|
||||
$null = Swap-Directory $ctx.WorkerDir $WorkerSource 'Worker' $false
|
||||
$swapped = $true
|
||||
Start-Gateway $ctx
|
||||
Invoke-Verify $ctx
|
||||
} catch {
|
||||
$err = $_
|
||||
Say "[FAIL] $($err.Exception.Message)"
|
||||
if ($script:Dry -or $NoAutoRollback) { throw }
|
||||
Warn 'upgrade failed - rolling back automatically'
|
||||
try {
|
||||
if (-not $swapped) { Warn 'swap did not complete; rollback swaps whatever .bak sets exist for this stamp' }
|
||||
Invoke-RollbackSwap $ctx $script:Stamp $false
|
||||
Invoke-Verify $ctx
|
||||
Warn 'ROLLED BACK to the previous build; original failure follows'
|
||||
} catch { Warn "AUTOMATIC ROLLBACK FAILED: $($_.Exception.Message). Manual intervention required (backups: *.bak-$(Get-Suffix))." }
|
||||
throw $err
|
||||
}
|
||||
}
|
||||
Say '== DONE =='
|
||||
} catch {
|
||||
$exit = 1
|
||||
Write-Host "ERROR: $($_.Exception.Message)" -ForegroundColor Red
|
||||
} finally {
|
||||
if ($script:LockStream) { $script:LockStream.Dispose() }
|
||||
function Invoke-Main {
|
||||
$exit = 0
|
||||
$ctx = $null
|
||||
try {
|
||||
if (-not $script:Dry -and $ctx) {
|
||||
$notes = Join-Path (Split-Path -Parent $ctx.ServerDir) "_DEPLOY-NOTES-$($script:Stamp).txt"
|
||||
Set-Content -LiteralPath $notes -Value $script:Notes -Encoding UTF8
|
||||
Write-Host "deploy notes: $notes"
|
||||
Say "Upgrade-MxGatewayService stamp=$($script:Stamp) mode=$(if ($Rollback) {'ROLLBACK'} else {'UPGRADE'}) dry=$($script:Dry) host=$env:COMPUTERNAME"
|
||||
$ctx = Get-Context
|
||||
Invoke-Preflight $ctx
|
||||
|
||||
if (-not $script:Dry) {
|
||||
$lockPath = Join-Path (Split-Path -Parent $ctx.ServerDir) '.mxgw-deploy.lock'
|
||||
try { $script:LockStream = [IO.File]::Open($lockPath, 'OpenOrCreate', 'ReadWrite', 'None') }
|
||||
catch { Fail "another deploy run holds $lockPath" }
|
||||
}
|
||||
} catch { }
|
||||
|
||||
if ($Rollback) {
|
||||
Invoke-RollbackSwap $ctx $BackupStamp $RestoreStores.IsPresent $false
|
||||
Invoke-Verify $ctx
|
||||
} else {
|
||||
Stop-Gateway $ctx
|
||||
try { Backup-State $ctx }
|
||||
catch {
|
||||
$err = $_
|
||||
if (-not $script:Dry) {
|
||||
Warn 'backup failed before anything was swapped - starting the unchanged build again'
|
||||
try { Start-Gateway $ctx } catch { Warn "restart of the unchanged build FAILED: $($_.Exception.Message)" }
|
||||
}
|
||||
throw $err
|
||||
}
|
||||
$touched = $false
|
||||
try {
|
||||
$touched = $true
|
||||
$null = Swap-Directory $ctx.ServerDir $ServerSource 'Server' (-not $TakeNewAppSettings)
|
||||
$null = Swap-Directory $ctx.WorkerDir $WorkerSource 'Worker' $false
|
||||
if (-not $script:Dry) { Assert-EnvUnchanged $ctx 'after the swap' }
|
||||
Start-Gateway $ctx
|
||||
Invoke-Verify $ctx
|
||||
} catch {
|
||||
$err = $_
|
||||
Say "[FAIL] $(Hide $err.Exception.Message)"
|
||||
if ($script:Dry -or $NoAutoRollback -or -not $touched) { throw }
|
||||
Warn 'upgrade failed - rolling back automatically'
|
||||
$script:EnvWarnOnly = $true
|
||||
try {
|
||||
Invoke-RollbackSwap $ctx $script:Stamp $false $true
|
||||
Invoke-Verify $ctx $true
|
||||
Warn 'ROLLED BACK to the previous build; original failure follows'
|
||||
} catch { Warn "AUTOMATIC ROLLBACK FAILED: $($_.Exception.Message). Manual intervention required (backups: *.bak-$(Get-Suffix))." }
|
||||
throw $err
|
||||
}
|
||||
}
|
||||
Say '== DONE =='
|
||||
} catch {
|
||||
$exit = 1
|
||||
$m = Hide $_.Exception.Message
|
||||
$script:Notes.Add("ERROR: $m")
|
||||
Write-Host "ERROR: $m" -ForegroundColor Red
|
||||
} finally {
|
||||
if ($script:LockStream) { $script:LockStream.Dispose(); $script:LockStream = $null }
|
||||
try {
|
||||
if (-not $script:Dry -and $ctx) {
|
||||
$notes = Join-Path (Split-Path -Parent $ctx.ServerDir) "_DEPLOY-NOTES-$($script:Stamp).txt"
|
||||
Set-Content -LiteralPath $notes -Value $script:Notes -Encoding UTF8
|
||||
Write-Host "deploy notes: $notes"
|
||||
}
|
||||
} catch { }
|
||||
$script:Redact.Clear()
|
||||
}
|
||||
$exit
|
||||
}
|
||||
exit $exit
|
||||
|
||||
# Dot-sourcing (the dry-run harness) only defines the functions.
|
||||
if ($MyInvocation.InvocationName -ne '.') { $rc = Invoke-Main | Select-Object -Last 1; exit $rc }
|
||||
|
||||
Reference in New Issue
Block a user