fix(deploy): Upgrade-MxGatewayService.ps1 safe on the wonder layout, guards fail closed (#230)
The process guards matched only ZB.MOM.WW.MxGateway.*; the wonder boxes run MxGateway.Server.exe / MxGateway.Worker.exe, so the one-gateway guard, the pre-start OrphanWorkerTerminator guard and the STOP_PENDING kill passed vacuously. - Image names come from the installed exes (NSSM Application, Worker env path) plus both known schemes; processes are also matched by exact path (Win32_Process). - One-gateway guard fails closed: a RUNNING service whose gateway is not visible, at the Application path, as the service host's child, is an abort. - STOP_PENDING kill is scoped to this install's processes. - Staged checks use the installed exe names; staged Worker must be x86; a staged self-contained runtime older than the live one is refused (-AllowRuntimeDowngrade). - Carry-over: appsettings*.json and certificates/keys (dashboard.pfx) always keep the live copy; other live-only non-binary top-level files are carried; hash-verified. - Env block read from the registry and fingerprinted (names + SHA-256 prefixes, never values) after stop, swap, start and the stability window; any change aborts. Sensitive values are redacted from all output and notes. - File-based master key (vd03) accepted and backed up; a missing env master key (windev) is a warning. Backup dir is ACL-restricted before anything is written. - Rollback accepts .bak-<stamp> and .bak.<yyyyMMddTHHmmss> sets and refuses a mixed Server/Worker stamp; auto-rollback no longer fails on the old SHA. - -WhatIf now works on Windows PowerShell 5.1 (Get-FileHash returned nothing under WhatIfPreference); an empty Win32_Service result no longer reads as "unavailable". - Install-MxGatewayService.ps1 process guard matches both schemes. - Dry-run harness scripts/deploy/tests (fake box in both layouts, 140 assertions, passes on pwsh/macOS and Windows PowerShell 5.1) plus a mutation runner (12/12 killed). - Runbook updated. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
@@ -123,7 +123,9 @@ try {
|
||||
# --- one-gateway guard + existence -------------------------------------------------------------------
|
||||
$st = Invoke-Nssm -NssmArgs @('status', $ServiceName) -AllowFail
|
||||
if ($st.Code -eq 0) { Fail "service '$ServiceName' already exists (state $($st.Text)); use Upgrade-MxGatewayService.ps1" }
|
||||
$procs = @(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in @($ServerProc, $WorkerProc) })
|
||||
# Both naming schemes (ZB.MOM.WW.MxGateway.* on windev, MxGateway.* on the wonder boxes) plus the exe being installed (#230).
|
||||
$gwImages = @($ServerProc, $WorkerProc, 'MxGateway.Server', 'MxGateway.Worker', [IO.Path]::GetFileNameWithoutExtension($ServerExeName)) | Select-Object -Unique
|
||||
$procs = @(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in $gwImages })
|
||||
if ($procs.Count -gt 0) { Fail "gateway process(es) already running (PIDs $(($procs.Id) -join ',')); installing/starting a second gateway kills their Workers" }
|
||||
|
||||
if (-not $LogDir) { $LogDir = Join-Path $DataDir 'logs' }
|
||||
|
||||
@@ -2,32 +2,38 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
On-box upgrade / rollback of the NSSM-wrapped MxAccess Gateway (Server + x86 Worker).
|
||||
Implements docs/runbooks/DeployUpgradeNssmService.md sections 3, 5, 6, 7 and 8 (mxaccessgw#220).
|
||||
Implements docs/runbooks/DeployUpgradeNssmService.md sections 3, 5, 6, 7 and 8 (mxaccessgw#220, #230).
|
||||
|
||||
.DESCRIPTION
|
||||
RUN THIS ON THE WINDOWS BOX ITSELF (Windows PowerShell 5.1 is enough). The new Server publish and the
|
||||
new x86 Worker build must already be staged on the box (UNC/SMB copy; build both from the same SHA).
|
||||
|
||||
Upgrade flow: preflight -> stop (STOP_PENDING handled) -> backup -> swap Server + Worker -> start ->
|
||||
verify (/health/ready, VersionInfo SHA, service stable) -> automatic rollback if start/verify fails.
|
||||
Rollback flow (-Rollback): stop -> swap the .bak directories back -> optional store restore -> start -> verify.
|
||||
verify (/health/ready, VersionInfo SHA, service stable, env block unchanged) -> automatic rollback if
|
||||
start/verify fails. Rollback flow (-Rollback): stop -> swap the .bak directories back -> optional store
|
||||
restore -> start -> verify.
|
||||
|
||||
Safety properties:
|
||||
* Exactly one gateway per box. Refuses to run if a second gateway process / service points at the same
|
||||
exe, if the service is stopped but a gateway process exists, or if a Server/Worker process is still
|
||||
alive when a start is about to happen (OrphanWorkerTerminator trap, runbook section 2.1).
|
||||
* Never calls `nssm set ... AppEnvironmentExtra` (that replaces the whole block). The env block is only
|
||||
READ (saved to the backup dir, entry count compared before/after). The Application/AppDirectory
|
||||
are not changed either: the swap is rename-in-place, so the NSSM config is untouched.
|
||||
* API-key DB, Secrets store and master key live outside the content root and are never moved or
|
||||
deleted. They are only COPIED (db + -wal + -shm) into *.bak-<stamp> siblings while the service is stopped.
|
||||
The script refuses to run if a store path is inside the content root (the 2026-08-09 incident).
|
||||
* Exactly one gateway per box. The gateway and Worker image names are taken from the INSTALLED exes
|
||||
(NSSM Application and env MxGateway__Worker__ExecutablePath) plus both known naming schemes
|
||||
(ZB.MOM.WW.MxGateway.* on windev, MxGateway.* on the wonder boxes), and processes are also matched by
|
||||
exact path. The guards FAIL CLOSED: a RUNNING service whose gateway process cannot be found is an
|
||||
abort, not a pass (mxaccessgw#230: the guards used to pass vacuously on the wonder boxes).
|
||||
* Never writes the NSSM env block (AppEnvironmentExtra), Application or AppDirectory: the swap is
|
||||
rename-in-place. The env block is fingerprinted (entry names + SHA-256 prefix of each value; values are
|
||||
never printed) at preflight, after the stop, after the swap, after the start and after the stability
|
||||
window. Any added, removed or changed entry aborts (and triggers the automatic rollback).
|
||||
* Live-only top-level files in Server\ and Worker\ are carried into the new directories (appsettings*.json,
|
||||
dashboard.pfx and other certificates/keys always keep the LIVE copy), hash-verified.
|
||||
* API-key DB, Secrets store and a file-based master key must live outside the swapped directories; they are
|
||||
only COPIED (db + -wal + -shm) into *.bak-<stamp> siblings while the service is stopped.
|
||||
* A self-contained Server is refused if its .NET runtime is older than the live one (no silent downgrade).
|
||||
|
||||
-DryRun (or -WhatIf) prints every mutating action and performs none. Read-only probing (nssm get/status,
|
||||
process list, disk space, file existence) still runs so the decision logic is exercised.
|
||||
-DryRun (or -WhatIf) prints every mutating action and performs none. Read-only probing (service state,
|
||||
registry env block, process list, disk space, file hashes) still runs, so the guards are exercised for real.
|
||||
|
||||
.PARAMETER ServerSource Staged Server publish output (contains ZB.MOM.WW.MxGateway.Server.exe).
|
||||
.PARAMETER WorkerSource Staged x86 Worker output (contains ZB.MOM.WW.MxGateway.Worker.exe).
|
||||
.PARAMETER ServerSource Staged Server publish output (must contain the same exe name as NSSM Application).
|
||||
.PARAMETER WorkerSource Staged x86 Worker output (must contain the exe name of MxGateway__Worker__ExecutablePath).
|
||||
.PARAMETER ServiceName NSSM service name. Default MxAccessGw.
|
||||
.PARAMETER NssmPath nssm executable. Default 'nssm' (on PATH).
|
||||
.PARAMETER ServerDir Live Server directory. Default: NSSM AppDirectory.
|
||||
@@ -36,11 +42,12 @@
|
||||
must carry it in ProductVersion ('0.1.2+<sha>'). Builds from 2026-07-09..08-10 have a bogus stamp.
|
||||
.PARAMETER Label Free-text suffix for backup names (e.g. 'planwrites').
|
||||
.PARAMETER HealthBaseUrl Default http://localhost:5130 (dashboard port; /health/ready, /healthz anonymous).
|
||||
.PARAMETER Rollback Roll back instead of upgrading. Uses newest Server.bak-* unless -BackupStamp is given.
|
||||
.PARAMETER BackupStamp Stamp (yyyyMMdd-HHmmss) of the backup set to roll back to.
|
||||
.PARAMETER Rollback Roll back instead of upgrading. Uses the newest Server.bak-* / Server.bak.* unless -BackupStamp.
|
||||
.PARAMETER BackupStamp Stamp of the backup set to roll back to (yyyyMMdd-HHmmss, or yyyyMMddTHHmmss for .bak.<stamp> sets).
|
||||
.PARAMETER RestoreStores With -Rollback: also restore gateway-auth.db / secrets db from *.bak-<stamp> (schema-bump case).
|
||||
.PARAMETER NoAutoRollback Do not roll back automatically when start/verification fails.
|
||||
.PARAMETER TakeNewAppSettings Do not carry the live appsettings*.json over into the new Server dir.
|
||||
.PARAMETER AllowRuntimeDowngrade Accept a staged self-contained runtime older than the live one.
|
||||
.PARAMETER SkipFreeSpaceCheck / MinFreeGB / StopTimeoutSeconds / StableSeconds / ReadyTimeoutSeconds tunables.
|
||||
.PARAMETER DryRun Print, do not act. -WhatIf is equivalent.
|
||||
#>
|
||||
@@ -60,6 +67,7 @@ param(
|
||||
[switch]$RestoreStores,
|
||||
[switch]$NoAutoRollback,
|
||||
[switch]$TakeNewAppSettings,
|
||||
[switch]$AllowRuntimeDowngrade,
|
||||
[switch]$SkipFreeSpaceCheck,
|
||||
[double]$MinFreeGB = 2,
|
||||
[int]$StopTimeoutSeconds = 30,
|
||||
@@ -70,16 +78,38 @@ param(
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$script:Dry = $DryRun.IsPresent -or $WhatIfPreference
|
||||
# -WhatIf is handled by $script:Dry (Step). Leaving the preference on makes Windows PowerShell 5.1 cmdlets used for READ-ONLY
|
||||
# probing misbehave (Get-FileHash returns nothing under -WhatIf), so the plan would abort; found on windev, #230.
|
||||
$WhatIfPreference = $false
|
||||
$script:Stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
|
||||
$script:ServerProc = 'ZB.MOM.WW.MxGateway.Server'
|
||||
$script:WorkerProc = 'ZB.MOM.WW.MxGateway.Worker'
|
||||
# Both naming schemes seen in the field: windev runs ZB.MOM.WW.MxGateway.*.exe, the wonder boxes MxGateway.*.exe.
|
||||
# The installed exe names (NSSM Application, MxGateway__Worker__ExecutablePath) are added to these in Get-Context.
|
||||
$script:KnownServerImages = @('ZB.MOM.WW.MxGateway.Server', 'MxGateway.Server')
|
||||
$script:KnownWorkerImages = @('ZB.MOM.WW.MxGateway.Worker', 'MxGateway.Worker')
|
||||
$script:ServerImages = $script:KnownServerImages
|
||||
$script:WorkerImages = $script:KnownWorkerImages
|
||||
$script:EnvWorkerExe = 'MxGateway__Worker__ExecutablePath'
|
||||
# Live copy always wins for these (host config, certificates, keys), even when the staged build ships one.
|
||||
$script:PreservePatterns = @('appsettings*.json', '*.pfx', '*.p12', '*.pem', '*.key', '*.crt', '*.cer')
|
||||
# Never carried from the old build (they are build output; a live-only binary is a leftover of an older build).
|
||||
$script:BinaryExtensions = @('.dll', '.exe', '.pdb', '.so', '.dylib', '.winmd')
|
||||
$script:Notes = New-Object System.Collections.Generic.List[string]
|
||||
$script:Redact = New-Object System.Collections.Generic.List[string]
|
||||
$script:LockStream = $null
|
||||
$script:BackupDir = $null
|
||||
$script:EnvWarnOnly = $false
|
||||
|
||||
function Say([string]$m) { Write-Host $m; $script:Notes.Add($m) }
|
||||
# --- output (secret values are redacted from every line) ----------------------------------------------------
|
||||
|
||||
function Hide([string]$m) {
|
||||
if ($null -eq $m) { return '' }
|
||||
foreach ($s in $script:Redact) { if ($s) { $m = $m.Replace($s, '<redacted>') } }
|
||||
$m
|
||||
}
|
||||
function Say([string]$m) { $m = Hide $m; Write-Host $m; $script:Notes.Add($m) }
|
||||
function Info([string]$m) { Say "[info] $m" }
|
||||
function Warn([string]$m) { Write-Warning $m; $script:Notes.Add("[WARN] $m") }
|
||||
function Fail([string]$m) { throw "ABORT: $m" }
|
||||
function Warn([string]$m) { $m = Hide $m; Write-Warning $m; $script:Notes.Add("[WARN] $m") }
|
||||
function Fail([string]$m) { throw "ABORT: $(Hide $m)" }
|
||||
|
||||
# Mutating step: print in dry-run, run otherwise.
|
||||
function Step([string]$Description, [scriptblock]$Action) {
|
||||
@@ -88,7 +118,9 @@ function Step([string]$Description, [scriptblock]$Action) {
|
||||
& $Action
|
||||
}
|
||||
|
||||
# --- helpers --------------------------------------------------------------------------------------------
|
||||
# --- environment seams (the dry-run harness in scripts/deploy/tests replaces these) ---------------------------
|
||||
|
||||
function Test-IsWindowsHost { $env:OS -eq 'Windows_NT' }
|
||||
|
||||
function Invoke-Nssm {
|
||||
param([string[]]$NssmArgs, [switch]$AllowFail)
|
||||
@@ -107,20 +139,50 @@ function Get-ServiceState {
|
||||
($r.Text -split "`n")[0].Trim()
|
||||
}
|
||||
|
||||
function Parse-EnvBlock([string]$Text) {
|
||||
$map = [ordered]@{}
|
||||
foreach ($line in ($Text -split "[`r`n]+")) {
|
||||
$t = $line.Trim()
|
||||
if ($t -match '^([^=\s]+)=(.*)$') { $map[$Matches[1]] = $Matches[2] }
|
||||
# The env block exactly as NSSM stores it (REG_MULTI_SZ), one 'NAME=value' per element. Falls back to nssm get.
|
||||
function Read-EnvBlockLines {
|
||||
$key = "HKLM:\SYSTEM\CurrentControlSet\Services\$ServiceName\Parameters"
|
||||
if ((Test-IsWindowsHost) -and (Test-Path -LiteralPath $key)) {
|
||||
$p = Get-ItemProperty -LiteralPath $key -Name AppEnvironmentExtra -ErrorAction SilentlyContinue
|
||||
if ($p) { return @($p.AppEnvironmentExtra | Where-Object { $_ }) }
|
||||
return @()
|
||||
}
|
||||
$map
|
||||
@((Get-NssmValue 'AppEnvironmentExtra') -split "[`r`n]+" | Where-Object { $_.Trim() })
|
||||
}
|
||||
|
||||
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
|
||||
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
|
||||
function Test-Inside([string]$Path, [string]$Root) {
|
||||
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
|
||||
$a -eq $b -or $a.StartsWith("$b/")
|
||||
# Id, ParentId, Name (image name without .exe), Path (may be $null when unreadable).
|
||||
function Get-ProcessTable {
|
||||
if (Get-Command Get-CimInstance -ErrorAction SilentlyContinue) {
|
||||
return @(Get-CimInstance Win32_Process | ForEach-Object {
|
||||
[pscustomobject]@{ Id = [int]$_.ProcessId; ParentId = [int]$_.ParentProcessId; Name = [IO.Path]::GetFileNameWithoutExtension([string]$_.Name); Path = $_.ExecutablePath }
|
||||
})
|
||||
}
|
||||
@(Get-Process -ErrorAction SilentlyContinue | ForEach-Object {
|
||||
$p = $null; try { $p = $_.Path } catch { }
|
||||
[pscustomobject]@{ Id = $_.Id; ParentId = 0; Name = $_.ProcessName; Path = $p }
|
||||
})
|
||||
}
|
||||
|
||||
# PID of the NSSM service host (the gateway Server is its child). $null when it cannot be read.
|
||||
function Get-ServiceHostPid {
|
||||
if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null }
|
||||
$s = Get-CimInstance Win32_Service -Filter "Name='$ServiceName'" -ErrorAction SilentlyContinue
|
||||
if ($s -and $s.ProcessId) { return [int]$s.ProcessId }
|
||||
$null
|
||||
}
|
||||
|
||||
# Other services whose command line launches the gateway exe. $null when the check is unavailable.
|
||||
function Get-OtherGatewayServices([string]$ExeName) {
|
||||
if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null }
|
||||
# The leading comma keeps an EMPTY result an array: a bare @() reaches the caller as $null ('check unavailable').
|
||||
, @(Get-CimInstance Win32_Service | Where-Object { $_.Name -ne $ServiceName -and $_.PathName -and $_.PathName -like "*$ExeName*" } | ForEach-Object { $_.Name })
|
||||
}
|
||||
|
||||
function Stop-ProcessById([int]$Id) { Stop-Process -Id $Id -Force -ErrorAction SilentlyContinue }
|
||||
|
||||
function Get-HttpStatus([string]$Url) {
|
||||
try { return [int](Invoke-WebRequest -Uri $Url -UseBasicParsing -TimeoutSec 10).StatusCode }
|
||||
catch { if ($_.Exception.Response) { return [int]$_.Exception.Response.StatusCode }; return 0 }
|
||||
}
|
||||
|
||||
function Get-ProductSha([string]$Exe) {
|
||||
@@ -130,25 +192,200 @@ function Get-ProductSha([string]$Exe) {
|
||||
} catch { }
|
||||
$null
|
||||
}
|
||||
|
||||
# 'X86', 'Amd64', 'MSIL', ... or $null when it cannot be read.
|
||||
function Get-ExeArchitecture([string]$Exe) {
|
||||
try { return "$([Reflection.AssemblyName]::GetAssemblyName($Exe).ProcessorArchitecture)" } catch { return $null }
|
||||
}
|
||||
|
||||
# Restrict a directory to SYSTEM + Administrators, inheritance off. Throws if that fails.
|
||||
function Protect-Directory([string]$Path) {
|
||||
if (-not (Test-IsWindowsHost)) { Warn "not Windows: $Path is not ACL-restricted"; return }
|
||||
$null = & icacls.exe $Path /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' 2>&1
|
||||
if ($LASTEXITCODE -ne 0) { Fail "icacls could not restrict $Path (exit $LASTEXITCODE); refusing to write the env block there" }
|
||||
}
|
||||
|
||||
function Export-ServiceKey([string]$Destination) {
|
||||
if (-not (Test-IsWindowsHost)) { Info 'not Windows: service registry export skipped'; return }
|
||||
$null = & reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$ServiceName" $Destination /y 2>&1
|
||||
if ($LASTEXITCODE -ne 0) { Fail "reg export of the service key failed (exit $LASTEXITCODE)" }
|
||||
}
|
||||
|
||||
function Get-DirSddl([string]$Path) {
|
||||
if (-not (Test-IsWindowsHost)) { return $null }
|
||||
try { (Get-Acl -LiteralPath $Path).Sddl } catch { $null }
|
||||
}
|
||||
|
||||
# Give a carried file the old file's ACL when the old one had explicit (non-inherited) entries.
|
||||
function Copy-ExplicitAcl([string]$From, [string]$To) {
|
||||
if (-not (Test-IsWindowsHost)) { return }
|
||||
try {
|
||||
$a = Get-Acl -LiteralPath $From
|
||||
if (@($a.Access | Where-Object { -not $_.IsInherited }).Count -gt 0 -or $a.AreAccessRulesProtected) {
|
||||
Set-Acl -LiteralPath $To -AclObject $a
|
||||
Info " ACL of $(Split-Path -Leaf $From) copied (it had explicit entries)"
|
||||
}
|
||||
} catch { Warn "could not copy the ACL onto ${To}: $($_.Exception.Message)" }
|
||||
}
|
||||
|
||||
# --- pure helpers ---------------------------------------------------------------------------------------------
|
||||
|
||||
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
|
||||
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
|
||||
function Test-Inside([string]$Path, [string]$Root) {
|
||||
if (-not $Path -or -not $Root) { return $false }
|
||||
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
|
||||
$a -eq $b -or $a.StartsWith("$b/")
|
||||
}
|
||||
function Test-SamePath([string]$A, [string]$B) { $A -and $B -and ((Norm $A).ToLowerInvariant() -eq (Norm $B).ToLowerInvariant()) }
|
||||
function Get-Leaf([string]$p) { if (-not $p) { return $null }; ((Norm $p) -split '/')[-1] }
|
||||
function Get-Stem([string]$p) { $l = Get-Leaf $p; if (-not $l) { return $null }; [IO.Path]::GetFileNameWithoutExtension($l) }
|
||||
|
||||
function Test-ShaMatch([string]$Actual, [string]$Expected) {
|
||||
if (-not $Actual -or -not $Expected) { return $false }
|
||||
$e = $Expected.ToLowerInvariant()
|
||||
$Actual.StartsWith($e) -or $e.StartsWith($Actual)
|
||||
}
|
||||
|
||||
function Get-GatewayProcesses {
|
||||
@(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in @($script:ServerProc, $script:WorkerProc) })
|
||||
function Get-TextHashPrefix([string]$Text) {
|
||||
$sha = [Security.Cryptography.SHA256]::Create()
|
||||
try { (($sha.ComputeHash([Text.Encoding]::UTF8.GetBytes($Text)) | ForEach-Object { $_.ToString('x2') }) -join '').Substring(0, 12) }
|
||||
finally { $sha.Dispose() }
|
||||
}
|
||||
function Get-FileSha([string]$Path) {
|
||||
$sha = [Security.Cryptography.SHA256]::Create(); $fs = [IO.File]::OpenRead($Path)
|
||||
try { (($sha.ComputeHash($fs) | ForEach-Object { $_.ToString('X2') }) -join '') } finally { $fs.Dispose(); $sha.Dispose() }
|
||||
}
|
||||
|
||||
# Store file set: db + -wal + -shm
|
||||
function Get-StoreFiles([string]$Db) { @($Db, "$Db-wal", "$Db-shm") | Where-Object { Test-Path -LiteralPath $_ } }
|
||||
function Test-SensitiveName([string]$Name) { ($Name -match 'KEY|PEPPER|PASS|SECRET|TOKEN|PWD|CREDENTIAL') -and ($Name -notmatch '(Path|Source|EnvVarName|Name)$') }
|
||||
|
||||
# name -> value. Every sensitive value is registered for redaction.
|
||||
function ConvertTo-EnvMap([string[]]$Lines) {
|
||||
$map = [ordered]@{}
|
||||
foreach ($line in $Lines) {
|
||||
$i = $line.IndexOf('=')
|
||||
if ($i -le 0) { continue }
|
||||
$k = $line.Substring(0, $i).Trim(); $v = $line.Substring($i + 1)
|
||||
$map[$k] = $v
|
||||
if ((Test-SensitiveName $k) -and $v.Length -ge 4 -and -not $script:Redact.Contains($v)) { $script:Redact.Add($v) }
|
||||
}
|
||||
$map
|
||||
}
|
||||
|
||||
# Names + per-value SHA-256 prefixes. Never carries a value.
|
||||
function Get-EnvFingerprint([string[]]$Lines) {
|
||||
$map = ConvertTo-EnvMap $Lines
|
||||
$entries = [ordered]@{}
|
||||
foreach ($k in $map.Keys) { $entries[$k] = Get-TextHashPrefix ([string]$map[$k]) }
|
||||
[pscustomobject]@{ Count = @($Lines).Count; Names = @($map.Keys); Entries = $entries; BlockHash = (Get-TextHashPrefix (@($Lines) -join "`n")) }
|
||||
}
|
||||
|
||||
# List of differences (names only). Empty when identical.
|
||||
function Compare-EnvFingerprint($Before, $After) {
|
||||
$d = New-Object System.Collections.Generic.List[string]
|
||||
foreach ($k in $Before.Entries.Keys) {
|
||||
if (-not $After.Entries.Contains($k)) { $d.Add("removed $k") }
|
||||
elseif ($After.Entries[$k] -ne $Before.Entries[$k]) { $d.Add("changed $k (sha256 $($Before.Entries[$k]) -> $($After.Entries[$k]))") }
|
||||
}
|
||||
foreach ($k in $After.Entries.Keys) { if (-not $Before.Entries.Contains($k)) { $d.Add("added $k") } }
|
||||
if ($d.Count -eq 0 -and $Before.BlockHash -ne $After.BlockHash) { $d.Add("block hash $($Before.BlockHash) -> $($After.BlockHash) (order, duplicates or blank entries changed)") }
|
||||
, $d.ToArray()
|
||||
}
|
||||
|
||||
function Format-EnvFingerprint($Fp) { "$($Fp.Count) entries, block sha256 $($Fp.BlockHash): " + (($Fp.Entries.Keys | ForEach-Object { "$_=$($Fp.Entries[$_])" }) -join ', ') }
|
||||
|
||||
# Re-read the env block and abort on any difference from the preflight fingerprint.
|
||||
function Assert-EnvUnchanged($ctx, [string]$When) {
|
||||
$now = Get-EnvFingerprint (Read-EnvBlockLines)
|
||||
$diff = Compare-EnvFingerprint $ctx.EnvFp $now
|
||||
if ($diff.Count -gt 0) {
|
||||
# During an automatic rollback the difference is already the reported failure; restoring the old build still goes ahead.
|
||||
if ($script:EnvWarnOnly) { Warn "NSSM env block changed ${When}: $($diff -join '; ') (not written by this script; restore it from the backup dir's AppEnvironmentExtra.txt if unintended)"; return }
|
||||
Fail "NSSM env block changed ${When}: $($diff -join '; ')"
|
||||
}
|
||||
Info "env block unchanged $When ($($now.Count) entries, block sha256 $($now.BlockHash))"
|
||||
}
|
||||
|
||||
# Directory listings via .NET (FileInfo/DirectoryInfo, hidden files included). Get-ChildItem costs ~250 ms per call
|
||||
# under pwsh on macOS (per-file owner lookups), which made the dry-run harness 20x slower; on Windows both are fine.
|
||||
function Get-Files([string]$Dir, [string]$Filter = '*', [switch]$Recurse) {
|
||||
if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return }
|
||||
$o = if ($Recurse) { [IO.SearchOption]::AllDirectories } else { [IO.SearchOption]::TopDirectoryOnly }
|
||||
try { (New-Object IO.DirectoryInfo $Dir).GetFiles($Filter, $o) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" }
|
||||
}
|
||||
function Get-Dirs([string]$Dir, [string]$Filter = '*') {
|
||||
if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return }
|
||||
try { (New-Object IO.DirectoryInfo $Dir).GetDirectories($Filter) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" }
|
||||
}
|
||||
|
||||
function Get-StoreFiles([string]$Db) { @($Db, "$Db-wal", "$Db-shm") | Where-Object { $_ -and (Test-Path -LiteralPath $_) } }
|
||||
|
||||
function Get-DirSize([string]$p) {
|
||||
if (-not (Test-Path -LiteralPath $p)) { return 0 }
|
||||
$s = (Get-ChildItem -LiteralPath $p -Recurse -File -Force -ErrorAction SilentlyContinue | Measure-Object Length -Sum).Sum
|
||||
$s = (@(Get-Files $p -Recurse) | Measure-Object Length -Sum).Sum
|
||||
if ($s) { [double]$s } else { 0 }
|
||||
}
|
||||
|
||||
# Version of a self-contained .NET runtime in a directory (from mscordaccore_<arch>_<arch>_<ver>.dll), or $null (framework-dependent).
|
||||
function Get-BundledRuntimeVersion([string]$Dir) {
|
||||
$f = @(Get-Files $Dir 'mscordaccore_*.dll')
|
||||
foreach ($x in $f) { if ($x.Name -match '_(\d+\.\d+\.\d+(\.\d+)?)\.dll$') { return [version]$Matches[1] } }
|
||||
$null
|
||||
}
|
||||
|
||||
# --- processes --------------------------------------------------------------------------------------------------
|
||||
|
||||
function Test-GatewayServer($p, $ctx) { ($script:ServerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.Application) }
|
||||
function Test-GatewayWorker($p, $ctx) { ($script:WorkerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.WorkerExe) }
|
||||
|
||||
function Get-GatewayProcesses($ctx) {
|
||||
@(Get-ProcessTable | Where-Object { (Test-GatewayServer $_ $ctx) -or (Test-GatewayWorker $_ $ctx) } | ForEach-Object {
|
||||
$role = if (Test-GatewayServer $_ $ctx) { 'Server' } else { 'Worker' }
|
||||
$_ | Add-Member -NotePropertyName Role -NotePropertyValue $role -PassThru -Force
|
||||
})
|
||||
}
|
||||
function Format-Procs($procs) { (@($procs) | ForEach-Object { "$($_.Name)#$($_.Id)<-$($_.ParentId) [$(if ($_.Path) { $_.Path } else { 'path unreadable' })]" }) -join ', ' }
|
||||
|
||||
function Assert-NoGatewayProcess($ctx, [string]$When) {
|
||||
$left = @(Get-GatewayProcesses $ctx)
|
||||
if ($left.Count -gt 0) { Fail "${When}: gateway process(es) still alive: $(Format-Procs $left)" }
|
||||
}
|
||||
|
||||
# The one-gateway guard. Fails closed: a RUNNING service whose gateway cannot be seen is an abort.
|
||||
function Assert-SingleGateway($ctx, [string]$State) {
|
||||
$procs = @(Get-GatewayProcesses $ctx)
|
||||
$servers = @($procs | Where-Object { $_.Role -eq 'Server' })
|
||||
$workers = @($procs | Where-Object { $_.Role -eq 'Worker' })
|
||||
Info "gateway processes on the box (images $($script:ServerImages -join '|') / $($script:WorkerImages -join '|')): $(if ($procs.Count) { Format-Procs $procs } else { 'none' })"
|
||||
if ($servers.Count -gt 1) { Fail "$($servers.Count) gateway Server processes are running ($(Format-Procs $servers)). A second gateway is the OrphanWorkerTerminator trap - resolve manually." }
|
||||
switch ($State) {
|
||||
'SERVICE_STOPPED' {
|
||||
if ($procs.Count -gt 0) { Fail "service is STOPPED but gateway process(es) exist ($(Format-Procs $procs)): a rogue second gateway or an unclean stop. Starting would terminate those workers." }
|
||||
}
|
||||
'SERVICE_RUNNING' {
|
||||
if ($servers.Count -eq 0) {
|
||||
Fail "service is RUNNING but no gateway Server process was found (looked for image names $($script:ServerImages -join ', ') and path $($ctx.Application)). The process guards cannot see this gateway, so they would pass vacuously; refusing."
|
||||
}
|
||||
$s = $servers[0]
|
||||
if ($s.Path -and -not (Test-SamePath $s.Path $ctx.Application)) { Fail "the running gateway is $($s.Path), not the service's Application $($ctx.Application): a second gateway is running and the service's own is not" }
|
||||
if (-not $s.Path) { Warn "gateway Server #$($s.Id) path is unreadable; matched by image name only" }
|
||||
$hostPid = Get-ServiceHostPid
|
||||
if ($hostPid) {
|
||||
if ($s.ParentId -ne $hostPid) { Fail "gateway Server #$($s.Id) is not a child of the $ServiceName service host #${hostPid} (parent #$($s.ParentId)): it is another gateway" }
|
||||
Info "gateway Server #$($s.Id) is the service's own (child of service host #$hostPid)"
|
||||
} else { Warn 'service host PID unavailable; parent check skipped' }
|
||||
$foreign = @($workers | Where-Object { $_.ParentId -and $_.ParentId -ne $s.Id })
|
||||
if ($foreign.Count -gt 0) { Warn "Worker process(es) not owned by the service's gateway: $(Format-Procs $foreign). They are killed as orphans on the next start." }
|
||||
$wrongPath = @($workers | Where-Object { $_.Path -and $ctx.WorkerExe -and -not (Test-SamePath $_.Path $ctx.WorkerExe) })
|
||||
if ($wrongPath.Count -gt 0) { Warn "Worker process(es) not running the configured $($ctx.WorkerExe): $(Format-Procs $wrongPath)" }
|
||||
}
|
||||
'SERVICE_STOP_PENDING' { Warn 'service is STOP_PENDING; the stop step will finish it' }
|
||||
default {
|
||||
if ($servers.Count -gt 0) { Fail "a gateway Server process exists while the service is $State" }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# --- context discovery (read only) -------------------------------------------------------------------------
|
||||
|
||||
function Get-Context {
|
||||
@@ -158,25 +395,47 @@ function Get-Context {
|
||||
$ctx.State = $state
|
||||
$ctx.Application = Get-NssmValue 'Application'
|
||||
$ctx.AppDirectory = Get-NssmValue 'AppDirectory'
|
||||
$ctx.EnvText = Get-NssmValue 'AppEnvironmentExtra'
|
||||
$ctx.Env = Parse-EnvBlock $ctx.EnvText
|
||||
$ctx.EnvCount = $ctx.Env.Count
|
||||
if (-not $ServerDir) { $ctx.ServerDir = $ctx.AppDirectory } else { $ctx.ServerDir = $ServerDir }
|
||||
$wexe = $ctx.Env['MxGateway__Worker__ExecutablePath']
|
||||
$lines = @(Read-EnvBlockLines)
|
||||
$ctx.Env = ConvertTo-EnvMap $lines
|
||||
$ctx.EnvFp = Get-EnvFingerprint $lines
|
||||
$ctx.EnvCount = $ctx.EnvFp.Count
|
||||
if ($ServerDir) { $ctx.ServerDir = $ServerDir } else { $ctx.ServerDir = $ctx.AppDirectory }
|
||||
$wexe = $ctx.Env[$script:EnvWorkerExe]
|
||||
$ctx.WorkerExe = $wexe
|
||||
if ($WorkerDir) { $ctx.WorkerDir = $WorkerDir }
|
||||
elseif ($wexe) { $ctx.WorkerDir = Split-Path -Parent $wexe }
|
||||
else { Fail 'cannot determine the Worker directory: env MxGateway__Worker__ExecutablePath is not in the NSSM env block; pass -WorkerDir' }
|
||||
else { Fail "cannot determine the Worker directory: env $($script:EnvWorkerExe) is not in the NSSM env block; pass -WorkerDir" }
|
||||
$pd = 'C:\ProgramData\MxGateway'
|
||||
$auth = $ctx.Env['MxGateway__Authentication__SqlitePath']; if (-not $auth) { $auth = "$pd\gateway-auth.db" }
|
||||
$sec = $ctx.Env['Secrets__SqlitePath']; if (-not $sec) { $sec = "$pd\mxgateway-secrets.db" }
|
||||
$ctx.AuthDb = $auth; $ctx.SecretsDb = $sec
|
||||
$ctx.ServerExeName = Split-Path -Leaf $ctx.Application
|
||||
$ctx.KekFile = $ctx.Env['Secrets__MasterKey__FilePath']
|
||||
$ctx.ServerExeName = Get-Leaf $ctx.Application
|
||||
$ctx.WorkerExeName = Get-Leaf $wexe
|
||||
if (-not $ctx.WorkerExeName) { $ctx.WorkerExeName = 'ZB.MOM.WW.MxGateway.Worker.exe' }
|
||||
# Process image names: the installed ones first, then both known schemes.
|
||||
$script:ServerImages = @(@(Get-Stem $ctx.Application) + $script:KnownServerImages | Where-Object { $_ } | Select-Object -Unique)
|
||||
$script:WorkerImages = @(@(Get-Stem $wexe) + $script:KnownWorkerImages | Where-Object { $_ } | Select-Object -Unique)
|
||||
[pscustomobject]$ctx
|
||||
}
|
||||
|
||||
# --- preflight -----------------------------------------------------------------------------------------------
|
||||
|
||||
function Test-MasterKey($ctx) {
|
||||
$src = $ctx.Env['Secrets__MasterKey__Source']
|
||||
$envVar = $ctx.Env['Secrets__MasterKey__EnvVarName']; if (-not $envVar) { $envVar = 'ZB_SECRETS_MASTER_KEY' }
|
||||
if ($ctx.KekFile -or $src -eq 'File') {
|
||||
if (-not $ctx.KekFile) { Fail 'Secrets__MasterKey__Source=File but no Secrets__MasterKey__FilePath in the env block' }
|
||||
if (-not (Test-Rooted $ctx.KekFile)) { Fail "master key file '$($ctx.KekFile)' is not absolute" }
|
||||
foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) { if (Test-Inside $ctx.KekFile $root) { Fail "master key file '$($ctx.KekFile)' is INSIDE '$root' - the swap would lose it" } }
|
||||
if (Test-Path -LiteralPath $ctx.KekFile) { Info "master key: file $($ctx.KekFile) (present, outside the swap)" } else { Fail "master key file '$($ctx.KekFile)' does not exist (the Secrets store would be unreadable)" }
|
||||
} elseif ($ctx.Env.Contains($envVar)) {
|
||||
Info "master key: env block entry $envVar"
|
||||
} else {
|
||||
Warn "the env block carries no master key ($envVar or Secrets__MasterKey__FilePath); it comes from elsewhere (appsettings or the account environment). The upgrade does not change that, but confirm the Secrets store opens after the start."
|
||||
}
|
||||
}
|
||||
|
||||
function Invoke-Preflight($ctx) {
|
||||
Say '== Preflight =='
|
||||
Info "service '$ServiceName' state: $($ctx.State)"
|
||||
@@ -186,20 +445,19 @@ function Invoke-Preflight($ctx) {
|
||||
Info "Worker dir : $($ctx.WorkerDir) (exe env: $($ctx.WorkerExe))"
|
||||
Info "API-key DB : $($ctx.AuthDb)"
|
||||
Info "Secrets DB : $($ctx.SecretsDb)"
|
||||
Info "NSSM env block: $($ctx.EnvCount) entries (names only): $(($ctx.Env.Keys) -join ', ')"
|
||||
Info "NSSM env block fingerprint (names + value sha256 prefixes, no values): $(Format-EnvFingerprint $ctx.EnvFp)"
|
||||
|
||||
if ($ctx.EnvCount -eq 0) { Fail 'NSSM env block is empty - refusing: it holds Kestrel/store paths, the KEK and the pepper' }
|
||||
foreach ($need in 'ZB_SECRETS_MASTER_KEY') {
|
||||
if (-not $ctx.Env.Contains($need)) { Fail "env block has no $need (the Secrets store would be unreadable after the restart)" }
|
||||
}
|
||||
if ($ctx.EnvCount -eq 0) { Fail 'NSSM env block is empty - refusing: it holds Kestrel/store paths, the master key and the pepper' }
|
||||
if (-not ($ctx.Env.Keys | Where-Object { $_ -match 'Pepper' })) {
|
||||
Warn 'no *Pepper* entry in the env block; confirm the API-key pepper is supplied another way or issued tokens will not verify'
|
||||
}
|
||||
if ($ctx.Env['ASPNETCORE_ENVIRONMENT'] -ne 'Production' -and $ctx.Env['DOTNET_ENVIRONMENT'] -ne 'Production') {
|
||||
Warn 'neither ASPNETCORE_ENVIRONMENT nor DOTNET_ENVIRONMENT is exactly Production (runbook trap 6) - do not change it during an upgrade'
|
||||
}
|
||||
if (-not (Test-SamePath (Split-Path -Parent $ctx.Application) $ctx.ServerDir)) { Fail "NSSM Application '$($ctx.Application)' is not inside the Server dir '$($ctx.ServerDir)'" }
|
||||
if ($ctx.WorkerExe -and -not (Test-SamePath (Split-Path -Parent $ctx.WorkerExe) $ctx.WorkerDir)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' is not inside the Worker dir '$($ctx.WorkerDir)'" }
|
||||
|
||||
# State that must survive: rooted and outside the content root.
|
||||
# State that must survive: rooted and outside the swapped directories.
|
||||
foreach ($pair in @(@('API-key DB', $ctx.AuthDb), @('Secrets DB', $ctx.SecretsDb))) {
|
||||
if (-not (Test-Rooted $pair[1])) { Fail "$($pair[0]) path '$($pair[1])' is not absolute" }
|
||||
foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) {
|
||||
@@ -208,6 +466,7 @@ function Invoke-Preflight($ctx) {
|
||||
if (Test-Path -LiteralPath $pair[1]) { Info "$($pair[0]) present ($((Get-StoreFiles $pair[1]).Count) file(s) incl. wal/shm)" }
|
||||
else { Warn "$($pair[0]) '$($pair[1])' does not exist yet (fresh store will be created on start)" }
|
||||
}
|
||||
Test-MasterKey $ctx
|
||||
|
||||
# Directories
|
||||
foreach ($d in @($ctx.ServerDir, $ctx.WorkerDir)) {
|
||||
@@ -216,33 +475,34 @@ function Invoke-Preflight($ctx) {
|
||||
if ((Test-Inside $ctx.WorkerDir $ctx.ServerDir) -or (Test-Inside $ctx.ServerDir $ctx.WorkerDir)) {
|
||||
Fail 'Server and Worker directories are nested; the rename-in-place swap needs sibling directories'
|
||||
}
|
||||
if (-not (Test-Path -LiteralPath $ctx.Application)) { Fail "NSSM Application '$($ctx.Application)' does not exist" }
|
||||
if ($ctx.WorkerExe -and -not (Test-Path -LiteralPath $ctx.WorkerExe)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' does not exist" }
|
||||
$dbs = @(@(Get-Files $ctx.ServerDir -Recurse) + @(Get-Files $ctx.WorkerDir -Recurse) | Where-Object { $_.Name -match '\.db(-wal|-shm)?$' })
|
||||
if ($dbs.Count -gt 0) { Fail "SQLite file(s) inside the swapped directories would be orphaned by the rename: $(($dbs | ForEach-Object { $_.FullName }) -join ', ')" }
|
||||
|
||||
# One-gateway guard
|
||||
$procs = Get-GatewayProcesses
|
||||
$servers = @($procs | Where-Object { $_.ProcessName -eq $script:ServerProc })
|
||||
if ($servers.Count -gt 1) { Fail "$($servers.Count) gateway Server processes are running (PIDs $(($servers.Id) -join ',')). A second gateway is the OrphanWorkerTerminator trap - resolve manually." }
|
||||
if ($ctx.State -eq 'SERVICE_STOPPED' -and $procs.Count -gt 0) {
|
||||
Fail "service is STOPPED but gateway process(es) exist (PIDs $(($procs.Id) -join ',')): a rogue second gateway. Starting would terminate its live workers."
|
||||
}
|
||||
if ($servers.Count -eq 1 -and $ctx.State -ne 'SERVICE_RUNNING' -and $ctx.State -ne 'SERVICE_STOP_PENDING') {
|
||||
Fail "a Server process exists while the service is $($ctx.State)"
|
||||
}
|
||||
if (Get-Command Get-CimInstance -ErrorAction SilentlyContinue) {
|
||||
$svcs = @(Get-CimInstance Win32_Service | Where-Object { $_.Name -ne $ServiceName -and $_.PathName -and $_.PathName -like "*$($ctx.ServerExeName)*" })
|
||||
if ($svcs.Count -gt 0) { Fail "another service ($(($svcs.Name) -join ', ')) also launches $($ctx.ServerExeName); two gateways on one box is forbidden" }
|
||||
} else { Info 'Win32_Service cross-check unavailable on this host (skipped)' }
|
||||
# One-gateway guard (fail closed)
|
||||
Assert-SingleGateway $ctx $ctx.State
|
||||
$others = Get-OtherGatewayServices $ctx.ServerExeName
|
||||
if ($null -eq $others) { Info 'Win32_Service cross-check unavailable on this host (skipped)' }
|
||||
elseif (@($others).Count -gt 0) { Fail "another service ($(@($others) -join ', ')) also launches $($ctx.ServerExeName); two gateways on one box is forbidden" }
|
||||
else { Info "no other service launches $($ctx.ServerExeName)" }
|
||||
|
||||
if ($Rollback) { return }
|
||||
if ($Rollback) { Say '== Preflight OK =='; return }
|
||||
|
||||
# Sources
|
||||
if (-not $ServerSource -or -not $WorkerSource) { Fail '-ServerSource and -WorkerSource are required for an upgrade' }
|
||||
foreach ($s in @(@($ServerSource, $ctx.ServerExeName), @($WorkerSource, "$($script:WorkerProc).exe"))) {
|
||||
if (-not (Test-Path -LiteralPath (Join-Path $s[0] $s[1]))) { Fail "staged source '$($s[0])' does not contain $($s[1])" }
|
||||
$stagedServerExe = Join-Path $ServerSource $ctx.ServerExeName
|
||||
$stagedWorkerExe = Join-Path $WorkerSource $ctx.WorkerExeName
|
||||
foreach ($s in @(@($ServerSource, $stagedServerExe), @($WorkerSource, $stagedWorkerExe))) {
|
||||
if (-not (Test-Path -LiteralPath $s[1])) { Fail "staged source '$($s[0])' does not contain $(Get-Leaf $s[1]) (the installed exe name; NSSM and $($script:EnvWorkerExe) point at it)" }
|
||||
}
|
||||
if ((Test-Inside $ServerSource $ctx.ServerDir) -or (Test-Inside $WorkerSource $ctx.WorkerDir)) { Fail 'staging directory is inside a live directory' }
|
||||
$sSha = Get-ProductSha (Join-Path $ServerSource $ctx.ServerExeName)
|
||||
$wSha = Get-ProductSha (Join-Path $WorkerSource "$($script:WorkerProc).exe")
|
||||
Info "staged Server SHA: $sSha ; staged Worker SHA: $wSha"
|
||||
if ((Test-Inside $ServerSource $ctx.ServerDir) -or (Test-Inside $WorkerSource $ctx.WorkerDir) -or (Test-Inside $ServerSource $ctx.WorkerDir) -or (Test-Inside $WorkerSource $ctx.ServerDir)) { Fail 'staging directory is inside a live directory' }
|
||||
$arch = Get-ExeArchitecture $stagedWorkerExe
|
||||
if ($arch -ne 'X86') { Fail "staged Worker $stagedWorkerExe architecture is '$arch', not X86 (trap 5)" }
|
||||
Info "staged Worker architecture: X86"
|
||||
$sSha = Get-ProductSha $stagedServerExe
|
||||
$wSha = Get-ProductSha $stagedWorkerExe
|
||||
Info "staged Server SHA: $sSha ; staged Worker SHA: $wSha ; live Server SHA: $(Get-ProductSha $ctx.Application)"
|
||||
if ($sSha -and $wSha -and -not (Test-ShaMatch $sSha $wSha)) { Fail "Server ($sSha) and Worker ($wSha) were built from different commits (wire protocol risk)" }
|
||||
if ($ExpectedSha) {
|
||||
if (-not (Test-ShaMatch $sSha $ExpectedSha)) {
|
||||
@@ -250,6 +510,27 @@ function Invoke-Preflight($ctx) {
|
||||
else { Fail "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha'" }
|
||||
}
|
||||
} else { Warn 'no -ExpectedSha given: the post-deploy SHA check will only record the SHA' }
|
||||
$leafSha = if ((Get-Leaf $ctx.ServerDir) -match '([0-9a-f]{7,40})$') { $Matches[1] } else { $null }
|
||||
if ($leafSha -and $sSha -and -not (Test-ShaMatch $leafSha $sSha)) { Warn "the Server dir name '$(Get-Leaf $ctx.ServerDir)' carries SHA $leafSha but will hold $sSha after the rename-in-place swap" }
|
||||
|
||||
# Runtime: never silently downgrade a self-contained Server.
|
||||
$liveRt = Get-BundledRuntimeVersion $ctx.ServerDir; $newRt = Get-BundledRuntimeVersion $ServerSource
|
||||
Info "bundled .NET runtime: live $(if ($liveRt) { $liveRt } else { 'none (framework-dependent)' }), staged $(if ($newRt) { $newRt } else { 'none (framework-dependent)' })"
|
||||
if ($liveRt -and -not $newRt) { Warn 'live Server is self-contained, staged is framework-dependent: the box must have a matching .NET 10 runtime installed' }
|
||||
if ($liveRt -and $newRt -and $newRt -lt $liveRt) {
|
||||
if ($AllowRuntimeDowngrade) { Warn "staged runtime $newRt is OLDER than live $liveRt (-AllowRuntimeDowngrade)" }
|
||||
else { Fail "staged self-contained runtime $newRt is OLDER than the live $liveRt; rebuild with a newer SDK or pass -AllowRuntimeDowngrade" }
|
||||
}
|
||||
|
||||
# What the swap will carry over / drop (top-level files).
|
||||
foreach ($pair in @(@('Server', $ctx.ServerDir, $ServerSource), @('Worker', $ctx.WorkerDir, $WorkerSource))) {
|
||||
$plan = Get-CarryPlan $pair[1] $pair[2] ($pair[0] -eq 'Server' -and -not $TakeNewAppSettings)
|
||||
Info "$($pair[0]): carry from live -> $(if ($plan.Carry.Count) { $plan.Carry -join ', ' } else { 'nothing' }); live-only binaries dropped -> $(if ($plan.Dropped.Count) { $plan.Dropped -join ', ' } else { 'none' })"
|
||||
if ($plan.Dirs.Count) { Warn "$($pair[0]): live-only directories NOT carried (they stay in the .bak): $($plan.Dirs -join ', ')" }
|
||||
}
|
||||
$lw = @(Get-Files $ctx.WorkerDir | ForEach-Object { $_.Name }); $nw = @(Get-Files $WorkerSource | ForEach-Object { $_.Name })
|
||||
$onlyNew = @($nw | Where-Object { $lw -notcontains $_ })
|
||||
if ($onlyNew.Count) { Info "Worker: files only in the staged build: $($onlyNew -join ', ')" }
|
||||
|
||||
# Disk space: 2x (server + worker + stores + payload), at least MinFreeGB
|
||||
if (-not $SkipFreeSpaceCheck) {
|
||||
@@ -272,52 +553,67 @@ function Wait-State([string]$Want, [int]$TimeoutSec) {
|
||||
$false
|
||||
}
|
||||
|
||||
function Assert-NoGatewayProcess([string]$When) {
|
||||
$left = Get-GatewayProcesses
|
||||
if ($left.Count -gt 0) { Fail "${When}: gateway process(es) still alive: $(($left | ForEach-Object { "$($_.ProcessName)#$($_.Id)" }) -join ', ')" }
|
||||
function Wait-NoGatewayProcess($ctx, [int]$TimeoutSec) {
|
||||
$deadline = (Get-Date).AddSeconds($TimeoutSec)
|
||||
do { if (@(Get-GatewayProcesses $ctx).Count -eq 0) { return $true }; Start-Sleep -Seconds 1 } while ((Get-Date) -lt $deadline)
|
||||
$false
|
||||
}
|
||||
|
||||
function Stop-Gateway {
|
||||
# Kill only this install's gateway processes: path under the Server/Worker dir, or (path unreadable) the installed image name.
|
||||
function Stop-InstallProcesses($ctx, [string]$Why) {
|
||||
$installed = @((Get-Stem $ctx.Application), (Get-Stem $ctx.WorkerExe)) | Where-Object { $_ }
|
||||
$mine = @(Get-GatewayProcesses $ctx | Where-Object {
|
||||
if ($_.Path) { (Test-Inside $_.Path $ctx.ServerDir) -or (Test-Inside $_.Path $ctx.WorkerDir) } else { $installed -contains $_.Name }
|
||||
} | Sort-Object { if ($_.Role -eq 'Server') { 0 } else { 1 } })
|
||||
foreach ($p in $mine) { Say "[ACTION] Stop-Process -Id $($p.Id) -Force ($($p.Name), $Why)"; Stop-ProcessById $p.Id }
|
||||
$mine.Count
|
||||
}
|
||||
|
||||
function Stop-Gateway($ctx) {
|
||||
Say '-- Stop --'
|
||||
$state = Get-ServiceState
|
||||
if ($state -eq 'SERVICE_STOPPED') { Info 'already stopped' }
|
||||
else {
|
||||
Step "nssm stop $ServiceName (wait up to $StopTimeoutSeconds s for SERVICE_STOPPED)" {
|
||||
Step "nssm stop $ServiceName (wait up to $StopTimeoutSeconds s for SERVICE_STOPPED, then for every gateway/Worker process to exit)" {
|
||||
[void](Invoke-Nssm -NssmArgs @('stop', $ServiceName) -AllowFail)
|
||||
if (-not (Wait-State 'SERVICE_STOPPED' $StopTimeoutSeconds)) {
|
||||
# STOP_PENDING trap: workers did not exit in time. Kill the Server (then Workers) by image name -
|
||||
# safe because preflight proved there is exactly one gateway on the box.
|
||||
Warn "service still $(Get-ServiceState) after $StopTimeoutSeconds s (STOP_PENDING trap) - killing gateway processes"
|
||||
foreach ($n in @($script:ServerProc, $script:WorkerProc)) {
|
||||
Get-Process -Name $n -ErrorAction SilentlyContinue | ForEach-Object { Say "[ACTION] Stop-Process -Id $($_.Id) -Force ($n)"; Stop-Process -Id $_.Id -Force }
|
||||
}
|
||||
# STOP_PENDING trap: workers did not exit in time. Kill this install's Server, then its Workers.
|
||||
Warn "service still $(Get-ServiceState) after $StopTimeoutSeconds s (STOP_PENDING trap) - killing this install's gateway processes"
|
||||
$n = Stop-InstallProcesses $ctx 'STOP_PENDING'
|
||||
if ($n -eq 0) { Warn 'no process of this install was found to kill' }
|
||||
if (-not (Wait-State 'SERVICE_STOPPED' 20)) {
|
||||
Warn 'still not STOPPED; killing the NSSM service host process'
|
||||
try {
|
||||
$svcPid = (Get-CimInstance Win32_Service -Filter "Name='$ServiceName'").ProcessId
|
||||
if ($svcPid) { Stop-Process -Id $svcPid -Force }
|
||||
} catch { Warn "could not kill the service host: $_" }
|
||||
$svcPid = Get-ServiceHostPid
|
||||
if ($svcPid) { Say "[ACTION] Stop-Process -Id $svcPid -Force (service host)"; Stop-ProcessById $svcPid }
|
||||
if (-not (Wait-State 'SERVICE_STOPPED' 20)) { Fail "service '$ServiceName' will not stop; investigate manually (state $(Get-ServiceState))" }
|
||||
}
|
||||
}
|
||||
Start-Sleep -Seconds 2
|
||||
Assert-NoGatewayProcess 'after stop'
|
||||
if (-not (Wait-NoGatewayProcess $ctx 20)) {
|
||||
Warn "gateway/Worker process(es) outlived the service stop (unclean stop): $(Format-Procs (Get-GatewayProcesses $ctx))"
|
||||
[void](Stop-InstallProcesses $ctx 'left over after the stop')
|
||||
[void](Wait-NoGatewayProcess $ctx 10)
|
||||
}
|
||||
}
|
||||
}
|
||||
if (-not $script:Dry) { Assert-NoGatewayProcess 'after stop' }
|
||||
if (-not $script:Dry) {
|
||||
Assert-NoGatewayProcess $ctx 'after stop'
|
||||
Info 'stopped; no gateway/Worker process on the box'
|
||||
Assert-EnvUnchanged $ctx 'after the stop'
|
||||
}
|
||||
}
|
||||
|
||||
function Start-Gateway($ctx) {
|
||||
Say '-- Start --'
|
||||
if (-not $script:Dry) { Assert-NoGatewayProcess 'before start (OrphanWorkerTerminator guard)' }
|
||||
else { Say '[DRYRUN] assert no Server/Worker process exists (OrphanWorkerTerminator guard)' }
|
||||
if (-not $script:Dry) {
|
||||
Assert-NoGatewayProcess $ctx 'before start (OrphanWorkerTerminator guard)'
|
||||
Assert-EnvUnchanged $ctx 'before the start'
|
||||
} else { Say "[DRYRUN] assert no process named $($script:ServerImages -join '|') / $($script:WorkerImages -join '|') exists, and the env block fingerprint is unchanged (OrphanWorkerTerminator guard)" }
|
||||
Step "nssm start $ServiceName (once)" {
|
||||
[void](Invoke-Nssm -NssmArgs @('start', $ServiceName))
|
||||
}
|
||||
Step "wait for SERVICE_RUNNING and compare env entry count (expect $($ctx.EnvCount))" {
|
||||
Step "wait for SERVICE_RUNNING and compare the env block fingerprint" {
|
||||
if (-not (Wait-State 'SERVICE_RUNNING' 60)) { Fail "service did not reach RUNNING (state $(Get-ServiceState))" }
|
||||
$now = (Parse-EnvBlock (Get-NssmValue 'AppEnvironmentExtra')).Count
|
||||
if ($now -ne $ctx.EnvCount) { Fail "NSSM env block entry count changed: $($ctx.EnvCount) -> $now" }
|
||||
Assert-EnvUnchanged $ctx 'after the start'
|
||||
}
|
||||
}
|
||||
|
||||
@@ -329,94 +625,157 @@ function Backup-State($ctx) {
|
||||
Say '-- Backup (service stopped) --'
|
||||
$suffix = Get-Suffix
|
||||
$root = Split-Path -Parent $ctx.ServerDir
|
||||
$envFile = Join-Path $root "_env-AppEnvironmentExtra-$($script:Stamp).txt"
|
||||
Step "save the NSSM env block + Application/AppDirectory to $envFile (contains secrets: keep it ACL-restricted)" {
|
||||
$body = @("Application=$($ctx.Application)", "AppDirectory=$($ctx.AppDirectory)", '--- AppEnvironmentExtra ---', $ctx.EnvText)
|
||||
Set-Content -LiteralPath $envFile -Value $body -Encoding UTF8
|
||||
try { & icacls.exe $envFile /inheritance:r /grant:r 'SYSTEM:(F)' 'BUILTIN\Administrators:(F)' | Out-Null } catch { Warn "could not restrict ACL on ${envFile}: $_" }
|
||||
$dir = Join-Path $root "_deploy-backup-$suffix"
|
||||
$script:BackupDir = $dir
|
||||
Step "create $dir, restrict it to SYSTEM + Administrators, then save the env block (secrets), service registry key and env fingerprint" {
|
||||
New-Item -ItemType Directory -Path $dir -Force | Out-Null
|
||||
Protect-Directory $dir
|
||||
$body = @("Application=$($ctx.Application)", "AppDirectory=$($ctx.AppDirectory)", '--- AppEnvironmentExtra ---') + @(Read-EnvBlockLines)
|
||||
Set-Content -LiteralPath (Join-Path $dir 'AppEnvironmentExtra.txt') -Value $body -Encoding UTF8
|
||||
Export-ServiceKey (Join-Path $dir "$ServiceName-servicekey.reg")
|
||||
$fp = @("BlockHash=$($ctx.EnvFp.BlockHash)") + @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" })
|
||||
Set-Content -LiteralPath (Join-Path $dir 'env-fingerprint.txt') -Value $fp -Encoding ASCII
|
||||
}
|
||||
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
|
||||
foreach ($f in (Get-StoreFiles $db)) {
|
||||
$dst = "$f.bak-$suffix"
|
||||
Step "copy store file $f -> $dst (live file is never moved)" { Copy-Item -LiteralPath $f -Destination $dst -Force }
|
||||
Step "copy store file $f -> $dst (live file is never moved)" {
|
||||
Copy-Item -LiteralPath $f -Destination $dst -Force
|
||||
if ((Get-FileSha $f) -ne (Get-FileSha $dst)) { Fail "store backup hash mismatch for $f" }
|
||||
}
|
||||
}
|
||||
if (-not (Test-Path -LiteralPath $db)) { Info "no store at $db to back up" }
|
||||
}
|
||||
if ($ctx.KekFile -and (Test-Path -LiteralPath $ctx.KekFile)) {
|
||||
Step "copy master key file $($ctx.KekFile) into $dir (live file is never moved)" { Copy-Item -LiteralPath $ctx.KekFile -Destination (Join-Path $dir (Get-Leaf $ctx.KekFile)) -Force }
|
||||
}
|
||||
}
|
||||
|
||||
function Test-Preserved([string]$Name) { foreach ($p in $script:PreservePatterns) { if ($Name -like $p) { return $true } }; $false }
|
||||
|
||||
# Which top-level files of the live dir go into the new one. Carry = names copied from live (live wins).
|
||||
function Get-CarryPlan([string]$Live, [string]$Source, [bool]$KeepSettings) {
|
||||
$liveFiles = @(Get-Files $Live)
|
||||
$srcNames = @(Get-Files $Source | ForEach-Object { $_.Name })
|
||||
$srcDirs = @(Get-Dirs $Source | ForEach-Object { $_.Name })
|
||||
$carry = New-Object System.Collections.Generic.List[string]; $dropped = New-Object System.Collections.Generic.List[string]
|
||||
foreach ($f in $liveFiles) {
|
||||
$inSource = $srcNames -contains $f.Name
|
||||
if ($f.Name -like 'appsettings*.json') { if ($KeepSettings) { $carry.Add($f.Name) }; continue }
|
||||
if (Test-Preserved $f.Name) { $carry.Add($f.Name); continue }
|
||||
if ($inSource) { continue }
|
||||
if ($script:BinaryExtensions -contains $f.Extension.ToLowerInvariant()) { $dropped.Add($f.Name) } else { $carry.Add($f.Name) }
|
||||
}
|
||||
$dirs = @(Get-Dirs $Live | Where-Object { $srcDirs -notcontains $_.Name } | ForEach-Object { $_.Name })
|
||||
[pscustomobject]@{ Carry = $carry.ToArray(); Dropped = $dropped.ToArray(); Dirs = $dirs }
|
||||
}
|
||||
|
||||
function Swap-Directory([string]$Live, [string]$Source, [string]$Name, [bool]$KeepSettings) {
|
||||
$bak = "$Live.bak-$(Get-Suffix)"
|
||||
$plan = Get-CarryPlan $Live $Source $KeepSettings
|
||||
$hashes = @{}
|
||||
foreach ($f in $plan.Carry) { $hashes[$f] = Get-FileSha (Join-Path $Live $f) }
|
||||
$sddlOld = Get-DirSddl $Live
|
||||
Step "rename $Name dir: $Live -> $bak" { Rename-Item -LiteralPath $Live -NewName (Split-Path -Leaf $bak) }
|
||||
Step "copy new ${Name}: $Source -> $Live" { Copy-Item -LiteralPath $Source -Destination $Live -Recurse -Force }
|
||||
if ($KeepSettings) {
|
||||
$from = if ($script:Dry) { $Live } else { $bak } # in a dry run nothing was renamed yet
|
||||
foreach ($f in 'appsettings.json', 'appsettings.Development.json') {
|
||||
if (Test-Path -LiteralPath (Join-Path $from $f)) {
|
||||
Step "restore live $f byte-identical from $bak" { Copy-Item -LiteralPath (Join-Path $bak $f) -Destination (Join-Path $Live $f) -Force }
|
||||
} else { Info "$f not present in the old $Name dir (nothing to carry over)" }
|
||||
foreach ($f in $plan.Carry) {
|
||||
Step "carry live $Name\$f forward from $bak (sha256 $($hashes[$f].Substring(0, 12)))" {
|
||||
Copy-Item -LiteralPath (Join-Path $bak $f) -Destination (Join-Path $Live $f) -Force
|
||||
if ((Get-FileSha (Join-Path $Live $f)) -ne $hashes[$f]) { Fail "carried $Name\$f does not match the live copy" }
|
||||
Copy-ExplicitAcl (Join-Path $bak $f) (Join-Path $Live $f)
|
||||
}
|
||||
}
|
||||
foreach ($f in $plan.Dropped) { Info "$Name\$f exists only in the old build (binary) and is not carried; it stays in $bak" }
|
||||
if (-not $script:Dry) {
|
||||
$sddlNew = Get-DirSddl $Live
|
||||
if ($sddlOld -and $sddlNew -and $sddlOld -ne $sddlNew) { Warn "$Name dir ACL differs from the old one (new inherits from the parent). old: $sddlOld new: $sddlNew" }
|
||||
}
|
||||
$bak
|
||||
}
|
||||
|
||||
# --- verification ----------------------------------------------------------------------------------------------------
|
||||
|
||||
function Invoke-Verify($ctx) {
|
||||
function Invoke-Verify($ctx, [bool]$AfterRollback = $false) {
|
||||
Say '== Verify =='
|
||||
if ($script:Dry) {
|
||||
Say "[DRYRUN] GET $HealthBaseUrl/health/ready until 200 (<= $ReadyTimeoutSeconds s)"
|
||||
Say "[DRYRUN] GET $HealthBaseUrl/healthz ; read ProductVersion SHA of $(Join-Path $ctx.ServerDir $ctx.ServerExeName)$(if ($ExpectedSha) { " and require it to match $ExpectedSha" })"
|
||||
Say "[DRYRUN] service must stay SERVICE_RUNNING for $StableSeconds s with env entry count $($ctx.EnvCount)"
|
||||
Say "[DRYRUN] GET $HealthBaseUrl/healthz ; read ProductVersion SHA of $($ctx.Application)$(if ($ExpectedSha -and -not $Rollback) { " and require it to match $ExpectedSha" })"
|
||||
Say "[DRYRUN] exactly one gateway Server ($($ctx.Application)) on the box, same PID, service SERVICE_RUNNING for $StableSeconds s, env fingerprint unchanged"
|
||||
return
|
||||
}
|
||||
$ready = $false; $deadline = (Get-Date).AddSeconds($ReadyTimeoutSeconds)
|
||||
while (-not $ready -and (Get-Date) -lt $deadline) {
|
||||
try { $r = Invoke-WebRequest -Uri "$HealthBaseUrl/health/ready" -UseBasicParsing -TimeoutSec 10; $ready = ($r.StatusCode -eq 200) }
|
||||
catch { Start-Sleep -Seconds 3 }
|
||||
$ready = ((Get-HttpStatus "$HealthBaseUrl/health/ready") -eq 200)
|
||||
if (-not $ready) { Start-Sleep -Seconds 3 }
|
||||
}
|
||||
if (-not $ready) { Fail "$HealthBaseUrl/health/ready did not return 200 within $ReadyTimeoutSeconds s" }
|
||||
Info '/health/ready = 200'
|
||||
try { Info "/healthz = $((Invoke-WebRequest -Uri "$HealthBaseUrl/healthz" -UseBasicParsing -TimeoutSec 10).StatusCode)" } catch { Fail "/healthz failed: $_" }
|
||||
$hz = Get-HttpStatus "$HealthBaseUrl/healthz"
|
||||
if ($hz -ne 200) { Fail "/healthz = $hz" }
|
||||
Info '/healthz = 200'
|
||||
|
||||
$sha = Get-ProductSha (Join-Path $ctx.ServerDir $ctx.ServerExeName)
|
||||
$sha = Get-ProductSha $ctx.Application
|
||||
Info "deployed Server SHA (ProductVersion): $sha"
|
||||
if ($ExpectedSha -and -not $Rollback -and -not (Test-ShaMatch $sha $ExpectedSha)) { Fail "deployed SHA '$sha' != expected '$ExpectedSha'" }
|
||||
if ($ExpectedSha -and -not $Rollback -and -not $AfterRollback -and -not (Test-ShaMatch $sha $ExpectedSha)) { Fail "deployed SHA '$sha' != expected '$ExpectedSha'" }
|
||||
|
||||
Info "watching the service for $StableSeconds s"
|
||||
Assert-SingleGateway $ctx 'SERVICE_RUNNING'
|
||||
$pid0 = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' })[0].Id
|
||||
Info "watching the service for $StableSeconds s (gateway #$pid0)"
|
||||
$until = (Get-Date).AddSeconds($StableSeconds)
|
||||
while ((Get-Date) -lt $until) {
|
||||
do {
|
||||
if ((Get-ServiceState) -ne 'SERVICE_RUNNING') { Fail 'service left RUNNING during the stability window (restart loop?)' }
|
||||
Start-Sleep -Seconds 5
|
||||
}
|
||||
$n = (Parse-EnvBlock (Get-NssmValue 'AppEnvironmentExtra')).Count
|
||||
if ($n -ne $ctx.EnvCount) { Fail "env block entry count $n != preflight $($ctx.EnvCount)" }
|
||||
Info "env block still $n entries"
|
||||
} while ((Get-Date) -lt $until)
|
||||
$servers = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' })
|
||||
if ($servers.Count -ne 1 -or $servers[0].Id -ne $pid0) { Fail "gateway process changed during the stability window: #$pid0 -> $(Format-Procs $servers) (restart loop or second gateway)" }
|
||||
Assert-EnvUnchanged $ctx 'after the stability window'
|
||||
Warn 'health is green with zero sessions; still prove the Worker path: open a session with a PRE-EXISTING API key and read a known tag (runbook 7.3)'
|
||||
}
|
||||
|
||||
# --- rollback ------------------------------------------------------------------------------------------------------------
|
||||
|
||||
function Find-Backup([string]$Live, [string]$Stamp) {
|
||||
$leaf = Split-Path -Leaf $Live; $parent = Split-Path -Parent $Live
|
||||
$cands = @(Get-ChildItem -LiteralPath $parent -Directory -Filter "$leaf.bak-*" -ErrorAction SilentlyContinue)
|
||||
if ($Stamp) { $cands = @($cands | Where-Object { $_.Name -like "$leaf.bak-$Stamp*" }) }
|
||||
$cands | Sort-Object Name -Descending | Select-Object -First 1
|
||||
# Backup dirs come in two spellings: <leaf>.bak-yyyyMMdd-HHmmss[-label] (this script) and <leaf>.bak.yyyyMMddTHHmmss (manual / adapted scripts).
|
||||
function Get-BackupKey([string]$Name) {
|
||||
if ($Name -match '\.bak[-.](\d{8})[-T]?(\d{6})') { return "$($Matches[1])$($Matches[2])" }
|
||||
$null
|
||||
}
|
||||
|
||||
function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores) {
|
||||
function Find-Backup([string]$Live, [string]$Stamp) {
|
||||
$leaf = Split-Path -Leaf $Live; $parent = Split-Path -Parent $Live
|
||||
$cands = @(Get-Dirs $parent | Where-Object { ($_.Name -like "$leaf.bak-*" -or $_.Name -like "$leaf.bak.*") -and (Get-BackupKey $_.Name) })
|
||||
if ($Stamp) {
|
||||
$key = ($Stamp -replace '[^0-9]', '')
|
||||
$cands = @($cands | Where-Object { (Get-BackupKey $_.Name) -eq $key -or $_.Name -like "$leaf.bak-$Stamp*" -or $_.Name -like "$leaf.bak.$Stamp*" })
|
||||
}
|
||||
$cands | Sort-Object { Get-BackupKey $_.Name } -Descending | Select-Object -First 1
|
||||
}
|
||||
|
||||
function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores, [bool]$AllowPartial) {
|
||||
Say '== Rollback =='
|
||||
$sb = Find-Backup $ctx.ServerDir $Stamp
|
||||
$wb = Find-Backup $ctx.WorkerDir $Stamp
|
||||
if (-not $sb -or -not $wb) { Fail "no matching Server/Worker backup directories found (stamp '$Stamp')" }
|
||||
Info "rolling back to Server=$($sb.Name) Worker=$($wb.Name)"
|
||||
Stop-Gateway
|
||||
if (-not $sb -and -not $AllowPartial) { Fail "no Server backup directory found (stamp '$Stamp')" }
|
||||
$key = if ($sb) { Get-BackupKey $sb.Name } else { ($Stamp -replace '[^0-9]', '') }
|
||||
$wb = Find-Backup $ctx.WorkerDir $key
|
||||
if ($wb -and (Get-BackupKey $wb.Name) -ne $key) { $wb = $null }
|
||||
if (-not $wb -and -not $AllowPartial) { Fail "no Worker backup matching the Server backup $($sb.Name) (stamp key $key); refusing a mixed rollback" }
|
||||
if (-not $sb -and -not $wb) { Fail "nothing to roll back to (stamp '$Stamp')" }
|
||||
Info "rolling back to Server=$(if ($sb) { $sb.Name } else { '(unchanged)' }) Worker=$(if ($wb) { $wb.Name } else { '(unchanged)' })"
|
||||
$fpFile = @(Get-Dirs (Split-Path -Parent $ctx.ServerDir) '_deploy-backup-*' | Where-Object { ($_.Name -replace '[^0-9]', '').StartsWith($key) } | ForEach-Object { Join-Path $_.FullName 'env-fingerprint.txt' } | Where-Object { Test-Path -LiteralPath $_ })
|
||||
if ($fpFile.Count) {
|
||||
$saved = Get-Content -LiteralPath $fpFile[0] | Where-Object { $_ -notlike 'BlockHash=*' }
|
||||
$now = @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" })
|
||||
if (($saved -join '|') -ne ($now -join '|')) { Warn "the env block differs from the one saved before the upgrade ($($fpFile[0])); restore it from AppEnvironmentExtra.txt there if that was not intended (runbook section 8 step 4)" }
|
||||
else { Info "env block matches the pre-upgrade fingerprint in $($fpFile[0])" }
|
||||
}
|
||||
Stop-Gateway $ctx
|
||||
foreach ($pair in @(@($ctx.ServerDir, $sb), @($ctx.WorkerDir, $wb))) {
|
||||
if (-not $pair[1]) { Info "$($pair[0]) has no backup for this stamp; left unchanged"; continue }
|
||||
$failed = "$($pair[0]).failed-$($script:Stamp)"
|
||||
if (Test-Path -LiteralPath $pair[0]) { Step "rename $($pair[0]) -> $failed" { Rename-Item -LiteralPath $pair[0] -NewName (Split-Path -Leaf $failed) } }
|
||||
Step "rename $($pair[1].FullName) -> $($pair[0])" { Rename-Item -LiteralPath $pair[1].FullName -NewName (Split-Path -Leaf $pair[0]) }
|
||||
}
|
||||
if ($Stores) {
|
||||
$bstamp = ($sb.Name -replace '^.*\.bak-', '')
|
||||
$bstamp = ($sb.Name -replace '^.*\.bak[-.]', '')
|
||||
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
|
||||
foreach ($ext in '', '-wal', '-shm') {
|
||||
$live = "$db$ext"; $bak = "$db$ext.bak-$bstamp"
|
||||
@@ -434,57 +793,75 @@ function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores) {
|
||||
|
||||
# --- main -------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
$exit = 0
|
||||
try {
|
||||
Say "Upgrade-MxGatewayService stamp=$($script:Stamp) mode=$(if ($Rollback) {'ROLLBACK'} else {'UPGRADE'}) dry=$($script:Dry)"
|
||||
$ctx = Get-Context
|
||||
Invoke-Preflight $ctx
|
||||
|
||||
if (-not $script:Dry) {
|
||||
$lockPath = Join-Path (Split-Path -Parent $ctx.ServerDir) '.mxgw-deploy.lock'
|
||||
try { $script:LockStream = [IO.File]::Open($lockPath, 'OpenOrCreate', 'ReadWrite', 'None') }
|
||||
catch { Fail "another deploy run holds $lockPath" }
|
||||
}
|
||||
|
||||
if ($Rollback) {
|
||||
Invoke-RollbackSwap $ctx $BackupStamp $RestoreStores.IsPresent
|
||||
Invoke-Verify $ctx
|
||||
} else {
|
||||
Stop-Gateway
|
||||
Backup-State $ctx
|
||||
$swapped = $false
|
||||
try {
|
||||
$null = Swap-Directory $ctx.ServerDir $ServerSource 'Server' (-not $TakeNewAppSettings)
|
||||
$null = Swap-Directory $ctx.WorkerDir $WorkerSource 'Worker' $false
|
||||
$swapped = $true
|
||||
Start-Gateway $ctx
|
||||
Invoke-Verify $ctx
|
||||
} catch {
|
||||
$err = $_
|
||||
Say "[FAIL] $($err.Exception.Message)"
|
||||
if ($script:Dry -or $NoAutoRollback) { throw }
|
||||
Warn 'upgrade failed - rolling back automatically'
|
||||
try {
|
||||
if (-not $swapped) { Warn 'swap did not complete; rollback swaps whatever .bak sets exist for this stamp' }
|
||||
Invoke-RollbackSwap $ctx $script:Stamp $false
|
||||
Invoke-Verify $ctx
|
||||
Warn 'ROLLED BACK to the previous build; original failure follows'
|
||||
} catch { Warn "AUTOMATIC ROLLBACK FAILED: $($_.Exception.Message). Manual intervention required (backups: *.bak-$(Get-Suffix))." }
|
||||
throw $err
|
||||
}
|
||||
}
|
||||
Say '== DONE =='
|
||||
} catch {
|
||||
$exit = 1
|
||||
Write-Host "ERROR: $($_.Exception.Message)" -ForegroundColor Red
|
||||
} finally {
|
||||
if ($script:LockStream) { $script:LockStream.Dispose() }
|
||||
function Invoke-Main {
|
||||
$exit = 0
|
||||
$ctx = $null
|
||||
try {
|
||||
if (-not $script:Dry -and $ctx) {
|
||||
$notes = Join-Path (Split-Path -Parent $ctx.ServerDir) "_DEPLOY-NOTES-$($script:Stamp).txt"
|
||||
Set-Content -LiteralPath $notes -Value $script:Notes -Encoding UTF8
|
||||
Write-Host "deploy notes: $notes"
|
||||
Say "Upgrade-MxGatewayService stamp=$($script:Stamp) mode=$(if ($Rollback) {'ROLLBACK'} else {'UPGRADE'}) dry=$($script:Dry) host=$env:COMPUTERNAME"
|
||||
$ctx = Get-Context
|
||||
Invoke-Preflight $ctx
|
||||
|
||||
if (-not $script:Dry) {
|
||||
$lockPath = Join-Path (Split-Path -Parent $ctx.ServerDir) '.mxgw-deploy.lock'
|
||||
try { $script:LockStream = [IO.File]::Open($lockPath, 'OpenOrCreate', 'ReadWrite', 'None') }
|
||||
catch { Fail "another deploy run holds $lockPath" }
|
||||
}
|
||||
} catch { }
|
||||
|
||||
if ($Rollback) {
|
||||
Invoke-RollbackSwap $ctx $BackupStamp $RestoreStores.IsPresent $false
|
||||
Invoke-Verify $ctx
|
||||
} else {
|
||||
Stop-Gateway $ctx
|
||||
try { Backup-State $ctx }
|
||||
catch {
|
||||
$err = $_
|
||||
if (-not $script:Dry) {
|
||||
Warn 'backup failed before anything was swapped - starting the unchanged build again'
|
||||
try { Start-Gateway $ctx } catch { Warn "restart of the unchanged build FAILED: $($_.Exception.Message)" }
|
||||
}
|
||||
throw $err
|
||||
}
|
||||
$touched = $false
|
||||
try {
|
||||
$touched = $true
|
||||
$null = Swap-Directory $ctx.ServerDir $ServerSource 'Server' (-not $TakeNewAppSettings)
|
||||
$null = Swap-Directory $ctx.WorkerDir $WorkerSource 'Worker' $false
|
||||
if (-not $script:Dry) { Assert-EnvUnchanged $ctx 'after the swap' }
|
||||
Start-Gateway $ctx
|
||||
Invoke-Verify $ctx
|
||||
} catch {
|
||||
$err = $_
|
||||
Say "[FAIL] $(Hide $err.Exception.Message)"
|
||||
if ($script:Dry -or $NoAutoRollback -or -not $touched) { throw }
|
||||
Warn 'upgrade failed - rolling back automatically'
|
||||
$script:EnvWarnOnly = $true
|
||||
try {
|
||||
Invoke-RollbackSwap $ctx $script:Stamp $false $true
|
||||
Invoke-Verify $ctx $true
|
||||
Warn 'ROLLED BACK to the previous build; original failure follows'
|
||||
} catch { Warn "AUTOMATIC ROLLBACK FAILED: $($_.Exception.Message). Manual intervention required (backups: *.bak-$(Get-Suffix))." }
|
||||
throw $err
|
||||
}
|
||||
}
|
||||
Say '== DONE =='
|
||||
} catch {
|
||||
$exit = 1
|
||||
$m = Hide $_.Exception.Message
|
||||
$script:Notes.Add("ERROR: $m")
|
||||
Write-Host "ERROR: $m" -ForegroundColor Red
|
||||
} finally {
|
||||
if ($script:LockStream) { $script:LockStream.Dispose(); $script:LockStream = $null }
|
||||
try {
|
||||
if (-not $script:Dry -and $ctx) {
|
||||
$notes = Join-Path (Split-Path -Parent $ctx.ServerDir) "_DEPLOY-NOTES-$($script:Stamp).txt"
|
||||
Set-Content -LiteralPath $notes -Value $script:Notes -Encoding UTF8
|
||||
Write-Host "deploy notes: $notes"
|
||||
}
|
||||
} catch { }
|
||||
$script:Redact.Clear()
|
||||
}
|
||||
$exit
|
||||
}
|
||||
exit $exit
|
||||
|
||||
# Dot-sourcing (the dry-run harness) only defines the functions.
|
||||
if ($MyInvocation.InvocationName -ne '.') { $rc = Invoke-Main | Select-Object -Last 1; exit $rc }
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Mutation check for Test-UpgradeMxGatewayService.ps1 (mxaccessgw#230): re-introduces each guarded defect into a
|
||||
temp copy of Upgrade-MxGatewayService.ps1 and requires the harness to FAIL on it. A mutation the harness does not
|
||||
catch means an assertion is vacuous. Exits non-zero if any mutation survives (or does not apply).
|
||||
|
||||
.EXAMPLE
|
||||
pwsh -NoProfile -File scripts/deploy/tests/Invoke-UpgradeMutations.ps1
|
||||
#>
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$here = Split-Path -Parent $MyInvocation.MyCommand.Path
|
||||
$src = Get-Content -Raw (Join-Path (Split-Path -Parent $here) 'Upgrade-MxGatewayService.ps1')
|
||||
$harness = Join-Path $here 'Test-UpgradeMxGatewayService.ps1'
|
||||
|
||||
$mutations = [ordered]@{
|
||||
# The #230 defect: only the ZB.MOM.WW.* names, nothing derived from the installed exes.
|
||||
'only ZB.MOM.WW.* image names (#230 original)' = @(
|
||||
@("`$script:KnownServerImages = @('ZB.MOM.WW.MxGateway.Server', 'MxGateway.Server')", "`$script:KnownServerImages = @('ZB.MOM.WW.MxGateway.Server')"),
|
||||
@("`$script:KnownWorkerImages = @('ZB.MOM.WW.MxGateway.Worker', 'MxGateway.Worker')", "`$script:KnownWorkerImages = @('ZB.MOM.WW.MxGateway.Worker')"),
|
||||
@('$script:ServerImages = @(@(Get-Stem $ctx.Application) + $script:KnownServerImages', '$script:ServerImages = @(@() + $script:KnownServerImages'),
|
||||
@('$script:WorkerImages = @(@(Get-Stem $wexe) + $script:KnownWorkerImages', '$script:WorkerImages = @(@() + $script:KnownWorkerImages'),
|
||||
@('function Test-GatewayServer($p, $ctx) { ($script:ServerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.Application) }', 'function Test-GatewayServer($p, $ctx) { ($script:ServerImages -contains $p.Name) }'),
|
||||
@('function Test-GatewayWorker($p, $ctx) { ($script:WorkerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.WorkerExe) }', 'function Test-GatewayWorker($p, $ctx) { ($script:WorkerImages -contains $p.Name) }'))
|
||||
'guard passes when no gateway is visible' = @(
|
||||
@(' if ($servers.Count -eq 0) {', ' if ($false) {'))
|
||||
'other naming scheme not matched' = @(
|
||||
@("`$script:KnownServerImages = @('ZB.MOM.WW.MxGateway.Server', 'MxGateway.Server')", "`$script:KnownServerImages = @()"),
|
||||
@("`$script:KnownWorkerImages = @('ZB.MOM.WW.MxGateway.Worker', 'MxGateway.Worker')", "`$script:KnownWorkerImages = @()"))
|
||||
'no parent check' = @(
|
||||
@('if ($s.ParentId -ne $hostPid) { Fail', 'if ($false) { Fail'))
|
||||
'pfx and live-only files not carried (#230 original)' = @(
|
||||
@(' if (Test-Preserved $f.Name) { $carry.Add($f.Name); continue }', ''),
|
||||
@('if ($script:BinaryExtensions -contains $f.Extension.ToLowerInvariant()) { $dropped.Add($f.Name) } else { $carry.Add($f.Name) }', 'if ($script:BinaryExtensions -contains $f.Extension.ToLowerInvariant()) { $dropped.Add($f.Name) }'))
|
||||
'env check is a no-op' = @(
|
||||
@(' $diff = Compare-EnvFingerprint $ctx.EnvFp $now', ' $diff = @()'))
|
||||
'env compare looks at names only' = @(
|
||||
@('elseif ($After.Entries[$k] -ne $Before.Entries[$k])', 'elseif ($false)'),
|
||||
@('if ($d.Count -eq 0 -and $Before.BlockHash -ne $After.BlockHash)', 'if ($false)'))
|
||||
'values not redacted' = @(
|
||||
@('foreach ($s in $script:Redact) { if ($s) { $m = $m.Replace($s, ''<redacted>'') } }', ''),
|
||||
@('function Format-EnvFingerprint($Fp) { "$($Fp.Count) entries, block sha256 $($Fp.BlockHash): " + (($Fp.Entries.Keys | ForEach-Object { "$_=$($Fp.Entries[$_])" }) -join '', '') }', 'function Format-EnvFingerprint($Fp) { "$($Fp.Count) entries: " + ((ConvertTo-EnvMap (Read-EnvBlockLines)).GetEnumerator() | ForEach-Object { "$($_.Key)=$($_.Value)" }) -join '', '' }'))
|
||||
'STOP_PENDING kill by any name' = @(
|
||||
@('if ($_.Path) { (Test-Inside $_.Path $ctx.ServerDir) -or (Test-Inside $_.Path $ctx.WorkerDir) } else { $installed -contains $_.Name }', '$true'),
|
||||
@(' $mine = @(Get-GatewayProcesses $ctx | Where-Object {', ' $mine = @(Get-ProcessTable | Where-Object {'))
|
||||
'runtime downgrade allowed' = @(
|
||||
@('if ($liveRt -and $newRt -and $newRt -lt $liveRt) {', 'if ($false) {'))
|
||||
'mixed rollback allowed' = @(
|
||||
@('if ($wb -and (Get-BackupKey $wb.Name) -ne $key) { $wb = $null }', ''),
|
||||
@(' $wb = Find-Backup $ctx.WorkerDir $key', ' $wb = Find-Backup $ctx.WorkerDir $null'))
|
||||
'.bak.<stamp> sets not found' = @(
|
||||
@('($_.Name -like "$leaf.bak-*" -or $_.Name -like "$leaf.bak.*")', '($_.Name -like "$leaf.bak-*")'))
|
||||
}
|
||||
|
||||
$tmp = Join-Path ([IO.Path]::GetTempPath()) ("mxgw230-mut-" + [guid]::NewGuid().ToString('n').Substring(0, 8))
|
||||
New-Item -ItemType Directory -Path $tmp | Out-Null
|
||||
$survivors = 0
|
||||
try {
|
||||
foreach ($name in $mutations.Keys) {
|
||||
$m = $src
|
||||
$pairs = @($mutations[$name])
|
||||
if ($pairs[0] -is [string]) { $pairs = @(, $pairs) } # a single @(@(old, new)) flattens to @(old, new)
|
||||
foreach ($pair in $pairs) {
|
||||
if (-not $m.Contains($pair[0])) { Write-Host "MUTATION DID NOT APPLY: $name -> '$($pair[0])'" -ForegroundColor Red; $survivors++; continue }
|
||||
$m = $m.Replace($pair[0], $pair[1])
|
||||
}
|
||||
$path = Join-Path $tmp 'Upgrade-MxGatewayService.ps1'
|
||||
Set-Content -LiteralPath $path -Value $m -NoNewline
|
||||
$out = @(& pwsh -NoProfile -File $harness -ScriptPath $path 2>&1)
|
||||
$rc = $LASTEXITCODE
|
||||
$fails = @($out | Where-Object { "$_" -match '^\s+FAIL \[' })
|
||||
# Killed only if the harness ran to its verdict and reported failed assertions (a crash is not a kill).
|
||||
if ($rc -ne 0 -and $fails.Count -gt 0 -and ($out -match '^FAILED: ')) { Write-Host ("killed {0,-48} ({1} assertion(s) failed, e.g. {2})" -f $name, $fails.Count, ("$($fails | Select-Object -First 1)".Trim())) }
|
||||
else { Write-Host "SURVIVED $name (rc=$rc, $($fails.Count) failed assertion(s)): $(($out | Select-Object -Last 2) -join ' | ')" -ForegroundColor Red; $survivors++ }
|
||||
}
|
||||
} finally { Remove-Item -LiteralPath $tmp -Recurse -Force }
|
||||
if ($survivors -gt 0) { Write-Host "$survivors mutation(s) survived or did not apply" -ForegroundColor Red; exit 1 }
|
||||
Write-Host "all $($mutations.Count) mutations killed" -ForegroundColor Green
|
||||
@@ -0,0 +1,382 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Dry-run harness for scripts/deploy/Upgrade-MxGatewayService.ps1 (mxaccessgw#230). Runs the script's real logic
|
||||
against a fake box: a temp directory laid out like wonder-app-vd03 (MxGateway.Server.exe / MxGateway.Worker.exe,
|
||||
dashboard.pfx, file-based master key) or like windev (ZB.MOM.WW.MxGateway.*.exe, dated Server dir), with a fake
|
||||
nssm, process table, registry env block, health endpoint and clock (tests/UpgradeFakes.ps1). The repo has no
|
||||
Pester; this is a plain script that exits non-zero when any assertion fails.
|
||||
|
||||
Besides -DryRun scenarios it runs full upgrades and rollbacks for real against the temp directories (renames,
|
||||
copies, carry-over, hash checks, auto-rollback), because only the service manager and processes are faked.
|
||||
|
||||
.PARAMETER ScriptPath The upgrade script under test (default: ../Upgrade-MxGatewayService.ps1). The mutation runner
|
||||
points this at mutated copies to prove the assertions can fail.
|
||||
|
||||
.EXAMPLE
|
||||
pwsh -NoProfile -File scripts/deploy/tests/Test-UpgradeMxGatewayService.ps1
|
||||
#>
|
||||
param([string]$ScriptPath)
|
||||
Set-StrictMode -Off
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$here = Split-Path -Parent $MyInvocation.MyCommand.Path
|
||||
if (-not $ScriptPath) { $ScriptPath = Join-Path (Split-Path -Parent $here) 'Upgrade-MxGatewayService.ps1' }
|
||||
$FakesPath = Join-Path $here 'UpgradeFakes.ps1'
|
||||
|
||||
$script:failures = 0
|
||||
$script:passes = 0
|
||||
$script:group = ''
|
||||
function Assert([bool]$Condition, [string]$Message) {
|
||||
if ($Condition) { $script:passes++ } else { $script:failures++; Write-Host " FAIL [$($script:group)]: $Message" -ForegroundColor Red }
|
||||
}
|
||||
# Not 'Group': that is a built-in alias of Group-Object, and aliases win over functions.
|
||||
# An exception in a section is a failed assertion, and the run still reaches its verdict.
|
||||
trap { $script:failures++; Write-Host " FAIL [$($script:group)]: exception: $($_.Exception.Message)" -ForegroundColor Red; continue }
|
||||
function Section([string]$Name) { $script:group = $Name; Write-Host $Name }
|
||||
|
||||
$Pepper = 'pepper-VALUE-do-not-print-7f3a'
|
||||
$LdapPw = 'ldap-PASSWORD-do-not-print-91c2'
|
||||
$OldSha = '2b13564e'
|
||||
$NewSha = '76a71bc8'
|
||||
|
||||
function New-File([string]$Path, [string]$Content) {
|
||||
[void][IO.Directory]::CreateDirectory((Split-Path -Parent $Path)) # New-Item -Force costs ~70 ms per call on macOS
|
||||
[IO.File]::WriteAllText($Path, $Content)
|
||||
}
|
||||
|
||||
# Scheme 'wonder' = vd03 (MxGateway.*.exe, dashboard.pfx in Server\, file KEK in data\); 'windev' = ZB.MOM.WW.* in a dated dir.
|
||||
function New-FakeBox([string]$Scheme) {
|
||||
$root = Join-Path ([IO.Path]::GetTempPath()) ("mxgw230-" + [guid]::NewGuid().ToString('n').Substring(0, 8))
|
||||
if ($Scheme -eq 'wonder') { $srvName = 'MxGateway.Server.exe'; $wrkName = 'MxGateway.Worker.exe'; $srvDir = "$root/Server" }
|
||||
else { $srvName = 'ZB.MOM.WW.MxGateway.Server.exe'; $wrkName = 'ZB.MOM.WW.MxGateway.Worker.exe'; $srvDir = "$root/Server-20261112-$OldSha" }
|
||||
$wrkDir = "$root/Worker"
|
||||
# live install
|
||||
New-File "$srvDir/$srvName" "sha=$OldSha;arch=Amd64"
|
||||
New-File "$srvDir/ZB.MOM.WW.MxGateway.Server.dll" "old server dll $OldSha"
|
||||
New-File "$srvDir/Old.Removed.Dependency.dll" 'binary only in the old build'
|
||||
New-File "$srvDir/appsettings.json" '{ "live": "host config with no pepper" }'
|
||||
New-File "$srvDir/appsettings.Development.json" '{ "live": "dev overlay" }'
|
||||
New-File "$srvDir/mscordaccore_amd64_amd64_10.0.1426.12345.dll" 'runtime 10.0.14'
|
||||
if ($Scheme -eq 'wonder') { New-File "$srvDir/dashboard.pfx" 'PFX-BYTES-live-dashboard-cert' }
|
||||
New-File "$srvDir/notes-left-by-an-operator.txt" 'live-only text file'
|
||||
[void][IO.Directory]::CreateDirectory("$srvDir/logs")
|
||||
New-File "$wrkDir/$wrkName" "sha=$OldSha;arch=X86"
|
||||
New-File "$wrkDir/$wrkName.config" '<configuration old/>'
|
||||
New-File "$root/data/gateway-auth.db" 'auth-db'; New-File "$root/data/gateway-auth.db-wal" 'auth-wal'
|
||||
New-File "$root/data/mxgateway-secrets.db" 'secrets-db'; New-File "$root/data/kek.key" 'KEK-BYTES'
|
||||
# staged new build (the repo appsettings.json, no pfx)
|
||||
New-File "$root/stage/Server/$srvName" "sha=$NewSha;arch=Amd64"
|
||||
New-File "$root/stage/Server/ZB.MOM.WW.MxGateway.Server.dll" "new server dll $NewSha"
|
||||
New-File "$root/stage/Server/appsettings.json" '{ "repo": "default config" }'
|
||||
New-File "$root/stage/Server/mscordaccore_amd64_amd64_10.0.1426.12345.dll" 'runtime 10.0.14'
|
||||
New-File "$root/stage/Worker/$wrkName" "sha=$NewSha;arch=X86"
|
||||
New-File "$root/stage/Worker/$wrkName.config" '<configuration new/>'
|
||||
|
||||
$app = "$srvDir/$srvName"; $wexe = "$wrkDir/$wrkName"
|
||||
$env = @(
|
||||
'ASPNETCORE_ENVIRONMENT=Production',
|
||||
"MxGateway__Worker__ExecutablePath=$wexe",
|
||||
"MxGateway__Authentication__SqlitePath=$root/data/gateway-auth.db",
|
||||
"Secrets__SqlitePath=$root/data/mxgateway-secrets.db",
|
||||
"MxGateway__ApiKeyPepper=$Pepper",
|
||||
"MxGateway__Ldap__ServiceAccountPassword=$LdapPw")
|
||||
if ($Scheme -eq 'wonder') { $env += @("Secrets__MasterKey__FilePath=$root/data/kek.key", 'Secrets__MasterKey__Source=File') }
|
||||
$hostPid = 4100
|
||||
$box = @{
|
||||
Scheme = $Scheme; Root = $root; ServerDir = $srvDir; WorkerDir = $wrkDir; Application = $app; AppDirectory = $srvDir; WorkerExe = $wexe
|
||||
State = 'SERVICE_RUNNING'; Env = $env; HostPid = $hostPid; Health = 200; StopHangs = $false; Starts = 0; OnStart = $null
|
||||
OtherServices = @(); Survivors = @()
|
||||
Procs = @(
|
||||
[pscustomobject]@{ Id = 4200; ParentId = $hostPid; Name = [IO.Path]::GetFileNameWithoutExtension($srvName); Path = $app; Role = 'Server' },
|
||||
[pscustomobject]@{ Id = 4201; ParentId = 4200; Name = [IO.Path]::GetFileNameWithoutExtension($wrkName); Path = $wexe; Role = 'Worker' },
|
||||
[pscustomobject]@{ Id = 4202; ParentId = 4200; Name = [IO.Path]::GetFileNameWithoutExtension($wrkName); Path = $wexe; Role = 'Worker' },
|
||||
[pscustomobject]@{ Id = 900; ParentId = 1; Name = 'svchost'; Path = 'C:/Windows/System32/svchost.exe'; Role = 'Other' })
|
||||
NssmCalls = (New-Object System.Collections.Generic.List[string]); Killed = (New-Object System.Collections.Generic.List[int])
|
||||
HttpCalls = (New-Object System.Collections.Generic.List[string]); Protected = (New-Object System.Collections.Generic.List[string])
|
||||
StartedWith = (New-Object System.Collections.Generic.List[string])
|
||||
}
|
||||
$box
|
||||
}
|
||||
|
||||
function Get-TreeHash([string]$Root) {
|
||||
$d = New-Object IO.DirectoryInfo $Root
|
||||
((@($d.GetDirectories('*', 'AllDirectories') | ForEach-Object { "D $($_.FullName)" }) + @($d.GetFiles('*', 'AllDirectories') | ForEach-Object { "F $($_.FullName) $((Get-FileHash -LiteralPath $_.FullName).Hash)" })) | Sort-Object) -join "`n"
|
||||
}
|
||||
|
||||
# Runs Invoke-Main of the script under test against $Box with the fakes loaded. Returns rc + all output text.
|
||||
function Invoke-Scenario([hashtable]$Box, [hashtable]$P) {
|
||||
$script:Box = $Box
|
||||
$script:FakeNow = [datetime]'2026-11-12T12:00:00'
|
||||
$base = @{ ServerSource = "$($Box.Root)/stage/Server"; WorkerSource = "$($Box.Root)/stage/Worker"; ExpectedSha = $NewSha; MinFreeGB = 0.001; StableSeconds = 10 }
|
||||
foreach ($k in $P.Keys) { $base[$k] = $P[$k] }
|
||||
foreach ($k in @($base.Keys)) { if ($null -eq $base[$k]) { $base.Remove($k) } }
|
||||
. $ScriptPath @base
|
||||
. $FakesPath
|
||||
$records = @(Invoke-Main 6>&1 3>&1)
|
||||
$rc = @($records | Where-Object { $_ -is [int] })[-1]
|
||||
$text = (@($records | Where-Object { $_ -isnot [int] } | ForEach-Object { "$_" }) -join "`n")
|
||||
[pscustomobject]@{ Rc = $rc; Text = $text }
|
||||
}
|
||||
|
||||
function Assert-NoSecrets([string]$Text, [string]$Where) {
|
||||
Assert (-not $Text.Contains($Pepper)) "the pepper value does not appear in $Where"
|
||||
Assert (-not $Text.Contains($LdapPw)) "the LDAP password does not appear in $Where"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------------------------------------------
|
||||
foreach ($scheme in 'wonder', 'windev') {
|
||||
Section "Dry run on a $scheme layout sees the live gateway and changes nothing"
|
||||
$b = New-FakeBox $scheme
|
||||
$before = Get-TreeHash $b.Root
|
||||
$r = Invoke-Scenario $b @{ DryRun = $true }
|
||||
Assert ($r.Rc -eq 0) "dry run exits 0 (rc=$($r.Rc)) $(if ($r.Rc -ne 0) { $r.Text })"
|
||||
$stem = [IO.Path]::GetFileNameWithoutExtension($b.Application)
|
||||
Assert ($r.Text -match "gateway Server #4200 is the service's own") 'the guard found the service gateway by PID/parent'
|
||||
Assert ($r.Text -match [regex]::Escape("$stem#4200")) "the process listing shows $stem#4200"
|
||||
Assert ($r.Text -match "MxGateway.Worker.exe#4201|$([regex]::Escape([IO.Path]::GetFileNameWithoutExtension($b.WorkerExe)))#4201") 'the Worker processes are seen'
|
||||
Assert ((Get-TreeHash $b.Root) -eq $before) 'no file or directory changed'
|
||||
Assert (@($b.NssmCalls | Where-Object { $_ -match '^(stop|start|set)' }).Count -eq 0) "no nssm stop/start/set ($($b.NssmCalls -join '; '))"
|
||||
Assert ($b.Killed.Count -eq 0) 'no process killed'
|
||||
Assert ($r.Text -match '\[DRYRUN\] nssm stop') 'the stop is printed, not run'
|
||||
Assert ($r.Text -match 'no other service launches') 'the other-service cross-check ran (an empty result is not "unavailable")'
|
||||
Assert ($r.Text -match 'carry from live -> [^\r\n]*appsettings\.json') 'plan carries appsettings.json'
|
||||
Assert ($r.Text -match 'carry from live -> [^\r\n]*notes-left-by-an-operator\.txt') 'plan carries a live-only text file'
|
||||
Assert ($r.Text -match 'dropped -> Old\.Removed\.Dependency\.dll') 'plan drops a live-only binary'
|
||||
Assert ($r.Text -match 'live-only directories NOT carried[^\r\n]*logs') 'plan warns about the live-only logs dir'
|
||||
Assert ($r.Text -match 'MxGateway__ApiKeyPepper=[0-9a-f]{12}') 'the env fingerprint lists the pepper by name + hash prefix'
|
||||
Assert-NoSecrets $r.Text 'the dry-run output'
|
||||
if ($scheme -eq 'wonder') {
|
||||
Assert ($r.Text -match 'carry from live -> [^\r\n]*dashboard\.pfx') 'plan carries dashboard.pfx'
|
||||
Assert ($r.Text -match 'master key: file') 'file-based master key accepted'
|
||||
} else {
|
||||
Assert ($r.Text -match 'carries no master key') 'missing env master key is a warning on windev, not an abort'
|
||||
Assert ($r.Text -match 'carries SHA 2b13564e but will hold 76a71bc8') 'dated dir name mismatch is flagged'
|
||||
}
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
}
|
||||
|
||||
Section '-WhatIf is the same as -DryRun'
|
||||
$b = New-FakeBox 'windev'
|
||||
$before = Get-TreeHash $b.Root
|
||||
$r = Invoke-Scenario $b @{ WhatIf = $true }
|
||||
Assert ($r.Rc -eq 0 -and $r.Text -match 'dry=True' -and $r.Text -match '== DONE ==') "-WhatIf runs the whole plan (rc=$($r.Rc))"
|
||||
Assert ($r.Text -match '\[DRYRUN\] carry live Server\\appsettings\.json forward[^\r\n]*sha256 [0-9A-F]{12}') 'carry-over hashes are computed under -WhatIf'
|
||||
Assert ((Get-TreeHash $b.Root) -eq $before -and $b.Starts -eq 0) 'nothing changed'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'Fail closed: service RUNNING but no gateway process visible'
|
||||
$b = New-FakeBox 'wonder'
|
||||
$b.Procs = @($b.Procs | Where-Object { $_.Role -eq 'Other' })
|
||||
$r = Invoke-Scenario $b @{ DryRun = $true }
|
||||
Assert ($r.Rc -eq 1) 'refuses (rc 1)'
|
||||
Assert ($r.Text -match 'RUNNING but no gateway Server process was found') "says why ($($r.Text -split "`n" | Select-String ERROR))"
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'Fail closed: the gateway runs under an image name and path the script does not know'
|
||||
$b = New-FakeBox 'wonder'
|
||||
foreach ($p in $b.Procs) { if ($p.Role -ne 'Other') { $p.Name = 'Renamed.Gateway'; $p.Path = $null } }
|
||||
$r = Invoke-Scenario $b @{ DryRun = $true }
|
||||
Assert ($r.Rc -eq 1) 'refuses (rc 1)'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'Path match works when the image name is unexpected'
|
||||
$b = New-FakeBox 'wonder'
|
||||
foreach ($p in $b.Procs) { if ($p.Role -eq 'Server') { $p.Name = 'Renamed.Gateway' } }
|
||||
$r = Invoke-Scenario $b @{ DryRun = $true }
|
||||
Assert ($r.Rc -eq 0) "passes via the exact Application path (rc=$($r.Rc))"
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
foreach ($scheme in 'wonder', 'windev') {
|
||||
Section "Second gateway of the OTHER naming scheme is caught ($scheme box)"
|
||||
$b = New-FakeBox $scheme
|
||||
$otherName = if ($scheme -eq 'wonder') { 'ZB.MOM.WW.MxGateway.Server' } else { 'MxGateway.Server' }
|
||||
$b.Procs = @($b.Procs) + @([pscustomobject]@{ Id = 5500; ParentId = 77; Name = $otherName; Path = "C:/elsewhere/$otherName.exe"; Role = 'Other' })
|
||||
$r = Invoke-Scenario $b @{ DryRun = $true }
|
||||
Assert ($r.Rc -eq 1) 'refuses (rc 1)'
|
||||
Assert ($r.Text -match '2 gateway Server processes') 'reports two gateways'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
}
|
||||
|
||||
Section 'Another service launching the gateway exe'
|
||||
$b = New-FakeBox 'wonder'
|
||||
$b.OtherServices = @('MxAccessGwTest')
|
||||
$r = Invoke-Scenario $b @{ DryRun = $true }
|
||||
Assert ($r.Rc -eq 1 -and $r.Text -match 'another service \(MxAccessGwTest\)') 'refuses a second service'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'Service STOPPED with a stray Worker of the other scheme'
|
||||
$b = New-FakeBox 'wonder'
|
||||
$b.State = 'SERVICE_STOPPED'
|
||||
$b.Procs = @([pscustomobject]@{ Id = 5600; ParentId = 1; Name = 'ZB.MOM.WW.MxGateway.Worker'; Path = $null; Role = 'Other' })
|
||||
$r = Invoke-Scenario $b @{ DryRun = $true }
|
||||
Assert ($r.Rc -eq 1) 'refuses (rc 1)'
|
||||
Assert ($r.Text -match 'STOPPED but gateway process') 'says why'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'Gateway that is not the service host child'
|
||||
$b = New-FakeBox 'wonder'
|
||||
$b.Procs[0].ParentId = 1234
|
||||
$r = Invoke-Scenario $b @{ DryRun = $true }
|
||||
Assert ($r.Rc -eq 1 -and $r.Text -match 'is not a child of the MxAccessGw service host') 'refuses a gateway with the wrong parent'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'Staged Worker must carry the installed exe name'
|
||||
$b = New-FakeBox 'wonder'
|
||||
Rename-Item -LiteralPath "$($b.Root)/stage/Worker/MxGateway.Worker.exe" -NewName 'ZB.MOM.WW.MxGateway.Worker.exe'
|
||||
$r = Invoke-Scenario $b @{ DryRun = $true }
|
||||
Assert ($r.Rc -eq 1 -and $r.Text -match 'does not contain MxGateway\.Worker\.exe') 'refuses a staged Worker with the other exe name'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'Staged Worker must be x86'
|
||||
$b = New-FakeBox 'wonder'
|
||||
Set-Content -LiteralPath "$($b.Root)/stage/Worker/MxGateway.Worker.exe" -Value "sha=$NewSha;arch=Amd64"
|
||||
$r = Invoke-Scenario $b @{ DryRun = $true }
|
||||
Assert ($r.Rc -eq 1 -and $r.Text -match 'not X86') 'refuses an x64 Worker'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'Self-contained runtime downgrade is refused'
|
||||
$b = New-FakeBox 'wonder'
|
||||
Rename-Item -LiteralPath "$($b.Root)/stage/Server/mscordaccore_amd64_amd64_10.0.1426.12345.dll" -NewName 'mscordaccore_amd64_amd64_10.0.826.20000.dll'
|
||||
$r = Invoke-Scenario $b @{ DryRun = $true }
|
||||
Assert ($r.Rc -eq 1 -and $r.Text -match 'OLDER than the live') 'refuses 10.0.8 over 10.0.14'
|
||||
$r = Invoke-Scenario $b @{ DryRun = $true; AllowRuntimeDowngrade = $true }
|
||||
Assert ($r.Rc -eq 0) '-AllowRuntimeDowngrade overrides it'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'A store inside the swapped dir is refused'
|
||||
$b = New-FakeBox 'wonder'
|
||||
New-File "$($b.ServerDir)/stray.db" 'x'
|
||||
$r = Invoke-Scenario $b @{ DryRun = $true }
|
||||
Assert ($r.Rc -eq 1 -and $r.Text -match 'would be orphaned by the rename') 'refuses a .db under Server'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
# ---------------------------------------------------------------------------------------------------------------
|
||||
foreach ($scheme in 'wonder', 'windev') {
|
||||
Section "Real upgrade on a fake $scheme box (files really swapped)"
|
||||
$b = New-FakeBox $scheme
|
||||
$pfxHash = if ($scheme -eq 'wonder') { (Get-FileHash "$($b.ServerDir)/dashboard.pfx").Hash } else { $null }
|
||||
$cfgHash = (Get-FileHash "$($b.ServerDir)/appsettings.json").Hash
|
||||
$devHash = (Get-FileHash "$($b.ServerDir)/appsettings.Development.json").Hash
|
||||
$r = Invoke-Scenario $b @{ Label = 'test' }
|
||||
Assert ($r.Rc -eq 0) "upgrade exits 0 (rc=$($r.Rc)) $(if ($r.Rc -ne 0) { $r.Text })"
|
||||
Assert ((Get-Content -Raw $b.Application) -match "sha=$NewSha") 'live Server exe is the new build'
|
||||
Assert ((Get-Content -Raw $b.WorkerExe) -match "sha=$NewSha") 'live Worker exe is the new build'
|
||||
Assert ((Get-FileHash "$($b.ServerDir)/appsettings.json").Hash -eq $cfgHash) 'appsettings.json is the live copy, byte-identical'
|
||||
Assert ((Get-FileHash "$($b.ServerDir)/appsettings.Development.json").Hash -eq $devHash) 'appsettings.Development.json carried'
|
||||
if ($pfxHash) { Assert ((Test-Path "$($b.ServerDir)/dashboard.pfx") -and (Get-FileHash "$($b.ServerDir)/dashboard.pfx").Hash -eq $pfxHash) 'dashboard.pfx carried, byte-identical' }
|
||||
Assert (Test-Path "$($b.ServerDir)/notes-left-by-an-operator.txt") 'live-only text file carried'
|
||||
Assert (-not (Test-Path "$($b.ServerDir)/Old.Removed.Dependency.dll")) 'live-only binary not carried'
|
||||
Assert ((Get-Content -Raw "$($b.WorkerDir)/$(Split-Path -Leaf $b.WorkerExe).config") -match 'new') 'Worker exe.config is the new build output'
|
||||
$sbak = @(Get-ChildItem -LiteralPath $b.Root -Directory | Where-Object { $_.Name -like "$(Split-Path -Leaf $b.ServerDir).bak-*-test" })
|
||||
Assert ($sbak.Count -eq 1 -and ((Get-Content -Raw (Join-Path $sbak[0].FullName (Split-Path -Leaf $b.Application))) -match "sha=$OldSha")) 'old Server kept as .bak-<stamp>-test'
|
||||
Assert (@(Get-ChildItem -LiteralPath $b.Root -Directory -Filter 'Worker.bak-*-test').Count -eq 1) 'old Worker kept as .bak'
|
||||
Assert ((Test-Path "$($b.Root)/data/gateway-auth.db") -and (Get-Content -Raw "$($b.Root)/data/gateway-auth.db") -eq 'auth-db') 'live API-key store untouched'
|
||||
Assert (@(Get-ChildItem "$($b.Root)/data" -Filter 'gateway-auth.db*.bak-*').Count -eq 2) 'store db + wal backed up'
|
||||
$bk = @(Get-ChildItem -LiteralPath $b.Root -Directory -Filter '_deploy-backup-*')
|
||||
Assert ($bk.Count -eq 1 -and $b.Protected -contains $bk[0].FullName) 'backup dir created and ACL-restricted'
|
||||
$fp = Get-Content -Raw (Join-Path $bk[0].FullName 'env-fingerprint.txt')
|
||||
Assert ($fp -match 'MxGateway__ApiKeyPepper=[0-9a-f]{12}') 'fingerprint file has the pepper hash prefix'
|
||||
Assert-NoSecrets $fp 'env-fingerprint.txt'
|
||||
$notes = Get-Content -Raw (Get-ChildItem -LiteralPath $b.Root -Filter '_DEPLOY-NOTES-*.txt')[0].FullName
|
||||
Assert-NoSecrets $notes '_DEPLOY-NOTES'
|
||||
Assert-NoSecrets $r.Text 'the upgrade output'
|
||||
Assert ($b.Env -contains "MxGateway__ApiKeyPepper=$Pepper") 'env block unchanged'
|
||||
Assert ($b.Starts -eq 1) "started once ($($b.Starts))"
|
||||
Assert ($r.Text -match 'env block unchanged after the stability window') 'final env check ran'
|
||||
if ($scheme -eq 'wonder') { Assert ((Test-Path "$($bk[0].FullName)/kek.key")) 'file master key copied into the backup dir' }
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
}
|
||||
|
||||
Section 'Env block changed during the start -> abort + automatic rollback, values never printed'
|
||||
$b = New-FakeBox 'wonder'
|
||||
$b.OnStart = { param($x) if ($x.Starts -eq 1) { $x.Env = @($x.Env | ForEach-Object { if ($_ -like 'MxGateway__ApiKeyPepper=*') { 'MxGateway__ApiKeyPepper=a-different-pepper' } else { $_ } }) } }
|
||||
$r = Invoke-Scenario $b @{}
|
||||
Assert ($r.Rc -eq 1) 'upgrade fails (rc 1)'
|
||||
Assert ($r.Text -match 'changed MxGateway__ApiKeyPepper \(sha256 [0-9a-f]{12} -> [0-9a-f]{12}\)') 'names the changed entry with hash prefixes'
|
||||
Assert (-not $r.Text.Contains('a-different-pepper')) 'the new value is not printed'
|
||||
Assert-NoSecrets $r.Text 'the failure output'
|
||||
Assert ((Get-Content -Raw $b.Application) -match "sha=$OldSha") 'auto-rollback restored the old Server'
|
||||
Assert (($b.StartedWith -join ',') -eq "$NewSha,$OldSha") "started new, then old ($($b.StartedWith -join ','))"
|
||||
Assert ((Get-Content -Raw $b.WorkerExe) -match "sha=$OldSha") 'auto-rollback restored the old Worker'
|
||||
Assert (Test-Path "$($b.ServerDir)/dashboard.pfx") 'dashboard.pfx present after the rollback'
|
||||
Assert (@(Get-ChildItem -LiteralPath $b.Root -Directory -Filter 'Server.failed-*').Count -eq 1) 'failed build kept as Server.failed-<stamp>'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'Env entry removed between the stop and the swap check -> abort, the new build is never started'
|
||||
$b = New-FakeBox 'wonder'
|
||||
# Simulate something rewriting the block mid-deploy (e.g. install-service.bat): the backup's Export-ServiceKey seam drops the pepper.
|
||||
$r = & {
|
||||
$script:Box = $b
|
||||
$script:FakeNow = [datetime]'2026-11-12T12:00:00'
|
||||
. $ScriptPath -ServerSource "$($b.Root)/stage/Server" -WorkerSource "$($b.Root)/stage/Worker" -ExpectedSha $NewSha -MinFreeGB 0.001 -StableSeconds 10
|
||||
. $FakesPath
|
||||
function Export-ServiceKey([string]$Destination) { Set-Content -LiteralPath $Destination -Value 'fake'; $script:Box.Env = @($script:Box.Env | Where-Object { $_ -notlike 'MxGateway__ApiKeyPepper=*' }) }
|
||||
$rec = @(Invoke-Main 6>&1 3>&1)
|
||||
[pscustomobject]@{ Rc = @($rec | Where-Object { $_ -is [int] })[-1]; Text = (@($rec | Where-Object { $_ -isnot [int] } | ForEach-Object { "$_" }) -join "`n") }
|
||||
}
|
||||
Assert ($r.Rc -eq 1) 'upgrade fails (rc 1)'
|
||||
Assert ($r.Text -match 'env block changed after the swap: removed MxGateway__ApiKeyPepper') 'names the removed entry'
|
||||
Assert (@($b.StartedWith | Where-Object { $_ -eq $NewSha }).Count -eq 0) "the new build was never started (started: $($b.StartedWith -join ','))"
|
||||
Assert ((Get-Content -Raw $b.Application) -match "sha=$OldSha") 'auto-rollback put the old Server back'
|
||||
Assert ($r.Text -match 'env block changed before the start') 'the rollback reports the env difference again (warning)'
|
||||
Assert-NoSecrets $r.Text 'the failure output'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'STOP_PENDING: only this install''s processes are killed'
|
||||
$b = New-FakeBox 'wonder'
|
||||
$b.StopHangs = $true
|
||||
$r = Invoke-Scenario $b @{}
|
||||
Assert ($r.Rc -eq 0) "upgrade completes (rc=$($r.Rc)) $(if ($r.Rc -ne 0) { $r.Text })"
|
||||
Assert ($r.Text -match 'STOP_PENDING trap') 'STOP_PENDING path taken'
|
||||
Assert ($b.Killed.Count -ge 1 -and $b.Killed[0] -eq 4200) "the Server is killed first ($($b.Killed -join ','))"
|
||||
Assert (-not ($b.Killed -contains 900)) 'an unrelated process is not touched'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'Health never ready -> automatic rollback to the old build'
|
||||
$b = New-FakeBox 'windev'
|
||||
$b.OnStart = { param($x) if ($x.Starts -eq 1) { $x.Health = 503 } else { $x.Health = 200 } }
|
||||
$r = Invoke-Scenario $b @{ ReadyTimeoutSeconds = 10 }
|
||||
Assert ($r.Rc -eq 1) 'upgrade fails (rc 1)'
|
||||
Assert ($r.Text -match 'ROLLED BACK to the previous build') "rollback verified ($(($r.Text -split "`n" | Select-String 'ROLLBACK FAILED') -join ''))"
|
||||
Assert ((Get-Content -Raw $b.Application) -match "sha=$OldSha") 'old Server back in place'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'Manual rollback finds a .bak.<stamp> set left by the #229 adapted scripts'
|
||||
$b = New-FakeBox 'wonder'
|
||||
Copy-Item -LiteralPath $b.ServerDir -Destination "$($b.ServerDir).bak.20261112T211945" -Recurse
|
||||
Copy-Item -LiteralPath $b.WorkerDir -Destination "$($b.WorkerDir).bak.20261112T211945" -Recurse
|
||||
Set-Content -LiteralPath $b.Application -Value "sha=$NewSha;arch=Amd64"
|
||||
$r = Invoke-Scenario $b @{ Rollback = $true; DryRun = $true; ServerSource = $null; WorkerSource = $null }
|
||||
Assert ($r.Rc -eq 0 -and $r.Text -match 'Server=Server\.bak\.20261112T211945 Worker=Worker\.bak\.20261112T211945') "dry-run picks the matching pair ($($r.Rc))"
|
||||
$r = Invoke-Scenario $b @{ Rollback = $true; BackupStamp = '20261112T211945'; ServerSource = $null; WorkerSource = $null }
|
||||
Assert ($r.Rc -eq 0) "rollback exits 0 ($(if ($r.Rc -ne 0) { $r.Text }))"
|
||||
Assert ((Get-Content -Raw $b.Application) -match "sha=$OldSha") 'old Server restored'
|
||||
Assert (Test-Path "$($b.ServerDir)/dashboard.pfx") 'dashboard.pfx is there after the rollback'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'Rollback refuses a Server backup with no matching Worker backup'
|
||||
$b = New-FakeBox 'wonder'
|
||||
Copy-Item -LiteralPath $b.ServerDir -Destination "$($b.ServerDir).bak-20261101-100000" -Recurse
|
||||
Copy-Item -LiteralPath $b.WorkerDir -Destination "$($b.WorkerDir).bak-20261001-100000" -Recurse
|
||||
$r = Invoke-Scenario $b @{ Rollback = $true; DryRun = $true; ServerSource = $null; WorkerSource = $null }
|
||||
Assert ($r.Rc -eq 1 -and $r.Text -match 'refusing a mixed rollback') 'refuses mixed stamps'
|
||||
Remove-Item -LiteralPath $b.Root -Recurse -Force
|
||||
|
||||
Section 'Env fingerprint helpers'
|
||||
. $ScriptPath -DryRun
|
||||
$a = Get-EnvFingerprint @('A=1', "MxGateway__ApiKeyPepper=$Pepper", 'C=3')
|
||||
Assert (-not ((Format-EnvFingerprint $a).Contains($Pepper))) 'the formatted fingerprint carries no value'
|
||||
Assert ((Compare-EnvFingerprint $a (Get-EnvFingerprint @('A=1', "MxGateway__ApiKeyPepper=$Pepper", 'C=3'))).Count -eq 0) 'identical blocks compare equal'
|
||||
$d = Compare-EnvFingerprint $a (Get-EnvFingerprint @('A=1', 'MxGateway__ApiKeyPepper=other', 'C=3'))
|
||||
Assert ($d.Count -eq 1 -and $d[0] -like 'changed MxGateway__ApiKeyPepper*') "a changed value is caught ($d)"
|
||||
$d = Compare-EnvFingerprint $a (Get-EnvFingerprint @('A=1', "MxGateway__ApiKeyPepper=$Pepper"))
|
||||
Assert ($d.Count -eq 1 -and $d[0] -eq 'removed C') "a removed entry is caught ($d)"
|
||||
$d = Compare-EnvFingerprint $a (Get-EnvFingerprint @('A=1', "MxGateway__ApiKeyPepper=$Pepper", 'C=3', 'D=4'))
|
||||
Assert ($d.Count -eq 1 -and $d[0] -eq 'added D') "an added entry is caught ($d)"
|
||||
$d = Compare-EnvFingerprint $a (Get-EnvFingerprint @('C=3', 'A=1', "MxGateway__ApiKeyPepper=$Pepper"))
|
||||
Assert ($d.Count -eq 1 -and $d[0] -like 'block hash*') "a reorder is caught ($d)"
|
||||
Assert ((Hide "x $Pepper y") -eq 'x <redacted> y') 'a sensitive value is redacted from output'
|
||||
|
||||
Write-Host ''
|
||||
if ($script:failures -gt 0) { Write-Host "FAILED: $($script:failures) assertion(s) failed, $($script:passes) passed" -ForegroundColor Red; exit 1 }
|
||||
Write-Host "OK: $($script:passes) assertions passed" -ForegroundColor Green
|
||||
exit 0
|
||||
@@ -0,0 +1,75 @@
|
||||
# Fakes for Test-UpgradeMxGatewayService.ps1. Dot-sourced AFTER Upgrade-MxGatewayService.ps1 so these replace
|
||||
# its environment seams. All state lives in $script:Box (built by New-FakeBox in the test script).
|
||||
# Nothing here touches the real service manager, registry, process table or network.
|
||||
|
||||
function Invoke-Nssm {
|
||||
param([string[]]$NssmArgs, [switch]$AllowFail)
|
||||
$b = $script:Box
|
||||
$b.NssmCalls.Add(($NssmArgs -join ' '))
|
||||
$verb = $NssmArgs[0]
|
||||
switch ($verb) {
|
||||
'status' {
|
||||
if ($b.State -eq 'MISSING') { return [pscustomobject]@{ Code = 3; Text = "Can't open service!" } }
|
||||
if ($b.State -eq 'SERVICE_STOP_PENDING' -and -not @($b.Procs | Where-Object { $_.Role -eq 'Server' }).Count) { $b.State = 'SERVICE_STOPPED' }
|
||||
return [pscustomobject]@{ Code = 0; Text = $b.State }
|
||||
}
|
||||
'get' {
|
||||
switch ($NssmArgs[2]) {
|
||||
'Application' { return [pscustomobject]@{ Code = 0; Text = $b.Application } }
|
||||
'AppDirectory' { return [pscustomobject]@{ Code = 0; Text = $b.AppDirectory } }
|
||||
'AppEnvironmentExtra' { return [pscustomobject]@{ Code = 0; Text = ($b.Env -join "`r`n") } }
|
||||
}
|
||||
}
|
||||
'stop' {
|
||||
if ($b.StopHangs) { $b.State = 'SERVICE_STOP_PENDING'; return [pscustomobject]@{ Code = 0; Text = '' } }
|
||||
$b.State = 'SERVICE_STOPPED'
|
||||
$b.Procs = @($b.Procs | Where-Object { $_.Role -eq 'Other' -or $b.Survivors -contains $_.Id })
|
||||
return [pscustomobject]@{ Code = 0; Text = '' }
|
||||
}
|
||||
'start' {
|
||||
$b.Starts++
|
||||
$b.StartedWith.Add($(if ((Get-Content -LiteralPath $b.Application -Raw) -match 'sha=([0-9a-f]+)') { $Matches[1] } else { '?' }))
|
||||
$b.State = 'SERVICE_RUNNING'
|
||||
$sid = 7000 + $b.Starts * 10
|
||||
$b.Procs = @($b.Procs) + @(
|
||||
[pscustomobject]@{ Id = $sid; ParentId = $b.HostPid; Name = [IO.Path]::GetFileNameWithoutExtension($b.Application); Path = $b.Application; Role = 'Server' },
|
||||
[pscustomobject]@{ Id = $sid + 1; ParentId = $sid; Name = [IO.Path]::GetFileNameWithoutExtension($b.WorkerExe); Path = $b.WorkerExe; Role = 'Worker' })
|
||||
if ($b.OnStart) { & $b.OnStart $b }
|
||||
return [pscustomobject]@{ Code = 0; Text = '' }
|
||||
}
|
||||
}
|
||||
throw "fake nssm: unexpected call $($NssmArgs -join ' ')"
|
||||
}
|
||||
|
||||
function Read-EnvBlockLines { @($script:Box.Env) }
|
||||
|
||||
# Return copies so the script's Add-Member (Role) cannot leak back into the fake state.
|
||||
function Get-ProcessTable { @($script:Box.Procs | ForEach-Object { [pscustomobject]@{ Id = $_.Id; ParentId = $_.ParentId; Name = $_.Name; Path = $_.Path } }) }
|
||||
function Get-ServiceHostPid { $script:Box.HostPid }
|
||||
function Get-OtherGatewayServices([string]$ExeName) { , @($script:Box.OtherServices) }
|
||||
function Stop-ProcessById([int]$Id) {
|
||||
$script:Box.Killed.Add($Id)
|
||||
$script:Box.Procs = @($script:Box.Procs | Where-Object { $_.Id -ne $Id })
|
||||
}
|
||||
function Get-HttpStatus([string]$Url) { $script:Box.HttpCalls.Add($Url); $script:Box.Health }
|
||||
function Get-ProductSha([string]$Exe) {
|
||||
if (-not (Test-Path -LiteralPath $Exe)) { return $null }
|
||||
if ((Get-Content -LiteralPath $Exe -Raw) -match 'sha=([0-9a-f]+)') { return $Matches[1] }
|
||||
$null
|
||||
}
|
||||
function Get-ExeArchitecture([string]$Exe) {
|
||||
if ((Get-Content -LiteralPath $Exe -Raw) -match 'arch=(\w+)') { return $Matches[1] }
|
||||
$null
|
||||
}
|
||||
function Protect-Directory([string]$Path) { $script:Box.Protected.Add($Path) }
|
||||
function Export-ServiceKey([string]$Destination) { Set-Content -LiteralPath $Destination -Value 'fake reg export' }
|
||||
function Get-DirSddl([string]$Path) { $null }
|
||||
function Copy-ExplicitAcl([string]$From, [string]$To) { }
|
||||
|
||||
# Fake clock: Start-Sleep advances it, so timeouts expire instantly and deterministically.
|
||||
function Start-Sleep { param([int]$Seconds, [int]$Milliseconds) $script:FakeNow = $script:FakeNow.AddSeconds([math]::Max($Seconds, 1)) }
|
||||
function Get-Date {
|
||||
param([string]$Format)
|
||||
if ($Format) { return (Microsoft.PowerShell.Utility\Get-Date -Format $Format) }
|
||||
$script:FakeNow
|
||||
}
|
||||
Reference in New Issue
Block a user