feat(deploy): on-box NSSM upgrade/rollback and new-box install scripts, runbook updated (mxaccessgw#220)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m26s
ci / java (push) Successful in 2m12s
ci / portable (push) Failing after 4m11s

Add scripts/deploy/Upgrade-MxGatewayService.ps1 (preflight, stop with STOP_PENDING handling,
backup, Server+Worker swap, start, /health/ready + SHA verify, auto/manual rollback,
one-gateway guard, never touches the NSSM env block) and Install-MxGatewayService.ps1
(service install, restart/recovery settings, env block). Both support -DryRun/-WhatIf.
Dry-run validated on macOS pwsh against a fake layout; untested on a real box.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 15:10:26 -05:00
parent 259d375022
commit 477fa55998
3 changed files with 767 additions and 6 deletions
+75 -6
View File
@@ -1,7 +1,7 @@
# Deploy / Upgrade Runbook: NSSM-wrapped Gateway Service (Server + x86 Worker)
> Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). There is **no deploy script** in
> this repo; this runbook is the procedure. It is assembled only from what the repo contains
> Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). Since mxaccessgw#220 the procedure
> is also scripted (see "Scripts" below); this runbook remains the spec the scripts implement. It is assembled only from what the repo contains
> (`docs/GatewayConfiguration.md`, `docs/Authentication.md`, `docs/Diagnostics.md`,
> `docs/runbooks/*`, `docs/plans/2026-08-07-*`, `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`,
> `src/.../appsettings.json`, `scripts/ci/`) cross-checked against `scadaproj/deployment.md` §5 and §7.1.
@@ -17,6 +17,72 @@
> and direct SQL. Confirm the current rule before starting; this repo's note is dated 2026-09-11 and
> the umbrella index has since relaxed it to deploy-only **[UNVERIFIED from this repo]**.
## Scripts (mxaccessgw#220)
> **UNTESTED ON A REAL BOX until the next deploy.** The scripts were only exercised on a Mac with
> `pwsh` in `-DryRun` mode against a fake directory layout and a fake `nssm`
> (parsing, preflight refusals, the one-gateway guard, the step sequence). Service stop/start,
> STOP_PENDING handling, the health/SHA verification, `sc.exe failure`, the rollback swaps and the
> NSSM UTF-16 output handling have not run against real Windows/NSSM. Run the first real use with
> `-DryRun` on the box, read the plan, and have section 8 ready by hand.
| Script | Where it runs | Does |
|---|---|---|
| `scripts/deploy/Upgrade-MxGatewayService.ps1` | **On the Windows box**, Windows PowerShell 5.1 | Sections 3, 5, 6, 7, 8: preflight, stop, backup, swap Server + Worker, start, verify, rollback |
| `scripts/deploy/Install-MxGatewayService.ps1` | On a **new** box | NSSM service install: recovery/restart, stop grace, log rotation, env block |
Both take `-DryRun` (or `-WhatIf`): read-only probes (`nssm get/status`, process list, disk, file
existence) still run, every mutating action is printed instead of executed.
**Upgrade script** (`-ServerSource`, `-WorkerSource` are the staged outputs from section 4, built from one SHA):
- Parameters: `-ServerSource -WorkerSource -ServiceName MxAccessGw -NssmPath -ServerDir -WorkerDir
-ExpectedSha -Label -HealthBaseUrl http://localhost:5130 -Rollback -BackupStamp -RestoreStores
-NoAutoRollback -TakeNewAppSettings -SkipFreeSpaceCheck -MinFreeGB -StopTimeoutSeconds
-StableSeconds -ReadyTimeoutSeconds -DryRun`. `ServerDir` defaults to NSSM `AppDirectory`, `WorkerDir`
to the parent of `MxGateway__Worker__ExecutablePath` read from the env block.
- Preflight refuses on: missing service, empty env block, no `ZB_SECRETS_MASTER_KEY`, a store path
that is relative or inside the Server/Worker dir, staged Server/Worker built from different SHAs (or not
matching `-ExpectedSha`), too little disk, **more than one gateway Server process, a gateway process
while the service is STOPPED, or another service launching the same exe** (trap 1). It also asserts no
Server/Worker process exists immediately before every `nssm start`.
- Stop waits `-StopTimeoutSeconds`; on STOP_PENDING (trap 8) it kills the Server then Worker processes by
image name (safe because preflight proved a single gateway), then the service host as a last resort.
- Swap is **rename-in-place** (`Server` -> `Server.bak-<stamp>[-label]`, same for `Worker`), so NSSM
`Application`/`AppDirectory`/`ExecutablePath` never change. Live `appsettings*.json` are copied back
byte-identical. The stores are only **copied** (`.db`, `-wal`, `-shm` -> `*.bak-<stamp>`), never moved
or deleted. The script **never runs `nssm set AppEnvironmentExtra`**; it saves the block (secrets
included, ACL-restricted via `icacls`) to `_env-AppEnvironmentExtra-<stamp>.txt` and compares the entry count
after start. Notes (names only, no values) go to `_DEPLOY-NOTES-<stamp>.txt` beside the install.
- Verify: `/health/ready` = 200, `/healthz`, `ProductVersion` SHA of the deployed Server exe against
`-ExpectedSha` (still subject to the version-stamp caveat in trap 7), service stays RUNNING for
`-StableSeconds`, env count unchanged. Health with zero sessions does not prove the Worker path: do the
section 7.3 API-key session read yourself.
- A failed start or verify triggers an automatic rollback to the just-made backups (disable with
`-NoAutoRollback`). Manual rollback: `-Rollback [-BackupStamp <stamp>] [-RestoreStores]`; the failed
directories are kept as `*.failed-<stamp>`. `-RestoreStores` restores the db triplet from the matching
`*.bak-<stamp>` (stale wal/shm are moved aside, nothing is deleted); use it only after a schema bump.
**Install script**: `-ServerDir -WorkerDir [-ServiceName -NssmPath -DataDir C:\ProgramData\MxGateway
-LogDir -EnvFile -Env -ObjectName -ServiceAccountPassword -EnvironmentName -RestartDelayMs -ThrottleMs
-StopGraceMs -RotateBytes -Start -DryRun]`. It refuses if the service exists or any gateway process is
running, requires the KEK and a `*Pepper*` env entry, requires store paths absolute and outside the content
root, and sets the env block in **one** `nssm set AppEnvironmentExtra` on a service that has none. Defaults
it applies: auto start, `AppExit Default Restart`, restart delay 5 s, throttle 10 s, console stop grace 20 s
(covers `Worker:ShutdownTimeoutSeconds` = 10), stdout/stderr to `<DataDir>\logs` with rotation,
`sc failure ... restart/5000/restart/5000/restart/30000`, reset 1 day. Secret values are masked in output.
The service is left stopped unless `-Start` is given.
**Driving from the Mac (winrm skill, deployment-only).** Remote execution on the wonder boxes is allowed
only for the on-box steps of a deploy the owner asked for. The intended flow, none of which was run for #220:
1. Build both artifacts on windev from one SHA (section 4); `smb push` them to a staging dir on the box
(UNC/SMB copies are always allowed), together with the two scripts.
2. `winrm exec` the script with `-DryRun` first, read the printed plan, then run it for real:
`powershell -NoProfile -ExecutionPolicy Bypass -File <stage>\Upgrade-MxGatewayService.ps1 -ServerSource <stage>\Server -WorkerSource <stage>\Worker -ExpectedSha <sha> -Label <label>`.
One box at a time; the exit code is 0 on success, 1 on any abort or failed run.
3. Do the section 7.3 Worker-path check from the Mac over gRPC (no remote exec needed), then write the deploy note.
## 0. What is being deployed
Two artifacts, deployed together because they share a wire protocol (`WorkerProtocolVersion`,
@@ -267,7 +333,10 @@ window, API keys no longer authenticate, or a Worker session cannot be opened.
- Whether the Server on each box is framework-dependent or self-contained (a box without .NET 10
needs self-contained; the 2026-10-19 windev plan says framework-dependent).
- The `STOP_PENDING` behaviour and the NSSM service account (`LocalSystem` per `deployment.md`).
- NSSM install script: **none exists in the repo** - installing a *new* box (service creation,
`AppExit`, recovery, log redirection settings) is not covered here; the only reference is
`deployment.md`'s statement that VQ04 was cloned from VQ03's NSSM `AppEnvironmentExtra` config.
- Automating this runbook as a script was the other half of the original finding; not done here.
- New-box install and the upgrade are now scripted (see "Scripts"), but **untested on a real box**; the
recovery/stop-grace/log-rotation defaults in `Install-MxGatewayService.ps1` are chosen from the repo's
own settings (`Worker:ShutdownTimeoutSeconds`), not copied from an existing box. Compare against the
`nssm dump` of a known-good box (vd03/vq03) before relying on them.
- The Worker output folder and build-vs-publish question (section 4) is still **[UNVERIFIED]**: the upgrade
script copies whatever the staged `-WorkerSource` directory contains over `Worker`, so stage exactly what
the existing `Worker` folder holds. Confirm with a file listing of the live `Worker` dir on the next deploy.
+202
View File
@@ -0,0 +1,202 @@
#requires -Version 5.1
<#
.SYNOPSIS
On-box NEW-BOX install of the MxAccess Gateway as an NSSM service (mxaccessgw#220).
Companion of Upgrade-MxGatewayService.ps1; see docs/runbooks/DeployUpgradeNssmService.md.
.DESCRIPTION
RUN ON THE WINDOWS BOX (Windows PowerShell 5.1). Creates the service ONCE, with:
* Application / AppDirectory / DisplayName / Description, auto start, account (default LocalSystem);
* NSSM exit handling: restart on any exit with a throttle, graceful-stop timeouts that cover the
Worker shutdown grace (MxGateway:Worker:ShutdownTimeoutSeconds, default 10 s);
* stdout/stderr redirection with rotation;
* Windows SCM recovery (sc failure: restart, restart, restart + reset period);
* the environment block, set in ONE `nssm set AppEnvironmentExtra` call on a service that has none
(so there is nothing to wipe). The env comes from -EnvFile (KEY=VALUE lines, # comments) and/or
-Env (hashtable); -Env wins. Secret VALUES are never printed (dry-run shows names and masks values).
It does NOT start the service unless -Start is given (review the env first), and it refuses to run if
the service already exists (use Upgrade-MxGatewayService.ps1) or if any gateway process is running
(OrphanWorkerTerminator trap: a second gateway kills live Workers).
Required env (checked by name): ZB_SECRETS_MASTER_KEY, a *Pepper* entry (MxGateway__ApiKeyPepper),
MxGateway__Worker__ExecutablePath (defaults to <WorkerDir>\ZB.MOM.WW.MxGateway.Worker.exe),
MxGateway__Authentication__SqlitePath and Secrets__SqlitePath (default under -DataDir; must be absolute and
OUTSIDE the content root, the validator is fail-closed), ASPNETCORE_ENVIRONMENT/DOTNET_ENVIRONMENT.
Kestrel endpoints default to docs/GatewayConfiguration.md's cleartext h2c :5120 + dashboard :5130 unless
the env supplies Kestrel__Endpoints__*.
-DryRun (or -WhatIf) prints every mutating action and performs none.
#>
[CmdletBinding(SupportsShouldProcess = $true)]
param(
[Parameter(Mandatory = $true)][string]$ServerDir,
[Parameter(Mandatory = $true)][string]$WorkerDir,
[string]$ServiceName = 'MxAccessGw',
[string]$NssmPath = 'nssm',
[string]$DataDir = 'C:\ProgramData\MxGateway',
[string]$LogDir,
[string]$EnvFile,
[hashtable]$Env = @{},
[string]$ServerExeName = 'ZB.MOM.WW.MxGateway.Server.exe',
[string]$ObjectName = 'LocalSystem',
[string]$ServiceAccountPassword,
[string]$EnvironmentName = 'Production',
[int]$RestartDelayMs = 5000,
[int]$ThrottleMs = 10000,
[int]$StopGraceMs = 20000,
[int]$RotateBytes = 10485760,
[switch]$Start,
[switch]$DryRun
)
$ErrorActionPreference = 'Stop'
$script:Dry = $DryRun.IsPresent -or $WhatIfPreference
$ServerProc = 'ZB.MOM.WW.MxGateway.Server'
$WorkerProc = 'ZB.MOM.WW.MxGateway.Worker'
function Say([string]$m) { Write-Host $m }
function Fail([string]$m) { throw "ABORT: $m" }
function Step([string]$Description, [scriptblock]$Action) {
if ($script:Dry) { Say "[DRYRUN] $Description"; return }
Say "[ACTION] $Description"; & $Action
}
function Invoke-Nssm {
param([string[]]$NssmArgs, [switch]$AllowFail)
$raw = & $NssmPath @NssmArgs 2>&1; $code = $LASTEXITCODE
$text = (($raw | ForEach-Object { "$_" }) -join "`n") -replace "`0", ''
if ($code -ne 0 -and -not $AllowFail) { Fail "nssm $($NssmArgs[0..1] -join ' ') failed (exit $code): $text" }
[pscustomobject]@{ Code = $code; Text = $text.Trim() }
}
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
function Test-Inside([string]$Path, [string]$Root) {
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
$a -eq $b -or $a.StartsWith("$b/")
}
function Mask([string]$k, [string]$v) {
if ($k -match 'KEY|PASSWORD|SECRET|PEPPER|TOKEN' -and $k -notmatch 'Path$|Url$') { '****' } else { $v }
}
try {
Say "Install-MxGatewayService service=$ServiceName dry=$($script:Dry)"
# --- environment assembly -------------------------------------------------------------------------
$envMap = [ordered]@{}
if ($EnvFile) {
if (-not (Test-Path -LiteralPath $EnvFile)) { Fail "-EnvFile '$EnvFile' not found" }
foreach ($line in (Get-Content -LiteralPath $EnvFile)) {
$t = $line.Trim()
if (-not $t -or $t.StartsWith('#')) { continue }
if ($t -match '^([^=\s]+)=(.*)$') { $envMap[$Matches[1]] = $Matches[2] } else { Fail "-EnvFile line is not KEY=VALUE: '$t'" }
}
}
foreach ($k in $Env.Keys) { $envMap[[string]$k] = [string]$Env[$k] }
$workerExe = Join-Path $WorkerDir "$WorkerProc.exe"
if (-not $envMap.Contains('MxGateway__Worker__ExecutablePath')) { $envMap['MxGateway__Worker__ExecutablePath'] = $workerExe }
if (-not $envMap.Contains('MxGateway__Authentication__SqlitePath')) { $envMap['MxGateway__Authentication__SqlitePath'] = Join-Path $DataDir 'gateway-auth.db' }
if (-not $envMap.Contains('Secrets__SqlitePath')) { $envMap['Secrets__SqlitePath'] = Join-Path $DataDir 'mxgateway-secrets.db' }
if (-not $envMap.Contains('ASPNETCORE_ENVIRONMENT')) { $envMap['ASPNETCORE_ENVIRONMENT'] = $EnvironmentName }
if (-not $envMap.Contains('DOTNET_ENVIRONMENT')) { $envMap['DOTNET_ENVIRONMENT'] = $EnvironmentName }
if (-not ($envMap.Keys | Where-Object { $_ -like 'Kestrel__Endpoints__*' })) {
$envMap['Kestrel__Endpoints__Http__Url'] = 'http://0.0.0.0:5120'
$envMap['Kestrel__Endpoints__Http__Protocols'] = 'Http2'
$envMap['Kestrel__Endpoints__Dashboard__Url'] = 'http://0.0.0.0:5130'
Say '[info] no Kestrel__Endpoints__* supplied: using the documented cleartext defaults (5120 h2c, 5130 dashboard)'
}
# --- validation -------------------------------------------------------------------------------------
if (-not $envMap.Contains('ZB_SECRETS_MASTER_KEY') -or -not $envMap['ZB_SECRETS_MASTER_KEY']) { Fail 'env must supply ZB_SECRETS_MASTER_KEY (the Secrets KEK)' }
if (-not ($envMap.Keys | Where-Object { $_ -match 'Pepper' })) { Fail 'env must supply the API-key pepper (MxGateway__ApiKeyPepper); without it no issued token verifies' }
foreach ($k in 'MxGateway__Authentication__SqlitePath', 'Secrets__SqlitePath') {
$v = $envMap[$k]
if (-not (Test-Rooted $v)) { Fail "$k '$v' must be absolute" }
if ((Test-Inside $v $ServerDir) -or (Test-Inside $v $WorkerDir)) { Fail "$k '$v' is inside the content root; the validator is fail-closed and an upgrade swap would lose it" }
}
foreach ($d in @($ServerDir, $WorkerDir)) { if (-not (Test-Rooted $d)) { Fail "'$d' must be an absolute path" } }
if ((Test-Inside $WorkerDir $ServerDir) -or (Test-Inside $ServerDir $WorkerDir)) { Fail 'Server and Worker directories must be siblings, not nested (the upgrade swap renames them independently)' }
if (-not (Test-Path -LiteralPath (Join-Path $ServerDir $ServerExeName))) { Fail "$ServerExeName not found in $ServerDir" }
if (-not (Test-Path -LiteralPath $workerExe)) { Fail "$WorkerProc.exe not found in $WorkerDir" }
if ($envMap['MxGateway__Worker__ExecutablePath'] -ne $workerExe) { Say '[WARN] MxGateway__Worker__ExecutablePath differs from <WorkerDir>\Worker.exe; keep ONE canonical spelling' }
if ($envMap['ASPNETCORE_ENVIRONMENT'] -ne 'Production') { Say '[WARN] ASPNETCORE_ENVIRONMENT is not exactly Production: DisableLogin / plaintext-LDAP hard-stops are only enforced for Production' }
# --- one-gateway guard + existence -------------------------------------------------------------------
$st = Invoke-Nssm -NssmArgs @('status', $ServiceName) -AllowFail
if ($st.Code -eq 0) { Fail "service '$ServiceName' already exists (state $($st.Text)); use Upgrade-MxGatewayService.ps1" }
$procs = @(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in @($ServerProc, $WorkerProc) })
if ($procs.Count -gt 0) { Fail "gateway process(es) already running (PIDs $(($procs.Id) -join ',')); installing/starting a second gateway kills their Workers" }
if (-not $LogDir) { $LogDir = Join-Path $DataDir 'logs' }
$exe = Join-Path $ServerDir $ServerExeName
Say "[info] env entries ($($envMap.Count)):"
foreach ($k in $envMap.Keys) { Say " $k=$(Mask $k $envMap[$k])" }
# --- install ------------------------------------------------------------------------------------------------
Step "create $DataDir and $LogDir (stores/certs/logs live OUTSIDE the content root)" {
New-Item -ItemType Directory -Force -Path $DataDir, $LogDir | Out-Null
}
Step "nssm install $ServiceName `"$exe`"" { [void](Invoke-Nssm -NssmArgs @('install', $ServiceName, $exe)); $script:Installed = $true }
$settings = @(
@('AppDirectory', $ServerDir),
@('DisplayName', 'MxAccess Gateway'),
@('Description', 'MxAccess Gateway (gRPC) - Server; launches the x86 Worker per session'),
@('Start', 'SERVICE_AUTO_START'),
@('Type', 'SERVICE_WIN32_OWN_PROCESS'),
@('AppExit', 'Default', 'Restart'),
@('AppRestartDelay', "$RestartDelayMs"),
@('AppThrottle', "$ThrottleMs"),
@('AppStopMethodConsole', "$StopGraceMs"),
@('AppStopMethodWindow', '0'),
@('AppStopMethodThreads', '0'),
@('AppStdout', (Join-Path $LogDir 'service-stdout.log')),
@('AppStderr', (Join-Path $LogDir 'service-stderr.log')),
@('AppRotateFiles', '1'),
@('AppRotateOnline', '1'),
@('AppRotateBytes', "$RotateBytes")
)
foreach ($s in $settings) {
Step "nssm set $ServiceName $($s -join ' ')" { [void](Invoke-Nssm -NssmArgs (@('set', $ServiceName) + $s)) }
}
if ($ObjectName -and $ObjectName -ne 'LocalSystem') {
Step "nssm set $ServiceName ObjectName $ObjectName <password>" { [void](Invoke-Nssm -NssmArgs @('set', $ServiceName, 'ObjectName', $ObjectName, $ServiceAccountPassword)) }
} else {
Step "nssm set $ServiceName ObjectName LocalSystem" { [void](Invoke-Nssm -NssmArgs @('set', $ServiceName, 'ObjectName', 'LocalSystem')) }
}
# Env block: first and only AppEnvironmentExtra write; the service has none, so nothing is wiped.
$pairs = @($envMap.Keys | ForEach-Object { "$_=$($envMap[$_])" })
Step "nssm set $ServiceName AppEnvironmentExtra <$($envMap.Count) entries, values hidden>" {
[void](Invoke-Nssm -NssmArgs (@('set', $ServiceName, 'AppEnvironmentExtra') + $pairs))
}
# SCM recovery: restart x3 (5 s, 5 s, 30 s), reset the counter after 1 day; also on non-crash exits.
Step "sc.exe failure $ServiceName reset= 86400 actions= restart/5000/restart/5000/restart/30000" {
& sc.exe failure $ServiceName reset= 86400 actions= 'restart/5000/restart/5000/restart/30000' | Out-Null
if ($LASTEXITCODE -ne 0) { Fail "sc.exe failure exited $LASTEXITCODE" }
}
Step "sc.exe failureflag $ServiceName 1" {
& sc.exe failureflag $ServiceName 1 | Out-Null
if ($LASTEXITCODE -ne 0) { Fail "sc.exe failureflag exited $LASTEXITCODE" }
}
# Verify read-back (names only) so a silent miss is caught before the first start.
Step "verify: nssm get AppEnvironmentExtra has $($envMap.Count) entries" {
$txt = (Invoke-Nssm -NssmArgs @('get', $ServiceName, 'AppEnvironmentExtra')).Text
$n = @($txt -split "[`r`n]+" | Where-Object { $_.Trim() -match '^[^=\s]+=' }).Count
if ($n -ne $envMap.Count) { Fail "env read-back has $n entries, expected $($envMap.Count)" }
}
if ($Start) {
Step "nssm start $ServiceName (exactly once)" { [void](Invoke-Nssm -NssmArgs @('start', $ServiceName)) }
Say 'Next: GET /health/ready on the dashboard port, then prove a Worker session with an API key (runbook section 7).'
} else {
Say '[info] service installed STOPPED. Review the env, then `nssm start` (or re-run with -Start). Seed secrets with the `secret` CLI using the SAME Secrets__SqlitePath + ZB_SECRETS_MASTER_KEY.'
}
Say '== DONE =='
exit 0
} catch {
Write-Host "ERROR: $($_.Exception.Message)" -ForegroundColor Red
if ($script:Installed) { Write-Host "PARTIAL INSTALL: inspect, then `nssm remove $ServiceName confirm` and re-run." -ForegroundColor Yellow }
exit 1
}
+490
View File
@@ -0,0 +1,490 @@
#requires -Version 5.1
<#
.SYNOPSIS
On-box upgrade / rollback of the NSSM-wrapped MxAccess Gateway (Server + x86 Worker).
Implements docs/runbooks/DeployUpgradeNssmService.md sections 3, 5, 6, 7 and 8 (mxaccessgw#220).
.DESCRIPTION
RUN THIS ON THE WINDOWS BOX ITSELF (Windows PowerShell 5.1 is enough). The new Server publish and the
new x86 Worker build must already be staged on the box (UNC/SMB copy; build both from the same SHA).
Upgrade flow: preflight -> stop (STOP_PENDING handled) -> backup -> swap Server + Worker -> start ->
verify (/health/ready, VersionInfo SHA, service stable) -> automatic rollback if start/verify fails.
Rollback flow (-Rollback): stop -> swap the .bak directories back -> optional store restore -> start -> verify.
Safety properties:
* Exactly one gateway per box. Refuses to run if a second gateway process / service points at the same
exe, if the service is stopped but a gateway process exists, or if a Server/Worker process is still
alive when a start is about to happen (OrphanWorkerTerminator trap, runbook section 2.1).
* Never calls `nssm set ... AppEnvironmentExtra` (that replaces the whole block). The env block is only
READ (saved to the backup dir, entry count compared before/after). The Application/AppDirectory
are not changed either: the swap is rename-in-place, so the NSSM config is untouched.
* API-key DB, Secrets store and master key live outside the content root and are never moved or
deleted. They are only COPIED (db + -wal + -shm) into *.bak-<stamp> siblings while the service is stopped.
The script refuses to run if a store path is inside the content root (the 2026-08-09 incident).
-DryRun (or -WhatIf) prints every mutating action and performs none. Read-only probing (nssm get/status,
process list, disk space, file existence) still runs so the decision logic is exercised.
.PARAMETER ServerSource Staged Server publish output (contains ZB.MOM.WW.MxGateway.Server.exe).
.PARAMETER WorkerSource Staged x86 Worker output (contains ZB.MOM.WW.MxGateway.Worker.exe).
.PARAMETER ServiceName NSSM service name. Default MxAccessGw.
.PARAMETER NssmPath nssm executable. Default 'nssm' (on PATH).
.PARAMETER ServerDir Live Server directory. Default: NSSM AppDirectory.
.PARAMETER WorkerDir Live Worker directory. Default: parent of env MxGateway__Worker__ExecutablePath.
.PARAMETER ExpectedSha Short/long git SHA. Staged Server+Worker (preflight) and the deployed Server (verify)
must carry it in ProductVersion ('0.1.2+<sha>'). Builds from 2026-07-09..08-10 have a bogus stamp.
.PARAMETER Label Free-text suffix for backup names (e.g. 'planwrites').
.PARAMETER HealthBaseUrl Default http://localhost:5130 (dashboard port; /health/ready, /healthz anonymous).
.PARAMETER Rollback Roll back instead of upgrading. Uses newest Server.bak-* unless -BackupStamp is given.
.PARAMETER BackupStamp Stamp (yyyyMMdd-HHmmss) of the backup set to roll back to.
.PARAMETER RestoreStores With -Rollback: also restore gateway-auth.db / secrets db from *.bak-<stamp> (schema-bump case).
.PARAMETER NoAutoRollback Do not roll back automatically when start/verification fails.
.PARAMETER TakeNewAppSettings Do not carry the live appsettings*.json over into the new Server dir.
.PARAMETER SkipFreeSpaceCheck / MinFreeGB / StopTimeoutSeconds / StableSeconds / ReadyTimeoutSeconds tunables.
.PARAMETER DryRun Print, do not act. -WhatIf is equivalent.
#>
[CmdletBinding(SupportsShouldProcess = $true)]
param(
[string]$ServerSource,
[string]$WorkerSource,
[string]$ServiceName = 'MxAccessGw',
[string]$NssmPath = 'nssm',
[string]$ServerDir,
[string]$WorkerDir,
[string]$ExpectedSha,
[string]$Label,
[string]$HealthBaseUrl = 'http://localhost:5130',
[switch]$Rollback,
[string]$BackupStamp,
[switch]$RestoreStores,
[switch]$NoAutoRollback,
[switch]$TakeNewAppSettings,
[switch]$SkipFreeSpaceCheck,
[double]$MinFreeGB = 2,
[int]$StopTimeoutSeconds = 30,
[int]$StableSeconds = 60,
[int]$ReadyTimeoutSeconds = 120,
[switch]$DryRun
)
$ErrorActionPreference = 'Stop'
$script:Dry = $DryRun.IsPresent -or $WhatIfPreference
$script:Stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
$script:ServerProc = 'ZB.MOM.WW.MxGateway.Server'
$script:WorkerProc = 'ZB.MOM.WW.MxGateway.Worker'
$script:Notes = New-Object System.Collections.Generic.List[string]
$script:LockStream = $null
function Say([string]$m) { Write-Host $m; $script:Notes.Add($m) }
function Info([string]$m) { Say "[info] $m" }
function Warn([string]$m) { Write-Warning $m; $script:Notes.Add("[WARN] $m") }
function Fail([string]$m) { throw "ABORT: $m" }
# Mutating step: print in dry-run, run otherwise.
function Step([string]$Description, [scriptblock]$Action) {
if ($script:Dry) { Say "[DRYRUN] $Description"; return }
Say "[ACTION] $Description"
& $Action
}
# --- helpers --------------------------------------------------------------------------------------------
function Invoke-Nssm {
param([string[]]$NssmArgs, [switch]$AllowFail)
$raw = & $NssmPath @NssmArgs 2>&1
$code = $LASTEXITCODE
$text = (($raw | ForEach-Object { "$_" }) -join "`n") -replace "`0", ''
if ($code -ne 0 -and -not $AllowFail) { Fail "nssm $($NssmArgs -join ' ') failed (exit $code): $text" }
[pscustomobject]@{ Code = $code; Text = $text.Trim() }
}
function Get-NssmValue([string]$Name) { (Invoke-Nssm -NssmArgs @('get', $ServiceName, $Name)).Text }
function Get-ServiceState {
$r = Invoke-Nssm -NssmArgs @('status', $ServiceName) -AllowFail
if ($r.Code -ne 0) { return 'MISSING' }
($r.Text -split "`n")[0].Trim()
}
function Parse-EnvBlock([string]$Text) {
$map = [ordered]@{}
foreach ($line in ($Text -split "[`r`n]+")) {
$t = $line.Trim()
if ($t -match '^([^=\s]+)=(.*)$') { $map[$Matches[1]] = $Matches[2] }
}
$map
}
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
function Test-Inside([string]$Path, [string]$Root) {
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
$a -eq $b -or $a.StartsWith("$b/")
}
function Get-ProductSha([string]$Exe) {
try {
$pv = [System.Diagnostics.FileVersionInfo]::GetVersionInfo($Exe).ProductVersion
if ($pv -match '\+([0-9a-fA-F]{7,40})(\b|$)') { return $Matches[1].ToLowerInvariant() }
} catch { }
$null
}
function Test-ShaMatch([string]$Actual, [string]$Expected) {
if (-not $Actual -or -not $Expected) { return $false }
$e = $Expected.ToLowerInvariant()
$Actual.StartsWith($e) -or $e.StartsWith($Actual)
}
function Get-GatewayProcesses {
@(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in @($script:ServerProc, $script:WorkerProc) })
}
# Store file set: db + -wal + -shm
function Get-StoreFiles([string]$Db) { @($Db, "$Db-wal", "$Db-shm") | Where-Object { Test-Path -LiteralPath $_ } }
function Get-DirSize([string]$p) {
if (-not (Test-Path -LiteralPath $p)) { return 0 }
$s = (Get-ChildItem -LiteralPath $p -Recurse -File -Force -ErrorAction SilentlyContinue | Measure-Object Length -Sum).Sum
if ($s) { [double]$s } else { 0 }
}
# --- context discovery (read only) -------------------------------------------------------------------------
function Get-Context {
$state = Get-ServiceState
if ($state -eq 'MISSING') { Fail "service '$ServiceName' does not exist (use Install-MxGatewayService.ps1 for a new box)" }
$ctx = [ordered]@{}
$ctx.State = $state
$ctx.Application = Get-NssmValue 'Application'
$ctx.AppDirectory = Get-NssmValue 'AppDirectory'
$ctx.EnvText = Get-NssmValue 'AppEnvironmentExtra'
$ctx.Env = Parse-EnvBlock $ctx.EnvText
$ctx.EnvCount = $ctx.Env.Count
if (-not $ServerDir) { $ctx.ServerDir = $ctx.AppDirectory } else { $ctx.ServerDir = $ServerDir }
$wexe = $ctx.Env['MxGateway__Worker__ExecutablePath']
$ctx.WorkerExe = $wexe
if ($WorkerDir) { $ctx.WorkerDir = $WorkerDir }
elseif ($wexe) { $ctx.WorkerDir = Split-Path -Parent $wexe }
else { Fail 'cannot determine the Worker directory: env MxGateway__Worker__ExecutablePath is not in the NSSM env block; pass -WorkerDir' }
$pd = 'C:\ProgramData\MxGateway'
$auth = $ctx.Env['MxGateway__Authentication__SqlitePath']; if (-not $auth) { $auth = "$pd\gateway-auth.db" }
$sec = $ctx.Env['Secrets__SqlitePath']; if (-not $sec) { $sec = "$pd\mxgateway-secrets.db" }
$ctx.AuthDb = $auth; $ctx.SecretsDb = $sec
$ctx.ServerExeName = Split-Path -Leaf $ctx.Application
[pscustomobject]$ctx
}
# --- preflight -----------------------------------------------------------------------------------------------
function Invoke-Preflight($ctx) {
Say '== Preflight =='
Info "service '$ServiceName' state: $($ctx.State)"
Info "Application : $($ctx.Application)"
Info "AppDirectory: $($ctx.AppDirectory)"
Info "Server dir : $($ctx.ServerDir)"
Info "Worker dir : $($ctx.WorkerDir) (exe env: $($ctx.WorkerExe))"
Info "API-key DB : $($ctx.AuthDb)"
Info "Secrets DB : $($ctx.SecretsDb)"
Info "NSSM env block: $($ctx.EnvCount) entries (names only): $(($ctx.Env.Keys) -join ', ')"
if ($ctx.EnvCount -eq 0) { Fail 'NSSM env block is empty - refusing: it holds Kestrel/store paths, the KEK and the pepper' }
foreach ($need in 'ZB_SECRETS_MASTER_KEY') {
if (-not $ctx.Env.Contains($need)) { Fail "env block has no $need (the Secrets store would be unreadable after the restart)" }
}
if (-not ($ctx.Env.Keys | Where-Object { $_ -match 'Pepper' })) {
Warn 'no *Pepper* entry in the env block; confirm the API-key pepper is supplied another way or issued tokens will not verify'
}
if ($ctx.Env['ASPNETCORE_ENVIRONMENT'] -ne 'Production' -and $ctx.Env['DOTNET_ENVIRONMENT'] -ne 'Production') {
Warn 'neither ASPNETCORE_ENVIRONMENT nor DOTNET_ENVIRONMENT is exactly Production (runbook trap 6) - do not change it during an upgrade'
}
# State that must survive: rooted and outside the content root.
foreach ($pair in @(@('API-key DB', $ctx.AuthDb), @('Secrets DB', $ctx.SecretsDb))) {
if (-not (Test-Rooted $pair[1])) { Fail "$($pair[0]) path '$($pair[1])' is not absolute" }
foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) {
if (Test-Inside $pair[1] $root) { Fail "$($pair[0]) '$($pair[1])' is INSIDE '$root' - the swap would lose it. Move it out first (runbook section 1)." }
}
if (Test-Path -LiteralPath $pair[1]) { Info "$($pair[0]) present ($((Get-StoreFiles $pair[1]).Count) file(s) incl. wal/shm)" }
else { Warn "$($pair[0]) '$($pair[1])' does not exist yet (fresh store will be created on start)" }
}
# Directories
foreach ($d in @($ctx.ServerDir, $ctx.WorkerDir)) {
if (-not (Test-Path -LiteralPath $d -PathType Container)) { Fail "live directory '$d' does not exist" }
}
if ((Test-Inside $ctx.WorkerDir $ctx.ServerDir) -or (Test-Inside $ctx.ServerDir $ctx.WorkerDir)) {
Fail 'Server and Worker directories are nested; the rename-in-place swap needs sibling directories'
}
# One-gateway guard
$procs = Get-GatewayProcesses
$servers = @($procs | Where-Object { $_.ProcessName -eq $script:ServerProc })
if ($servers.Count -gt 1) { Fail "$($servers.Count) gateway Server processes are running (PIDs $(($servers.Id) -join ',')). A second gateway is the OrphanWorkerTerminator trap - resolve manually." }
if ($ctx.State -eq 'SERVICE_STOPPED' -and $procs.Count -gt 0) {
Fail "service is STOPPED but gateway process(es) exist (PIDs $(($procs.Id) -join ',')): a rogue second gateway. Starting would terminate its live workers."
}
if ($servers.Count -eq 1 -and $ctx.State -ne 'SERVICE_RUNNING' -and $ctx.State -ne 'SERVICE_STOP_PENDING') {
Fail "a Server process exists while the service is $($ctx.State)"
}
if (Get-Command Get-CimInstance -ErrorAction SilentlyContinue) {
$svcs = @(Get-CimInstance Win32_Service | Where-Object { $_.Name -ne $ServiceName -and $_.PathName -and $_.PathName -like "*$($ctx.ServerExeName)*" })
if ($svcs.Count -gt 0) { Fail "another service ($(($svcs.Name) -join ', ')) also launches $($ctx.ServerExeName); two gateways on one box is forbidden" }
} else { Info 'Win32_Service cross-check unavailable on this host (skipped)' }
if ($Rollback) { return }
# Sources
if (-not $ServerSource -or -not $WorkerSource) { Fail '-ServerSource and -WorkerSource are required for an upgrade' }
foreach ($s in @(@($ServerSource, $ctx.ServerExeName), @($WorkerSource, "$($script:WorkerProc).exe"))) {
if (-not (Test-Path -LiteralPath (Join-Path $s[0] $s[1]))) { Fail "staged source '$($s[0])' does not contain $($s[1])" }
}
if ((Test-Inside $ServerSource $ctx.ServerDir) -or (Test-Inside $WorkerSource $ctx.WorkerDir)) { Fail 'staging directory is inside a live directory' }
$sSha = Get-ProductSha (Join-Path $ServerSource $ctx.ServerExeName)
$wSha = Get-ProductSha (Join-Path $WorkerSource "$($script:WorkerProc).exe")
Info "staged Server SHA: $sSha ; staged Worker SHA: $wSha"
if ($sSha -and $wSha -and -not (Test-ShaMatch $sSha $wSha)) { Fail "Server ($sSha) and Worker ($wSha) were built from different commits (wire protocol risk)" }
if ($ExpectedSha) {
if (-not (Test-ShaMatch $sSha $ExpectedSha)) {
if ($script:Dry) { Warn "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha' (dry-run: continuing)" }
else { Fail "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha'" }
}
} else { Warn 'no -ExpectedSha given: the post-deploy SHA check will only record the SHA' }
# Disk space: 2x (server + worker + stores + payload), at least MinFreeGB
if (-not $SkipFreeSpaceCheck) {
$need = (Get-DirSize $ctx.ServerDir) + (Get-DirSize $ctx.WorkerDir) + (Get-DirSize $ServerSource) + (Get-DirSize $WorkerSource)
foreach ($f in (Get-StoreFiles $ctx.AuthDb) + (Get-StoreFiles $ctx.SecretsDb)) { $need += (Get-Item -LiteralPath $f).Length }
$need = [math]::Max($need * 2, $MinFreeGB * 1GB)
$root = [IO.Path]::GetPathRoot((Resolve-Path -LiteralPath $ctx.ServerDir).Path)
$free = (New-Object IO.DriveInfo $root).AvailableFreeSpace
Info ("free on {0}: {1:N2} GB, required {2:N2} GB" -f $root, ($free / 1GB), ($need / 1GB))
if ($free -lt $need) { Fail 'not enough free disk space for the backups' }
}
Say '== Preflight OK =='
}
# --- stop / start ---------------------------------------------------------------------------------------------
function Wait-State([string]$Want, [int]$TimeoutSec) {
$deadline = (Get-Date).AddSeconds($TimeoutSec)
do { $s = Get-ServiceState; if ($s -eq $Want) { return $true }; Start-Sleep -Seconds 1 } while ((Get-Date) -lt $deadline)
$false
}
function Assert-NoGatewayProcess([string]$When) {
$left = Get-GatewayProcesses
if ($left.Count -gt 0) { Fail "${When}: gateway process(es) still alive: $(($left | ForEach-Object { "$($_.ProcessName)#$($_.Id)" }) -join ', ')" }
}
function Stop-Gateway {
Say '-- Stop --'
$state = Get-ServiceState
if ($state -eq 'SERVICE_STOPPED') { Info 'already stopped' }
else {
Step "nssm stop $ServiceName (wait up to $StopTimeoutSeconds s for SERVICE_STOPPED)" {
[void](Invoke-Nssm -NssmArgs @('stop', $ServiceName) -AllowFail)
if (-not (Wait-State 'SERVICE_STOPPED' $StopTimeoutSeconds)) {
# STOP_PENDING trap: workers did not exit in time. Kill the Server (then Workers) by image name -
# safe because preflight proved there is exactly one gateway on the box.
Warn "service still $(Get-ServiceState) after $StopTimeoutSeconds s (STOP_PENDING trap) - killing gateway processes"
foreach ($n in @($script:ServerProc, $script:WorkerProc)) {
Get-Process -Name $n -ErrorAction SilentlyContinue | ForEach-Object { Say "[ACTION] Stop-Process -Id $($_.Id) -Force ($n)"; Stop-Process -Id $_.Id -Force }
}
if (-not (Wait-State 'SERVICE_STOPPED' 20)) {
Warn 'still not STOPPED; killing the NSSM service host process'
try {
$svcPid = (Get-CimInstance Win32_Service -Filter "Name='$ServiceName'").ProcessId
if ($svcPid) { Stop-Process -Id $svcPid -Force }
} catch { Warn "could not kill the service host: $_" }
if (-not (Wait-State 'SERVICE_STOPPED' 20)) { Fail "service '$ServiceName' will not stop; investigate manually (state $(Get-ServiceState))" }
}
}
Start-Sleep -Seconds 2
Assert-NoGatewayProcess 'after stop'
}
}
if (-not $script:Dry) { Assert-NoGatewayProcess 'after stop' }
}
function Start-Gateway($ctx) {
Say '-- Start --'
if (-not $script:Dry) { Assert-NoGatewayProcess 'before start (OrphanWorkerTerminator guard)' }
else { Say '[DRYRUN] assert no Server/Worker process exists (OrphanWorkerTerminator guard)' }
Step "nssm start $ServiceName (once)" {
[void](Invoke-Nssm -NssmArgs @('start', $ServiceName))
}
Step "wait for SERVICE_RUNNING and compare env entry count (expect $($ctx.EnvCount))" {
if (-not (Wait-State 'SERVICE_RUNNING' 60)) { Fail "service did not reach RUNNING (state $(Get-ServiceState))" }
$now = (Parse-EnvBlock (Get-NssmValue 'AppEnvironmentExtra')).Count
if ($now -ne $ctx.EnvCount) { Fail "NSSM env block entry count changed: $($ctx.EnvCount) -> $now" }
}
}
# --- backup / swap -----------------------------------------------------------------------------------------------
function Get-Suffix { if ($Label) { "$($script:Stamp)-$Label" } else { $script:Stamp } }
function Backup-State($ctx) {
Say '-- Backup (service stopped) --'
$suffix = Get-Suffix
$root = Split-Path -Parent $ctx.ServerDir
$envFile = Join-Path $root "_env-AppEnvironmentExtra-$($script:Stamp).txt"
Step "save the NSSM env block + Application/AppDirectory to $envFile (contains secrets: keep it ACL-restricted)" {
$body = @("Application=$($ctx.Application)", "AppDirectory=$($ctx.AppDirectory)", '--- AppEnvironmentExtra ---', $ctx.EnvText)
Set-Content -LiteralPath $envFile -Value $body -Encoding UTF8
try { & icacls.exe $envFile /inheritance:r /grant:r 'SYSTEM:(F)' 'BUILTIN\Administrators:(F)' | Out-Null } catch { Warn "could not restrict ACL on ${envFile}: $_" }
}
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
foreach ($f in (Get-StoreFiles $db)) {
$dst = "$f.bak-$suffix"
Step "copy store file $f -> $dst (live file is never moved)" { Copy-Item -LiteralPath $f -Destination $dst -Force }
}
if (-not (Test-Path -LiteralPath $db)) { Info "no store at $db to back up" }
}
}
function Swap-Directory([string]$Live, [string]$Source, [string]$Name, [bool]$KeepSettings) {
$bak = "$Live.bak-$(Get-Suffix)"
Step "rename $Name dir: $Live -> $bak" { Rename-Item -LiteralPath $Live -NewName (Split-Path -Leaf $bak) }
Step "copy new ${Name}: $Source -> $Live" { Copy-Item -LiteralPath $Source -Destination $Live -Recurse -Force }
if ($KeepSettings) {
$from = if ($script:Dry) { $Live } else { $bak } # in a dry run nothing was renamed yet
foreach ($f in 'appsettings.json', 'appsettings.Development.json') {
if (Test-Path -LiteralPath (Join-Path $from $f)) {
Step "restore live $f byte-identical from $bak" { Copy-Item -LiteralPath (Join-Path $bak $f) -Destination (Join-Path $Live $f) -Force }
} else { Info "$f not present in the old $Name dir (nothing to carry over)" }
}
}
$bak
}
# --- verification ----------------------------------------------------------------------------------------------------
function Invoke-Verify($ctx) {
Say '== Verify =='
if ($script:Dry) {
Say "[DRYRUN] GET $HealthBaseUrl/health/ready until 200 (<= $ReadyTimeoutSeconds s)"
Say "[DRYRUN] GET $HealthBaseUrl/healthz ; read ProductVersion SHA of $(Join-Path $ctx.ServerDir $ctx.ServerExeName)$(if ($ExpectedSha) { " and require it to match $ExpectedSha" })"
Say "[DRYRUN] service must stay SERVICE_RUNNING for $StableSeconds s with env entry count $($ctx.EnvCount)"
return
}
$ready = $false; $deadline = (Get-Date).AddSeconds($ReadyTimeoutSeconds)
while (-not $ready -and (Get-Date) -lt $deadline) {
try { $r = Invoke-WebRequest -Uri "$HealthBaseUrl/health/ready" -UseBasicParsing -TimeoutSec 10; $ready = ($r.StatusCode -eq 200) }
catch { Start-Sleep -Seconds 3 }
}
if (-not $ready) { Fail "$HealthBaseUrl/health/ready did not return 200 within $ReadyTimeoutSeconds s" }
Info '/health/ready = 200'
try { Info "/healthz = $((Invoke-WebRequest -Uri "$HealthBaseUrl/healthz" -UseBasicParsing -TimeoutSec 10).StatusCode)" } catch { Fail "/healthz failed: $_" }
$sha = Get-ProductSha (Join-Path $ctx.ServerDir $ctx.ServerExeName)
Info "deployed Server SHA (ProductVersion): $sha"
if ($ExpectedSha -and -not $Rollback -and -not (Test-ShaMatch $sha $ExpectedSha)) { Fail "deployed SHA '$sha' != expected '$ExpectedSha'" }
Info "watching the service for $StableSeconds s"
$until = (Get-Date).AddSeconds($StableSeconds)
while ((Get-Date) -lt $until) {
if ((Get-ServiceState) -ne 'SERVICE_RUNNING') { Fail 'service left RUNNING during the stability window (restart loop?)' }
Start-Sleep -Seconds 5
}
$n = (Parse-EnvBlock (Get-NssmValue 'AppEnvironmentExtra')).Count
if ($n -ne $ctx.EnvCount) { Fail "env block entry count $n != preflight $($ctx.EnvCount)" }
Info "env block still $n entries"
Warn 'health is green with zero sessions; still prove the Worker path: open a session with a PRE-EXISTING API key and read a known tag (runbook 7.3)'
}
# --- rollback ------------------------------------------------------------------------------------------------------------
function Find-Backup([string]$Live, [string]$Stamp) {
$leaf = Split-Path -Leaf $Live; $parent = Split-Path -Parent $Live
$cands = @(Get-ChildItem -LiteralPath $parent -Directory -Filter "$leaf.bak-*" -ErrorAction SilentlyContinue)
if ($Stamp) { $cands = @($cands | Where-Object { $_.Name -like "$leaf.bak-$Stamp*" }) }
$cands | Sort-Object Name -Descending | Select-Object -First 1
}
function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores) {
Say '== Rollback =='
$sb = Find-Backup $ctx.ServerDir $Stamp
$wb = Find-Backup $ctx.WorkerDir $Stamp
if (-not $sb -or -not $wb) { Fail "no matching Server/Worker backup directories found (stamp '$Stamp')" }
Info "rolling back to Server=$($sb.Name) Worker=$($wb.Name)"
Stop-Gateway
foreach ($pair in @(@($ctx.ServerDir, $sb), @($ctx.WorkerDir, $wb))) {
$failed = "$($pair[0]).failed-$($script:Stamp)"
if (Test-Path -LiteralPath $pair[0]) { Step "rename $($pair[0]) -> $failed" { Rename-Item -LiteralPath $pair[0] -NewName (Split-Path -Leaf $failed) } }
Step "rename $($pair[1].FullName) -> $($pair[0])" { Rename-Item -LiteralPath $pair[1].FullName -NewName (Split-Path -Leaf $pair[0]) }
}
if ($Stores) {
$bstamp = ($sb.Name -replace '^.*\.bak-', '')
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
foreach ($ext in '', '-wal', '-shm') {
$live = "$db$ext"; $bak = "$db$ext.bak-$bstamp"
if (Test-Path -LiteralPath $bak) { Step "restore store file $bak -> $live" { Copy-Item -LiteralPath $bak -Destination $live -Force } }
elseif ($ext -and (Test-Path -LiteralPath $live) -and (Test-Path -LiteralPath "$db.bak-$bstamp")) {
# a stale wal/shm must not pair with the restored db: move it aside, never delete
Step "move stale $live aside -> $live.stale-$($script:Stamp)" { Rename-Item -LiteralPath $live -NewName ("$(Split-Path -Leaf $live).stale-$($script:Stamp)") }
}
elseif (-not $ext) { Warn "no backup $bak for $db; live store left untouched" }
}
}
} else { Info 'stores left untouched (they live outside the swap); pass -RestoreStores after a schema bump' }
Start-Gateway $ctx
}
# --- main -------------------------------------------------------------------------------------------------------------------
$exit = 0
try {
Say "Upgrade-MxGatewayService stamp=$($script:Stamp) mode=$(if ($Rollback) {'ROLLBACK'} else {'UPGRADE'}) dry=$($script:Dry)"
$ctx = Get-Context
Invoke-Preflight $ctx
if (-not $script:Dry) {
$lockPath = Join-Path (Split-Path -Parent $ctx.ServerDir) '.mxgw-deploy.lock'
try { $script:LockStream = [IO.File]::Open($lockPath, 'OpenOrCreate', 'ReadWrite', 'None') }
catch { Fail "another deploy run holds $lockPath" }
}
if ($Rollback) {
Invoke-RollbackSwap $ctx $BackupStamp $RestoreStores.IsPresent
Invoke-Verify $ctx
} else {
Stop-Gateway
Backup-State $ctx
$swapped = $false
try {
$null = Swap-Directory $ctx.ServerDir $ServerSource 'Server' (-not $TakeNewAppSettings)
$null = Swap-Directory $ctx.WorkerDir $WorkerSource 'Worker' $false
$swapped = $true
Start-Gateway $ctx
Invoke-Verify $ctx
} catch {
$err = $_
Say "[FAIL] $($err.Exception.Message)"
if ($script:Dry -or $NoAutoRollback) { throw }
Warn 'upgrade failed - rolling back automatically'
try {
if (-not $swapped) { Warn 'swap did not complete; rollback swaps whatever .bak sets exist for this stamp' }
Invoke-RollbackSwap $ctx $script:Stamp $false
Invoke-Verify $ctx
Warn 'ROLLED BACK to the previous build; original failure follows'
} catch { Warn "AUTOMATIC ROLLBACK FAILED: $($_.Exception.Message). Manual intervention required (backups: *.bak-$(Get-Suffix))." }
throw $err
}
}
Say '== DONE =='
} catch {
$exit = 1
Write-Host "ERROR: $($_.Exception.Message)" -ForegroundColor Red
} finally {
if ($script:LockStream) { $script:LockStream.Dispose() }
try {
if (-not $script:Dry -and $ctx) {
$notes = Join-Path (Split-Path -Parent $ctx.ServerDir) "_DEPLOY-NOTES-$($script:Stamp).txt"
Set-Content -LiteralPath $notes -Value $script:Notes -Encoding UTF8
Write-Host "deploy notes: $notes"
}
} catch { }
}
exit $exit