feat(deploy): on-box NSSM upgrade/rollback and new-box install scripts, runbook updated (mxaccessgw#220)
Add scripts/deploy/Upgrade-MxGatewayService.ps1 (preflight, stop with STOP_PENDING handling, backup, Server+Worker swap, start, /health/ready + SHA verify, auto/manual rollback, one-gateway guard, never touches the NSSM env block) and Install-MxGatewayService.ps1 (service install, restart/recovery settings, env block). Both support -DryRun/-WhatIf. Dry-run validated on macOS pwsh against a fake layout; untested on a real box. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
# Deploy / Upgrade Runbook: NSSM-wrapped Gateway Service (Server + x86 Worker)
|
||||
|
||||
> Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). There is **no deploy script** in
|
||||
> this repo; this runbook is the procedure. It is assembled only from what the repo contains
|
||||
> Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). Since mxaccessgw#220 the procedure
|
||||
> is also scripted (see "Scripts" below); this runbook remains the spec the scripts implement. It is assembled only from what the repo contains
|
||||
> (`docs/GatewayConfiguration.md`, `docs/Authentication.md`, `docs/Diagnostics.md`,
|
||||
> `docs/runbooks/*`, `docs/plans/2026-08-07-*`, `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`,
|
||||
> `src/.../appsettings.json`, `scripts/ci/`) cross-checked against `scadaproj/deployment.md` §5 and §7.1.
|
||||
@@ -17,6 +17,72 @@
|
||||
> and direct SQL. Confirm the current rule before starting; this repo's note is dated 2026-09-11 and
|
||||
> the umbrella index has since relaxed it to deploy-only **[UNVERIFIED from this repo]**.
|
||||
|
||||
## Scripts (mxaccessgw#220)
|
||||
|
||||
> **UNTESTED ON A REAL BOX until the next deploy.** The scripts were only exercised on a Mac with
|
||||
> `pwsh` in `-DryRun` mode against a fake directory layout and a fake `nssm`
|
||||
> (parsing, preflight refusals, the one-gateway guard, the step sequence). Service stop/start,
|
||||
> STOP_PENDING handling, the health/SHA verification, `sc.exe failure`, the rollback swaps and the
|
||||
> NSSM UTF-16 output handling have not run against real Windows/NSSM. Run the first real use with
|
||||
> `-DryRun` on the box, read the plan, and have section 8 ready by hand.
|
||||
|
||||
| Script | Where it runs | Does |
|
||||
|---|---|---|
|
||||
| `scripts/deploy/Upgrade-MxGatewayService.ps1` | **On the Windows box**, Windows PowerShell 5.1 | Sections 3, 5, 6, 7, 8: preflight, stop, backup, swap Server + Worker, start, verify, rollback |
|
||||
| `scripts/deploy/Install-MxGatewayService.ps1` | On a **new** box | NSSM service install: recovery/restart, stop grace, log rotation, env block |
|
||||
|
||||
Both take `-DryRun` (or `-WhatIf`): read-only probes (`nssm get/status`, process list, disk, file
|
||||
existence) still run, every mutating action is printed instead of executed.
|
||||
|
||||
**Upgrade script** (`-ServerSource`, `-WorkerSource` are the staged outputs from section 4, built from one SHA):
|
||||
|
||||
- Parameters: `-ServerSource -WorkerSource -ServiceName MxAccessGw -NssmPath -ServerDir -WorkerDir
|
||||
-ExpectedSha -Label -HealthBaseUrl http://localhost:5130 -Rollback -BackupStamp -RestoreStores
|
||||
-NoAutoRollback -TakeNewAppSettings -SkipFreeSpaceCheck -MinFreeGB -StopTimeoutSeconds
|
||||
-StableSeconds -ReadyTimeoutSeconds -DryRun`. `ServerDir` defaults to NSSM `AppDirectory`, `WorkerDir`
|
||||
to the parent of `MxGateway__Worker__ExecutablePath` read from the env block.
|
||||
- Preflight refuses on: missing service, empty env block, no `ZB_SECRETS_MASTER_KEY`, a store path
|
||||
that is relative or inside the Server/Worker dir, staged Server/Worker built from different SHAs (or not
|
||||
matching `-ExpectedSha`), too little disk, **more than one gateway Server process, a gateway process
|
||||
while the service is STOPPED, or another service launching the same exe** (trap 1). It also asserts no
|
||||
Server/Worker process exists immediately before every `nssm start`.
|
||||
- Stop waits `-StopTimeoutSeconds`; on STOP_PENDING (trap 8) it kills the Server then Worker processes by
|
||||
image name (safe because preflight proved a single gateway), then the service host as a last resort.
|
||||
- Swap is **rename-in-place** (`Server` -> `Server.bak-<stamp>[-label]`, same for `Worker`), so NSSM
|
||||
`Application`/`AppDirectory`/`ExecutablePath` never change. Live `appsettings*.json` are copied back
|
||||
byte-identical. The stores are only **copied** (`.db`, `-wal`, `-shm` -> `*.bak-<stamp>`), never moved
|
||||
or deleted. The script **never runs `nssm set AppEnvironmentExtra`**; it saves the block (secrets
|
||||
included, ACL-restricted via `icacls`) to `_env-AppEnvironmentExtra-<stamp>.txt` and compares the entry count
|
||||
after start. Notes (names only, no values) go to `_DEPLOY-NOTES-<stamp>.txt` beside the install.
|
||||
- Verify: `/health/ready` = 200, `/healthz`, `ProductVersion` SHA of the deployed Server exe against
|
||||
`-ExpectedSha` (still subject to the version-stamp caveat in trap 7), service stays RUNNING for
|
||||
`-StableSeconds`, env count unchanged. Health with zero sessions does not prove the Worker path: do the
|
||||
section 7.3 API-key session read yourself.
|
||||
- A failed start or verify triggers an automatic rollback to the just-made backups (disable with
|
||||
`-NoAutoRollback`). Manual rollback: `-Rollback [-BackupStamp <stamp>] [-RestoreStores]`; the failed
|
||||
directories are kept as `*.failed-<stamp>`. `-RestoreStores` restores the db triplet from the matching
|
||||
`*.bak-<stamp>` (stale wal/shm are moved aside, nothing is deleted); use it only after a schema bump.
|
||||
|
||||
**Install script**: `-ServerDir -WorkerDir [-ServiceName -NssmPath -DataDir C:\ProgramData\MxGateway
|
||||
-LogDir -EnvFile -Env -ObjectName -ServiceAccountPassword -EnvironmentName -RestartDelayMs -ThrottleMs
|
||||
-StopGraceMs -RotateBytes -Start -DryRun]`. It refuses if the service exists or any gateway process is
|
||||
running, requires the KEK and a `*Pepper*` env entry, requires store paths absolute and outside the content
|
||||
root, and sets the env block in **one** `nssm set AppEnvironmentExtra` on a service that has none. Defaults
|
||||
it applies: auto start, `AppExit Default Restart`, restart delay 5 s, throttle 10 s, console stop grace 20 s
|
||||
(covers `Worker:ShutdownTimeoutSeconds` = 10), stdout/stderr to `<DataDir>\logs` with rotation,
|
||||
`sc failure ... restart/5000/restart/5000/restart/30000`, reset 1 day. Secret values are masked in output.
|
||||
The service is left stopped unless `-Start` is given.
|
||||
|
||||
**Driving from the Mac (winrm skill, deployment-only).** Remote execution on the wonder boxes is allowed
|
||||
only for the on-box steps of a deploy the owner asked for. The intended flow, none of which was run for #220:
|
||||
|
||||
1. Build both artifacts on windev from one SHA (section 4); `smb push` them to a staging dir on the box
|
||||
(UNC/SMB copies are always allowed), together with the two scripts.
|
||||
2. `winrm exec` the script with `-DryRun` first, read the printed plan, then run it for real:
|
||||
`powershell -NoProfile -ExecutionPolicy Bypass -File <stage>\Upgrade-MxGatewayService.ps1 -ServerSource <stage>\Server -WorkerSource <stage>\Worker -ExpectedSha <sha> -Label <label>`.
|
||||
One box at a time; the exit code is 0 on success, 1 on any abort or failed run.
|
||||
3. Do the section 7.3 Worker-path check from the Mac over gRPC (no remote exec needed), then write the deploy note.
|
||||
|
||||
## 0. What is being deployed
|
||||
|
||||
Two artifacts, deployed together because they share a wire protocol (`WorkerProtocolVersion`,
|
||||
@@ -267,7 +333,10 @@ window, API keys no longer authenticate, or a Worker session cannot be opened.
|
||||
- Whether the Server on each box is framework-dependent or self-contained (a box without .NET 10
|
||||
needs self-contained; the 2026-10-19 windev plan says framework-dependent).
|
||||
- The `STOP_PENDING` behaviour and the NSSM service account (`LocalSystem` per `deployment.md`).
|
||||
- NSSM install script: **none exists in the repo** - installing a *new* box (service creation,
|
||||
`AppExit`, recovery, log redirection settings) is not covered here; the only reference is
|
||||
`deployment.md`'s statement that VQ04 was cloned from VQ03's NSSM `AppEnvironmentExtra` config.
|
||||
- Automating this runbook as a script was the other half of the original finding; not done here.
|
||||
- New-box install and the upgrade are now scripted (see "Scripts"), but **untested on a real box**; the
|
||||
recovery/stop-grace/log-rotation defaults in `Install-MxGatewayService.ps1` are chosen from the repo's
|
||||
own settings (`Worker:ShutdownTimeoutSeconds`), not copied from an existing box. Compare against the
|
||||
`nssm dump` of a known-good box (vd03/vq03) before relying on them.
|
||||
- The Worker output folder and build-vs-publish question (section 4) is still **[UNVERIFIED]**: the upgrade
|
||||
script copies whatever the staged `-WorkerSource` directory contains over `Worker`, so stage exactly what
|
||||
the existing `Worker` folder holds. Confirm with a file listing of the live `Worker` dir on the next deploy.
|
||||
|
||||
@@ -0,0 +1,202 @@
|
||||
#requires -Version 5.1
|
||||
<#
|
||||
.SYNOPSIS
|
||||
On-box NEW-BOX install of the MxAccess Gateway as an NSSM service (mxaccessgw#220).
|
||||
Companion of Upgrade-MxGatewayService.ps1; see docs/runbooks/DeployUpgradeNssmService.md.
|
||||
|
||||
.DESCRIPTION
|
||||
RUN ON THE WINDOWS BOX (Windows PowerShell 5.1). Creates the service ONCE, with:
|
||||
* Application / AppDirectory / DisplayName / Description, auto start, account (default LocalSystem);
|
||||
* NSSM exit handling: restart on any exit with a throttle, graceful-stop timeouts that cover the
|
||||
Worker shutdown grace (MxGateway:Worker:ShutdownTimeoutSeconds, default 10 s);
|
||||
* stdout/stderr redirection with rotation;
|
||||
* Windows SCM recovery (sc failure: restart, restart, restart + reset period);
|
||||
* the environment block, set in ONE `nssm set AppEnvironmentExtra` call on a service that has none
|
||||
(so there is nothing to wipe). The env comes from -EnvFile (KEY=VALUE lines, # comments) and/or
|
||||
-Env (hashtable); -Env wins. Secret VALUES are never printed (dry-run shows names and masks values).
|
||||
It does NOT start the service unless -Start is given (review the env first), and it refuses to run if
|
||||
the service already exists (use Upgrade-MxGatewayService.ps1) or if any gateway process is running
|
||||
(OrphanWorkerTerminator trap: a second gateway kills live Workers).
|
||||
|
||||
Required env (checked by name): ZB_SECRETS_MASTER_KEY, a *Pepper* entry (MxGateway__ApiKeyPepper),
|
||||
MxGateway__Worker__ExecutablePath (defaults to <WorkerDir>\ZB.MOM.WW.MxGateway.Worker.exe),
|
||||
MxGateway__Authentication__SqlitePath and Secrets__SqlitePath (default under -DataDir; must be absolute and
|
||||
OUTSIDE the content root, the validator is fail-closed), ASPNETCORE_ENVIRONMENT/DOTNET_ENVIRONMENT.
|
||||
Kestrel endpoints default to docs/GatewayConfiguration.md's cleartext h2c :5120 + dashboard :5130 unless
|
||||
the env supplies Kestrel__Endpoints__*.
|
||||
|
||||
-DryRun (or -WhatIf) prints every mutating action and performs none.
|
||||
#>
|
||||
[CmdletBinding(SupportsShouldProcess = $true)]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)][string]$ServerDir,
|
||||
[Parameter(Mandatory = $true)][string]$WorkerDir,
|
||||
[string]$ServiceName = 'MxAccessGw',
|
||||
[string]$NssmPath = 'nssm',
|
||||
[string]$DataDir = 'C:\ProgramData\MxGateway',
|
||||
[string]$LogDir,
|
||||
[string]$EnvFile,
|
||||
[hashtable]$Env = @{},
|
||||
[string]$ServerExeName = 'ZB.MOM.WW.MxGateway.Server.exe',
|
||||
[string]$ObjectName = 'LocalSystem',
|
||||
[string]$ServiceAccountPassword,
|
||||
[string]$EnvironmentName = 'Production',
|
||||
[int]$RestartDelayMs = 5000,
|
||||
[int]$ThrottleMs = 10000,
|
||||
[int]$StopGraceMs = 20000,
|
||||
[int]$RotateBytes = 10485760,
|
||||
[switch]$Start,
|
||||
[switch]$DryRun
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$script:Dry = $DryRun.IsPresent -or $WhatIfPreference
|
||||
$ServerProc = 'ZB.MOM.WW.MxGateway.Server'
|
||||
$WorkerProc = 'ZB.MOM.WW.MxGateway.Worker'
|
||||
|
||||
function Say([string]$m) { Write-Host $m }
|
||||
function Fail([string]$m) { throw "ABORT: $m" }
|
||||
function Step([string]$Description, [scriptblock]$Action) {
|
||||
if ($script:Dry) { Say "[DRYRUN] $Description"; return }
|
||||
Say "[ACTION] $Description"; & $Action
|
||||
}
|
||||
function Invoke-Nssm {
|
||||
param([string[]]$NssmArgs, [switch]$AllowFail)
|
||||
$raw = & $NssmPath @NssmArgs 2>&1; $code = $LASTEXITCODE
|
||||
$text = (($raw | ForEach-Object { "$_" }) -join "`n") -replace "`0", ''
|
||||
if ($code -ne 0 -and -not $AllowFail) { Fail "nssm $($NssmArgs[0..1] -join ' ') failed (exit $code): $text" }
|
||||
[pscustomobject]@{ Code = $code; Text = $text.Trim() }
|
||||
}
|
||||
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
|
||||
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
|
||||
function Test-Inside([string]$Path, [string]$Root) {
|
||||
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
|
||||
$a -eq $b -or $a.StartsWith("$b/")
|
||||
}
|
||||
function Mask([string]$k, [string]$v) {
|
||||
if ($k -match 'KEY|PASSWORD|SECRET|PEPPER|TOKEN' -and $k -notmatch 'Path$|Url$') { '****' } else { $v }
|
||||
}
|
||||
|
||||
try {
|
||||
Say "Install-MxGatewayService service=$ServiceName dry=$($script:Dry)"
|
||||
|
||||
# --- environment assembly -------------------------------------------------------------------------
|
||||
$envMap = [ordered]@{}
|
||||
if ($EnvFile) {
|
||||
if (-not (Test-Path -LiteralPath $EnvFile)) { Fail "-EnvFile '$EnvFile' not found" }
|
||||
foreach ($line in (Get-Content -LiteralPath $EnvFile)) {
|
||||
$t = $line.Trim()
|
||||
if (-not $t -or $t.StartsWith('#')) { continue }
|
||||
if ($t -match '^([^=\s]+)=(.*)$') { $envMap[$Matches[1]] = $Matches[2] } else { Fail "-EnvFile line is not KEY=VALUE: '$t'" }
|
||||
}
|
||||
}
|
||||
foreach ($k in $Env.Keys) { $envMap[[string]$k] = [string]$Env[$k] }
|
||||
|
||||
$workerExe = Join-Path $WorkerDir "$WorkerProc.exe"
|
||||
if (-not $envMap.Contains('MxGateway__Worker__ExecutablePath')) { $envMap['MxGateway__Worker__ExecutablePath'] = $workerExe }
|
||||
if (-not $envMap.Contains('MxGateway__Authentication__SqlitePath')) { $envMap['MxGateway__Authentication__SqlitePath'] = Join-Path $DataDir 'gateway-auth.db' }
|
||||
if (-not $envMap.Contains('Secrets__SqlitePath')) { $envMap['Secrets__SqlitePath'] = Join-Path $DataDir 'mxgateway-secrets.db' }
|
||||
if (-not $envMap.Contains('ASPNETCORE_ENVIRONMENT')) { $envMap['ASPNETCORE_ENVIRONMENT'] = $EnvironmentName }
|
||||
if (-not $envMap.Contains('DOTNET_ENVIRONMENT')) { $envMap['DOTNET_ENVIRONMENT'] = $EnvironmentName }
|
||||
if (-not ($envMap.Keys | Where-Object { $_ -like 'Kestrel__Endpoints__*' })) {
|
||||
$envMap['Kestrel__Endpoints__Http__Url'] = 'http://0.0.0.0:5120'
|
||||
$envMap['Kestrel__Endpoints__Http__Protocols'] = 'Http2'
|
||||
$envMap['Kestrel__Endpoints__Dashboard__Url'] = 'http://0.0.0.0:5130'
|
||||
Say '[info] no Kestrel__Endpoints__* supplied: using the documented cleartext defaults (5120 h2c, 5130 dashboard)'
|
||||
}
|
||||
|
||||
# --- validation -------------------------------------------------------------------------------------
|
||||
if (-not $envMap.Contains('ZB_SECRETS_MASTER_KEY') -or -not $envMap['ZB_SECRETS_MASTER_KEY']) { Fail 'env must supply ZB_SECRETS_MASTER_KEY (the Secrets KEK)' }
|
||||
if (-not ($envMap.Keys | Where-Object { $_ -match 'Pepper' })) { Fail 'env must supply the API-key pepper (MxGateway__ApiKeyPepper); without it no issued token verifies' }
|
||||
foreach ($k in 'MxGateway__Authentication__SqlitePath', 'Secrets__SqlitePath') {
|
||||
$v = $envMap[$k]
|
||||
if (-not (Test-Rooted $v)) { Fail "$k '$v' must be absolute" }
|
||||
if ((Test-Inside $v $ServerDir) -or (Test-Inside $v $WorkerDir)) { Fail "$k '$v' is inside the content root; the validator is fail-closed and an upgrade swap would lose it" }
|
||||
}
|
||||
foreach ($d in @($ServerDir, $WorkerDir)) { if (-not (Test-Rooted $d)) { Fail "'$d' must be an absolute path" } }
|
||||
if ((Test-Inside $WorkerDir $ServerDir) -or (Test-Inside $ServerDir $WorkerDir)) { Fail 'Server and Worker directories must be siblings, not nested (the upgrade swap renames them independently)' }
|
||||
if (-not (Test-Path -LiteralPath (Join-Path $ServerDir $ServerExeName))) { Fail "$ServerExeName not found in $ServerDir" }
|
||||
if (-not (Test-Path -LiteralPath $workerExe)) { Fail "$WorkerProc.exe not found in $WorkerDir" }
|
||||
if ($envMap['MxGateway__Worker__ExecutablePath'] -ne $workerExe) { Say '[WARN] MxGateway__Worker__ExecutablePath differs from <WorkerDir>\Worker.exe; keep ONE canonical spelling' }
|
||||
if ($envMap['ASPNETCORE_ENVIRONMENT'] -ne 'Production') { Say '[WARN] ASPNETCORE_ENVIRONMENT is not exactly Production: DisableLogin / plaintext-LDAP hard-stops are only enforced for Production' }
|
||||
|
||||
# --- one-gateway guard + existence -------------------------------------------------------------------
|
||||
$st = Invoke-Nssm -NssmArgs @('status', $ServiceName) -AllowFail
|
||||
if ($st.Code -eq 0) { Fail "service '$ServiceName' already exists (state $($st.Text)); use Upgrade-MxGatewayService.ps1" }
|
||||
$procs = @(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in @($ServerProc, $WorkerProc) })
|
||||
if ($procs.Count -gt 0) { Fail "gateway process(es) already running (PIDs $(($procs.Id) -join ',')); installing/starting a second gateway kills their Workers" }
|
||||
|
||||
if (-not $LogDir) { $LogDir = Join-Path $DataDir 'logs' }
|
||||
$exe = Join-Path $ServerDir $ServerExeName
|
||||
|
||||
Say "[info] env entries ($($envMap.Count)):"
|
||||
foreach ($k in $envMap.Keys) { Say " $k=$(Mask $k $envMap[$k])" }
|
||||
|
||||
# --- install ------------------------------------------------------------------------------------------------
|
||||
Step "create $DataDir and $LogDir (stores/certs/logs live OUTSIDE the content root)" {
|
||||
New-Item -ItemType Directory -Force -Path $DataDir, $LogDir | Out-Null
|
||||
}
|
||||
Step "nssm install $ServiceName `"$exe`"" { [void](Invoke-Nssm -NssmArgs @('install', $ServiceName, $exe)); $script:Installed = $true }
|
||||
$settings = @(
|
||||
@('AppDirectory', $ServerDir),
|
||||
@('DisplayName', 'MxAccess Gateway'),
|
||||
@('Description', 'MxAccess Gateway (gRPC) - Server; launches the x86 Worker per session'),
|
||||
@('Start', 'SERVICE_AUTO_START'),
|
||||
@('Type', 'SERVICE_WIN32_OWN_PROCESS'),
|
||||
@('AppExit', 'Default', 'Restart'),
|
||||
@('AppRestartDelay', "$RestartDelayMs"),
|
||||
@('AppThrottle', "$ThrottleMs"),
|
||||
@('AppStopMethodConsole', "$StopGraceMs"),
|
||||
@('AppStopMethodWindow', '0'),
|
||||
@('AppStopMethodThreads', '0'),
|
||||
@('AppStdout', (Join-Path $LogDir 'service-stdout.log')),
|
||||
@('AppStderr', (Join-Path $LogDir 'service-stderr.log')),
|
||||
@('AppRotateFiles', '1'),
|
||||
@('AppRotateOnline', '1'),
|
||||
@('AppRotateBytes', "$RotateBytes")
|
||||
)
|
||||
foreach ($s in $settings) {
|
||||
Step "nssm set $ServiceName $($s -join ' ')" { [void](Invoke-Nssm -NssmArgs (@('set', $ServiceName) + $s)) }
|
||||
}
|
||||
if ($ObjectName -and $ObjectName -ne 'LocalSystem') {
|
||||
Step "nssm set $ServiceName ObjectName $ObjectName <password>" { [void](Invoke-Nssm -NssmArgs @('set', $ServiceName, 'ObjectName', $ObjectName, $ServiceAccountPassword)) }
|
||||
} else {
|
||||
Step "nssm set $ServiceName ObjectName LocalSystem" { [void](Invoke-Nssm -NssmArgs @('set', $ServiceName, 'ObjectName', 'LocalSystem')) }
|
||||
}
|
||||
|
||||
# Env block: first and only AppEnvironmentExtra write; the service has none, so nothing is wiped.
|
||||
$pairs = @($envMap.Keys | ForEach-Object { "$_=$($envMap[$_])" })
|
||||
Step "nssm set $ServiceName AppEnvironmentExtra <$($envMap.Count) entries, values hidden>" {
|
||||
[void](Invoke-Nssm -NssmArgs (@('set', $ServiceName, 'AppEnvironmentExtra') + $pairs))
|
||||
}
|
||||
|
||||
# SCM recovery: restart x3 (5 s, 5 s, 30 s), reset the counter after 1 day; also on non-crash exits.
|
||||
Step "sc.exe failure $ServiceName reset= 86400 actions= restart/5000/restart/5000/restart/30000" {
|
||||
& sc.exe failure $ServiceName reset= 86400 actions= 'restart/5000/restart/5000/restart/30000' | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { Fail "sc.exe failure exited $LASTEXITCODE" }
|
||||
}
|
||||
Step "sc.exe failureflag $ServiceName 1" {
|
||||
& sc.exe failureflag $ServiceName 1 | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { Fail "sc.exe failureflag exited $LASTEXITCODE" }
|
||||
}
|
||||
|
||||
# Verify read-back (names only) so a silent miss is caught before the first start.
|
||||
Step "verify: nssm get AppEnvironmentExtra has $($envMap.Count) entries" {
|
||||
$txt = (Invoke-Nssm -NssmArgs @('get', $ServiceName, 'AppEnvironmentExtra')).Text
|
||||
$n = @($txt -split "[`r`n]+" | Where-Object { $_.Trim() -match '^[^=\s]+=' }).Count
|
||||
if ($n -ne $envMap.Count) { Fail "env read-back has $n entries, expected $($envMap.Count)" }
|
||||
}
|
||||
|
||||
if ($Start) {
|
||||
Step "nssm start $ServiceName (exactly once)" { [void](Invoke-Nssm -NssmArgs @('start', $ServiceName)) }
|
||||
Say 'Next: GET /health/ready on the dashboard port, then prove a Worker session with an API key (runbook section 7).'
|
||||
} else {
|
||||
Say '[info] service installed STOPPED. Review the env, then `nssm start` (or re-run with -Start). Seed secrets with the `secret` CLI using the SAME Secrets__SqlitePath + ZB_SECRETS_MASTER_KEY.'
|
||||
}
|
||||
Say '== DONE =='
|
||||
exit 0
|
||||
} catch {
|
||||
Write-Host "ERROR: $($_.Exception.Message)" -ForegroundColor Red
|
||||
if ($script:Installed) { Write-Host "PARTIAL INSTALL: inspect, then `nssm remove $ServiceName confirm` and re-run." -ForegroundColor Yellow }
|
||||
exit 1
|
||||
}
|
||||
@@ -0,0 +1,490 @@
|
||||
#requires -Version 5.1
|
||||
<#
|
||||
.SYNOPSIS
|
||||
On-box upgrade / rollback of the NSSM-wrapped MxAccess Gateway (Server + x86 Worker).
|
||||
Implements docs/runbooks/DeployUpgradeNssmService.md sections 3, 5, 6, 7 and 8 (mxaccessgw#220).
|
||||
|
||||
.DESCRIPTION
|
||||
RUN THIS ON THE WINDOWS BOX ITSELF (Windows PowerShell 5.1 is enough). The new Server publish and the
|
||||
new x86 Worker build must already be staged on the box (UNC/SMB copy; build both from the same SHA).
|
||||
|
||||
Upgrade flow: preflight -> stop (STOP_PENDING handled) -> backup -> swap Server + Worker -> start ->
|
||||
verify (/health/ready, VersionInfo SHA, service stable) -> automatic rollback if start/verify fails.
|
||||
Rollback flow (-Rollback): stop -> swap the .bak directories back -> optional store restore -> start -> verify.
|
||||
|
||||
Safety properties:
|
||||
* Exactly one gateway per box. Refuses to run if a second gateway process / service points at the same
|
||||
exe, if the service is stopped but a gateway process exists, or if a Server/Worker process is still
|
||||
alive when a start is about to happen (OrphanWorkerTerminator trap, runbook section 2.1).
|
||||
* Never calls `nssm set ... AppEnvironmentExtra` (that replaces the whole block). The env block is only
|
||||
READ (saved to the backup dir, entry count compared before/after). The Application/AppDirectory
|
||||
are not changed either: the swap is rename-in-place, so the NSSM config is untouched.
|
||||
* API-key DB, Secrets store and master key live outside the content root and are never moved or
|
||||
deleted. They are only COPIED (db + -wal + -shm) into *.bak-<stamp> siblings while the service is stopped.
|
||||
The script refuses to run if a store path is inside the content root (the 2026-08-09 incident).
|
||||
|
||||
-DryRun (or -WhatIf) prints every mutating action and performs none. Read-only probing (nssm get/status,
|
||||
process list, disk space, file existence) still runs so the decision logic is exercised.
|
||||
|
||||
.PARAMETER ServerSource Staged Server publish output (contains ZB.MOM.WW.MxGateway.Server.exe).
|
||||
.PARAMETER WorkerSource Staged x86 Worker output (contains ZB.MOM.WW.MxGateway.Worker.exe).
|
||||
.PARAMETER ServiceName NSSM service name. Default MxAccessGw.
|
||||
.PARAMETER NssmPath nssm executable. Default 'nssm' (on PATH).
|
||||
.PARAMETER ServerDir Live Server directory. Default: NSSM AppDirectory.
|
||||
.PARAMETER WorkerDir Live Worker directory. Default: parent of env MxGateway__Worker__ExecutablePath.
|
||||
.PARAMETER ExpectedSha Short/long git SHA. Staged Server+Worker (preflight) and the deployed Server (verify)
|
||||
must carry it in ProductVersion ('0.1.2+<sha>'). Builds from 2026-07-09..08-10 have a bogus stamp.
|
||||
.PARAMETER Label Free-text suffix for backup names (e.g. 'planwrites').
|
||||
.PARAMETER HealthBaseUrl Default http://localhost:5130 (dashboard port; /health/ready, /healthz anonymous).
|
||||
.PARAMETER Rollback Roll back instead of upgrading. Uses newest Server.bak-* unless -BackupStamp is given.
|
||||
.PARAMETER BackupStamp Stamp (yyyyMMdd-HHmmss) of the backup set to roll back to.
|
||||
.PARAMETER RestoreStores With -Rollback: also restore gateway-auth.db / secrets db from *.bak-<stamp> (schema-bump case).
|
||||
.PARAMETER NoAutoRollback Do not roll back automatically when start/verification fails.
|
||||
.PARAMETER TakeNewAppSettings Do not carry the live appsettings*.json over into the new Server dir.
|
||||
.PARAMETER SkipFreeSpaceCheck / MinFreeGB / StopTimeoutSeconds / StableSeconds / ReadyTimeoutSeconds tunables.
|
||||
.PARAMETER DryRun Print, do not act. -WhatIf is equivalent.
|
||||
#>
|
||||
[CmdletBinding(SupportsShouldProcess = $true)]
|
||||
param(
|
||||
[string]$ServerSource,
|
||||
[string]$WorkerSource,
|
||||
[string]$ServiceName = 'MxAccessGw',
|
||||
[string]$NssmPath = 'nssm',
|
||||
[string]$ServerDir,
|
||||
[string]$WorkerDir,
|
||||
[string]$ExpectedSha,
|
||||
[string]$Label,
|
||||
[string]$HealthBaseUrl = 'http://localhost:5130',
|
||||
[switch]$Rollback,
|
||||
[string]$BackupStamp,
|
||||
[switch]$RestoreStores,
|
||||
[switch]$NoAutoRollback,
|
||||
[switch]$TakeNewAppSettings,
|
||||
[switch]$SkipFreeSpaceCheck,
|
||||
[double]$MinFreeGB = 2,
|
||||
[int]$StopTimeoutSeconds = 30,
|
||||
[int]$StableSeconds = 60,
|
||||
[int]$ReadyTimeoutSeconds = 120,
|
||||
[switch]$DryRun
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$script:Dry = $DryRun.IsPresent -or $WhatIfPreference
|
||||
$script:Stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
|
||||
$script:ServerProc = 'ZB.MOM.WW.MxGateway.Server'
|
||||
$script:WorkerProc = 'ZB.MOM.WW.MxGateway.Worker'
|
||||
$script:Notes = New-Object System.Collections.Generic.List[string]
|
||||
$script:LockStream = $null
|
||||
|
||||
function Say([string]$m) { Write-Host $m; $script:Notes.Add($m) }
|
||||
function Info([string]$m) { Say "[info] $m" }
|
||||
function Warn([string]$m) { Write-Warning $m; $script:Notes.Add("[WARN] $m") }
|
||||
function Fail([string]$m) { throw "ABORT: $m" }
|
||||
|
||||
# Mutating step: print in dry-run, run otherwise.
|
||||
function Step([string]$Description, [scriptblock]$Action) {
|
||||
if ($script:Dry) { Say "[DRYRUN] $Description"; return }
|
||||
Say "[ACTION] $Description"
|
||||
& $Action
|
||||
}
|
||||
|
||||
# --- helpers --------------------------------------------------------------------------------------------
|
||||
|
||||
function Invoke-Nssm {
|
||||
param([string[]]$NssmArgs, [switch]$AllowFail)
|
||||
$raw = & $NssmPath @NssmArgs 2>&1
|
||||
$code = $LASTEXITCODE
|
||||
$text = (($raw | ForEach-Object { "$_" }) -join "`n") -replace "`0", ''
|
||||
if ($code -ne 0 -and -not $AllowFail) { Fail "nssm $($NssmArgs -join ' ') failed (exit $code): $text" }
|
||||
[pscustomobject]@{ Code = $code; Text = $text.Trim() }
|
||||
}
|
||||
|
||||
function Get-NssmValue([string]$Name) { (Invoke-Nssm -NssmArgs @('get', $ServiceName, $Name)).Text }
|
||||
|
||||
function Get-ServiceState {
|
||||
$r = Invoke-Nssm -NssmArgs @('status', $ServiceName) -AllowFail
|
||||
if ($r.Code -ne 0) { return 'MISSING' }
|
||||
($r.Text -split "`n")[0].Trim()
|
||||
}
|
||||
|
||||
function Parse-EnvBlock([string]$Text) {
|
||||
$map = [ordered]@{}
|
||||
foreach ($line in ($Text -split "[`r`n]+")) {
|
||||
$t = $line.Trim()
|
||||
if ($t -match '^([^=\s]+)=(.*)$') { $map[$Matches[1]] = $Matches[2] }
|
||||
}
|
||||
$map
|
||||
}
|
||||
|
||||
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
|
||||
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
|
||||
function Test-Inside([string]$Path, [string]$Root) {
|
||||
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
|
||||
$a -eq $b -or $a.StartsWith("$b/")
|
||||
}
|
||||
|
||||
function Get-ProductSha([string]$Exe) {
|
||||
try {
|
||||
$pv = [System.Diagnostics.FileVersionInfo]::GetVersionInfo($Exe).ProductVersion
|
||||
if ($pv -match '\+([0-9a-fA-F]{7,40})(\b|$)') { return $Matches[1].ToLowerInvariant() }
|
||||
} catch { }
|
||||
$null
|
||||
}
|
||||
function Test-ShaMatch([string]$Actual, [string]$Expected) {
|
||||
if (-not $Actual -or -not $Expected) { return $false }
|
||||
$e = $Expected.ToLowerInvariant()
|
||||
$Actual.StartsWith($e) -or $e.StartsWith($Actual)
|
||||
}
|
||||
|
||||
function Get-GatewayProcesses {
|
||||
@(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in @($script:ServerProc, $script:WorkerProc) })
|
||||
}
|
||||
|
||||
# Store file set: db + -wal + -shm
|
||||
function Get-StoreFiles([string]$Db) { @($Db, "$Db-wal", "$Db-shm") | Where-Object { Test-Path -LiteralPath $_ } }
|
||||
|
||||
function Get-DirSize([string]$p) {
|
||||
if (-not (Test-Path -LiteralPath $p)) { return 0 }
|
||||
$s = (Get-ChildItem -LiteralPath $p -Recurse -File -Force -ErrorAction SilentlyContinue | Measure-Object Length -Sum).Sum
|
||||
if ($s) { [double]$s } else { 0 }
|
||||
}
|
||||
|
||||
# --- context discovery (read only) -------------------------------------------------------------------------
|
||||
|
||||
function Get-Context {
|
||||
$state = Get-ServiceState
|
||||
if ($state -eq 'MISSING') { Fail "service '$ServiceName' does not exist (use Install-MxGatewayService.ps1 for a new box)" }
|
||||
$ctx = [ordered]@{}
|
||||
$ctx.State = $state
|
||||
$ctx.Application = Get-NssmValue 'Application'
|
||||
$ctx.AppDirectory = Get-NssmValue 'AppDirectory'
|
||||
$ctx.EnvText = Get-NssmValue 'AppEnvironmentExtra'
|
||||
$ctx.Env = Parse-EnvBlock $ctx.EnvText
|
||||
$ctx.EnvCount = $ctx.Env.Count
|
||||
if (-not $ServerDir) { $ctx.ServerDir = $ctx.AppDirectory } else { $ctx.ServerDir = $ServerDir }
|
||||
$wexe = $ctx.Env['MxGateway__Worker__ExecutablePath']
|
||||
$ctx.WorkerExe = $wexe
|
||||
if ($WorkerDir) { $ctx.WorkerDir = $WorkerDir }
|
||||
elseif ($wexe) { $ctx.WorkerDir = Split-Path -Parent $wexe }
|
||||
else { Fail 'cannot determine the Worker directory: env MxGateway__Worker__ExecutablePath is not in the NSSM env block; pass -WorkerDir' }
|
||||
$pd = 'C:\ProgramData\MxGateway'
|
||||
$auth = $ctx.Env['MxGateway__Authentication__SqlitePath']; if (-not $auth) { $auth = "$pd\gateway-auth.db" }
|
||||
$sec = $ctx.Env['Secrets__SqlitePath']; if (-not $sec) { $sec = "$pd\mxgateway-secrets.db" }
|
||||
$ctx.AuthDb = $auth; $ctx.SecretsDb = $sec
|
||||
$ctx.ServerExeName = Split-Path -Leaf $ctx.Application
|
||||
[pscustomobject]$ctx
|
||||
}
|
||||
|
||||
# --- preflight -----------------------------------------------------------------------------------------------
|
||||
|
||||
function Invoke-Preflight($ctx) {
|
||||
Say '== Preflight =='
|
||||
Info "service '$ServiceName' state: $($ctx.State)"
|
||||
Info "Application : $($ctx.Application)"
|
||||
Info "AppDirectory: $($ctx.AppDirectory)"
|
||||
Info "Server dir : $($ctx.ServerDir)"
|
||||
Info "Worker dir : $($ctx.WorkerDir) (exe env: $($ctx.WorkerExe))"
|
||||
Info "API-key DB : $($ctx.AuthDb)"
|
||||
Info "Secrets DB : $($ctx.SecretsDb)"
|
||||
Info "NSSM env block: $($ctx.EnvCount) entries (names only): $(($ctx.Env.Keys) -join ', ')"
|
||||
|
||||
if ($ctx.EnvCount -eq 0) { Fail 'NSSM env block is empty - refusing: it holds Kestrel/store paths, the KEK and the pepper' }
|
||||
foreach ($need in 'ZB_SECRETS_MASTER_KEY') {
|
||||
if (-not $ctx.Env.Contains($need)) { Fail "env block has no $need (the Secrets store would be unreadable after the restart)" }
|
||||
}
|
||||
if (-not ($ctx.Env.Keys | Where-Object { $_ -match 'Pepper' })) {
|
||||
Warn 'no *Pepper* entry in the env block; confirm the API-key pepper is supplied another way or issued tokens will not verify'
|
||||
}
|
||||
if ($ctx.Env['ASPNETCORE_ENVIRONMENT'] -ne 'Production' -and $ctx.Env['DOTNET_ENVIRONMENT'] -ne 'Production') {
|
||||
Warn 'neither ASPNETCORE_ENVIRONMENT nor DOTNET_ENVIRONMENT is exactly Production (runbook trap 6) - do not change it during an upgrade'
|
||||
}
|
||||
|
||||
# State that must survive: rooted and outside the content root.
|
||||
foreach ($pair in @(@('API-key DB', $ctx.AuthDb), @('Secrets DB', $ctx.SecretsDb))) {
|
||||
if (-not (Test-Rooted $pair[1])) { Fail "$($pair[0]) path '$($pair[1])' is not absolute" }
|
||||
foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) {
|
||||
if (Test-Inside $pair[1] $root) { Fail "$($pair[0]) '$($pair[1])' is INSIDE '$root' - the swap would lose it. Move it out first (runbook section 1)." }
|
||||
}
|
||||
if (Test-Path -LiteralPath $pair[1]) { Info "$($pair[0]) present ($((Get-StoreFiles $pair[1]).Count) file(s) incl. wal/shm)" }
|
||||
else { Warn "$($pair[0]) '$($pair[1])' does not exist yet (fresh store will be created on start)" }
|
||||
}
|
||||
|
||||
# Directories
|
||||
foreach ($d in @($ctx.ServerDir, $ctx.WorkerDir)) {
|
||||
if (-not (Test-Path -LiteralPath $d -PathType Container)) { Fail "live directory '$d' does not exist" }
|
||||
}
|
||||
if ((Test-Inside $ctx.WorkerDir $ctx.ServerDir) -or (Test-Inside $ctx.ServerDir $ctx.WorkerDir)) {
|
||||
Fail 'Server and Worker directories are nested; the rename-in-place swap needs sibling directories'
|
||||
}
|
||||
|
||||
# One-gateway guard
|
||||
$procs = Get-GatewayProcesses
|
||||
$servers = @($procs | Where-Object { $_.ProcessName -eq $script:ServerProc })
|
||||
if ($servers.Count -gt 1) { Fail "$($servers.Count) gateway Server processes are running (PIDs $(($servers.Id) -join ',')). A second gateway is the OrphanWorkerTerminator trap - resolve manually." }
|
||||
if ($ctx.State -eq 'SERVICE_STOPPED' -and $procs.Count -gt 0) {
|
||||
Fail "service is STOPPED but gateway process(es) exist (PIDs $(($procs.Id) -join ',')): a rogue second gateway. Starting would terminate its live workers."
|
||||
}
|
||||
if ($servers.Count -eq 1 -and $ctx.State -ne 'SERVICE_RUNNING' -and $ctx.State -ne 'SERVICE_STOP_PENDING') {
|
||||
Fail "a Server process exists while the service is $($ctx.State)"
|
||||
}
|
||||
if (Get-Command Get-CimInstance -ErrorAction SilentlyContinue) {
|
||||
$svcs = @(Get-CimInstance Win32_Service | Where-Object { $_.Name -ne $ServiceName -and $_.PathName -and $_.PathName -like "*$($ctx.ServerExeName)*" })
|
||||
if ($svcs.Count -gt 0) { Fail "another service ($(($svcs.Name) -join ', ')) also launches $($ctx.ServerExeName); two gateways on one box is forbidden" }
|
||||
} else { Info 'Win32_Service cross-check unavailable on this host (skipped)' }
|
||||
|
||||
if ($Rollback) { return }
|
||||
|
||||
# Sources
|
||||
if (-not $ServerSource -or -not $WorkerSource) { Fail '-ServerSource and -WorkerSource are required for an upgrade' }
|
||||
foreach ($s in @(@($ServerSource, $ctx.ServerExeName), @($WorkerSource, "$($script:WorkerProc).exe"))) {
|
||||
if (-not (Test-Path -LiteralPath (Join-Path $s[0] $s[1]))) { Fail "staged source '$($s[0])' does not contain $($s[1])" }
|
||||
}
|
||||
if ((Test-Inside $ServerSource $ctx.ServerDir) -or (Test-Inside $WorkerSource $ctx.WorkerDir)) { Fail 'staging directory is inside a live directory' }
|
||||
$sSha = Get-ProductSha (Join-Path $ServerSource $ctx.ServerExeName)
|
||||
$wSha = Get-ProductSha (Join-Path $WorkerSource "$($script:WorkerProc).exe")
|
||||
Info "staged Server SHA: $sSha ; staged Worker SHA: $wSha"
|
||||
if ($sSha -and $wSha -and -not (Test-ShaMatch $sSha $wSha)) { Fail "Server ($sSha) and Worker ($wSha) were built from different commits (wire protocol risk)" }
|
||||
if ($ExpectedSha) {
|
||||
if (-not (Test-ShaMatch $sSha $ExpectedSha)) {
|
||||
if ($script:Dry) { Warn "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha' (dry-run: continuing)" }
|
||||
else { Fail "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha'" }
|
||||
}
|
||||
} else { Warn 'no -ExpectedSha given: the post-deploy SHA check will only record the SHA' }
|
||||
|
||||
# Disk space: 2x (server + worker + stores + payload), at least MinFreeGB
|
||||
if (-not $SkipFreeSpaceCheck) {
|
||||
$need = (Get-DirSize $ctx.ServerDir) + (Get-DirSize $ctx.WorkerDir) + (Get-DirSize $ServerSource) + (Get-DirSize $WorkerSource)
|
||||
foreach ($f in (Get-StoreFiles $ctx.AuthDb) + (Get-StoreFiles $ctx.SecretsDb)) { $need += (Get-Item -LiteralPath $f).Length }
|
||||
$need = [math]::Max($need * 2, $MinFreeGB * 1GB)
|
||||
$root = [IO.Path]::GetPathRoot((Resolve-Path -LiteralPath $ctx.ServerDir).Path)
|
||||
$free = (New-Object IO.DriveInfo $root).AvailableFreeSpace
|
||||
Info ("free on {0}: {1:N2} GB, required {2:N2} GB" -f $root, ($free / 1GB), ($need / 1GB))
|
||||
if ($free -lt $need) { Fail 'not enough free disk space for the backups' }
|
||||
}
|
||||
Say '== Preflight OK =='
|
||||
}
|
||||
|
||||
# --- stop / start ---------------------------------------------------------------------------------------------
|
||||
|
||||
function Wait-State([string]$Want, [int]$TimeoutSec) {
|
||||
$deadline = (Get-Date).AddSeconds($TimeoutSec)
|
||||
do { $s = Get-ServiceState; if ($s -eq $Want) { return $true }; Start-Sleep -Seconds 1 } while ((Get-Date) -lt $deadline)
|
||||
$false
|
||||
}
|
||||
|
||||
function Assert-NoGatewayProcess([string]$When) {
|
||||
$left = Get-GatewayProcesses
|
||||
if ($left.Count -gt 0) { Fail "${When}: gateway process(es) still alive: $(($left | ForEach-Object { "$($_.ProcessName)#$($_.Id)" }) -join ', ')" }
|
||||
}
|
||||
|
||||
function Stop-Gateway {
|
||||
Say '-- Stop --'
|
||||
$state = Get-ServiceState
|
||||
if ($state -eq 'SERVICE_STOPPED') { Info 'already stopped' }
|
||||
else {
|
||||
Step "nssm stop $ServiceName (wait up to $StopTimeoutSeconds s for SERVICE_STOPPED)" {
|
||||
[void](Invoke-Nssm -NssmArgs @('stop', $ServiceName) -AllowFail)
|
||||
if (-not (Wait-State 'SERVICE_STOPPED' $StopTimeoutSeconds)) {
|
||||
# STOP_PENDING trap: workers did not exit in time. Kill the Server (then Workers) by image name -
|
||||
# safe because preflight proved there is exactly one gateway on the box.
|
||||
Warn "service still $(Get-ServiceState) after $StopTimeoutSeconds s (STOP_PENDING trap) - killing gateway processes"
|
||||
foreach ($n in @($script:ServerProc, $script:WorkerProc)) {
|
||||
Get-Process -Name $n -ErrorAction SilentlyContinue | ForEach-Object { Say "[ACTION] Stop-Process -Id $($_.Id) -Force ($n)"; Stop-Process -Id $_.Id -Force }
|
||||
}
|
||||
if (-not (Wait-State 'SERVICE_STOPPED' 20)) {
|
||||
Warn 'still not STOPPED; killing the NSSM service host process'
|
||||
try {
|
||||
$svcPid = (Get-CimInstance Win32_Service -Filter "Name='$ServiceName'").ProcessId
|
||||
if ($svcPid) { Stop-Process -Id $svcPid -Force }
|
||||
} catch { Warn "could not kill the service host: $_" }
|
||||
if (-not (Wait-State 'SERVICE_STOPPED' 20)) { Fail "service '$ServiceName' will not stop; investigate manually (state $(Get-ServiceState))" }
|
||||
}
|
||||
}
|
||||
Start-Sleep -Seconds 2
|
||||
Assert-NoGatewayProcess 'after stop'
|
||||
}
|
||||
}
|
||||
if (-not $script:Dry) { Assert-NoGatewayProcess 'after stop' }
|
||||
}
|
||||
|
||||
function Start-Gateway($ctx) {
|
||||
Say '-- Start --'
|
||||
if (-not $script:Dry) { Assert-NoGatewayProcess 'before start (OrphanWorkerTerminator guard)' }
|
||||
else { Say '[DRYRUN] assert no Server/Worker process exists (OrphanWorkerTerminator guard)' }
|
||||
Step "nssm start $ServiceName (once)" {
|
||||
[void](Invoke-Nssm -NssmArgs @('start', $ServiceName))
|
||||
}
|
||||
Step "wait for SERVICE_RUNNING and compare env entry count (expect $($ctx.EnvCount))" {
|
||||
if (-not (Wait-State 'SERVICE_RUNNING' 60)) { Fail "service did not reach RUNNING (state $(Get-ServiceState))" }
|
||||
$now = (Parse-EnvBlock (Get-NssmValue 'AppEnvironmentExtra')).Count
|
||||
if ($now -ne $ctx.EnvCount) { Fail "NSSM env block entry count changed: $($ctx.EnvCount) -> $now" }
|
||||
}
|
||||
}
|
||||
|
||||
# --- backup / swap -----------------------------------------------------------------------------------------------
|
||||
|
||||
function Get-Suffix { if ($Label) { "$($script:Stamp)-$Label" } else { $script:Stamp } }
|
||||
|
||||
function Backup-State($ctx) {
|
||||
Say '-- Backup (service stopped) --'
|
||||
$suffix = Get-Suffix
|
||||
$root = Split-Path -Parent $ctx.ServerDir
|
||||
$envFile = Join-Path $root "_env-AppEnvironmentExtra-$($script:Stamp).txt"
|
||||
Step "save the NSSM env block + Application/AppDirectory to $envFile (contains secrets: keep it ACL-restricted)" {
|
||||
$body = @("Application=$($ctx.Application)", "AppDirectory=$($ctx.AppDirectory)", '--- AppEnvironmentExtra ---', $ctx.EnvText)
|
||||
Set-Content -LiteralPath $envFile -Value $body -Encoding UTF8
|
||||
try { & icacls.exe $envFile /inheritance:r /grant:r 'SYSTEM:(F)' 'BUILTIN\Administrators:(F)' | Out-Null } catch { Warn "could not restrict ACL on ${envFile}: $_" }
|
||||
}
|
||||
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
|
||||
foreach ($f in (Get-StoreFiles $db)) {
|
||||
$dst = "$f.bak-$suffix"
|
||||
Step "copy store file $f -> $dst (live file is never moved)" { Copy-Item -LiteralPath $f -Destination $dst -Force }
|
||||
}
|
||||
if (-not (Test-Path -LiteralPath $db)) { Info "no store at $db to back up" }
|
||||
}
|
||||
}
|
||||
|
||||
function Swap-Directory([string]$Live, [string]$Source, [string]$Name, [bool]$KeepSettings) {
|
||||
$bak = "$Live.bak-$(Get-Suffix)"
|
||||
Step "rename $Name dir: $Live -> $bak" { Rename-Item -LiteralPath $Live -NewName (Split-Path -Leaf $bak) }
|
||||
Step "copy new ${Name}: $Source -> $Live" { Copy-Item -LiteralPath $Source -Destination $Live -Recurse -Force }
|
||||
if ($KeepSettings) {
|
||||
$from = if ($script:Dry) { $Live } else { $bak } # in a dry run nothing was renamed yet
|
||||
foreach ($f in 'appsettings.json', 'appsettings.Development.json') {
|
||||
if (Test-Path -LiteralPath (Join-Path $from $f)) {
|
||||
Step "restore live $f byte-identical from $bak" { Copy-Item -LiteralPath (Join-Path $bak $f) -Destination (Join-Path $Live $f) -Force }
|
||||
} else { Info "$f not present in the old $Name dir (nothing to carry over)" }
|
||||
}
|
||||
}
|
||||
$bak
|
||||
}
|
||||
|
||||
# --- verification ----------------------------------------------------------------------------------------------------
|
||||
|
||||
function Invoke-Verify($ctx) {
|
||||
Say '== Verify =='
|
||||
if ($script:Dry) {
|
||||
Say "[DRYRUN] GET $HealthBaseUrl/health/ready until 200 (<= $ReadyTimeoutSeconds s)"
|
||||
Say "[DRYRUN] GET $HealthBaseUrl/healthz ; read ProductVersion SHA of $(Join-Path $ctx.ServerDir $ctx.ServerExeName)$(if ($ExpectedSha) { " and require it to match $ExpectedSha" })"
|
||||
Say "[DRYRUN] service must stay SERVICE_RUNNING for $StableSeconds s with env entry count $($ctx.EnvCount)"
|
||||
return
|
||||
}
|
||||
$ready = $false; $deadline = (Get-Date).AddSeconds($ReadyTimeoutSeconds)
|
||||
while (-not $ready -and (Get-Date) -lt $deadline) {
|
||||
try { $r = Invoke-WebRequest -Uri "$HealthBaseUrl/health/ready" -UseBasicParsing -TimeoutSec 10; $ready = ($r.StatusCode -eq 200) }
|
||||
catch { Start-Sleep -Seconds 3 }
|
||||
}
|
||||
if (-not $ready) { Fail "$HealthBaseUrl/health/ready did not return 200 within $ReadyTimeoutSeconds s" }
|
||||
Info '/health/ready = 200'
|
||||
try { Info "/healthz = $((Invoke-WebRequest -Uri "$HealthBaseUrl/healthz" -UseBasicParsing -TimeoutSec 10).StatusCode)" } catch { Fail "/healthz failed: $_" }
|
||||
|
||||
$sha = Get-ProductSha (Join-Path $ctx.ServerDir $ctx.ServerExeName)
|
||||
Info "deployed Server SHA (ProductVersion): $sha"
|
||||
if ($ExpectedSha -and -not $Rollback -and -not (Test-ShaMatch $sha $ExpectedSha)) { Fail "deployed SHA '$sha' != expected '$ExpectedSha'" }
|
||||
|
||||
Info "watching the service for $StableSeconds s"
|
||||
$until = (Get-Date).AddSeconds($StableSeconds)
|
||||
while ((Get-Date) -lt $until) {
|
||||
if ((Get-ServiceState) -ne 'SERVICE_RUNNING') { Fail 'service left RUNNING during the stability window (restart loop?)' }
|
||||
Start-Sleep -Seconds 5
|
||||
}
|
||||
$n = (Parse-EnvBlock (Get-NssmValue 'AppEnvironmentExtra')).Count
|
||||
if ($n -ne $ctx.EnvCount) { Fail "env block entry count $n != preflight $($ctx.EnvCount)" }
|
||||
Info "env block still $n entries"
|
||||
Warn 'health is green with zero sessions; still prove the Worker path: open a session with a PRE-EXISTING API key and read a known tag (runbook 7.3)'
|
||||
}
|
||||
|
||||
# --- rollback ------------------------------------------------------------------------------------------------------------
|
||||
|
||||
function Find-Backup([string]$Live, [string]$Stamp) {
|
||||
$leaf = Split-Path -Leaf $Live; $parent = Split-Path -Parent $Live
|
||||
$cands = @(Get-ChildItem -LiteralPath $parent -Directory -Filter "$leaf.bak-*" -ErrorAction SilentlyContinue)
|
||||
if ($Stamp) { $cands = @($cands | Where-Object { $_.Name -like "$leaf.bak-$Stamp*" }) }
|
||||
$cands | Sort-Object Name -Descending | Select-Object -First 1
|
||||
}
|
||||
|
||||
function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores) {
|
||||
Say '== Rollback =='
|
||||
$sb = Find-Backup $ctx.ServerDir $Stamp
|
||||
$wb = Find-Backup $ctx.WorkerDir $Stamp
|
||||
if (-not $sb -or -not $wb) { Fail "no matching Server/Worker backup directories found (stamp '$Stamp')" }
|
||||
Info "rolling back to Server=$($sb.Name) Worker=$($wb.Name)"
|
||||
Stop-Gateway
|
||||
foreach ($pair in @(@($ctx.ServerDir, $sb), @($ctx.WorkerDir, $wb))) {
|
||||
$failed = "$($pair[0]).failed-$($script:Stamp)"
|
||||
if (Test-Path -LiteralPath $pair[0]) { Step "rename $($pair[0]) -> $failed" { Rename-Item -LiteralPath $pair[0] -NewName (Split-Path -Leaf $failed) } }
|
||||
Step "rename $($pair[1].FullName) -> $($pair[0])" { Rename-Item -LiteralPath $pair[1].FullName -NewName (Split-Path -Leaf $pair[0]) }
|
||||
}
|
||||
if ($Stores) {
|
||||
$bstamp = ($sb.Name -replace '^.*\.bak-', '')
|
||||
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
|
||||
foreach ($ext in '', '-wal', '-shm') {
|
||||
$live = "$db$ext"; $bak = "$db$ext.bak-$bstamp"
|
||||
if (Test-Path -LiteralPath $bak) { Step "restore store file $bak -> $live" { Copy-Item -LiteralPath $bak -Destination $live -Force } }
|
||||
elseif ($ext -and (Test-Path -LiteralPath $live) -and (Test-Path -LiteralPath "$db.bak-$bstamp")) {
|
||||
# a stale wal/shm must not pair with the restored db: move it aside, never delete
|
||||
Step "move stale $live aside -> $live.stale-$($script:Stamp)" { Rename-Item -LiteralPath $live -NewName ("$(Split-Path -Leaf $live).stale-$($script:Stamp)") }
|
||||
}
|
||||
elseif (-not $ext) { Warn "no backup $bak for $db; live store left untouched" }
|
||||
}
|
||||
}
|
||||
} else { Info 'stores left untouched (they live outside the swap); pass -RestoreStores after a schema bump' }
|
||||
Start-Gateway $ctx
|
||||
}
|
||||
|
||||
# --- main -------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
$exit = 0
|
||||
try {
|
||||
Say "Upgrade-MxGatewayService stamp=$($script:Stamp) mode=$(if ($Rollback) {'ROLLBACK'} else {'UPGRADE'}) dry=$($script:Dry)"
|
||||
$ctx = Get-Context
|
||||
Invoke-Preflight $ctx
|
||||
|
||||
if (-not $script:Dry) {
|
||||
$lockPath = Join-Path (Split-Path -Parent $ctx.ServerDir) '.mxgw-deploy.lock'
|
||||
try { $script:LockStream = [IO.File]::Open($lockPath, 'OpenOrCreate', 'ReadWrite', 'None') }
|
||||
catch { Fail "another deploy run holds $lockPath" }
|
||||
}
|
||||
|
||||
if ($Rollback) {
|
||||
Invoke-RollbackSwap $ctx $BackupStamp $RestoreStores.IsPresent
|
||||
Invoke-Verify $ctx
|
||||
} else {
|
||||
Stop-Gateway
|
||||
Backup-State $ctx
|
||||
$swapped = $false
|
||||
try {
|
||||
$null = Swap-Directory $ctx.ServerDir $ServerSource 'Server' (-not $TakeNewAppSettings)
|
||||
$null = Swap-Directory $ctx.WorkerDir $WorkerSource 'Worker' $false
|
||||
$swapped = $true
|
||||
Start-Gateway $ctx
|
||||
Invoke-Verify $ctx
|
||||
} catch {
|
||||
$err = $_
|
||||
Say "[FAIL] $($err.Exception.Message)"
|
||||
if ($script:Dry -or $NoAutoRollback) { throw }
|
||||
Warn 'upgrade failed - rolling back automatically'
|
||||
try {
|
||||
if (-not $swapped) { Warn 'swap did not complete; rollback swaps whatever .bak sets exist for this stamp' }
|
||||
Invoke-RollbackSwap $ctx $script:Stamp $false
|
||||
Invoke-Verify $ctx
|
||||
Warn 'ROLLED BACK to the previous build; original failure follows'
|
||||
} catch { Warn "AUTOMATIC ROLLBACK FAILED: $($_.Exception.Message). Manual intervention required (backups: *.bak-$(Get-Suffix))." }
|
||||
throw $err
|
||||
}
|
||||
}
|
||||
Say '== DONE =='
|
||||
} catch {
|
||||
$exit = 1
|
||||
Write-Host "ERROR: $($_.Exception.Message)" -ForegroundColor Red
|
||||
} finally {
|
||||
if ($script:LockStream) { $script:LockStream.Dispose() }
|
||||
try {
|
||||
if (-not $script:Dry -and $ctx) {
|
||||
$notes = Join-Path (Split-Path -Parent $ctx.ServerDir) "_DEPLOY-NOTES-$($script:Stamp).txt"
|
||||
Set-Content -LiteralPath $notes -Value $script:Notes -Encoding UTF8
|
||||
Write-Host "deploy notes: $notes"
|
||||
}
|
||||
} catch { }
|
||||
}
|
||||
exit $exit
|
||||
Reference in New Issue
Block a user