diff --git a/docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md b/docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md index f7b8dc8d..6265c023 100644 --- a/docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md +++ b/docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md @@ -34,7 +34,9 @@ The deploy used `preflight/swap/rollback` scripts adapted from the proven `2b135 (`E:\psrem-stage\mxgw-229-76a71bc8\{common,pre,swap,rollback}229.ps1`). They match both exe names, carry `appsettings.json` (hash-checked), `appsettings.Development.json` and `dashboard.pfx` forward, never write NSSM `Application`/`AppDirectory`/`AppEnvironmentExtra`, and keep backups in the protected stage folder. The script gaps are filed -as #230. +as #230. **Fixed in #230:** the repo script now matches both exe naming schemes and fails closed, carries `dashboard.pfx`, +accepts the file-based master key, fingerprints the env block, refuses a runtime downgrade, and can roll back these +`.bak.20261112T211945` sets. Use it for QA. ## Timeline (box-local, EST) diff --git a/docs/runbooks/DeployUpgradeNssmService.md b/docs/runbooks/DeployUpgradeNssmService.md index 46006c27..62ae7729 100644 --- a/docs/runbooks/DeployUpgradeNssmService.md +++ b/docs/runbooks/DeployUpgradeNssmService.md @@ -19,12 +19,25 @@ ## Scripts (mxaccessgw#220) -> **UNTESTED ON A REAL BOX until the next deploy.** The scripts were only exercised on a Mac with -> `pwsh` in `-DryRun` mode against a fake directory layout and a fake `nssm` -> (parsing, preflight refusals, the one-gateway guard, the step sequence). Service stop/start, -> STOP_PENDING handling, the health/SHA verification, `sc.exe failure`, the rollback swaps and the -> NSSM UTF-16 output handling have not run against real Windows/NSSM. Run the first real use with -> `-DryRun` on the box, read the plan, and have section 8 ready by hand. +> **Test status (mxaccessgw#230, 2026-11-12).** The upgrade script has run in `-DryRun` against the real +> windev install (Windows PowerShell 5.1, live service, live processes): the guards found the running +> gateway and its Workers, and a copy with the image names removed failed closed. It has **not yet run a real +> upgrade on any box**: the first real use (QA, #229) should start with `-DryRun` on the box, read the plan, and +> have section 8 ready by hand. Stop/start, STOP_PENDING, carry-over, the env-block checks and both rollbacks are +> covered by the harness in `scripts/deploy/tests/` against a fake box (temp directories really renamed and copied, +> fake `nssm`/process table/registry/clock), in both naming schemes: +> `pwsh -NoProfile -File scripts/deploy/tests/Test-UpgradeMxGatewayService.ps1` (assertions) and +> `pwsh -NoProfile -File scripts/deploy/tests/Invoke-UpgradeMutations.ps1` (re-introduces each guarded defect +> and requires the harness to fail). The harness also passes under Windows PowerShell 5.1 on windev. The windev +> run caught two defects the Mac harness could not: an empty `Win32_Service` result read as "check unavailable", +> and Windows PowerShell 5.1's `Get-FileHash` returning nothing under `-WhatIf`, which aborted every `-WhatIf` plan. +> Both are fixed. `Install-MxGatewayService.ps1` is still untested on a real box. +> +> **Before #230 the upgrade script was not safe on the wonder boxes**: its process guards matched only +> `ZB.MOM.WW.MxGateway.*`, but vd03/vq03/vq04 run `MxGateway.Server.exe` / `MxGateway.Worker.exe`, so the +> one-gateway guard, the pre-start OrphanWorkerTerminator guard and the STOP_PENDING kill passed vacuously; +> `Server\dashboard.pfx` was dropped; and preflight refused a file-based master key. Do not use a copy of the +> script older than the #230 commit. | Script | Where it runs | Does | |---|---|---| @@ -38,30 +51,65 @@ existence) still run, every mutating action is printed instead of executed. - Parameters: `-ServerSource -WorkerSource -ServiceName MxAccessGw -NssmPath -ServerDir -WorkerDir -ExpectedSha -Label -HealthBaseUrl http://localhost:5130 -Rollback -BackupStamp -RestoreStores - -NoAutoRollback -TakeNewAppSettings -SkipFreeSpaceCheck -MinFreeGB -StopTimeoutSeconds + -NoAutoRollback -TakeNewAppSettings -AllowRuntimeDowngrade -SkipFreeSpaceCheck -MinFreeGB -StopTimeoutSeconds -StableSeconds -ReadyTimeoutSeconds -DryRun`. `ServerDir` defaults to NSSM `AppDirectory`, `WorkerDir` to the parent of `MxGateway__Worker__ExecutablePath` read from the env block. -- Preflight refuses on: missing service, empty env block, no `ZB_SECRETS_MASTER_KEY`, a store path - that is relative or inside the Server/Worker dir, staged Server/Worker built from different SHAs (or not - matching `-ExpectedSha`), too little disk, **more than one gateway Server process, a gateway process - while the service is STOPPED, or another service launching the same exe** (trap 1). It also asserts no - Server/Worker process exists immediately before every `nssm start`. -- Stop waits `-StopTimeoutSeconds`; on STOP_PENDING (trap 8) it kills the Server then Worker processes by - image name (safe because preflight proved a single gateway), then the service host as a last resort. +- **Process matching (trap 1).** Gateway processes are recognised by image name **and** by path. The names + are those of the installed exes (the leaf of NSSM `Application` and of `MxGateway__Worker__ExecutablePath`) + plus both naming schemes in the field: `ZB.MOM.WW.MxGateway.Server`/`.Worker` (windev) and + `MxGateway.Server`/`.Worker` (wonder boxes), so a stray gateway of the other scheme is caught too. The + process table comes from `Win32_Process` (path + parent PID). +- **The one-gateway guard fails closed.** With the service RUNNING it requires exactly one gateway Server, at + the `Application` path, whose parent is the NSSM service host PID. A RUNNING service whose gateway it cannot + find is an **abort**, not a pass (that is how the pre-#230 guard went blind). It also refuses: more than one + gateway Server; any gateway/Worker process while the service is STOPPED; a gateway while the service is in + another state; another service launching the same exe. It asserts no Server/Worker process exists right + before every `nssm start`. +- Preflight also refuses on: missing service, empty env block, a store path (or `Secrets__MasterKey__FilePath`) + that is relative or inside the Server/Worker dir, a `*.db`/`-wal`/`-shm` inside the Server/Worker dir, a + staged dir that does not contain the **installed** exe names (`Split-Path -Leaf` of `Application` and of the + Worker path), a staged Worker that is not x86, staged Server/Worker from different SHAs (or not matching + `-ExpectedSha`), too little disk, and a staged **self-contained runtime older than the live one** (compares + `mscordaccore_amd64_amd64_.dll`; `-AllowRuntimeDowngrade` overrides; see section 4). A master key that + is not in the env block (neither `ZB_SECRETS_MASTER_KEY` nor `Secrets__MasterKey__FilePath`, as on windev) + is a warning, since the upgrade does not change where it comes from. +- **Env block (trap 4): read, fingerprinted, never written.** The block is read from the service's registry + `Parameters\AppEnvironmentExtra` (what NSSM uses; `nssm get` is the fallback). The fingerprint is the entry + names plus a 12-hex SHA-256 prefix of each value, and the whole-block hash; values are never printed and + sensitive values are redacted from all output and notes. It is compared after the stop, after the swap, + before and after the start, and after the stability window; any added, removed or changed entry (or a + reorder) aborts and triggers the automatic rollback. On vd03 the pepper lives **only** in this block (#171), + so this check is what proves it survived. +- Stop waits `-StopTimeoutSeconds`; on STOP_PENDING (trap 8) it kills **this install's** Server, then its + Workers (path under the Server/Worker dir, or the installed image name when the path is unreadable; never by + a generic name), then the service host as a last resort. It then waits for every gateway/Worker process on + the box to be gone. +- Backup goes to `\_deploy-backup-[-label]\`, restricted to SYSTEM + Administrators + (inheritance off) **before** anything is written; if `icacls` fails nothing is written. It holds + `AppEnvironmentExtra.txt` (the raw block: secrets), the service registry key export, `env-fingerprint.txt` + (names + hash prefixes only) and a copy of a file-based master key. The stores are only **copied** (`.db`, + `-wal`, `-shm` -> `*.bak-`, hash-checked), never moved or deleted. - Swap is **rename-in-place** (`Server` -> `Server.bak-[-label]`, same for `Worker`), so NSSM - `Application`/`AppDirectory`/`ExecutablePath` never change. Live `appsettings*.json` are copied back - byte-identical. The stores are only **copied** (`.db`, `-wal`, `-shm` -> `*.bak-`), never moved - or deleted. The script **never runs `nssm set AppEnvironmentExtra`**; it saves the block (secrets - included, ACL-restricted via `icacls`) to `_env-AppEnvironmentExtra-.txt` and compares the entry count - after start. Notes (names only, no values) go to `_DEPLOY-NOTES-.txt` beside the install. -- Verify: `/health/ready` = 200, `/healthz`, `ProductVersion` SHA of the deployed Server exe against - `-ExpectedSha` (still subject to the version-stamp caveat in trap 7), service stays RUNNING for - `-StableSeconds`, env count unchanged. Health with zero sessions does not prove the Worker path: do the + `Application`/`AppDirectory`/`ExecutablePath` never change. **Carried forward** from the live dir + (hash-verified, explicit ACLs copied): `appsettings*.json` (unless `-TakeNewAppSettings`) and certificates/keys + (`*.pfx`, `*.p12`, `*.pem`, `*.key`, `*.crt`, `*.cer`, e.g. vd03's `Server\dashboard.pfx`) always as the live + copy, plus every other live-only top-level file that is not a binary. Live-only binaries + (`.dll`/`.exe`/`.pdb`) are leftovers of the old build and stay in the `.bak`; live-only sub-directories are + listed as a warning. `data\` (the wonder store dir) sits beside `Server\`, outside the swap, and is not touched. + Preflight prints the carry plan. +- Verify: `/health/ready` = 200, `/healthz` = 200, `ProductVersion` SHA of the deployed Server exe against + `-ExpectedSha` (still subject to the version-stamp caveat in trap 7), exactly one gateway at the + `Application` path that is the service host's child, the same PID and SERVICE_RUNNING through + `-StableSeconds`, env fingerprint unchanged. Health with zero sessions does not prove the Worker path: do the section 7.3 API-key session read yourself. - A failed start or verify triggers an automatic rollback to the just-made backups (disable with - `-NoAutoRollback`). Manual rollback: `-Rollback [-BackupStamp ] [-RestoreStores]`; the failed - directories are kept as `*.failed-`. `-RestoreStores` restores the db triplet from the matching - `*.bak-` (stale wal/shm are moved aside, nothing is deleted); use it only after a schema bump. + `-NoAutoRollback`); a failed backup restarts the unchanged build. Manual rollback: + `-Rollback [-BackupStamp ] [-RestoreStores]`. It accepts both backup spellings, `Server.bak-` + (this script) and `Server.bak.` (the adapted scripts used for #229 on vd03), and refuses + a Server backup without a Worker backup of the same stamp. It compares the env block to the saved + `env-fingerprint.txt` (section 8 step 4). The failed directories are kept as `*.failed-`. + `-RestoreStores` restores the db triplet from the matching `*.bak-` (stale wal/shm are moved aside, + nothing is deleted); use it only after a schema bump. **Install script**: `-ServerDir -WorkerDir [-ServiceName -NssmPath -DataDir C:\ProgramData\MxGateway -LogDir -EnvFile -Env -ObjectName -ServiceAccountPassword -EnvironmentName -RestartDelayMs -ThrottleMs @@ -135,7 +183,7 @@ days. The SQLite DB must also never be opened with a host-side `sqlite3` while t 1. **`OrphanWorkerTerminator` kills live workers if a second gateway starts.** On startup the Server enumerates processes named after the configured Worker exe - (`ZB.MOM.WW.MxGateway.Worker`) and kills every one whose path equals the configured + (`ZB.MOM.WW.MxGateway.Worker` on windev, `MxGateway.Worker` on the wonder boxes) and kills every one whose path equals the configured `ExecutablePath` - or whose path it cannot read, which is the normal case for an x86 process seen from an x64 gateway, so it falls back to **image-name match only** (`src/ZB.MOM.WW.MxGateway.Server/Workers/OrphanWorkerTerminator.cs`). It skips only its own PID's @@ -174,7 +222,8 @@ days. The SQLite DB must also never be opened with a host-side `sqlite3` while t 8. **Stuck `STOP_PENDING`.** Stopping the service can hang in `STOP_PENDING` (workers must exit within `Worker:ShutdownTimeoutSeconds`, default 10). Observed on the QA boxes per deploy notes **[UNVERIFIED from this repo]**; if it hangs, wait, then kill the Server process by PID. Do not start the new - build until `Get-Process` shows no `ZB.MOM.WW.MxGateway.Server` and no `ZB.MOM.WW.MxGateway.Worker`. + build until no gateway Server and no Worker process is left. Check **both** naming schemes: + `ZB.MOM.WW.MxGateway.Server`/`.Worker` (windev) and `MxGateway.Server`/`.Worker` (wonder boxes). ## 3. Preflight (no changes yet) @@ -230,6 +279,14 @@ the host **[UNVERIFIED]**. Note: the Worker is net48 and must be built where the pack and MSBuild are present (windev; `scadaproj` notes say Build Tools 2022 + net48 reference assemblies verified 2026-09-10 **[UNVERIFIED from this repo]**). +**Self-contained runtime.** A self-contained Server carries its own .NET runtime, so the SDK that builds it +decides the runtime on the box. On 2026-11-12 the windev SDK (10.0.204) produced runtime 10.0.8 while vd03 ran +10.0.14, which would have been a downgrade; the Mac SDK (10.0.403) produced 10.0.14. Build where the SDK carries +the newest runtime. The upgrade script refuses a downgrade (it compares `mscordaccore_amd64_amd64_.dll` +between the live and staged dirs). The wonder boxes use the `MxGateway.Server.exe` / `MxGateway.Worker.exe` +names: stage a byte copy of the apphost under that name and rename the Worker apphost and its `.config` +(`docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md`). + For the wonder boxes the staged output is copied over SMB (UNC copy is always allowed) into a staging directory such as `E:\psrem-stage\-\` per `scadaproj/deployment.md` §7.1 (**[UNVERIFIED]** path convention); the on-box steps below are then the only remote-exec part. @@ -258,7 +315,7 @@ the other (`deployment.md` §7.1 records the 2026-09-10 round as "rolling, one s 1. **Stop and confirm gone.** ```powershell nssm stop MxAccessGw # or Stop-Service MxAccessGw -Force - Get-Process ZB.MOM.WW.MxGateway.Server, ZB.MOM.WW.MxGateway.Worker -ErrorAction SilentlyContinue + Get-Process ZB.MOM.WW.MxGateway.Server, ZB.MOM.WW.MxGateway.Worker, MxGateway.Server, MxGateway.Worker -ErrorAction SilentlyContinue ``` Both lists must be empty before continuing (traps 1 and 8). If a Worker is left over, the next start will kill it as an orphan (by design) - but it signals an unclean stop worth noting. @@ -327,9 +384,11 @@ window, API keys no longer authenticate, or a Worker session cannot be opened. ## 9. Open gaps (not verifiable from this repo) -- Exact install paths on the wonder boxes (`E:\ApiInstall\MxAccessGw\...`), the staging convention and - the `_DEPLOY-NOTES` file format - taken from `scadaproj/deployment.md`, whose live content could not - be cross-checked here. +- Install layout on the wonder boxes, confirmed on vd03 (2026-11-12, #229/#230): `E:\ApiInstall\MxGateway\Server\MxGateway.Server.exe` + (NSSM `Application`), `Worker\MxGateway.Worker.exe` (`MxGateway__Worker__ExecutablePath`), stores and `kek.key` in + `E:\ApiInstall\MxGateway\data\`, `Server\dashboard.pfx`, master key `Secrets__MasterKey__Source=File`. windev: + `C:\publish\mxaccessgw\Server--\ZB.MOM.WW.MxGateway.Server.exe` and `C:\publish\mxaccessgw\Worker\`, stores in + `C:\ProgramData\MxGateway\`. vq03/vq04 are assumed to match vd03; the dry run shows it. - Whether the Server on each box is framework-dependent or self-contained (a box without .NET 10 needs self-contained; the 2026-10-19 windev plan says framework-dependent). - The `STOP_PENDING` behaviour and the NSSM service account (`LocalSystem` per `deployment.md`). diff --git a/scripts/deploy/Install-MxGatewayService.ps1 b/scripts/deploy/Install-MxGatewayService.ps1 index 6b756841..d7f07d65 100644 --- a/scripts/deploy/Install-MxGatewayService.ps1 +++ b/scripts/deploy/Install-MxGatewayService.ps1 @@ -123,7 +123,9 @@ try { # --- one-gateway guard + existence ------------------------------------------------------------------- $st = Invoke-Nssm -NssmArgs @('status', $ServiceName) -AllowFail if ($st.Code -eq 0) { Fail "service '$ServiceName' already exists (state $($st.Text)); use Upgrade-MxGatewayService.ps1" } - $procs = @(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in @($ServerProc, $WorkerProc) }) + # Both naming schemes (ZB.MOM.WW.MxGateway.* on windev, MxGateway.* on the wonder boxes) plus the exe being installed (#230). + $gwImages = @($ServerProc, $WorkerProc, 'MxGateway.Server', 'MxGateway.Worker', [IO.Path]::GetFileNameWithoutExtension($ServerExeName)) | Select-Object -Unique + $procs = @(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in $gwImages }) if ($procs.Count -gt 0) { Fail "gateway process(es) already running (PIDs $(($procs.Id) -join ',')); installing/starting a second gateway kills their Workers" } if (-not $LogDir) { $LogDir = Join-Path $DataDir 'logs' } diff --git a/scripts/deploy/Upgrade-MxGatewayService.ps1 b/scripts/deploy/Upgrade-MxGatewayService.ps1 index afc1373f..486b22d8 100644 --- a/scripts/deploy/Upgrade-MxGatewayService.ps1 +++ b/scripts/deploy/Upgrade-MxGatewayService.ps1 @@ -2,32 +2,38 @@ <# .SYNOPSIS On-box upgrade / rollback of the NSSM-wrapped MxAccess Gateway (Server + x86 Worker). - Implements docs/runbooks/DeployUpgradeNssmService.md sections 3, 5, 6, 7 and 8 (mxaccessgw#220). + Implements docs/runbooks/DeployUpgradeNssmService.md sections 3, 5, 6, 7 and 8 (mxaccessgw#220, #230). .DESCRIPTION RUN THIS ON THE WINDOWS BOX ITSELF (Windows PowerShell 5.1 is enough). The new Server publish and the new x86 Worker build must already be staged on the box (UNC/SMB copy; build both from the same SHA). Upgrade flow: preflight -> stop (STOP_PENDING handled) -> backup -> swap Server + Worker -> start -> - verify (/health/ready, VersionInfo SHA, service stable) -> automatic rollback if start/verify fails. - Rollback flow (-Rollback): stop -> swap the .bak directories back -> optional store restore -> start -> verify. + verify (/health/ready, VersionInfo SHA, service stable, env block unchanged) -> automatic rollback if + start/verify fails. Rollback flow (-Rollback): stop -> swap the .bak directories back -> optional store + restore -> start -> verify. Safety properties: - * Exactly one gateway per box. Refuses to run if a second gateway process / service points at the same - exe, if the service is stopped but a gateway process exists, or if a Server/Worker process is still - alive when a start is about to happen (OrphanWorkerTerminator trap, runbook section 2.1). - * Never calls `nssm set ... AppEnvironmentExtra` (that replaces the whole block). The env block is only - READ (saved to the backup dir, entry count compared before/after). The Application/AppDirectory - are not changed either: the swap is rename-in-place, so the NSSM config is untouched. - * API-key DB, Secrets store and master key live outside the content root and are never moved or - deleted. They are only COPIED (db + -wal + -shm) into *.bak- siblings while the service is stopped. - The script refuses to run if a store path is inside the content root (the 2026-08-09 incident). + * Exactly one gateway per box. The gateway and Worker image names are taken from the INSTALLED exes + (NSSM Application and env MxGateway__Worker__ExecutablePath) plus both known naming schemes + (ZB.MOM.WW.MxGateway.* on windev, MxGateway.* on the wonder boxes), and processes are also matched by + exact path. The guards FAIL CLOSED: a RUNNING service whose gateway process cannot be found is an + abort, not a pass (mxaccessgw#230: the guards used to pass vacuously on the wonder boxes). + * Never writes the NSSM env block (AppEnvironmentExtra), Application or AppDirectory: the swap is + rename-in-place. The env block is fingerprinted (entry names + SHA-256 prefix of each value; values are + never printed) at preflight, after the stop, after the swap, after the start and after the stability + window. Any added, removed or changed entry aborts (and triggers the automatic rollback). + * Live-only top-level files in Server\ and Worker\ are carried into the new directories (appsettings*.json, + dashboard.pfx and other certificates/keys always keep the LIVE copy), hash-verified. + * API-key DB, Secrets store and a file-based master key must live outside the swapped directories; they are + only COPIED (db + -wal + -shm) into *.bak- siblings while the service is stopped. + * A self-contained Server is refused if its .NET runtime is older than the live one (no silent downgrade). - -DryRun (or -WhatIf) prints every mutating action and performs none. Read-only probing (nssm get/status, - process list, disk space, file existence) still runs so the decision logic is exercised. + -DryRun (or -WhatIf) prints every mutating action and performs none. Read-only probing (service state, + registry env block, process list, disk space, file hashes) still runs, so the guards are exercised for real. -.PARAMETER ServerSource Staged Server publish output (contains ZB.MOM.WW.MxGateway.Server.exe). -.PARAMETER WorkerSource Staged x86 Worker output (contains ZB.MOM.WW.MxGateway.Worker.exe). +.PARAMETER ServerSource Staged Server publish output (must contain the same exe name as NSSM Application). +.PARAMETER WorkerSource Staged x86 Worker output (must contain the exe name of MxGateway__Worker__ExecutablePath). .PARAMETER ServiceName NSSM service name. Default MxAccessGw. .PARAMETER NssmPath nssm executable. Default 'nssm' (on PATH). .PARAMETER ServerDir Live Server directory. Default: NSSM AppDirectory. @@ -36,11 +42,12 @@ must carry it in ProductVersion ('0.1.2+'). Builds from 2026-07-09..08-10 have a bogus stamp. .PARAMETER Label Free-text suffix for backup names (e.g. 'planwrites'). .PARAMETER HealthBaseUrl Default http://localhost:5130 (dashboard port; /health/ready, /healthz anonymous). -.PARAMETER Rollback Roll back instead of upgrading. Uses newest Server.bak-* unless -BackupStamp is given. -.PARAMETER BackupStamp Stamp (yyyyMMdd-HHmmss) of the backup set to roll back to. +.PARAMETER Rollback Roll back instead of upgrading. Uses the newest Server.bak-* / Server.bak.* unless -BackupStamp. +.PARAMETER BackupStamp Stamp of the backup set to roll back to (yyyyMMdd-HHmmss, or yyyyMMddTHHmmss for .bak. sets). .PARAMETER RestoreStores With -Rollback: also restore gateway-auth.db / secrets db from *.bak- (schema-bump case). .PARAMETER NoAutoRollback Do not roll back automatically when start/verification fails. .PARAMETER TakeNewAppSettings Do not carry the live appsettings*.json over into the new Server dir. +.PARAMETER AllowRuntimeDowngrade Accept a staged self-contained runtime older than the live one. .PARAMETER SkipFreeSpaceCheck / MinFreeGB / StopTimeoutSeconds / StableSeconds / ReadyTimeoutSeconds tunables. .PARAMETER DryRun Print, do not act. -WhatIf is equivalent. #> @@ -60,6 +67,7 @@ param( [switch]$RestoreStores, [switch]$NoAutoRollback, [switch]$TakeNewAppSettings, + [switch]$AllowRuntimeDowngrade, [switch]$SkipFreeSpaceCheck, [double]$MinFreeGB = 2, [int]$StopTimeoutSeconds = 30, @@ -70,16 +78,38 @@ param( $ErrorActionPreference = 'Stop' $script:Dry = $DryRun.IsPresent -or $WhatIfPreference +# -WhatIf is handled by $script:Dry (Step). Leaving the preference on makes Windows PowerShell 5.1 cmdlets used for READ-ONLY +# probing misbehave (Get-FileHash returns nothing under -WhatIf), so the plan would abort; found on windev, #230. +$WhatIfPreference = $false $script:Stamp = Get-Date -Format 'yyyyMMdd-HHmmss' -$script:ServerProc = 'ZB.MOM.WW.MxGateway.Server' -$script:WorkerProc = 'ZB.MOM.WW.MxGateway.Worker' +# Both naming schemes seen in the field: windev runs ZB.MOM.WW.MxGateway.*.exe, the wonder boxes MxGateway.*.exe. +# The installed exe names (NSSM Application, MxGateway__Worker__ExecutablePath) are added to these in Get-Context. +$script:KnownServerImages = @('ZB.MOM.WW.MxGateway.Server', 'MxGateway.Server') +$script:KnownWorkerImages = @('ZB.MOM.WW.MxGateway.Worker', 'MxGateway.Worker') +$script:ServerImages = $script:KnownServerImages +$script:WorkerImages = $script:KnownWorkerImages +$script:EnvWorkerExe = 'MxGateway__Worker__ExecutablePath' +# Live copy always wins for these (host config, certificates, keys), even when the staged build ships one. +$script:PreservePatterns = @('appsettings*.json', '*.pfx', '*.p12', '*.pem', '*.key', '*.crt', '*.cer') +# Never carried from the old build (they are build output; a live-only binary is a leftover of an older build). +$script:BinaryExtensions = @('.dll', '.exe', '.pdb', '.so', '.dylib', '.winmd') $script:Notes = New-Object System.Collections.Generic.List[string] +$script:Redact = New-Object System.Collections.Generic.List[string] $script:LockStream = $null +$script:BackupDir = $null +$script:EnvWarnOnly = $false -function Say([string]$m) { Write-Host $m; $script:Notes.Add($m) } +# --- output (secret values are redacted from every line) ---------------------------------------------------- + +function Hide([string]$m) { + if ($null -eq $m) { return '' } + foreach ($s in $script:Redact) { if ($s) { $m = $m.Replace($s, '') } } + $m +} +function Say([string]$m) { $m = Hide $m; Write-Host $m; $script:Notes.Add($m) } function Info([string]$m) { Say "[info] $m" } -function Warn([string]$m) { Write-Warning $m; $script:Notes.Add("[WARN] $m") } -function Fail([string]$m) { throw "ABORT: $m" } +function Warn([string]$m) { $m = Hide $m; Write-Warning $m; $script:Notes.Add("[WARN] $m") } +function Fail([string]$m) { throw "ABORT: $(Hide $m)" } # Mutating step: print in dry-run, run otherwise. function Step([string]$Description, [scriptblock]$Action) { @@ -88,7 +118,9 @@ function Step([string]$Description, [scriptblock]$Action) { & $Action } -# --- helpers -------------------------------------------------------------------------------------------- +# --- environment seams (the dry-run harness in scripts/deploy/tests replaces these) --------------------------- + +function Test-IsWindowsHost { $env:OS -eq 'Windows_NT' } function Invoke-Nssm { param([string[]]$NssmArgs, [switch]$AllowFail) @@ -107,20 +139,50 @@ function Get-ServiceState { ($r.Text -split "`n")[0].Trim() } -function Parse-EnvBlock([string]$Text) { - $map = [ordered]@{} - foreach ($line in ($Text -split "[`r`n]+")) { - $t = $line.Trim() - if ($t -match '^([^=\s]+)=(.*)$') { $map[$Matches[1]] = $Matches[2] } +# The env block exactly as NSSM stores it (REG_MULTI_SZ), one 'NAME=value' per element. Falls back to nssm get. +function Read-EnvBlockLines { + $key = "HKLM:\SYSTEM\CurrentControlSet\Services\$ServiceName\Parameters" + if ((Test-IsWindowsHost) -and (Test-Path -LiteralPath $key)) { + $p = Get-ItemProperty -LiteralPath $key -Name AppEnvironmentExtra -ErrorAction SilentlyContinue + if ($p) { return @($p.AppEnvironmentExtra | Where-Object { $_ }) } + return @() } - $map + @((Get-NssmValue 'AppEnvironmentExtra') -split "[`r`n]+" | Where-Object { $_.Trim() }) } -function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') } -function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') } -function Test-Inside([string]$Path, [string]$Root) { - $a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant() - $a -eq $b -or $a.StartsWith("$b/") +# Id, ParentId, Name (image name without .exe), Path (may be $null when unreadable). +function Get-ProcessTable { + if (Get-Command Get-CimInstance -ErrorAction SilentlyContinue) { + return @(Get-CimInstance Win32_Process | ForEach-Object { + [pscustomobject]@{ Id = [int]$_.ProcessId; ParentId = [int]$_.ParentProcessId; Name = [IO.Path]::GetFileNameWithoutExtension([string]$_.Name); Path = $_.ExecutablePath } + }) + } + @(Get-Process -ErrorAction SilentlyContinue | ForEach-Object { + $p = $null; try { $p = $_.Path } catch { } + [pscustomobject]@{ Id = $_.Id; ParentId = 0; Name = $_.ProcessName; Path = $p } + }) +} + +# PID of the NSSM service host (the gateway Server is its child). $null when it cannot be read. +function Get-ServiceHostPid { + if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null } + $s = Get-CimInstance Win32_Service -Filter "Name='$ServiceName'" -ErrorAction SilentlyContinue + if ($s -and $s.ProcessId) { return [int]$s.ProcessId } + $null +} + +# Other services whose command line launches the gateway exe. $null when the check is unavailable. +function Get-OtherGatewayServices([string]$ExeName) { + if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null } + # The leading comma keeps an EMPTY result an array: a bare @() reaches the caller as $null ('check unavailable'). + , @(Get-CimInstance Win32_Service | Where-Object { $_.Name -ne $ServiceName -and $_.PathName -and $_.PathName -like "*$ExeName*" } | ForEach-Object { $_.Name }) +} + +function Stop-ProcessById([int]$Id) { Stop-Process -Id $Id -Force -ErrorAction SilentlyContinue } + +function Get-HttpStatus([string]$Url) { + try { return [int](Invoke-WebRequest -Uri $Url -UseBasicParsing -TimeoutSec 10).StatusCode } + catch { if ($_.Exception.Response) { return [int]$_.Exception.Response.StatusCode }; return 0 } } function Get-ProductSha([string]$Exe) { @@ -130,25 +192,200 @@ function Get-ProductSha([string]$Exe) { } catch { } $null } + +# 'X86', 'Amd64', 'MSIL', ... or $null when it cannot be read. +function Get-ExeArchitecture([string]$Exe) { + try { return "$([Reflection.AssemblyName]::GetAssemblyName($Exe).ProcessorArchitecture)" } catch { return $null } +} + +# Restrict a directory to SYSTEM + Administrators, inheritance off. Throws if that fails. +function Protect-Directory([string]$Path) { + if (-not (Test-IsWindowsHost)) { Warn "not Windows: $Path is not ACL-restricted"; return } + $null = & icacls.exe $Path /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' 2>&1 + if ($LASTEXITCODE -ne 0) { Fail "icacls could not restrict $Path (exit $LASTEXITCODE); refusing to write the env block there" } +} + +function Export-ServiceKey([string]$Destination) { + if (-not (Test-IsWindowsHost)) { Info 'not Windows: service registry export skipped'; return } + $null = & reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$ServiceName" $Destination /y 2>&1 + if ($LASTEXITCODE -ne 0) { Fail "reg export of the service key failed (exit $LASTEXITCODE)" } +} + +function Get-DirSddl([string]$Path) { + if (-not (Test-IsWindowsHost)) { return $null } + try { (Get-Acl -LiteralPath $Path).Sddl } catch { $null } +} + +# Give a carried file the old file's ACL when the old one had explicit (non-inherited) entries. +function Copy-ExplicitAcl([string]$From, [string]$To) { + if (-not (Test-IsWindowsHost)) { return } + try { + $a = Get-Acl -LiteralPath $From + if (@($a.Access | Where-Object { -not $_.IsInherited }).Count -gt 0 -or $a.AreAccessRulesProtected) { + Set-Acl -LiteralPath $To -AclObject $a + Info " ACL of $(Split-Path -Leaf $From) copied (it had explicit entries)" + } + } catch { Warn "could not copy the ACL onto ${To}: $($_.Exception.Message)" } +} + +# --- pure helpers --------------------------------------------------------------------------------------------- + +function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') } +function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') } +function Test-Inside([string]$Path, [string]$Root) { + if (-not $Path -or -not $Root) { return $false } + $a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant() + $a -eq $b -or $a.StartsWith("$b/") +} +function Test-SamePath([string]$A, [string]$B) { $A -and $B -and ((Norm $A).ToLowerInvariant() -eq (Norm $B).ToLowerInvariant()) } +function Get-Leaf([string]$p) { if (-not $p) { return $null }; ((Norm $p) -split '/')[-1] } +function Get-Stem([string]$p) { $l = Get-Leaf $p; if (-not $l) { return $null }; [IO.Path]::GetFileNameWithoutExtension($l) } + function Test-ShaMatch([string]$Actual, [string]$Expected) { if (-not $Actual -or -not $Expected) { return $false } $e = $Expected.ToLowerInvariant() $Actual.StartsWith($e) -or $e.StartsWith($Actual) } -function Get-GatewayProcesses { - @(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in @($script:ServerProc, $script:WorkerProc) }) +function Get-TextHashPrefix([string]$Text) { + $sha = [Security.Cryptography.SHA256]::Create() + try { (($sha.ComputeHash([Text.Encoding]::UTF8.GetBytes($Text)) | ForEach-Object { $_.ToString('x2') }) -join '').Substring(0, 12) } + finally { $sha.Dispose() } +} +function Get-FileSha([string]$Path) { + $sha = [Security.Cryptography.SHA256]::Create(); $fs = [IO.File]::OpenRead($Path) + try { (($sha.ComputeHash($fs) | ForEach-Object { $_.ToString('X2') }) -join '') } finally { $fs.Dispose(); $sha.Dispose() } } -# Store file set: db + -wal + -shm -function Get-StoreFiles([string]$Db) { @($Db, "$Db-wal", "$Db-shm") | Where-Object { Test-Path -LiteralPath $_ } } +function Test-SensitiveName([string]$Name) { ($Name -match 'KEY|PEPPER|PASS|SECRET|TOKEN|PWD|CREDENTIAL') -and ($Name -notmatch '(Path|Source|EnvVarName|Name)$') } + +# name -> value. Every sensitive value is registered for redaction. +function ConvertTo-EnvMap([string[]]$Lines) { + $map = [ordered]@{} + foreach ($line in $Lines) { + $i = $line.IndexOf('=') + if ($i -le 0) { continue } + $k = $line.Substring(0, $i).Trim(); $v = $line.Substring($i + 1) + $map[$k] = $v + if ((Test-SensitiveName $k) -and $v.Length -ge 4 -and -not $script:Redact.Contains($v)) { $script:Redact.Add($v) } + } + $map +} + +# Names + per-value SHA-256 prefixes. Never carries a value. +function Get-EnvFingerprint([string[]]$Lines) { + $map = ConvertTo-EnvMap $Lines + $entries = [ordered]@{} + foreach ($k in $map.Keys) { $entries[$k] = Get-TextHashPrefix ([string]$map[$k]) } + [pscustomobject]@{ Count = @($Lines).Count; Names = @($map.Keys); Entries = $entries; BlockHash = (Get-TextHashPrefix (@($Lines) -join "`n")) } +} + +# List of differences (names only). Empty when identical. +function Compare-EnvFingerprint($Before, $After) { + $d = New-Object System.Collections.Generic.List[string] + foreach ($k in $Before.Entries.Keys) { + if (-not $After.Entries.Contains($k)) { $d.Add("removed $k") } + elseif ($After.Entries[$k] -ne $Before.Entries[$k]) { $d.Add("changed $k (sha256 $($Before.Entries[$k]) -> $($After.Entries[$k]))") } + } + foreach ($k in $After.Entries.Keys) { if (-not $Before.Entries.Contains($k)) { $d.Add("added $k") } } + if ($d.Count -eq 0 -and $Before.BlockHash -ne $After.BlockHash) { $d.Add("block hash $($Before.BlockHash) -> $($After.BlockHash) (order, duplicates or blank entries changed)") } + , $d.ToArray() +} + +function Format-EnvFingerprint($Fp) { "$($Fp.Count) entries, block sha256 $($Fp.BlockHash): " + (($Fp.Entries.Keys | ForEach-Object { "$_=$($Fp.Entries[$_])" }) -join ', ') } + +# Re-read the env block and abort on any difference from the preflight fingerprint. +function Assert-EnvUnchanged($ctx, [string]$When) { + $now = Get-EnvFingerprint (Read-EnvBlockLines) + $diff = Compare-EnvFingerprint $ctx.EnvFp $now + if ($diff.Count -gt 0) { + # During an automatic rollback the difference is already the reported failure; restoring the old build still goes ahead. + if ($script:EnvWarnOnly) { Warn "NSSM env block changed ${When}: $($diff -join '; ') (not written by this script; restore it from the backup dir's AppEnvironmentExtra.txt if unintended)"; return } + Fail "NSSM env block changed ${When}: $($diff -join '; ')" + } + Info "env block unchanged $When ($($now.Count) entries, block sha256 $($now.BlockHash))" +} + +# Directory listings via .NET (FileInfo/DirectoryInfo, hidden files included). Get-ChildItem costs ~250 ms per call +# under pwsh on macOS (per-file owner lookups), which made the dry-run harness 20x slower; on Windows both are fine. +function Get-Files([string]$Dir, [string]$Filter = '*', [switch]$Recurse) { + if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return } + $o = if ($Recurse) { [IO.SearchOption]::AllDirectories } else { [IO.SearchOption]::TopDirectoryOnly } + try { (New-Object IO.DirectoryInfo $Dir).GetFiles($Filter, $o) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" } +} +function Get-Dirs([string]$Dir, [string]$Filter = '*') { + if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return } + try { (New-Object IO.DirectoryInfo $Dir).GetDirectories($Filter) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" } +} + +function Get-StoreFiles([string]$Db) { @($Db, "$Db-wal", "$Db-shm") | Where-Object { $_ -and (Test-Path -LiteralPath $_) } } function Get-DirSize([string]$p) { if (-not (Test-Path -LiteralPath $p)) { return 0 } - $s = (Get-ChildItem -LiteralPath $p -Recurse -File -Force -ErrorAction SilentlyContinue | Measure-Object Length -Sum).Sum + $s = (@(Get-Files $p -Recurse) | Measure-Object Length -Sum).Sum if ($s) { [double]$s } else { 0 } } +# Version of a self-contained .NET runtime in a directory (from mscordaccore___.dll), or $null (framework-dependent). +function Get-BundledRuntimeVersion([string]$Dir) { + $f = @(Get-Files $Dir 'mscordaccore_*.dll') + foreach ($x in $f) { if ($x.Name -match '_(\d+\.\d+\.\d+(\.\d+)?)\.dll$') { return [version]$Matches[1] } } + $null +} + +# --- processes -------------------------------------------------------------------------------------------------- + +function Test-GatewayServer($p, $ctx) { ($script:ServerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.Application) } +function Test-GatewayWorker($p, $ctx) { ($script:WorkerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.WorkerExe) } + +function Get-GatewayProcesses($ctx) { + @(Get-ProcessTable | Where-Object { (Test-GatewayServer $_ $ctx) -or (Test-GatewayWorker $_ $ctx) } | ForEach-Object { + $role = if (Test-GatewayServer $_ $ctx) { 'Server' } else { 'Worker' } + $_ | Add-Member -NotePropertyName Role -NotePropertyValue $role -PassThru -Force + }) +} +function Format-Procs($procs) { (@($procs) | ForEach-Object { "$($_.Name)#$($_.Id)<-$($_.ParentId) [$(if ($_.Path) { $_.Path } else { 'path unreadable' })]" }) -join ', ' } + +function Assert-NoGatewayProcess($ctx, [string]$When) { + $left = @(Get-GatewayProcesses $ctx) + if ($left.Count -gt 0) { Fail "${When}: gateway process(es) still alive: $(Format-Procs $left)" } +} + +# The one-gateway guard. Fails closed: a RUNNING service whose gateway cannot be seen is an abort. +function Assert-SingleGateway($ctx, [string]$State) { + $procs = @(Get-GatewayProcesses $ctx) + $servers = @($procs | Where-Object { $_.Role -eq 'Server' }) + $workers = @($procs | Where-Object { $_.Role -eq 'Worker' }) + Info "gateway processes on the box (images $($script:ServerImages -join '|') / $($script:WorkerImages -join '|')): $(if ($procs.Count) { Format-Procs $procs } else { 'none' })" + if ($servers.Count -gt 1) { Fail "$($servers.Count) gateway Server processes are running ($(Format-Procs $servers)). A second gateway is the OrphanWorkerTerminator trap - resolve manually." } + switch ($State) { + 'SERVICE_STOPPED' { + if ($procs.Count -gt 0) { Fail "service is STOPPED but gateway process(es) exist ($(Format-Procs $procs)): a rogue second gateway or an unclean stop. Starting would terminate those workers." } + } + 'SERVICE_RUNNING' { + if ($servers.Count -eq 0) { + Fail "service is RUNNING but no gateway Server process was found (looked for image names $($script:ServerImages -join ', ') and path $($ctx.Application)). The process guards cannot see this gateway, so they would pass vacuously; refusing." + } + $s = $servers[0] + if ($s.Path -and -not (Test-SamePath $s.Path $ctx.Application)) { Fail "the running gateway is $($s.Path), not the service's Application $($ctx.Application): a second gateway is running and the service's own is not" } + if (-not $s.Path) { Warn "gateway Server #$($s.Id) path is unreadable; matched by image name only" } + $hostPid = Get-ServiceHostPid + if ($hostPid) { + if ($s.ParentId -ne $hostPid) { Fail "gateway Server #$($s.Id) is not a child of the $ServiceName service host #${hostPid} (parent #$($s.ParentId)): it is another gateway" } + Info "gateway Server #$($s.Id) is the service's own (child of service host #$hostPid)" + } else { Warn 'service host PID unavailable; parent check skipped' } + $foreign = @($workers | Where-Object { $_.ParentId -and $_.ParentId -ne $s.Id }) + if ($foreign.Count -gt 0) { Warn "Worker process(es) not owned by the service's gateway: $(Format-Procs $foreign). They are killed as orphans on the next start." } + $wrongPath = @($workers | Where-Object { $_.Path -and $ctx.WorkerExe -and -not (Test-SamePath $_.Path $ctx.WorkerExe) }) + if ($wrongPath.Count -gt 0) { Warn "Worker process(es) not running the configured $($ctx.WorkerExe): $(Format-Procs $wrongPath)" } + } + 'SERVICE_STOP_PENDING' { Warn 'service is STOP_PENDING; the stop step will finish it' } + default { + if ($servers.Count -gt 0) { Fail "a gateway Server process exists while the service is $State" } + } + } +} + # --- context discovery (read only) ------------------------------------------------------------------------- function Get-Context { @@ -158,25 +395,47 @@ function Get-Context { $ctx.State = $state $ctx.Application = Get-NssmValue 'Application' $ctx.AppDirectory = Get-NssmValue 'AppDirectory' - $ctx.EnvText = Get-NssmValue 'AppEnvironmentExtra' - $ctx.Env = Parse-EnvBlock $ctx.EnvText - $ctx.EnvCount = $ctx.Env.Count - if (-not $ServerDir) { $ctx.ServerDir = $ctx.AppDirectory } else { $ctx.ServerDir = $ServerDir } - $wexe = $ctx.Env['MxGateway__Worker__ExecutablePath'] + $lines = @(Read-EnvBlockLines) + $ctx.Env = ConvertTo-EnvMap $lines + $ctx.EnvFp = Get-EnvFingerprint $lines + $ctx.EnvCount = $ctx.EnvFp.Count + if ($ServerDir) { $ctx.ServerDir = $ServerDir } else { $ctx.ServerDir = $ctx.AppDirectory } + $wexe = $ctx.Env[$script:EnvWorkerExe] $ctx.WorkerExe = $wexe if ($WorkerDir) { $ctx.WorkerDir = $WorkerDir } elseif ($wexe) { $ctx.WorkerDir = Split-Path -Parent $wexe } - else { Fail 'cannot determine the Worker directory: env MxGateway__Worker__ExecutablePath is not in the NSSM env block; pass -WorkerDir' } + else { Fail "cannot determine the Worker directory: env $($script:EnvWorkerExe) is not in the NSSM env block; pass -WorkerDir" } $pd = 'C:\ProgramData\MxGateway' $auth = $ctx.Env['MxGateway__Authentication__SqlitePath']; if (-not $auth) { $auth = "$pd\gateway-auth.db" } $sec = $ctx.Env['Secrets__SqlitePath']; if (-not $sec) { $sec = "$pd\mxgateway-secrets.db" } $ctx.AuthDb = $auth; $ctx.SecretsDb = $sec - $ctx.ServerExeName = Split-Path -Leaf $ctx.Application + $ctx.KekFile = $ctx.Env['Secrets__MasterKey__FilePath'] + $ctx.ServerExeName = Get-Leaf $ctx.Application + $ctx.WorkerExeName = Get-Leaf $wexe + if (-not $ctx.WorkerExeName) { $ctx.WorkerExeName = 'ZB.MOM.WW.MxGateway.Worker.exe' } + # Process image names: the installed ones first, then both known schemes. + $script:ServerImages = @(@(Get-Stem $ctx.Application) + $script:KnownServerImages | Where-Object { $_ } | Select-Object -Unique) + $script:WorkerImages = @(@(Get-Stem $wexe) + $script:KnownWorkerImages | Where-Object { $_ } | Select-Object -Unique) [pscustomobject]$ctx } # --- preflight ----------------------------------------------------------------------------------------------- +function Test-MasterKey($ctx) { + $src = $ctx.Env['Secrets__MasterKey__Source'] + $envVar = $ctx.Env['Secrets__MasterKey__EnvVarName']; if (-not $envVar) { $envVar = 'ZB_SECRETS_MASTER_KEY' } + if ($ctx.KekFile -or $src -eq 'File') { + if (-not $ctx.KekFile) { Fail 'Secrets__MasterKey__Source=File but no Secrets__MasterKey__FilePath in the env block' } + if (-not (Test-Rooted $ctx.KekFile)) { Fail "master key file '$($ctx.KekFile)' is not absolute" } + foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) { if (Test-Inside $ctx.KekFile $root) { Fail "master key file '$($ctx.KekFile)' is INSIDE '$root' - the swap would lose it" } } + if (Test-Path -LiteralPath $ctx.KekFile) { Info "master key: file $($ctx.KekFile) (present, outside the swap)" } else { Fail "master key file '$($ctx.KekFile)' does not exist (the Secrets store would be unreadable)" } + } elseif ($ctx.Env.Contains($envVar)) { + Info "master key: env block entry $envVar" + } else { + Warn "the env block carries no master key ($envVar or Secrets__MasterKey__FilePath); it comes from elsewhere (appsettings or the account environment). The upgrade does not change that, but confirm the Secrets store opens after the start." + } +} + function Invoke-Preflight($ctx) { Say '== Preflight ==' Info "service '$ServiceName' state: $($ctx.State)" @@ -186,20 +445,19 @@ function Invoke-Preflight($ctx) { Info "Worker dir : $($ctx.WorkerDir) (exe env: $($ctx.WorkerExe))" Info "API-key DB : $($ctx.AuthDb)" Info "Secrets DB : $($ctx.SecretsDb)" - Info "NSSM env block: $($ctx.EnvCount) entries (names only): $(($ctx.Env.Keys) -join ', ')" + Info "NSSM env block fingerprint (names + value sha256 prefixes, no values): $(Format-EnvFingerprint $ctx.EnvFp)" - if ($ctx.EnvCount -eq 0) { Fail 'NSSM env block is empty - refusing: it holds Kestrel/store paths, the KEK and the pepper' } - foreach ($need in 'ZB_SECRETS_MASTER_KEY') { - if (-not $ctx.Env.Contains($need)) { Fail "env block has no $need (the Secrets store would be unreadable after the restart)" } - } + if ($ctx.EnvCount -eq 0) { Fail 'NSSM env block is empty - refusing: it holds Kestrel/store paths, the master key and the pepper' } if (-not ($ctx.Env.Keys | Where-Object { $_ -match 'Pepper' })) { Warn 'no *Pepper* entry in the env block; confirm the API-key pepper is supplied another way or issued tokens will not verify' } if ($ctx.Env['ASPNETCORE_ENVIRONMENT'] -ne 'Production' -and $ctx.Env['DOTNET_ENVIRONMENT'] -ne 'Production') { Warn 'neither ASPNETCORE_ENVIRONMENT nor DOTNET_ENVIRONMENT is exactly Production (runbook trap 6) - do not change it during an upgrade' } + if (-not (Test-SamePath (Split-Path -Parent $ctx.Application) $ctx.ServerDir)) { Fail "NSSM Application '$($ctx.Application)' is not inside the Server dir '$($ctx.ServerDir)'" } + if ($ctx.WorkerExe -and -not (Test-SamePath (Split-Path -Parent $ctx.WorkerExe) $ctx.WorkerDir)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' is not inside the Worker dir '$($ctx.WorkerDir)'" } - # State that must survive: rooted and outside the content root. + # State that must survive: rooted and outside the swapped directories. foreach ($pair in @(@('API-key DB', $ctx.AuthDb), @('Secrets DB', $ctx.SecretsDb))) { if (-not (Test-Rooted $pair[1])) { Fail "$($pair[0]) path '$($pair[1])' is not absolute" } foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) { @@ -208,6 +466,7 @@ function Invoke-Preflight($ctx) { if (Test-Path -LiteralPath $pair[1]) { Info "$($pair[0]) present ($((Get-StoreFiles $pair[1]).Count) file(s) incl. wal/shm)" } else { Warn "$($pair[0]) '$($pair[1])' does not exist yet (fresh store will be created on start)" } } + Test-MasterKey $ctx # Directories foreach ($d in @($ctx.ServerDir, $ctx.WorkerDir)) { @@ -216,33 +475,34 @@ function Invoke-Preflight($ctx) { if ((Test-Inside $ctx.WorkerDir $ctx.ServerDir) -or (Test-Inside $ctx.ServerDir $ctx.WorkerDir)) { Fail 'Server and Worker directories are nested; the rename-in-place swap needs sibling directories' } + if (-not (Test-Path -LiteralPath $ctx.Application)) { Fail "NSSM Application '$($ctx.Application)' does not exist" } + if ($ctx.WorkerExe -and -not (Test-Path -LiteralPath $ctx.WorkerExe)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' does not exist" } + $dbs = @(@(Get-Files $ctx.ServerDir -Recurse) + @(Get-Files $ctx.WorkerDir -Recurse) | Where-Object { $_.Name -match '\.db(-wal|-shm)?$' }) + if ($dbs.Count -gt 0) { Fail "SQLite file(s) inside the swapped directories would be orphaned by the rename: $(($dbs | ForEach-Object { $_.FullName }) -join ', ')" } - # One-gateway guard - $procs = Get-GatewayProcesses - $servers = @($procs | Where-Object { $_.ProcessName -eq $script:ServerProc }) - if ($servers.Count -gt 1) { Fail "$($servers.Count) gateway Server processes are running (PIDs $(($servers.Id) -join ',')). A second gateway is the OrphanWorkerTerminator trap - resolve manually." } - if ($ctx.State -eq 'SERVICE_STOPPED' -and $procs.Count -gt 0) { - Fail "service is STOPPED but gateway process(es) exist (PIDs $(($procs.Id) -join ',')): a rogue second gateway. Starting would terminate its live workers." - } - if ($servers.Count -eq 1 -and $ctx.State -ne 'SERVICE_RUNNING' -and $ctx.State -ne 'SERVICE_STOP_PENDING') { - Fail "a Server process exists while the service is $($ctx.State)" - } - if (Get-Command Get-CimInstance -ErrorAction SilentlyContinue) { - $svcs = @(Get-CimInstance Win32_Service | Where-Object { $_.Name -ne $ServiceName -and $_.PathName -and $_.PathName -like "*$($ctx.ServerExeName)*" }) - if ($svcs.Count -gt 0) { Fail "another service ($(($svcs.Name) -join ', ')) also launches $($ctx.ServerExeName); two gateways on one box is forbidden" } - } else { Info 'Win32_Service cross-check unavailable on this host (skipped)' } + # One-gateway guard (fail closed) + Assert-SingleGateway $ctx $ctx.State + $others = Get-OtherGatewayServices $ctx.ServerExeName + if ($null -eq $others) { Info 'Win32_Service cross-check unavailable on this host (skipped)' } + elseif (@($others).Count -gt 0) { Fail "another service ($(@($others) -join ', ')) also launches $($ctx.ServerExeName); two gateways on one box is forbidden" } + else { Info "no other service launches $($ctx.ServerExeName)" } - if ($Rollback) { return } + if ($Rollback) { Say '== Preflight OK =='; return } # Sources if (-not $ServerSource -or -not $WorkerSource) { Fail '-ServerSource and -WorkerSource are required for an upgrade' } - foreach ($s in @(@($ServerSource, $ctx.ServerExeName), @($WorkerSource, "$($script:WorkerProc).exe"))) { - if (-not (Test-Path -LiteralPath (Join-Path $s[0] $s[1]))) { Fail "staged source '$($s[0])' does not contain $($s[1])" } + $stagedServerExe = Join-Path $ServerSource $ctx.ServerExeName + $stagedWorkerExe = Join-Path $WorkerSource $ctx.WorkerExeName + foreach ($s in @(@($ServerSource, $stagedServerExe), @($WorkerSource, $stagedWorkerExe))) { + if (-not (Test-Path -LiteralPath $s[1])) { Fail "staged source '$($s[0])' does not contain $(Get-Leaf $s[1]) (the installed exe name; NSSM and $($script:EnvWorkerExe) point at it)" } } - if ((Test-Inside $ServerSource $ctx.ServerDir) -or (Test-Inside $WorkerSource $ctx.WorkerDir)) { Fail 'staging directory is inside a live directory' } - $sSha = Get-ProductSha (Join-Path $ServerSource $ctx.ServerExeName) - $wSha = Get-ProductSha (Join-Path $WorkerSource "$($script:WorkerProc).exe") - Info "staged Server SHA: $sSha ; staged Worker SHA: $wSha" + if ((Test-Inside $ServerSource $ctx.ServerDir) -or (Test-Inside $WorkerSource $ctx.WorkerDir) -or (Test-Inside $ServerSource $ctx.WorkerDir) -or (Test-Inside $WorkerSource $ctx.ServerDir)) { Fail 'staging directory is inside a live directory' } + $arch = Get-ExeArchitecture $stagedWorkerExe + if ($arch -ne 'X86') { Fail "staged Worker $stagedWorkerExe architecture is '$arch', not X86 (trap 5)" } + Info "staged Worker architecture: X86" + $sSha = Get-ProductSha $stagedServerExe + $wSha = Get-ProductSha $stagedWorkerExe + Info "staged Server SHA: $sSha ; staged Worker SHA: $wSha ; live Server SHA: $(Get-ProductSha $ctx.Application)" if ($sSha -and $wSha -and -not (Test-ShaMatch $sSha $wSha)) { Fail "Server ($sSha) and Worker ($wSha) were built from different commits (wire protocol risk)" } if ($ExpectedSha) { if (-not (Test-ShaMatch $sSha $ExpectedSha)) { @@ -250,6 +510,27 @@ function Invoke-Preflight($ctx) { else { Fail "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha'" } } } else { Warn 'no -ExpectedSha given: the post-deploy SHA check will only record the SHA' } + $leafSha = if ((Get-Leaf $ctx.ServerDir) -match '([0-9a-f]{7,40})$') { $Matches[1] } else { $null } + if ($leafSha -and $sSha -and -not (Test-ShaMatch $leafSha $sSha)) { Warn "the Server dir name '$(Get-Leaf $ctx.ServerDir)' carries SHA $leafSha but will hold $sSha after the rename-in-place swap" } + + # Runtime: never silently downgrade a self-contained Server. + $liveRt = Get-BundledRuntimeVersion $ctx.ServerDir; $newRt = Get-BundledRuntimeVersion $ServerSource + Info "bundled .NET runtime: live $(if ($liveRt) { $liveRt } else { 'none (framework-dependent)' }), staged $(if ($newRt) { $newRt } else { 'none (framework-dependent)' })" + if ($liveRt -and -not $newRt) { Warn 'live Server is self-contained, staged is framework-dependent: the box must have a matching .NET 10 runtime installed' } + if ($liveRt -and $newRt -and $newRt -lt $liveRt) { + if ($AllowRuntimeDowngrade) { Warn "staged runtime $newRt is OLDER than live $liveRt (-AllowRuntimeDowngrade)" } + else { Fail "staged self-contained runtime $newRt is OLDER than the live $liveRt; rebuild with a newer SDK or pass -AllowRuntimeDowngrade" } + } + + # What the swap will carry over / drop (top-level files). + foreach ($pair in @(@('Server', $ctx.ServerDir, $ServerSource), @('Worker', $ctx.WorkerDir, $WorkerSource))) { + $plan = Get-CarryPlan $pair[1] $pair[2] ($pair[0] -eq 'Server' -and -not $TakeNewAppSettings) + Info "$($pair[0]): carry from live -> $(if ($plan.Carry.Count) { $plan.Carry -join ', ' } else { 'nothing' }); live-only binaries dropped -> $(if ($plan.Dropped.Count) { $plan.Dropped -join ', ' } else { 'none' })" + if ($plan.Dirs.Count) { Warn "$($pair[0]): live-only directories NOT carried (they stay in the .bak): $($plan.Dirs -join ', ')" } + } + $lw = @(Get-Files $ctx.WorkerDir | ForEach-Object { $_.Name }); $nw = @(Get-Files $WorkerSource | ForEach-Object { $_.Name }) + $onlyNew = @($nw | Where-Object { $lw -notcontains $_ }) + if ($onlyNew.Count) { Info "Worker: files only in the staged build: $($onlyNew -join ', ')" } # Disk space: 2x (server + worker + stores + payload), at least MinFreeGB if (-not $SkipFreeSpaceCheck) { @@ -272,52 +553,67 @@ function Wait-State([string]$Want, [int]$TimeoutSec) { $false } -function Assert-NoGatewayProcess([string]$When) { - $left = Get-GatewayProcesses - if ($left.Count -gt 0) { Fail "${When}: gateway process(es) still alive: $(($left | ForEach-Object { "$($_.ProcessName)#$($_.Id)" }) -join ', ')" } +function Wait-NoGatewayProcess($ctx, [int]$TimeoutSec) { + $deadline = (Get-Date).AddSeconds($TimeoutSec) + do { if (@(Get-GatewayProcesses $ctx).Count -eq 0) { return $true }; Start-Sleep -Seconds 1 } while ((Get-Date) -lt $deadline) + $false } -function Stop-Gateway { +# Kill only this install's gateway processes: path under the Server/Worker dir, or (path unreadable) the installed image name. +function Stop-InstallProcesses($ctx, [string]$Why) { + $installed = @((Get-Stem $ctx.Application), (Get-Stem $ctx.WorkerExe)) | Where-Object { $_ } + $mine = @(Get-GatewayProcesses $ctx | Where-Object { + if ($_.Path) { (Test-Inside $_.Path $ctx.ServerDir) -or (Test-Inside $_.Path $ctx.WorkerDir) } else { $installed -contains $_.Name } + } | Sort-Object { if ($_.Role -eq 'Server') { 0 } else { 1 } }) + foreach ($p in $mine) { Say "[ACTION] Stop-Process -Id $($p.Id) -Force ($($p.Name), $Why)"; Stop-ProcessById $p.Id } + $mine.Count +} + +function Stop-Gateway($ctx) { Say '-- Stop --' $state = Get-ServiceState if ($state -eq 'SERVICE_STOPPED') { Info 'already stopped' } else { - Step "nssm stop $ServiceName (wait up to $StopTimeoutSeconds s for SERVICE_STOPPED)" { + Step "nssm stop $ServiceName (wait up to $StopTimeoutSeconds s for SERVICE_STOPPED, then for every gateway/Worker process to exit)" { [void](Invoke-Nssm -NssmArgs @('stop', $ServiceName) -AllowFail) if (-not (Wait-State 'SERVICE_STOPPED' $StopTimeoutSeconds)) { - # STOP_PENDING trap: workers did not exit in time. Kill the Server (then Workers) by image name - - # safe because preflight proved there is exactly one gateway on the box. - Warn "service still $(Get-ServiceState) after $StopTimeoutSeconds s (STOP_PENDING trap) - killing gateway processes" - foreach ($n in @($script:ServerProc, $script:WorkerProc)) { - Get-Process -Name $n -ErrorAction SilentlyContinue | ForEach-Object { Say "[ACTION] Stop-Process -Id $($_.Id) -Force ($n)"; Stop-Process -Id $_.Id -Force } - } + # STOP_PENDING trap: workers did not exit in time. Kill this install's Server, then its Workers. + Warn "service still $(Get-ServiceState) after $StopTimeoutSeconds s (STOP_PENDING trap) - killing this install's gateway processes" + $n = Stop-InstallProcesses $ctx 'STOP_PENDING' + if ($n -eq 0) { Warn 'no process of this install was found to kill' } if (-not (Wait-State 'SERVICE_STOPPED' 20)) { Warn 'still not STOPPED; killing the NSSM service host process' - try { - $svcPid = (Get-CimInstance Win32_Service -Filter "Name='$ServiceName'").ProcessId - if ($svcPid) { Stop-Process -Id $svcPid -Force } - } catch { Warn "could not kill the service host: $_" } + $svcPid = Get-ServiceHostPid + if ($svcPid) { Say "[ACTION] Stop-Process -Id $svcPid -Force (service host)"; Stop-ProcessById $svcPid } if (-not (Wait-State 'SERVICE_STOPPED' 20)) { Fail "service '$ServiceName' will not stop; investigate manually (state $(Get-ServiceState))" } } } - Start-Sleep -Seconds 2 - Assert-NoGatewayProcess 'after stop' + if (-not (Wait-NoGatewayProcess $ctx 20)) { + Warn "gateway/Worker process(es) outlived the service stop (unclean stop): $(Format-Procs (Get-GatewayProcesses $ctx))" + [void](Stop-InstallProcesses $ctx 'left over after the stop') + [void](Wait-NoGatewayProcess $ctx 10) + } } } - if (-not $script:Dry) { Assert-NoGatewayProcess 'after stop' } + if (-not $script:Dry) { + Assert-NoGatewayProcess $ctx 'after stop' + Info 'stopped; no gateway/Worker process on the box' + Assert-EnvUnchanged $ctx 'after the stop' + } } function Start-Gateway($ctx) { Say '-- Start --' - if (-not $script:Dry) { Assert-NoGatewayProcess 'before start (OrphanWorkerTerminator guard)' } - else { Say '[DRYRUN] assert no Server/Worker process exists (OrphanWorkerTerminator guard)' } + if (-not $script:Dry) { + Assert-NoGatewayProcess $ctx 'before start (OrphanWorkerTerminator guard)' + Assert-EnvUnchanged $ctx 'before the start' + } else { Say "[DRYRUN] assert no process named $($script:ServerImages -join '|') / $($script:WorkerImages -join '|') exists, and the env block fingerprint is unchanged (OrphanWorkerTerminator guard)" } Step "nssm start $ServiceName (once)" { [void](Invoke-Nssm -NssmArgs @('start', $ServiceName)) } - Step "wait for SERVICE_RUNNING and compare env entry count (expect $($ctx.EnvCount))" { + Step "wait for SERVICE_RUNNING and compare the env block fingerprint" { if (-not (Wait-State 'SERVICE_RUNNING' 60)) { Fail "service did not reach RUNNING (state $(Get-ServiceState))" } - $now = (Parse-EnvBlock (Get-NssmValue 'AppEnvironmentExtra')).Count - if ($now -ne $ctx.EnvCount) { Fail "NSSM env block entry count changed: $($ctx.EnvCount) -> $now" } + Assert-EnvUnchanged $ctx 'after the start' } } @@ -329,94 +625,157 @@ function Backup-State($ctx) { Say '-- Backup (service stopped) --' $suffix = Get-Suffix $root = Split-Path -Parent $ctx.ServerDir - $envFile = Join-Path $root "_env-AppEnvironmentExtra-$($script:Stamp).txt" - Step "save the NSSM env block + Application/AppDirectory to $envFile (contains secrets: keep it ACL-restricted)" { - $body = @("Application=$($ctx.Application)", "AppDirectory=$($ctx.AppDirectory)", '--- AppEnvironmentExtra ---', $ctx.EnvText) - Set-Content -LiteralPath $envFile -Value $body -Encoding UTF8 - try { & icacls.exe $envFile /inheritance:r /grant:r 'SYSTEM:(F)' 'BUILTIN\Administrators:(F)' | Out-Null } catch { Warn "could not restrict ACL on ${envFile}: $_" } + $dir = Join-Path $root "_deploy-backup-$suffix" + $script:BackupDir = $dir + Step "create $dir, restrict it to SYSTEM + Administrators, then save the env block (secrets), service registry key and env fingerprint" { + New-Item -ItemType Directory -Path $dir -Force | Out-Null + Protect-Directory $dir + $body = @("Application=$($ctx.Application)", "AppDirectory=$($ctx.AppDirectory)", '--- AppEnvironmentExtra ---') + @(Read-EnvBlockLines) + Set-Content -LiteralPath (Join-Path $dir 'AppEnvironmentExtra.txt') -Value $body -Encoding UTF8 + Export-ServiceKey (Join-Path $dir "$ServiceName-servicekey.reg") + $fp = @("BlockHash=$($ctx.EnvFp.BlockHash)") + @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" }) + Set-Content -LiteralPath (Join-Path $dir 'env-fingerprint.txt') -Value $fp -Encoding ASCII } foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) { foreach ($f in (Get-StoreFiles $db)) { $dst = "$f.bak-$suffix" - Step "copy store file $f -> $dst (live file is never moved)" { Copy-Item -LiteralPath $f -Destination $dst -Force } + Step "copy store file $f -> $dst (live file is never moved)" { + Copy-Item -LiteralPath $f -Destination $dst -Force + if ((Get-FileSha $f) -ne (Get-FileSha $dst)) { Fail "store backup hash mismatch for $f" } + } } if (-not (Test-Path -LiteralPath $db)) { Info "no store at $db to back up" } } + if ($ctx.KekFile -and (Test-Path -LiteralPath $ctx.KekFile)) { + Step "copy master key file $($ctx.KekFile) into $dir (live file is never moved)" { Copy-Item -LiteralPath $ctx.KekFile -Destination (Join-Path $dir (Get-Leaf $ctx.KekFile)) -Force } + } +} + +function Test-Preserved([string]$Name) { foreach ($p in $script:PreservePatterns) { if ($Name -like $p) { return $true } }; $false } + +# Which top-level files of the live dir go into the new one. Carry = names copied from live (live wins). +function Get-CarryPlan([string]$Live, [string]$Source, [bool]$KeepSettings) { + $liveFiles = @(Get-Files $Live) + $srcNames = @(Get-Files $Source | ForEach-Object { $_.Name }) + $srcDirs = @(Get-Dirs $Source | ForEach-Object { $_.Name }) + $carry = New-Object System.Collections.Generic.List[string]; $dropped = New-Object System.Collections.Generic.List[string] + foreach ($f in $liveFiles) { + $inSource = $srcNames -contains $f.Name + if ($f.Name -like 'appsettings*.json') { if ($KeepSettings) { $carry.Add($f.Name) }; continue } + if (Test-Preserved $f.Name) { $carry.Add($f.Name); continue } + if ($inSource) { continue } + if ($script:BinaryExtensions -contains $f.Extension.ToLowerInvariant()) { $dropped.Add($f.Name) } else { $carry.Add($f.Name) } + } + $dirs = @(Get-Dirs $Live | Where-Object { $srcDirs -notcontains $_.Name } | ForEach-Object { $_.Name }) + [pscustomobject]@{ Carry = $carry.ToArray(); Dropped = $dropped.ToArray(); Dirs = $dirs } } function Swap-Directory([string]$Live, [string]$Source, [string]$Name, [bool]$KeepSettings) { $bak = "$Live.bak-$(Get-Suffix)" + $plan = Get-CarryPlan $Live $Source $KeepSettings + $hashes = @{} + foreach ($f in $plan.Carry) { $hashes[$f] = Get-FileSha (Join-Path $Live $f) } + $sddlOld = Get-DirSddl $Live Step "rename $Name dir: $Live -> $bak" { Rename-Item -LiteralPath $Live -NewName (Split-Path -Leaf $bak) } Step "copy new ${Name}: $Source -> $Live" { Copy-Item -LiteralPath $Source -Destination $Live -Recurse -Force } - if ($KeepSettings) { - $from = if ($script:Dry) { $Live } else { $bak } # in a dry run nothing was renamed yet - foreach ($f in 'appsettings.json', 'appsettings.Development.json') { - if (Test-Path -LiteralPath (Join-Path $from $f)) { - Step "restore live $f byte-identical from $bak" { Copy-Item -LiteralPath (Join-Path $bak $f) -Destination (Join-Path $Live $f) -Force } - } else { Info "$f not present in the old $Name dir (nothing to carry over)" } + foreach ($f in $plan.Carry) { + Step "carry live $Name\$f forward from $bak (sha256 $($hashes[$f].Substring(0, 12)))" { + Copy-Item -LiteralPath (Join-Path $bak $f) -Destination (Join-Path $Live $f) -Force + if ((Get-FileSha (Join-Path $Live $f)) -ne $hashes[$f]) { Fail "carried $Name\$f does not match the live copy" } + Copy-ExplicitAcl (Join-Path $bak $f) (Join-Path $Live $f) } } + foreach ($f in $plan.Dropped) { Info "$Name\$f exists only in the old build (binary) and is not carried; it stays in $bak" } + if (-not $script:Dry) { + $sddlNew = Get-DirSddl $Live + if ($sddlOld -and $sddlNew -and $sddlOld -ne $sddlNew) { Warn "$Name dir ACL differs from the old one (new inherits from the parent). old: $sddlOld new: $sddlNew" } + } $bak } # --- verification ---------------------------------------------------------------------------------------------------- -function Invoke-Verify($ctx) { +function Invoke-Verify($ctx, [bool]$AfterRollback = $false) { Say '== Verify ==' if ($script:Dry) { Say "[DRYRUN] GET $HealthBaseUrl/health/ready until 200 (<= $ReadyTimeoutSeconds s)" - Say "[DRYRUN] GET $HealthBaseUrl/healthz ; read ProductVersion SHA of $(Join-Path $ctx.ServerDir $ctx.ServerExeName)$(if ($ExpectedSha) { " and require it to match $ExpectedSha" })" - Say "[DRYRUN] service must stay SERVICE_RUNNING for $StableSeconds s with env entry count $($ctx.EnvCount)" + Say "[DRYRUN] GET $HealthBaseUrl/healthz ; read ProductVersion SHA of $($ctx.Application)$(if ($ExpectedSha -and -not $Rollback) { " and require it to match $ExpectedSha" })" + Say "[DRYRUN] exactly one gateway Server ($($ctx.Application)) on the box, same PID, service SERVICE_RUNNING for $StableSeconds s, env fingerprint unchanged" return } $ready = $false; $deadline = (Get-Date).AddSeconds($ReadyTimeoutSeconds) while (-not $ready -and (Get-Date) -lt $deadline) { - try { $r = Invoke-WebRequest -Uri "$HealthBaseUrl/health/ready" -UseBasicParsing -TimeoutSec 10; $ready = ($r.StatusCode -eq 200) } - catch { Start-Sleep -Seconds 3 } + $ready = ((Get-HttpStatus "$HealthBaseUrl/health/ready") -eq 200) + if (-not $ready) { Start-Sleep -Seconds 3 } } if (-not $ready) { Fail "$HealthBaseUrl/health/ready did not return 200 within $ReadyTimeoutSeconds s" } Info '/health/ready = 200' - try { Info "/healthz = $((Invoke-WebRequest -Uri "$HealthBaseUrl/healthz" -UseBasicParsing -TimeoutSec 10).StatusCode)" } catch { Fail "/healthz failed: $_" } + $hz = Get-HttpStatus "$HealthBaseUrl/healthz" + if ($hz -ne 200) { Fail "/healthz = $hz" } + Info '/healthz = 200' - $sha = Get-ProductSha (Join-Path $ctx.ServerDir $ctx.ServerExeName) + $sha = Get-ProductSha $ctx.Application Info "deployed Server SHA (ProductVersion): $sha" - if ($ExpectedSha -and -not $Rollback -and -not (Test-ShaMatch $sha $ExpectedSha)) { Fail "deployed SHA '$sha' != expected '$ExpectedSha'" } + if ($ExpectedSha -and -not $Rollback -and -not $AfterRollback -and -not (Test-ShaMatch $sha $ExpectedSha)) { Fail "deployed SHA '$sha' != expected '$ExpectedSha'" } - Info "watching the service for $StableSeconds s" + Assert-SingleGateway $ctx 'SERVICE_RUNNING' + $pid0 = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' })[0].Id + Info "watching the service for $StableSeconds s (gateway #$pid0)" $until = (Get-Date).AddSeconds($StableSeconds) - while ((Get-Date) -lt $until) { + do { if ((Get-ServiceState) -ne 'SERVICE_RUNNING') { Fail 'service left RUNNING during the stability window (restart loop?)' } Start-Sleep -Seconds 5 - } - $n = (Parse-EnvBlock (Get-NssmValue 'AppEnvironmentExtra')).Count - if ($n -ne $ctx.EnvCount) { Fail "env block entry count $n != preflight $($ctx.EnvCount)" } - Info "env block still $n entries" + } while ((Get-Date) -lt $until) + $servers = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' }) + if ($servers.Count -ne 1 -or $servers[0].Id -ne $pid0) { Fail "gateway process changed during the stability window: #$pid0 -> $(Format-Procs $servers) (restart loop or second gateway)" } + Assert-EnvUnchanged $ctx 'after the stability window' Warn 'health is green with zero sessions; still prove the Worker path: open a session with a PRE-EXISTING API key and read a known tag (runbook 7.3)' } # --- rollback ------------------------------------------------------------------------------------------------------------ -function Find-Backup([string]$Live, [string]$Stamp) { - $leaf = Split-Path -Leaf $Live; $parent = Split-Path -Parent $Live - $cands = @(Get-ChildItem -LiteralPath $parent -Directory -Filter "$leaf.bak-*" -ErrorAction SilentlyContinue) - if ($Stamp) { $cands = @($cands | Where-Object { $_.Name -like "$leaf.bak-$Stamp*" }) } - $cands | Sort-Object Name -Descending | Select-Object -First 1 +# Backup dirs come in two spellings: .bak-yyyyMMdd-HHmmss[-label] (this script) and .bak.yyyyMMddTHHmmss (manual / adapted scripts). +function Get-BackupKey([string]$Name) { + if ($Name -match '\.bak[-.](\d{8})[-T]?(\d{6})') { return "$($Matches[1])$($Matches[2])" } + $null } -function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores) { +function Find-Backup([string]$Live, [string]$Stamp) { + $leaf = Split-Path -Leaf $Live; $parent = Split-Path -Parent $Live + $cands = @(Get-Dirs $parent | Where-Object { ($_.Name -like "$leaf.bak-*" -or $_.Name -like "$leaf.bak.*") -and (Get-BackupKey $_.Name) }) + if ($Stamp) { + $key = ($Stamp -replace '[^0-9]', '') + $cands = @($cands | Where-Object { (Get-BackupKey $_.Name) -eq $key -or $_.Name -like "$leaf.bak-$Stamp*" -or $_.Name -like "$leaf.bak.$Stamp*" }) + } + $cands | Sort-Object { Get-BackupKey $_.Name } -Descending | Select-Object -First 1 +} + +function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores, [bool]$AllowPartial) { Say '== Rollback ==' $sb = Find-Backup $ctx.ServerDir $Stamp - $wb = Find-Backup $ctx.WorkerDir $Stamp - if (-not $sb -or -not $wb) { Fail "no matching Server/Worker backup directories found (stamp '$Stamp')" } - Info "rolling back to Server=$($sb.Name) Worker=$($wb.Name)" - Stop-Gateway + if (-not $sb -and -not $AllowPartial) { Fail "no Server backup directory found (stamp '$Stamp')" } + $key = if ($sb) { Get-BackupKey $sb.Name } else { ($Stamp -replace '[^0-9]', '') } + $wb = Find-Backup $ctx.WorkerDir $key + if ($wb -and (Get-BackupKey $wb.Name) -ne $key) { $wb = $null } + if (-not $wb -and -not $AllowPartial) { Fail "no Worker backup matching the Server backup $($sb.Name) (stamp key $key); refusing a mixed rollback" } + if (-not $sb -and -not $wb) { Fail "nothing to roll back to (stamp '$Stamp')" } + Info "rolling back to Server=$(if ($sb) { $sb.Name } else { '(unchanged)' }) Worker=$(if ($wb) { $wb.Name } else { '(unchanged)' })" + $fpFile = @(Get-Dirs (Split-Path -Parent $ctx.ServerDir) '_deploy-backup-*' | Where-Object { ($_.Name -replace '[^0-9]', '').StartsWith($key) } | ForEach-Object { Join-Path $_.FullName 'env-fingerprint.txt' } | Where-Object { Test-Path -LiteralPath $_ }) + if ($fpFile.Count) { + $saved = Get-Content -LiteralPath $fpFile[0] | Where-Object { $_ -notlike 'BlockHash=*' } + $now = @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" }) + if (($saved -join '|') -ne ($now -join '|')) { Warn "the env block differs from the one saved before the upgrade ($($fpFile[0])); restore it from AppEnvironmentExtra.txt there if that was not intended (runbook section 8 step 4)" } + else { Info "env block matches the pre-upgrade fingerprint in $($fpFile[0])" } + } + Stop-Gateway $ctx foreach ($pair in @(@($ctx.ServerDir, $sb), @($ctx.WorkerDir, $wb))) { + if (-not $pair[1]) { Info "$($pair[0]) has no backup for this stamp; left unchanged"; continue } $failed = "$($pair[0]).failed-$($script:Stamp)" if (Test-Path -LiteralPath $pair[0]) { Step "rename $($pair[0]) -> $failed" { Rename-Item -LiteralPath $pair[0] -NewName (Split-Path -Leaf $failed) } } Step "rename $($pair[1].FullName) -> $($pair[0])" { Rename-Item -LiteralPath $pair[1].FullName -NewName (Split-Path -Leaf $pair[0]) } } if ($Stores) { - $bstamp = ($sb.Name -replace '^.*\.bak-', '') + $bstamp = ($sb.Name -replace '^.*\.bak[-.]', '') foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) { foreach ($ext in '', '-wal', '-shm') { $live = "$db$ext"; $bak = "$db$ext.bak-$bstamp" @@ -434,57 +793,75 @@ function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores) { # --- main ------------------------------------------------------------------------------------------------------------------- -$exit = 0 -try { - Say "Upgrade-MxGatewayService stamp=$($script:Stamp) mode=$(if ($Rollback) {'ROLLBACK'} else {'UPGRADE'}) dry=$($script:Dry)" - $ctx = Get-Context - Invoke-Preflight $ctx - - if (-not $script:Dry) { - $lockPath = Join-Path (Split-Path -Parent $ctx.ServerDir) '.mxgw-deploy.lock' - try { $script:LockStream = [IO.File]::Open($lockPath, 'OpenOrCreate', 'ReadWrite', 'None') } - catch { Fail "another deploy run holds $lockPath" } - } - - if ($Rollback) { - Invoke-RollbackSwap $ctx $BackupStamp $RestoreStores.IsPresent - Invoke-Verify $ctx - } else { - Stop-Gateway - Backup-State $ctx - $swapped = $false - try { - $null = Swap-Directory $ctx.ServerDir $ServerSource 'Server' (-not $TakeNewAppSettings) - $null = Swap-Directory $ctx.WorkerDir $WorkerSource 'Worker' $false - $swapped = $true - Start-Gateway $ctx - Invoke-Verify $ctx - } catch { - $err = $_ - Say "[FAIL] $($err.Exception.Message)" - if ($script:Dry -or $NoAutoRollback) { throw } - Warn 'upgrade failed - rolling back automatically' - try { - if (-not $swapped) { Warn 'swap did not complete; rollback swaps whatever .bak sets exist for this stamp' } - Invoke-RollbackSwap $ctx $script:Stamp $false - Invoke-Verify $ctx - Warn 'ROLLED BACK to the previous build; original failure follows' - } catch { Warn "AUTOMATIC ROLLBACK FAILED: $($_.Exception.Message). Manual intervention required (backups: *.bak-$(Get-Suffix))." } - throw $err - } - } - Say '== DONE ==' -} catch { - $exit = 1 - Write-Host "ERROR: $($_.Exception.Message)" -ForegroundColor Red -} finally { - if ($script:LockStream) { $script:LockStream.Dispose() } +function Invoke-Main { + $exit = 0 + $ctx = $null try { - if (-not $script:Dry -and $ctx) { - $notes = Join-Path (Split-Path -Parent $ctx.ServerDir) "_DEPLOY-NOTES-$($script:Stamp).txt" - Set-Content -LiteralPath $notes -Value $script:Notes -Encoding UTF8 - Write-Host "deploy notes: $notes" + Say "Upgrade-MxGatewayService stamp=$($script:Stamp) mode=$(if ($Rollback) {'ROLLBACK'} else {'UPGRADE'}) dry=$($script:Dry) host=$env:COMPUTERNAME" + $ctx = Get-Context + Invoke-Preflight $ctx + + if (-not $script:Dry) { + $lockPath = Join-Path (Split-Path -Parent $ctx.ServerDir) '.mxgw-deploy.lock' + try { $script:LockStream = [IO.File]::Open($lockPath, 'OpenOrCreate', 'ReadWrite', 'None') } + catch { Fail "another deploy run holds $lockPath" } } - } catch { } + + if ($Rollback) { + Invoke-RollbackSwap $ctx $BackupStamp $RestoreStores.IsPresent $false + Invoke-Verify $ctx + } else { + Stop-Gateway $ctx + try { Backup-State $ctx } + catch { + $err = $_ + if (-not $script:Dry) { + Warn 'backup failed before anything was swapped - starting the unchanged build again' + try { Start-Gateway $ctx } catch { Warn "restart of the unchanged build FAILED: $($_.Exception.Message)" } + } + throw $err + } + $touched = $false + try { + $touched = $true + $null = Swap-Directory $ctx.ServerDir $ServerSource 'Server' (-not $TakeNewAppSettings) + $null = Swap-Directory $ctx.WorkerDir $WorkerSource 'Worker' $false + if (-not $script:Dry) { Assert-EnvUnchanged $ctx 'after the swap' } + Start-Gateway $ctx + Invoke-Verify $ctx + } catch { + $err = $_ + Say "[FAIL] $(Hide $err.Exception.Message)" + if ($script:Dry -or $NoAutoRollback -or -not $touched) { throw } + Warn 'upgrade failed - rolling back automatically' + $script:EnvWarnOnly = $true + try { + Invoke-RollbackSwap $ctx $script:Stamp $false $true + Invoke-Verify $ctx $true + Warn 'ROLLED BACK to the previous build; original failure follows' + } catch { Warn "AUTOMATIC ROLLBACK FAILED: $($_.Exception.Message). Manual intervention required (backups: *.bak-$(Get-Suffix))." } + throw $err + } + } + Say '== DONE ==' + } catch { + $exit = 1 + $m = Hide $_.Exception.Message + $script:Notes.Add("ERROR: $m") + Write-Host "ERROR: $m" -ForegroundColor Red + } finally { + if ($script:LockStream) { $script:LockStream.Dispose(); $script:LockStream = $null } + try { + if (-not $script:Dry -and $ctx) { + $notes = Join-Path (Split-Path -Parent $ctx.ServerDir) "_DEPLOY-NOTES-$($script:Stamp).txt" + Set-Content -LiteralPath $notes -Value $script:Notes -Encoding UTF8 + Write-Host "deploy notes: $notes" + } + } catch { } + $script:Redact.Clear() + } + $exit } -exit $exit + +# Dot-sourcing (the dry-run harness) only defines the functions. +if ($MyInvocation.InvocationName -ne '.') { $rc = Invoke-Main | Select-Object -Last 1; exit $rc } diff --git a/scripts/deploy/tests/Invoke-UpgradeMutations.ps1 b/scripts/deploy/tests/Invoke-UpgradeMutations.ps1 new file mode 100644 index 00000000..8a844f6c --- /dev/null +++ b/scripts/deploy/tests/Invoke-UpgradeMutations.ps1 @@ -0,0 +1,77 @@ +<# +.SYNOPSIS + Mutation check for Test-UpgradeMxGatewayService.ps1 (mxaccessgw#230): re-introduces each guarded defect into a + temp copy of Upgrade-MxGatewayService.ps1 and requires the harness to FAIL on it. A mutation the harness does not + catch means an assertion is vacuous. Exits non-zero if any mutation survives (or does not apply). + +.EXAMPLE + pwsh -NoProfile -File scripts/deploy/tests/Invoke-UpgradeMutations.ps1 +#> +$ErrorActionPreference = 'Stop' +$here = Split-Path -Parent $MyInvocation.MyCommand.Path +$src = Get-Content -Raw (Join-Path (Split-Path -Parent $here) 'Upgrade-MxGatewayService.ps1') +$harness = Join-Path $here 'Test-UpgradeMxGatewayService.ps1' + +$mutations = [ordered]@{ + # The #230 defect: only the ZB.MOM.WW.* names, nothing derived from the installed exes. + 'only ZB.MOM.WW.* image names (#230 original)' = @( + @("`$script:KnownServerImages = @('ZB.MOM.WW.MxGateway.Server', 'MxGateway.Server')", "`$script:KnownServerImages = @('ZB.MOM.WW.MxGateway.Server')"), + @("`$script:KnownWorkerImages = @('ZB.MOM.WW.MxGateway.Worker', 'MxGateway.Worker')", "`$script:KnownWorkerImages = @('ZB.MOM.WW.MxGateway.Worker')"), + @('$script:ServerImages = @(@(Get-Stem $ctx.Application) + $script:KnownServerImages', '$script:ServerImages = @(@() + $script:KnownServerImages'), + @('$script:WorkerImages = @(@(Get-Stem $wexe) + $script:KnownWorkerImages', '$script:WorkerImages = @(@() + $script:KnownWorkerImages'), + @('function Test-GatewayServer($p, $ctx) { ($script:ServerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.Application) }', 'function Test-GatewayServer($p, $ctx) { ($script:ServerImages -contains $p.Name) }'), + @('function Test-GatewayWorker($p, $ctx) { ($script:WorkerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.WorkerExe) }', 'function Test-GatewayWorker($p, $ctx) { ($script:WorkerImages -contains $p.Name) }')) + 'guard passes when no gateway is visible' = @( + @(' if ($servers.Count -eq 0) {', ' if ($false) {')) + 'other naming scheme not matched' = @( + @("`$script:KnownServerImages = @('ZB.MOM.WW.MxGateway.Server', 'MxGateway.Server')", "`$script:KnownServerImages = @()"), + @("`$script:KnownWorkerImages = @('ZB.MOM.WW.MxGateway.Worker', 'MxGateway.Worker')", "`$script:KnownWorkerImages = @()")) + 'no parent check' = @( + @('if ($s.ParentId -ne $hostPid) { Fail', 'if ($false) { Fail')) + 'pfx and live-only files not carried (#230 original)' = @( + @(' if (Test-Preserved $f.Name) { $carry.Add($f.Name); continue }', ''), + @('if ($script:BinaryExtensions -contains $f.Extension.ToLowerInvariant()) { $dropped.Add($f.Name) } else { $carry.Add($f.Name) }', 'if ($script:BinaryExtensions -contains $f.Extension.ToLowerInvariant()) { $dropped.Add($f.Name) }')) + 'env check is a no-op' = @( + @(' $diff = Compare-EnvFingerprint $ctx.EnvFp $now', ' $diff = @()')) + 'env compare looks at names only' = @( + @('elseif ($After.Entries[$k] -ne $Before.Entries[$k])', 'elseif ($false)'), + @('if ($d.Count -eq 0 -and $Before.BlockHash -ne $After.BlockHash)', 'if ($false)')) + 'values not redacted' = @( + @('foreach ($s in $script:Redact) { if ($s) { $m = $m.Replace($s, '''') } }', ''), + @('function Format-EnvFingerprint($Fp) { "$($Fp.Count) entries, block sha256 $($Fp.BlockHash): " + (($Fp.Entries.Keys | ForEach-Object { "$_=$($Fp.Entries[$_])" }) -join '', '') }', 'function Format-EnvFingerprint($Fp) { "$($Fp.Count) entries: " + ((ConvertTo-EnvMap (Read-EnvBlockLines)).GetEnumerator() | ForEach-Object { "$($_.Key)=$($_.Value)" }) -join '', '' }')) + 'STOP_PENDING kill by any name' = @( + @('if ($_.Path) { (Test-Inside $_.Path $ctx.ServerDir) -or (Test-Inside $_.Path $ctx.WorkerDir) } else { $installed -contains $_.Name }', '$true'), + @(' $mine = @(Get-GatewayProcesses $ctx | Where-Object {', ' $mine = @(Get-ProcessTable | Where-Object {')) + 'runtime downgrade allowed' = @( + @('if ($liveRt -and $newRt -and $newRt -lt $liveRt) {', 'if ($false) {')) + 'mixed rollback allowed' = @( + @('if ($wb -and (Get-BackupKey $wb.Name) -ne $key) { $wb = $null }', ''), + @(' $wb = Find-Backup $ctx.WorkerDir $key', ' $wb = Find-Backup $ctx.WorkerDir $null')) + '.bak. sets not found' = @( + @('($_.Name -like "$leaf.bak-*" -or $_.Name -like "$leaf.bak.*")', '($_.Name -like "$leaf.bak-*")')) +} + +$tmp = Join-Path ([IO.Path]::GetTempPath()) ("mxgw230-mut-" + [guid]::NewGuid().ToString('n').Substring(0, 8)) +New-Item -ItemType Directory -Path $tmp | Out-Null +$survivors = 0 +try { + foreach ($name in $mutations.Keys) { + $m = $src + $pairs = @($mutations[$name]) + if ($pairs[0] -is [string]) { $pairs = @(, $pairs) } # a single @(@(old, new)) flattens to @(old, new) + foreach ($pair in $pairs) { + if (-not $m.Contains($pair[0])) { Write-Host "MUTATION DID NOT APPLY: $name -> '$($pair[0])'" -ForegroundColor Red; $survivors++; continue } + $m = $m.Replace($pair[0], $pair[1]) + } + $path = Join-Path $tmp 'Upgrade-MxGatewayService.ps1' + Set-Content -LiteralPath $path -Value $m -NoNewline + $out = @(& pwsh -NoProfile -File $harness -ScriptPath $path 2>&1) + $rc = $LASTEXITCODE + $fails = @($out | Where-Object { "$_" -match '^\s+FAIL \[' }) + # Killed only if the harness ran to its verdict and reported failed assertions (a crash is not a kill). + if ($rc -ne 0 -and $fails.Count -gt 0 -and ($out -match '^FAILED: ')) { Write-Host ("killed {0,-48} ({1} assertion(s) failed, e.g. {2})" -f $name, $fails.Count, ("$($fails | Select-Object -First 1)".Trim())) } + else { Write-Host "SURVIVED $name (rc=$rc, $($fails.Count) failed assertion(s)): $(($out | Select-Object -Last 2) -join ' | ')" -ForegroundColor Red; $survivors++ } + } +} finally { Remove-Item -LiteralPath $tmp -Recurse -Force } +if ($survivors -gt 0) { Write-Host "$survivors mutation(s) survived or did not apply" -ForegroundColor Red; exit 1 } +Write-Host "all $($mutations.Count) mutations killed" -ForegroundColor Green diff --git a/scripts/deploy/tests/Test-UpgradeMxGatewayService.ps1 b/scripts/deploy/tests/Test-UpgradeMxGatewayService.ps1 new file mode 100644 index 00000000..4b091ecc --- /dev/null +++ b/scripts/deploy/tests/Test-UpgradeMxGatewayService.ps1 @@ -0,0 +1,382 @@ +<# +.SYNOPSIS + Dry-run harness for scripts/deploy/Upgrade-MxGatewayService.ps1 (mxaccessgw#230). Runs the script's real logic + against a fake box: a temp directory laid out like wonder-app-vd03 (MxGateway.Server.exe / MxGateway.Worker.exe, + dashboard.pfx, file-based master key) or like windev (ZB.MOM.WW.MxGateway.*.exe, dated Server dir), with a fake + nssm, process table, registry env block, health endpoint and clock (tests/UpgradeFakes.ps1). The repo has no + Pester; this is a plain script that exits non-zero when any assertion fails. + + Besides -DryRun scenarios it runs full upgrades and rollbacks for real against the temp directories (renames, + copies, carry-over, hash checks, auto-rollback), because only the service manager and processes are faked. + +.PARAMETER ScriptPath The upgrade script under test (default: ../Upgrade-MxGatewayService.ps1). The mutation runner + points this at mutated copies to prove the assertions can fail. + +.EXAMPLE + pwsh -NoProfile -File scripts/deploy/tests/Test-UpgradeMxGatewayService.ps1 +#> +param([string]$ScriptPath) +Set-StrictMode -Off +$ErrorActionPreference = 'Stop' +$here = Split-Path -Parent $MyInvocation.MyCommand.Path +if (-not $ScriptPath) { $ScriptPath = Join-Path (Split-Path -Parent $here) 'Upgrade-MxGatewayService.ps1' } +$FakesPath = Join-Path $here 'UpgradeFakes.ps1' + +$script:failures = 0 +$script:passes = 0 +$script:group = '' +function Assert([bool]$Condition, [string]$Message) { + if ($Condition) { $script:passes++ } else { $script:failures++; Write-Host " FAIL [$($script:group)]: $Message" -ForegroundColor Red } +} +# Not 'Group': that is a built-in alias of Group-Object, and aliases win over functions. +# An exception in a section is a failed assertion, and the run still reaches its verdict. +trap { $script:failures++; Write-Host " FAIL [$($script:group)]: exception: $($_.Exception.Message)" -ForegroundColor Red; continue } +function Section([string]$Name) { $script:group = $Name; Write-Host $Name } + +$Pepper = 'pepper-VALUE-do-not-print-7f3a' +$LdapPw = 'ldap-PASSWORD-do-not-print-91c2' +$OldSha = '2b13564e' +$NewSha = '76a71bc8' + +function New-File([string]$Path, [string]$Content) { + [void][IO.Directory]::CreateDirectory((Split-Path -Parent $Path)) # New-Item -Force costs ~70 ms per call on macOS + [IO.File]::WriteAllText($Path, $Content) +} + +# Scheme 'wonder' = vd03 (MxGateway.*.exe, dashboard.pfx in Server\, file KEK in data\); 'windev' = ZB.MOM.WW.* in a dated dir. +function New-FakeBox([string]$Scheme) { + $root = Join-Path ([IO.Path]::GetTempPath()) ("mxgw230-" + [guid]::NewGuid().ToString('n').Substring(0, 8)) + if ($Scheme -eq 'wonder') { $srvName = 'MxGateway.Server.exe'; $wrkName = 'MxGateway.Worker.exe'; $srvDir = "$root/Server" } + else { $srvName = 'ZB.MOM.WW.MxGateway.Server.exe'; $wrkName = 'ZB.MOM.WW.MxGateway.Worker.exe'; $srvDir = "$root/Server-20261112-$OldSha" } + $wrkDir = "$root/Worker" + # live install + New-File "$srvDir/$srvName" "sha=$OldSha;arch=Amd64" + New-File "$srvDir/ZB.MOM.WW.MxGateway.Server.dll" "old server dll $OldSha" + New-File "$srvDir/Old.Removed.Dependency.dll" 'binary only in the old build' + New-File "$srvDir/appsettings.json" '{ "live": "host config with no pepper" }' + New-File "$srvDir/appsettings.Development.json" '{ "live": "dev overlay" }' + New-File "$srvDir/mscordaccore_amd64_amd64_10.0.1426.12345.dll" 'runtime 10.0.14' + if ($Scheme -eq 'wonder') { New-File "$srvDir/dashboard.pfx" 'PFX-BYTES-live-dashboard-cert' } + New-File "$srvDir/notes-left-by-an-operator.txt" 'live-only text file' + [void][IO.Directory]::CreateDirectory("$srvDir/logs") + New-File "$wrkDir/$wrkName" "sha=$OldSha;arch=X86" + New-File "$wrkDir/$wrkName.config" '' + New-File "$root/data/gateway-auth.db" 'auth-db'; New-File "$root/data/gateway-auth.db-wal" 'auth-wal' + New-File "$root/data/mxgateway-secrets.db" 'secrets-db'; New-File "$root/data/kek.key" 'KEK-BYTES' + # staged new build (the repo appsettings.json, no pfx) + New-File "$root/stage/Server/$srvName" "sha=$NewSha;arch=Amd64" + New-File "$root/stage/Server/ZB.MOM.WW.MxGateway.Server.dll" "new server dll $NewSha" + New-File "$root/stage/Server/appsettings.json" '{ "repo": "default config" }' + New-File "$root/stage/Server/mscordaccore_amd64_amd64_10.0.1426.12345.dll" 'runtime 10.0.14' + New-File "$root/stage/Worker/$wrkName" "sha=$NewSha;arch=X86" + New-File "$root/stage/Worker/$wrkName.config" '' + + $app = "$srvDir/$srvName"; $wexe = "$wrkDir/$wrkName" + $env = @( + 'ASPNETCORE_ENVIRONMENT=Production', + "MxGateway__Worker__ExecutablePath=$wexe", + "MxGateway__Authentication__SqlitePath=$root/data/gateway-auth.db", + "Secrets__SqlitePath=$root/data/mxgateway-secrets.db", + "MxGateway__ApiKeyPepper=$Pepper", + "MxGateway__Ldap__ServiceAccountPassword=$LdapPw") + if ($Scheme -eq 'wonder') { $env += @("Secrets__MasterKey__FilePath=$root/data/kek.key", 'Secrets__MasterKey__Source=File') } + $hostPid = 4100 + $box = @{ + Scheme = $Scheme; Root = $root; ServerDir = $srvDir; WorkerDir = $wrkDir; Application = $app; AppDirectory = $srvDir; WorkerExe = $wexe + State = 'SERVICE_RUNNING'; Env = $env; HostPid = $hostPid; Health = 200; StopHangs = $false; Starts = 0; OnStart = $null + OtherServices = @(); Survivors = @() + Procs = @( + [pscustomobject]@{ Id = 4200; ParentId = $hostPid; Name = [IO.Path]::GetFileNameWithoutExtension($srvName); Path = $app; Role = 'Server' }, + [pscustomobject]@{ Id = 4201; ParentId = 4200; Name = [IO.Path]::GetFileNameWithoutExtension($wrkName); Path = $wexe; Role = 'Worker' }, + [pscustomobject]@{ Id = 4202; ParentId = 4200; Name = [IO.Path]::GetFileNameWithoutExtension($wrkName); Path = $wexe; Role = 'Worker' }, + [pscustomobject]@{ Id = 900; ParentId = 1; Name = 'svchost'; Path = 'C:/Windows/System32/svchost.exe'; Role = 'Other' }) + NssmCalls = (New-Object System.Collections.Generic.List[string]); Killed = (New-Object System.Collections.Generic.List[int]) + HttpCalls = (New-Object System.Collections.Generic.List[string]); Protected = (New-Object System.Collections.Generic.List[string]) + StartedWith = (New-Object System.Collections.Generic.List[string]) + } + $box +} + +function Get-TreeHash([string]$Root) { + $d = New-Object IO.DirectoryInfo $Root + ((@($d.GetDirectories('*', 'AllDirectories') | ForEach-Object { "D $($_.FullName)" }) + @($d.GetFiles('*', 'AllDirectories') | ForEach-Object { "F $($_.FullName) $((Get-FileHash -LiteralPath $_.FullName).Hash)" })) | Sort-Object) -join "`n" +} + +# Runs Invoke-Main of the script under test against $Box with the fakes loaded. Returns rc + all output text. +function Invoke-Scenario([hashtable]$Box, [hashtable]$P) { + $script:Box = $Box + $script:FakeNow = [datetime]'2026-11-12T12:00:00' + $base = @{ ServerSource = "$($Box.Root)/stage/Server"; WorkerSource = "$($Box.Root)/stage/Worker"; ExpectedSha = $NewSha; MinFreeGB = 0.001; StableSeconds = 10 } + foreach ($k in $P.Keys) { $base[$k] = $P[$k] } + foreach ($k in @($base.Keys)) { if ($null -eq $base[$k]) { $base.Remove($k) } } + . $ScriptPath @base + . $FakesPath + $records = @(Invoke-Main 6>&1 3>&1) + $rc = @($records | Where-Object { $_ -is [int] })[-1] + $text = (@($records | Where-Object { $_ -isnot [int] } | ForEach-Object { "$_" }) -join "`n") + [pscustomobject]@{ Rc = $rc; Text = $text } +} + +function Assert-NoSecrets([string]$Text, [string]$Where) { + Assert (-not $Text.Contains($Pepper)) "the pepper value does not appear in $Where" + Assert (-not $Text.Contains($LdapPw)) "the LDAP password does not appear in $Where" +} + +# --------------------------------------------------------------------------------------------------------------- +foreach ($scheme in 'wonder', 'windev') { + Section "Dry run on a $scheme layout sees the live gateway and changes nothing" + $b = New-FakeBox $scheme + $before = Get-TreeHash $b.Root + $r = Invoke-Scenario $b @{ DryRun = $true } + Assert ($r.Rc -eq 0) "dry run exits 0 (rc=$($r.Rc)) $(if ($r.Rc -ne 0) { $r.Text })" + $stem = [IO.Path]::GetFileNameWithoutExtension($b.Application) + Assert ($r.Text -match "gateway Server #4200 is the service's own") 'the guard found the service gateway by PID/parent' + Assert ($r.Text -match [regex]::Escape("$stem#4200")) "the process listing shows $stem#4200" + Assert ($r.Text -match "MxGateway.Worker.exe#4201|$([regex]::Escape([IO.Path]::GetFileNameWithoutExtension($b.WorkerExe)))#4201") 'the Worker processes are seen' + Assert ((Get-TreeHash $b.Root) -eq $before) 'no file or directory changed' + Assert (@($b.NssmCalls | Where-Object { $_ -match '^(stop|start|set)' }).Count -eq 0) "no nssm stop/start/set ($($b.NssmCalls -join '; '))" + Assert ($b.Killed.Count -eq 0) 'no process killed' + Assert ($r.Text -match '\[DRYRUN\] nssm stop') 'the stop is printed, not run' + Assert ($r.Text -match 'no other service launches') 'the other-service cross-check ran (an empty result is not "unavailable")' + Assert ($r.Text -match 'carry from live -> [^\r\n]*appsettings\.json') 'plan carries appsettings.json' + Assert ($r.Text -match 'carry from live -> [^\r\n]*notes-left-by-an-operator\.txt') 'plan carries a live-only text file' + Assert ($r.Text -match 'dropped -> Old\.Removed\.Dependency\.dll') 'plan drops a live-only binary' + Assert ($r.Text -match 'live-only directories NOT carried[^\r\n]*logs') 'plan warns about the live-only logs dir' + Assert ($r.Text -match 'MxGateway__ApiKeyPepper=[0-9a-f]{12}') 'the env fingerprint lists the pepper by name + hash prefix' + Assert-NoSecrets $r.Text 'the dry-run output' + if ($scheme -eq 'wonder') { + Assert ($r.Text -match 'carry from live -> [^\r\n]*dashboard\.pfx') 'plan carries dashboard.pfx' + Assert ($r.Text -match 'master key: file') 'file-based master key accepted' + } else { + Assert ($r.Text -match 'carries no master key') 'missing env master key is a warning on windev, not an abort' + Assert ($r.Text -match 'carries SHA 2b13564e but will hold 76a71bc8') 'dated dir name mismatch is flagged' + } + Remove-Item -LiteralPath $b.Root -Recurse -Force +} + +Section '-WhatIf is the same as -DryRun' +$b = New-FakeBox 'windev' +$before = Get-TreeHash $b.Root +$r = Invoke-Scenario $b @{ WhatIf = $true } +Assert ($r.Rc -eq 0 -and $r.Text -match 'dry=True' -and $r.Text -match '== DONE ==') "-WhatIf runs the whole plan (rc=$($r.Rc))" +Assert ($r.Text -match '\[DRYRUN\] carry live Server\\appsettings\.json forward[^\r\n]*sha256 [0-9A-F]{12}') 'carry-over hashes are computed under -WhatIf' +Assert ((Get-TreeHash $b.Root) -eq $before -and $b.Starts -eq 0) 'nothing changed' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'Fail closed: service RUNNING but no gateway process visible' +$b = New-FakeBox 'wonder' +$b.Procs = @($b.Procs | Where-Object { $_.Role -eq 'Other' }) +$r = Invoke-Scenario $b @{ DryRun = $true } +Assert ($r.Rc -eq 1) 'refuses (rc 1)' +Assert ($r.Text -match 'RUNNING but no gateway Server process was found') "says why ($($r.Text -split "`n" | Select-String ERROR))" +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'Fail closed: the gateway runs under an image name and path the script does not know' +$b = New-FakeBox 'wonder' +foreach ($p in $b.Procs) { if ($p.Role -ne 'Other') { $p.Name = 'Renamed.Gateway'; $p.Path = $null } } +$r = Invoke-Scenario $b @{ DryRun = $true } +Assert ($r.Rc -eq 1) 'refuses (rc 1)' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'Path match works when the image name is unexpected' +$b = New-FakeBox 'wonder' +foreach ($p in $b.Procs) { if ($p.Role -eq 'Server') { $p.Name = 'Renamed.Gateway' } } +$r = Invoke-Scenario $b @{ DryRun = $true } +Assert ($r.Rc -eq 0) "passes via the exact Application path (rc=$($r.Rc))" +Remove-Item -LiteralPath $b.Root -Recurse -Force + +foreach ($scheme in 'wonder', 'windev') { + Section "Second gateway of the OTHER naming scheme is caught ($scheme box)" + $b = New-FakeBox $scheme + $otherName = if ($scheme -eq 'wonder') { 'ZB.MOM.WW.MxGateway.Server' } else { 'MxGateway.Server' } + $b.Procs = @($b.Procs) + @([pscustomobject]@{ Id = 5500; ParentId = 77; Name = $otherName; Path = "C:/elsewhere/$otherName.exe"; Role = 'Other' }) + $r = Invoke-Scenario $b @{ DryRun = $true } + Assert ($r.Rc -eq 1) 'refuses (rc 1)' + Assert ($r.Text -match '2 gateway Server processes') 'reports two gateways' + Remove-Item -LiteralPath $b.Root -Recurse -Force +} + +Section 'Another service launching the gateway exe' +$b = New-FakeBox 'wonder' +$b.OtherServices = @('MxAccessGwTest') +$r = Invoke-Scenario $b @{ DryRun = $true } +Assert ($r.Rc -eq 1 -and $r.Text -match 'another service \(MxAccessGwTest\)') 'refuses a second service' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'Service STOPPED with a stray Worker of the other scheme' +$b = New-FakeBox 'wonder' +$b.State = 'SERVICE_STOPPED' +$b.Procs = @([pscustomobject]@{ Id = 5600; ParentId = 1; Name = 'ZB.MOM.WW.MxGateway.Worker'; Path = $null; Role = 'Other' }) +$r = Invoke-Scenario $b @{ DryRun = $true } +Assert ($r.Rc -eq 1) 'refuses (rc 1)' +Assert ($r.Text -match 'STOPPED but gateway process') 'says why' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'Gateway that is not the service host child' +$b = New-FakeBox 'wonder' +$b.Procs[0].ParentId = 1234 +$r = Invoke-Scenario $b @{ DryRun = $true } +Assert ($r.Rc -eq 1 -and $r.Text -match 'is not a child of the MxAccessGw service host') 'refuses a gateway with the wrong parent' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'Staged Worker must carry the installed exe name' +$b = New-FakeBox 'wonder' +Rename-Item -LiteralPath "$($b.Root)/stage/Worker/MxGateway.Worker.exe" -NewName 'ZB.MOM.WW.MxGateway.Worker.exe' +$r = Invoke-Scenario $b @{ DryRun = $true } +Assert ($r.Rc -eq 1 -and $r.Text -match 'does not contain MxGateway\.Worker\.exe') 'refuses a staged Worker with the other exe name' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'Staged Worker must be x86' +$b = New-FakeBox 'wonder' +Set-Content -LiteralPath "$($b.Root)/stage/Worker/MxGateway.Worker.exe" -Value "sha=$NewSha;arch=Amd64" +$r = Invoke-Scenario $b @{ DryRun = $true } +Assert ($r.Rc -eq 1 -and $r.Text -match 'not X86') 'refuses an x64 Worker' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'Self-contained runtime downgrade is refused' +$b = New-FakeBox 'wonder' +Rename-Item -LiteralPath "$($b.Root)/stage/Server/mscordaccore_amd64_amd64_10.0.1426.12345.dll" -NewName 'mscordaccore_amd64_amd64_10.0.826.20000.dll' +$r = Invoke-Scenario $b @{ DryRun = $true } +Assert ($r.Rc -eq 1 -and $r.Text -match 'OLDER than the live') 'refuses 10.0.8 over 10.0.14' +$r = Invoke-Scenario $b @{ DryRun = $true; AllowRuntimeDowngrade = $true } +Assert ($r.Rc -eq 0) '-AllowRuntimeDowngrade overrides it' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'A store inside the swapped dir is refused' +$b = New-FakeBox 'wonder' +New-File "$($b.ServerDir)/stray.db" 'x' +$r = Invoke-Scenario $b @{ DryRun = $true } +Assert ($r.Rc -eq 1 -and $r.Text -match 'would be orphaned by the rename') 'refuses a .db under Server' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +# --------------------------------------------------------------------------------------------------------------- +foreach ($scheme in 'wonder', 'windev') { + Section "Real upgrade on a fake $scheme box (files really swapped)" + $b = New-FakeBox $scheme + $pfxHash = if ($scheme -eq 'wonder') { (Get-FileHash "$($b.ServerDir)/dashboard.pfx").Hash } else { $null } + $cfgHash = (Get-FileHash "$($b.ServerDir)/appsettings.json").Hash + $devHash = (Get-FileHash "$($b.ServerDir)/appsettings.Development.json").Hash + $r = Invoke-Scenario $b @{ Label = 'test' } + Assert ($r.Rc -eq 0) "upgrade exits 0 (rc=$($r.Rc)) $(if ($r.Rc -ne 0) { $r.Text })" + Assert ((Get-Content -Raw $b.Application) -match "sha=$NewSha") 'live Server exe is the new build' + Assert ((Get-Content -Raw $b.WorkerExe) -match "sha=$NewSha") 'live Worker exe is the new build' + Assert ((Get-FileHash "$($b.ServerDir)/appsettings.json").Hash -eq $cfgHash) 'appsettings.json is the live copy, byte-identical' + Assert ((Get-FileHash "$($b.ServerDir)/appsettings.Development.json").Hash -eq $devHash) 'appsettings.Development.json carried' + if ($pfxHash) { Assert ((Test-Path "$($b.ServerDir)/dashboard.pfx") -and (Get-FileHash "$($b.ServerDir)/dashboard.pfx").Hash -eq $pfxHash) 'dashboard.pfx carried, byte-identical' } + Assert (Test-Path "$($b.ServerDir)/notes-left-by-an-operator.txt") 'live-only text file carried' + Assert (-not (Test-Path "$($b.ServerDir)/Old.Removed.Dependency.dll")) 'live-only binary not carried' + Assert ((Get-Content -Raw "$($b.WorkerDir)/$(Split-Path -Leaf $b.WorkerExe).config") -match 'new') 'Worker exe.config is the new build output' + $sbak = @(Get-ChildItem -LiteralPath $b.Root -Directory | Where-Object { $_.Name -like "$(Split-Path -Leaf $b.ServerDir).bak-*-test" }) + Assert ($sbak.Count -eq 1 -and ((Get-Content -Raw (Join-Path $sbak[0].FullName (Split-Path -Leaf $b.Application))) -match "sha=$OldSha")) 'old Server kept as .bak--test' + Assert (@(Get-ChildItem -LiteralPath $b.Root -Directory -Filter 'Worker.bak-*-test').Count -eq 1) 'old Worker kept as .bak' + Assert ((Test-Path "$($b.Root)/data/gateway-auth.db") -and (Get-Content -Raw "$($b.Root)/data/gateway-auth.db") -eq 'auth-db') 'live API-key store untouched' + Assert (@(Get-ChildItem "$($b.Root)/data" -Filter 'gateway-auth.db*.bak-*').Count -eq 2) 'store db + wal backed up' + $bk = @(Get-ChildItem -LiteralPath $b.Root -Directory -Filter '_deploy-backup-*') + Assert ($bk.Count -eq 1 -and $b.Protected -contains $bk[0].FullName) 'backup dir created and ACL-restricted' + $fp = Get-Content -Raw (Join-Path $bk[0].FullName 'env-fingerprint.txt') + Assert ($fp -match 'MxGateway__ApiKeyPepper=[0-9a-f]{12}') 'fingerprint file has the pepper hash prefix' + Assert-NoSecrets $fp 'env-fingerprint.txt' + $notes = Get-Content -Raw (Get-ChildItem -LiteralPath $b.Root -Filter '_DEPLOY-NOTES-*.txt')[0].FullName + Assert-NoSecrets $notes '_DEPLOY-NOTES' + Assert-NoSecrets $r.Text 'the upgrade output' + Assert ($b.Env -contains "MxGateway__ApiKeyPepper=$Pepper") 'env block unchanged' + Assert ($b.Starts -eq 1) "started once ($($b.Starts))" + Assert ($r.Text -match 'env block unchanged after the stability window') 'final env check ran' + if ($scheme -eq 'wonder') { Assert ((Test-Path "$($bk[0].FullName)/kek.key")) 'file master key copied into the backup dir' } + Remove-Item -LiteralPath $b.Root -Recurse -Force +} + +Section 'Env block changed during the start -> abort + automatic rollback, values never printed' +$b = New-FakeBox 'wonder' +$b.OnStart = { param($x) if ($x.Starts -eq 1) { $x.Env = @($x.Env | ForEach-Object { if ($_ -like 'MxGateway__ApiKeyPepper=*') { 'MxGateway__ApiKeyPepper=a-different-pepper' } else { $_ } }) } } +$r = Invoke-Scenario $b @{} +Assert ($r.Rc -eq 1) 'upgrade fails (rc 1)' +Assert ($r.Text -match 'changed MxGateway__ApiKeyPepper \(sha256 [0-9a-f]{12} -> [0-9a-f]{12}\)') 'names the changed entry with hash prefixes' +Assert (-not $r.Text.Contains('a-different-pepper')) 'the new value is not printed' +Assert-NoSecrets $r.Text 'the failure output' +Assert ((Get-Content -Raw $b.Application) -match "sha=$OldSha") 'auto-rollback restored the old Server' +Assert (($b.StartedWith -join ',') -eq "$NewSha,$OldSha") "started new, then old ($($b.StartedWith -join ','))" +Assert ((Get-Content -Raw $b.WorkerExe) -match "sha=$OldSha") 'auto-rollback restored the old Worker' +Assert (Test-Path "$($b.ServerDir)/dashboard.pfx") 'dashboard.pfx present after the rollback' +Assert (@(Get-ChildItem -LiteralPath $b.Root -Directory -Filter 'Server.failed-*').Count -eq 1) 'failed build kept as Server.failed-' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'Env entry removed between the stop and the swap check -> abort, the new build is never started' +$b = New-FakeBox 'wonder' +# Simulate something rewriting the block mid-deploy (e.g. install-service.bat): the backup's Export-ServiceKey seam drops the pepper. +$r = & { + $script:Box = $b + $script:FakeNow = [datetime]'2026-11-12T12:00:00' + . $ScriptPath -ServerSource "$($b.Root)/stage/Server" -WorkerSource "$($b.Root)/stage/Worker" -ExpectedSha $NewSha -MinFreeGB 0.001 -StableSeconds 10 + . $FakesPath + function Export-ServiceKey([string]$Destination) { Set-Content -LiteralPath $Destination -Value 'fake'; $script:Box.Env = @($script:Box.Env | Where-Object { $_ -notlike 'MxGateway__ApiKeyPepper=*' }) } + $rec = @(Invoke-Main 6>&1 3>&1) + [pscustomobject]@{ Rc = @($rec | Where-Object { $_ -is [int] })[-1]; Text = (@($rec | Where-Object { $_ -isnot [int] } | ForEach-Object { "$_" }) -join "`n") } +} +Assert ($r.Rc -eq 1) 'upgrade fails (rc 1)' +Assert ($r.Text -match 'env block changed after the swap: removed MxGateway__ApiKeyPepper') 'names the removed entry' +Assert (@($b.StartedWith | Where-Object { $_ -eq $NewSha }).Count -eq 0) "the new build was never started (started: $($b.StartedWith -join ','))" +Assert ((Get-Content -Raw $b.Application) -match "sha=$OldSha") 'auto-rollback put the old Server back' +Assert ($r.Text -match 'env block changed before the start') 'the rollback reports the env difference again (warning)' +Assert-NoSecrets $r.Text 'the failure output' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'STOP_PENDING: only this install''s processes are killed' +$b = New-FakeBox 'wonder' +$b.StopHangs = $true +$r = Invoke-Scenario $b @{} +Assert ($r.Rc -eq 0) "upgrade completes (rc=$($r.Rc)) $(if ($r.Rc -ne 0) { $r.Text })" +Assert ($r.Text -match 'STOP_PENDING trap') 'STOP_PENDING path taken' +Assert ($b.Killed.Count -ge 1 -and $b.Killed[0] -eq 4200) "the Server is killed first ($($b.Killed -join ','))" +Assert (-not ($b.Killed -contains 900)) 'an unrelated process is not touched' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'Health never ready -> automatic rollback to the old build' +$b = New-FakeBox 'windev' +$b.OnStart = { param($x) if ($x.Starts -eq 1) { $x.Health = 503 } else { $x.Health = 200 } } +$r = Invoke-Scenario $b @{ ReadyTimeoutSeconds = 10 } +Assert ($r.Rc -eq 1) 'upgrade fails (rc 1)' +Assert ($r.Text -match 'ROLLED BACK to the previous build') "rollback verified ($(($r.Text -split "`n" | Select-String 'ROLLBACK FAILED') -join ''))" +Assert ((Get-Content -Raw $b.Application) -match "sha=$OldSha") 'old Server back in place' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'Manual rollback finds a .bak. set left by the #229 adapted scripts' +$b = New-FakeBox 'wonder' +Copy-Item -LiteralPath $b.ServerDir -Destination "$($b.ServerDir).bak.20261112T211945" -Recurse +Copy-Item -LiteralPath $b.WorkerDir -Destination "$($b.WorkerDir).bak.20261112T211945" -Recurse +Set-Content -LiteralPath $b.Application -Value "sha=$NewSha;arch=Amd64" +$r = Invoke-Scenario $b @{ Rollback = $true; DryRun = $true; ServerSource = $null; WorkerSource = $null } +Assert ($r.Rc -eq 0 -and $r.Text -match 'Server=Server\.bak\.20261112T211945 Worker=Worker\.bak\.20261112T211945') "dry-run picks the matching pair ($($r.Rc))" +$r = Invoke-Scenario $b @{ Rollback = $true; BackupStamp = '20261112T211945'; ServerSource = $null; WorkerSource = $null } +Assert ($r.Rc -eq 0) "rollback exits 0 ($(if ($r.Rc -ne 0) { $r.Text }))" +Assert ((Get-Content -Raw $b.Application) -match "sha=$OldSha") 'old Server restored' +Assert (Test-Path "$($b.ServerDir)/dashboard.pfx") 'dashboard.pfx is there after the rollback' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'Rollback refuses a Server backup with no matching Worker backup' +$b = New-FakeBox 'wonder' +Copy-Item -LiteralPath $b.ServerDir -Destination "$($b.ServerDir).bak-20261101-100000" -Recurse +Copy-Item -LiteralPath $b.WorkerDir -Destination "$($b.WorkerDir).bak-20261001-100000" -Recurse +$r = Invoke-Scenario $b @{ Rollback = $true; DryRun = $true; ServerSource = $null; WorkerSource = $null } +Assert ($r.Rc -eq 1 -and $r.Text -match 'refusing a mixed rollback') 'refuses mixed stamps' +Remove-Item -LiteralPath $b.Root -Recurse -Force + +Section 'Env fingerprint helpers' +. $ScriptPath -DryRun +$a = Get-EnvFingerprint @('A=1', "MxGateway__ApiKeyPepper=$Pepper", 'C=3') +Assert (-not ((Format-EnvFingerprint $a).Contains($Pepper))) 'the formatted fingerprint carries no value' +Assert ((Compare-EnvFingerprint $a (Get-EnvFingerprint @('A=1', "MxGateway__ApiKeyPepper=$Pepper", 'C=3'))).Count -eq 0) 'identical blocks compare equal' +$d = Compare-EnvFingerprint $a (Get-EnvFingerprint @('A=1', 'MxGateway__ApiKeyPepper=other', 'C=3')) +Assert ($d.Count -eq 1 -and $d[0] -like 'changed MxGateway__ApiKeyPepper*') "a changed value is caught ($d)" +$d = Compare-EnvFingerprint $a (Get-EnvFingerprint @('A=1', "MxGateway__ApiKeyPepper=$Pepper")) +Assert ($d.Count -eq 1 -and $d[0] -eq 'removed C') "a removed entry is caught ($d)" +$d = Compare-EnvFingerprint $a (Get-EnvFingerprint @('A=1', "MxGateway__ApiKeyPepper=$Pepper", 'C=3', 'D=4')) +Assert ($d.Count -eq 1 -and $d[0] -eq 'added D') "an added entry is caught ($d)" +$d = Compare-EnvFingerprint $a (Get-EnvFingerprint @('C=3', 'A=1', "MxGateway__ApiKeyPepper=$Pepper")) +Assert ($d.Count -eq 1 -and $d[0] -like 'block hash*') "a reorder is caught ($d)" +Assert ((Hide "x $Pepper y") -eq 'x y') 'a sensitive value is redacted from output' + +Write-Host '' +if ($script:failures -gt 0) { Write-Host "FAILED: $($script:failures) assertion(s) failed, $($script:passes) passed" -ForegroundColor Red; exit 1 } +Write-Host "OK: $($script:passes) assertions passed" -ForegroundColor Green +exit 0 diff --git a/scripts/deploy/tests/UpgradeFakes.ps1 b/scripts/deploy/tests/UpgradeFakes.ps1 new file mode 100644 index 00000000..c628e7fc --- /dev/null +++ b/scripts/deploy/tests/UpgradeFakes.ps1 @@ -0,0 +1,75 @@ +# Fakes for Test-UpgradeMxGatewayService.ps1. Dot-sourced AFTER Upgrade-MxGatewayService.ps1 so these replace +# its environment seams. All state lives in $script:Box (built by New-FakeBox in the test script). +# Nothing here touches the real service manager, registry, process table or network. + +function Invoke-Nssm { + param([string[]]$NssmArgs, [switch]$AllowFail) + $b = $script:Box + $b.NssmCalls.Add(($NssmArgs -join ' ')) + $verb = $NssmArgs[0] + switch ($verb) { + 'status' { + if ($b.State -eq 'MISSING') { return [pscustomobject]@{ Code = 3; Text = "Can't open service!" } } + if ($b.State -eq 'SERVICE_STOP_PENDING' -and -not @($b.Procs | Where-Object { $_.Role -eq 'Server' }).Count) { $b.State = 'SERVICE_STOPPED' } + return [pscustomobject]@{ Code = 0; Text = $b.State } + } + 'get' { + switch ($NssmArgs[2]) { + 'Application' { return [pscustomobject]@{ Code = 0; Text = $b.Application } } + 'AppDirectory' { return [pscustomobject]@{ Code = 0; Text = $b.AppDirectory } } + 'AppEnvironmentExtra' { return [pscustomobject]@{ Code = 0; Text = ($b.Env -join "`r`n") } } + } + } + 'stop' { + if ($b.StopHangs) { $b.State = 'SERVICE_STOP_PENDING'; return [pscustomobject]@{ Code = 0; Text = '' } } + $b.State = 'SERVICE_STOPPED' + $b.Procs = @($b.Procs | Where-Object { $_.Role -eq 'Other' -or $b.Survivors -contains $_.Id }) + return [pscustomobject]@{ Code = 0; Text = '' } + } + 'start' { + $b.Starts++ + $b.StartedWith.Add($(if ((Get-Content -LiteralPath $b.Application -Raw) -match 'sha=([0-9a-f]+)') { $Matches[1] } else { '?' })) + $b.State = 'SERVICE_RUNNING' + $sid = 7000 + $b.Starts * 10 + $b.Procs = @($b.Procs) + @( + [pscustomobject]@{ Id = $sid; ParentId = $b.HostPid; Name = [IO.Path]::GetFileNameWithoutExtension($b.Application); Path = $b.Application; Role = 'Server' }, + [pscustomobject]@{ Id = $sid + 1; ParentId = $sid; Name = [IO.Path]::GetFileNameWithoutExtension($b.WorkerExe); Path = $b.WorkerExe; Role = 'Worker' }) + if ($b.OnStart) { & $b.OnStart $b } + return [pscustomobject]@{ Code = 0; Text = '' } + } + } + throw "fake nssm: unexpected call $($NssmArgs -join ' ')" +} + +function Read-EnvBlockLines { @($script:Box.Env) } + +# Return copies so the script's Add-Member (Role) cannot leak back into the fake state. +function Get-ProcessTable { @($script:Box.Procs | ForEach-Object { [pscustomobject]@{ Id = $_.Id; ParentId = $_.ParentId; Name = $_.Name; Path = $_.Path } }) } +function Get-ServiceHostPid { $script:Box.HostPid } +function Get-OtherGatewayServices([string]$ExeName) { , @($script:Box.OtherServices) } +function Stop-ProcessById([int]$Id) { + $script:Box.Killed.Add($Id) + $script:Box.Procs = @($script:Box.Procs | Where-Object { $_.Id -ne $Id }) +} +function Get-HttpStatus([string]$Url) { $script:Box.HttpCalls.Add($Url); $script:Box.Health } +function Get-ProductSha([string]$Exe) { + if (-not (Test-Path -LiteralPath $Exe)) { return $null } + if ((Get-Content -LiteralPath $Exe -Raw) -match 'sha=([0-9a-f]+)') { return $Matches[1] } + $null +} +function Get-ExeArchitecture([string]$Exe) { + if ((Get-Content -LiteralPath $Exe -Raw) -match 'arch=(\w+)') { return $Matches[1] } + $null +} +function Protect-Directory([string]$Path) { $script:Box.Protected.Add($Path) } +function Export-ServiceKey([string]$Destination) { Set-Content -LiteralPath $Destination -Value 'fake reg export' } +function Get-DirSddl([string]$Path) { $null } +function Copy-ExplicitAcl([string]$From, [string]$To) { } + +# Fake clock: Start-Sleep advances it, so timeouts expire instantly and deterministically. +function Start-Sleep { param([int]$Seconds, [int]$Milliseconds) $script:FakeNow = $script:FakeNow.AddSeconds([math]::Max($Seconds, 1)) } +function Get-Date { + param([string]$Format) + if ($Format) { return (Microsoft.PowerShell.Utility\Get-Date -Format $Format) } + $script:FakeNow +}