fix(deploy): Upgrade-MxGatewayService.ps1 safe on the wonder layout, guards fail closed (#230)
ci / windows-x86 (push) Successful in 2m41s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Failing after 12m34s
ci / java (push) Failing after 14m50s

The process guards matched only ZB.MOM.WW.MxGateway.*; the wonder boxes run
MxGateway.Server.exe / MxGateway.Worker.exe, so the one-gateway guard, the
pre-start OrphanWorkerTerminator guard and the STOP_PENDING kill passed vacuously.

- Image names come from the installed exes (NSSM Application, Worker env path) plus
  both known schemes; processes are also matched by exact path (Win32_Process).
- One-gateway guard fails closed: a RUNNING service whose gateway is not visible, at
  the Application path, as the service host's child, is an abort.
- STOP_PENDING kill is scoped to this install's processes.
- Staged checks use the installed exe names; staged Worker must be x86; a staged
  self-contained runtime older than the live one is refused (-AllowRuntimeDowngrade).
- Carry-over: appsettings*.json and certificates/keys (dashboard.pfx) always keep the
  live copy; other live-only non-binary top-level files are carried; hash-verified.
- Env block read from the registry and fingerprinted (names + SHA-256 prefixes, never
  values) after stop, swap, start and the stability window; any change aborts.
  Sensitive values are redacted from all output and notes.
- File-based master key (vd03) accepted and backed up; a missing env master key
  (windev) is a warning. Backup dir is ACL-restricted before anything is written.
- Rollback accepts .bak-<stamp> and .bak.<yyyyMMddTHHmmss> sets and refuses a
  mixed Server/Worker stamp; auto-rollback no longer fails on the old SHA.
- -WhatIf now works on Windows PowerShell 5.1 (Get-FileHash returned nothing under
  WhatIfPreference); an empty Win32_Service result no longer reads as "unavailable".
- Install-MxGatewayService.ps1 process guard matches both schemes.
- Dry-run harness scripts/deploy/tests (fake box in both layouts, 140 assertions,
  passes on pwsh/macOS and Windows PowerShell 5.1) plus a mutation runner (12/12 killed).
- Runbook updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 22:23:14 -05:00
parent 453fe01525
commit 4890a3d16e
7 changed files with 1192 additions and 218 deletions
+90 -31
View File
@@ -19,12 +19,25 @@
## Scripts (mxaccessgw#220)
> **UNTESTED ON A REAL BOX until the next deploy.** The scripts were only exercised on a Mac with
> `pwsh` in `-DryRun` mode against a fake directory layout and a fake `nssm`
> (parsing, preflight refusals, the one-gateway guard, the step sequence). Service stop/start,
> STOP_PENDING handling, the health/SHA verification, `sc.exe failure`, the rollback swaps and the
> NSSM UTF-16 output handling have not run against real Windows/NSSM. Run the first real use with
> `-DryRun` on the box, read the plan, and have section 8 ready by hand.
> **Test status (mxaccessgw#230, 2026-11-12).** The upgrade script has run in `-DryRun` against the real
> windev install (Windows PowerShell 5.1, live service, live processes): the guards found the running
> gateway and its Workers, and a copy with the image names removed failed closed. It has **not yet run a real
> upgrade on any box**: the first real use (QA, #229) should start with `-DryRun` on the box, read the plan, and
> have section 8 ready by hand. Stop/start, STOP_PENDING, carry-over, the env-block checks and both rollbacks are
> covered by the harness in `scripts/deploy/tests/` against a fake box (temp directories really renamed and copied,
> fake `nssm`/process table/registry/clock), in both naming schemes:
> `pwsh -NoProfile -File scripts/deploy/tests/Test-UpgradeMxGatewayService.ps1` (assertions) and
> `pwsh -NoProfile -File scripts/deploy/tests/Invoke-UpgradeMutations.ps1` (re-introduces each guarded defect
> and requires the harness to fail). The harness also passes under Windows PowerShell 5.1 on windev. The windev
> run caught two defects the Mac harness could not: an empty `Win32_Service` result read as "check unavailable",
> and Windows PowerShell 5.1's `Get-FileHash` returning nothing under `-WhatIf`, which aborted every `-WhatIf` plan.
> Both are fixed. `Install-MxGatewayService.ps1` is still untested on a real box.
>
> **Before #230 the upgrade script was not safe on the wonder boxes**: its process guards matched only
> `ZB.MOM.WW.MxGateway.*`, but vd03/vq03/vq04 run `MxGateway.Server.exe` / `MxGateway.Worker.exe`, so the
> one-gateway guard, the pre-start OrphanWorkerTerminator guard and the STOP_PENDING kill passed vacuously;
> `Server\dashboard.pfx` was dropped; and preflight refused a file-based master key. Do not use a copy of the
> script older than the #230 commit.
| Script | Where it runs | Does |
|---|---|---|
@@ -38,30 +51,65 @@ existence) still run, every mutating action is printed instead of executed.
- Parameters: `-ServerSource -WorkerSource -ServiceName MxAccessGw -NssmPath -ServerDir -WorkerDir
-ExpectedSha -Label -HealthBaseUrl http://localhost:5130 -Rollback -BackupStamp -RestoreStores
-NoAutoRollback -TakeNewAppSettings -SkipFreeSpaceCheck -MinFreeGB -StopTimeoutSeconds
-NoAutoRollback -TakeNewAppSettings -AllowRuntimeDowngrade -SkipFreeSpaceCheck -MinFreeGB -StopTimeoutSeconds
-StableSeconds -ReadyTimeoutSeconds -DryRun`. `ServerDir` defaults to NSSM `AppDirectory`, `WorkerDir`
to the parent of `MxGateway__Worker__ExecutablePath` read from the env block.
- Preflight refuses on: missing service, empty env block, no `ZB_SECRETS_MASTER_KEY`, a store path
that is relative or inside the Server/Worker dir, staged Server/Worker built from different SHAs (or not
matching `-ExpectedSha`), too little disk, **more than one gateway Server process, a gateway process
while the service is STOPPED, or another service launching the same exe** (trap 1). It also asserts no
Server/Worker process exists immediately before every `nssm start`.
- Stop waits `-StopTimeoutSeconds`; on STOP_PENDING (trap 8) it kills the Server then Worker processes by
image name (safe because preflight proved a single gateway), then the service host as a last resort.
- **Process matching (trap 1).** Gateway processes are recognised by image name **and** by path. The names
are those of the installed exes (the leaf of NSSM `Application` and of `MxGateway__Worker__ExecutablePath`)
plus both naming schemes in the field: `ZB.MOM.WW.MxGateway.Server`/`.Worker` (windev) and
`MxGateway.Server`/`.Worker` (wonder boxes), so a stray gateway of the other scheme is caught too. The
process table comes from `Win32_Process` (path + parent PID).
- **The one-gateway guard fails closed.** With the service RUNNING it requires exactly one gateway Server, at
the `Application` path, whose parent is the NSSM service host PID. A RUNNING service whose gateway it cannot
find is an **abort**, not a pass (that is how the pre-#230 guard went blind). It also refuses: more than one
gateway Server; any gateway/Worker process while the service is STOPPED; a gateway while the service is in
another state; another service launching the same exe. It asserts no Server/Worker process exists right
before every `nssm start`.
- Preflight also refuses on: missing service, empty env block, a store path (or `Secrets__MasterKey__FilePath`)
that is relative or inside the Server/Worker dir, a `*.db`/`-wal`/`-shm` inside the Server/Worker dir, a
staged dir that does not contain the **installed** exe names (`Split-Path -Leaf` of `Application` and of the
Worker path), a staged Worker that is not x86, staged Server/Worker from different SHAs (or not matching
`-ExpectedSha`), too little disk, and a staged **self-contained runtime older than the live one** (compares
`mscordaccore_amd64_amd64_<ver>.dll`; `-AllowRuntimeDowngrade` overrides; see section 4). A master key that
is not in the env block (neither `ZB_SECRETS_MASTER_KEY` nor `Secrets__MasterKey__FilePath`, as on windev)
is a warning, since the upgrade does not change where it comes from.
- **Env block (trap 4): read, fingerprinted, never written.** The block is read from the service's registry
`Parameters\AppEnvironmentExtra` (what NSSM uses; `nssm get` is the fallback). The fingerprint is the entry
names plus a 12-hex SHA-256 prefix of each value, and the whole-block hash; values are never printed and
sensitive values are redacted from all output and notes. It is compared after the stop, after the swap,
before and after the start, and after the stability window; any added, removed or changed entry (or a
reorder) aborts and triggers the automatic rollback. On vd03 the pepper lives **only** in this block (#171),
so this check is what proves it survived.
- Stop waits `-StopTimeoutSeconds`; on STOP_PENDING (trap 8) it kills **this install's** Server, then its
Workers (path under the Server/Worker dir, or the installed image name when the path is unreadable; never by
a generic name), then the service host as a last resort. It then waits for every gateway/Worker process on
the box to be gone.
- Backup goes to `<install root>\_deploy-backup-<stamp>[-label]\`, restricted to SYSTEM + Administrators
(inheritance off) **before** anything is written; if `icacls` fails nothing is written. It holds
`AppEnvironmentExtra.txt` (the raw block: secrets), the service registry key export, `env-fingerprint.txt`
(names + hash prefixes only) and a copy of a file-based master key. The stores are only **copied** (`.db`,
`-wal`, `-shm` -> `*.bak-<stamp>`, hash-checked), never moved or deleted.
- Swap is **rename-in-place** (`Server` -> `Server.bak-<stamp>[-label]`, same for `Worker`), so NSSM
`Application`/`AppDirectory`/`ExecutablePath` never change. Live `appsettings*.json` are copied back
byte-identical. The stores are only **copied** (`.db`, `-wal`, `-shm` -> `*.bak-<stamp>`), never moved
or deleted. The script **never runs `nssm set AppEnvironmentExtra`**; it saves the block (secrets
included, ACL-restricted via `icacls`) to `_env-AppEnvironmentExtra-<stamp>.txt` and compares the entry count
after start. Notes (names only, no values) go to `_DEPLOY-NOTES-<stamp>.txt` beside the install.
- Verify: `/health/ready` = 200, `/healthz`, `ProductVersion` SHA of the deployed Server exe against
`-ExpectedSha` (still subject to the version-stamp caveat in trap 7), service stays RUNNING for
`-StableSeconds`, env count unchanged. Health with zero sessions does not prove the Worker path: do the
`Application`/`AppDirectory`/`ExecutablePath` never change. **Carried forward** from the live dir
(hash-verified, explicit ACLs copied): `appsettings*.json` (unless `-TakeNewAppSettings`) and certificates/keys
(`*.pfx`, `*.p12`, `*.pem`, `*.key`, `*.crt`, `*.cer`, e.g. vd03's `Server\dashboard.pfx`) always as the live
copy, plus every other live-only top-level file that is not a binary. Live-only binaries
(`.dll`/`.exe`/`.pdb`) are leftovers of the old build and stay in the `.bak`; live-only sub-directories are
listed as a warning. `data\` (the wonder store dir) sits beside `Server\`, outside the swap, and is not touched.
Preflight prints the carry plan.
- Verify: `/health/ready` = 200, `/healthz` = 200, `ProductVersion` SHA of the deployed Server exe against
`-ExpectedSha` (still subject to the version-stamp caveat in trap 7), exactly one gateway at the
`Application` path that is the service host's child, the same PID and SERVICE_RUNNING through
`-StableSeconds`, env fingerprint unchanged. Health with zero sessions does not prove the Worker path: do the
section 7.3 API-key session read yourself.
- A failed start or verify triggers an automatic rollback to the just-made backups (disable with
`-NoAutoRollback`). Manual rollback: `-Rollback [-BackupStamp <stamp>] [-RestoreStores]`; the failed
directories are kept as `*.failed-<stamp>`. `-RestoreStores` restores the db triplet from the matching
`*.bak-<stamp>` (stale wal/shm are moved aside, nothing is deleted); use it only after a schema bump.
`-NoAutoRollback`); a failed backup restarts the unchanged build. Manual rollback:
`-Rollback [-BackupStamp <stamp>] [-RestoreStores]`. It accepts both backup spellings, `Server.bak-<stamp>`
(this script) and `Server.bak.<yyyyMMddTHHmmss>` (the adapted scripts used for #229 on vd03), and refuses
a Server backup without a Worker backup of the same stamp. It compares the env block to the saved
`env-fingerprint.txt` (section 8 step 4). The failed directories are kept as `*.failed-<stamp>`.
`-RestoreStores` restores the db triplet from the matching `*.bak-<stamp>` (stale wal/shm are moved aside,
nothing is deleted); use it only after a schema bump.
**Install script**: `-ServerDir -WorkerDir [-ServiceName -NssmPath -DataDir C:\ProgramData\MxGateway
-LogDir -EnvFile -Env -ObjectName -ServiceAccountPassword -EnvironmentName -RestartDelayMs -ThrottleMs
@@ -135,7 +183,7 @@ days. The SQLite DB must also never be opened with a host-side `sqlite3` while t
1. **`OrphanWorkerTerminator` kills live workers if a second gateway starts.**
On startup the Server enumerates processes named after the configured Worker exe
(`ZB.MOM.WW.MxGateway.Worker`) and kills every one whose path equals the configured
(`ZB.MOM.WW.MxGateway.Worker` on windev, `MxGateway.Worker` on the wonder boxes) and kills every one whose path equals the configured
`ExecutablePath` - or whose path it cannot read, which is the normal case for an x86 process seen
from an x64 gateway, so it falls back to **image-name match only**
(`src/ZB.MOM.WW.MxGateway.Server/Workers/OrphanWorkerTerminator.cs`). It skips only its own PID's
@@ -174,7 +222,8 @@ days. The SQLite DB must also never be opened with a host-side `sqlite3` while t
8. **Stuck `STOP_PENDING`.** Stopping the service can hang in `STOP_PENDING` (workers must exit within
`Worker:ShutdownTimeoutSeconds`, default 10). Observed on the QA boxes per deploy notes
**[UNVERIFIED from this repo]**; if it hangs, wait, then kill the Server process by PID. Do not start the new
build until `Get-Process` shows no `ZB.MOM.WW.MxGateway.Server` and no `ZB.MOM.WW.MxGateway.Worker`.
build until no gateway Server and no Worker process is left. Check **both** naming schemes:
`ZB.MOM.WW.MxGateway.Server`/`.Worker` (windev) and `MxGateway.Server`/`.Worker` (wonder boxes).
## 3. Preflight (no changes yet)
@@ -230,6 +279,14 @@ the host **[UNVERIFIED]**. Note: the Worker is net48 and must be built where the
pack and MSBuild are present (windev; `scadaproj` notes say Build Tools 2022 + net48 reference
assemblies verified 2026-09-10 **[UNVERIFIED from this repo]**).
**Self-contained runtime.** A self-contained Server carries its own .NET runtime, so the SDK that builds it
decides the runtime on the box. On 2026-11-12 the windev SDK (10.0.204) produced runtime 10.0.8 while vd03 ran
10.0.14, which would have been a downgrade; the Mac SDK (10.0.403) produced 10.0.14. Build where the SDK carries
the newest runtime. The upgrade script refuses a downgrade (it compares `mscordaccore_amd64_amd64_<ver>.dll`
between the live and staged dirs). The wonder boxes use the `MxGateway.Server.exe` / `MxGateway.Worker.exe`
names: stage a byte copy of the apphost under that name and rename the Worker apphost and its `.config`
(`docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md`).
For the wonder boxes the staged output is copied over SMB (UNC copy is always allowed) into a
staging directory such as `E:\psrem-stage\<app>-<sha>\` per `scadaproj/deployment.md` §7.1
(**[UNVERIFIED]** path convention); the on-box steps below are then the only remote-exec part.
@@ -258,7 +315,7 @@ the other (`deployment.md` §7.1 records the 2026-09-10 round as "rolling, one s
1. **Stop and confirm gone.**
```powershell
nssm stop MxAccessGw # or Stop-Service MxAccessGw -Force
Get-Process ZB.MOM.WW.MxGateway.Server, ZB.MOM.WW.MxGateway.Worker -ErrorAction SilentlyContinue
Get-Process ZB.MOM.WW.MxGateway.Server, ZB.MOM.WW.MxGateway.Worker, MxGateway.Server, MxGateway.Worker -ErrorAction SilentlyContinue
```
Both lists must be empty before continuing (traps 1 and 8). If a Worker is left over, the next
start will kill it as an orphan (by design) - but it signals an unclean stop worth noting.
@@ -327,9 +384,11 @@ window, API keys no longer authenticate, or a Worker session cannot be opened.
## 9. Open gaps (not verifiable from this repo)
- Exact install paths on the wonder boxes (`E:\ApiInstall\MxAccessGw\...`), the staging convention and
the `_DEPLOY-NOTES` file format - taken from `scadaproj/deployment.md`, whose live content could not
be cross-checked here.
- Install layout on the wonder boxes, confirmed on vd03 (2026-11-12, #229/#230): `E:\ApiInstall\MxGateway\Server\MxGateway.Server.exe`
(NSSM `Application`), `Worker\MxGateway.Worker.exe` (`MxGateway__Worker__ExecutablePath`), stores and `kek.key` in
`E:\ApiInstall\MxGateway\data\`, `Server\dashboard.pfx`, master key `Secrets__MasterKey__Source=File`. windev:
`C:\publish\mxaccessgw\Server-<date>-<sha>\ZB.MOM.WW.MxGateway.Server.exe` and `C:\publish\mxaccessgw\Worker\`, stores in
`C:\ProgramData\MxGateway\`. vq03/vq04 are assumed to match vd03; the dry run shows it.
- Whether the Server on each box is framework-dependent or self-contained (a box without .NET 10
needs self-contained; the 2026-10-19 windev plan says framework-dependent).
- The `STOP_PENDING` behaviour and the NSSM service account (`LocalSystem` per `deployment.md`).