Upgrade-MxGatewayService.ps1: process guards miss MxGateway.Server.exe/MxGateway.Worker.exe on the wonder boxes, dashboard.pfx not carried over #230

Closed
opened 2026-11-12 21:23:28 -05:00 by dohertj2 · 1 comment
Owner

Found while deploying #229 to DEV wonder-app-vd03, 2026-11-12. I read scripts/deploy/Upgrade-MxGatewayService.ps1 against the box, and it is not safe to run there as written. That deploy used adapted on-box scripts instead (docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md).

1. The process guards never match on the wonder boxes.

  • $script:ServerProc / $script:WorkerProc are ZB.MOM.WW.MxGateway.Server / ZB.MOM.WW.MxGateway.Worker. Get-GatewayProcesses filters on exactly those names.
  • vd03 runs E:\ApiInstall\MxGateway\Server\MxGateway.Server.exe (NSSM Application), with the Worker at Worker\MxGateway.Worker.exe (MxGateway__Worker__ExecutablePath). So the process names are MxGateway.Server / MxGateway.Worker.
  • As a result, the one-gateway preflight guard, Assert-NoGatewayProcess before every start (the OrphanWorkerTerminator trap), and the STOP_PENDING kill all pass vacuously or do nothing.
  • Fix: match both name families. Or better, take the image names from NSSM Application and from MxGateway__Worker__ExecutablePath.

2. The staged Worker check needs the wrong file name. Preflight requires <WorkerSource>\ZB.MOM.WW.MxGateway.Worker.exe, but the live layout (and the env path) uses MxGateway.Worker.exe. Fix: check Split-Path -Leaf $ctx.WorkerExe.

3. Files other than appsettings*.json are lost. Swap-Directory carries only appsettings.json and appsettings.Development.json forward. vd03's Server\ also holds dashboard.pfx. Fix: add dashboard.pfx, or carry forward every top-level file that is in the live directory but not in the staged one, and list them.

4. Note for the runbook (section 4): runtime pack. A self-contained Server published on windev (SDK 10.0.204) ships runtime 10.0.8, but vd03 ran 10.0.14, so it would have downgraded the runtime. The Mac SDK 10.0.403 gives 10.0.14. Preflight could compare mscordaccore_amd64_amd64_<ver>.dll between the live and staged directories and refuse a downgrade.

Each of these needs a -DryRun test against a fake vd03 layout (legacy exe names, dashboard.pfx).

Found while deploying #229 to DEV `wonder-app-vd03`, 2026-11-12. I read `scripts/deploy/Upgrade-MxGatewayService.ps1` against the box, and it is **not safe to run there as written**. That deploy used adapted on-box scripts instead (`docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md`). **1. The process guards never match on the wonder boxes.** - `$script:ServerProc` / `$script:WorkerProc` are `ZB.MOM.WW.MxGateway.Server` / `ZB.MOM.WW.MxGateway.Worker`. `Get-GatewayProcesses` filters on exactly those names. - vd03 runs `E:\ApiInstall\MxGateway\Server\MxGateway.Server.exe` (NSSM `Application`), with the Worker at `Worker\MxGateway.Worker.exe` (`MxGateway__Worker__ExecutablePath`). So the process names are `MxGateway.Server` / `MxGateway.Worker`. - As a result, the one-gateway preflight guard, `Assert-NoGatewayProcess` before every start (the OrphanWorkerTerminator trap), and the STOP_PENDING kill all pass vacuously or do nothing. - Fix: match both name families. Or better, take the image names from NSSM `Application` and from `MxGateway__Worker__ExecutablePath`. **2. The staged Worker check needs the wrong file name.** Preflight requires `<WorkerSource>\ZB.MOM.WW.MxGateway.Worker.exe`, but the live layout (and the env path) uses `MxGateway.Worker.exe`. Fix: check `Split-Path -Leaf $ctx.WorkerExe`. **3. Files other than `appsettings*.json` are lost.** `Swap-Directory` carries only `appsettings.json` and `appsettings.Development.json` forward. vd03's `Server\` also holds `dashboard.pfx`. Fix: add `dashboard.pfx`, or carry forward every top-level file that is in the live directory but not in the staged one, and list them. **4. Note for the runbook (section 4): runtime pack.** A self-contained Server published on windev (SDK 10.0.204) ships runtime **10.0.8**, but vd03 ran **10.0.14**, so it would have downgraded the runtime. The Mac SDK 10.0.403 gives 10.0.14. Preflight could compare `mscordaccore_amd64_amd64_<ver>.dll` between the live and staged directories and refuse a downgrade. Each of these needs a `-DryRun` test against a fake vd03 layout (legacy exe names, `dashboard.pfx`).
dohertj2 added the area:gatewaypriority:p1type:followup labels 2026-11-12 21:23:28 -05:00
Author
Owner

Fixed on main in 4890a3d1 (fix(deploy): Upgrade-MxGatewayService.ps1 safe on the wonder layout, guards fail closed).

What changed (scripts/deploy/Upgrade-MxGatewayService.ps1)

  1. Process guards (item 1). Image names come from the installed exes (the leaf of NSSM Application and of MxGateway__Worker__ExecutablePath), plus both known schemes, ZB.MOM.WW.MxGateway.* and MxGateway.*. Processes are also matched by exact path through Win32_Process, which gives path and parent PID.
    • Fail closed: if the service is RUNNING, there must be exactly one gateway Server, at the Application path, and it must be a child of the NSSM service host PID. Otherwise the script aborts ("the process guards cannot see this gateway, so they would pass vacuously; refusing").
    • The pre-start OrphanWorkerTerminator guard and the after-stop check now use the same matcher across the whole box.
    • The STOP_PENDING kill only touches this install's processes: those whose path is under the Server/Worker dir, or that carry the installed image name when the path can't be read.
  2. Staged checks (item 2). The staged dirs must contain the installed exe names. The staged Worker must be x86.
  3. Carry-over (item 3). appsettings*.json and certificates/keys (*.pfx, *.p12, *.pem, *.key, *.crt, *.cer, so vd03's dashboard.pfx) always keep the live copy. Every other live-only top-level file that isn't a binary is carried too. Carried files are hash-verified and keep any explicit ACLs. Live-only binaries stay in the .bak, live-only subdirectories are listed as a warning, and preflight prints the carry plan.
  4. Runtime pack (item 4). If the staged mscordaccore_amd64_amd64_<ver>.dll is older than the live one, the script refuses. -AllowRuntimeDowngrade overrides this.
  5. Env block, never written. It is read from the service's registry AppEnvironmentExtra and fingerprinted: entry names, a 12-hex SHA-256 prefix per value, and a whole-block hash. The fingerprint is compared after the stop, after the swap, before and after the start, and after the stability window. Any added, removed, changed or reordered entry aborts and triggers the automatic rollback. Values are never printed, and sensitive values are redacted from all output and the notes file.
  6. Other gaps, found by comparing with the #229 adapted scripts.
    • Master key: the old preflight required ZB_SECRETS_MASTER_KEY, which would also have blocked vd03 (Secrets__MasterKey__Source=File). A file key is now accepted if it is outside the swap, and it is backed up. A key that isn't in the env block at all (windev) is a warning.
    • No .db files may sit under Server/Worker.
    • Backup dir: _deploy-backup-<stamp> is restricted to SYSTEM + Administrators before anything is written, and nothing is written if icacls fails. It holds the raw env block, the service .reg export and the fingerprint.
    • Store backups are hash-checked.
    • Failed backup: the unchanged build is restarted.
    • Verify also requires the same gateway PID through the stability window.
    • Rollback (runbook section 8) accepts both .bak-<stamp> and #229's .bak.<yyyyMMddTHHmmss> sets. It refuses a Server backup without a Worker backup of the same stamp, and checks the env block against the saved fingerprint.
    • Automatic rollback no longer fails its own verify on the old SHA.
  7. -DryRun / -WhatIf print every mutating step. -WhatIf was broken on Windows PowerShell 5.1: Get-FileHash returns nothing under $WhatIfPreference, so the plan aborted. The windev run caught this and it is fixed.
  8. Install-MxGatewayService.ps1: its process guard now matches both schemes.

Tests

scripts/deploy/tests/. The repo has no Pester, so these are a plain harness plus a mutation runner.

  • Test-UpgradeMxGatewayService.ps1: 140 assertions, all passing on pwsh/macOS and on Windows PowerShell 5.1 (windev). It runs against a fake box in both layouts: vd03 (MxGateway.*.exe, dashboard.pfx, file master key) and windev (ZB.MOM.WW.*, dated Server dir). nssm, the process table, the registry env block, health and the clock are faked. Directories are really renamed and copied.
    • It covers dry runs in both schemes (nothing changes, live processes seen, carry plan correct), the fail-closed cases (no gateway visible; unknown name and path), path-only matching, a second gateway of the other scheme, a stray Worker while stopped, a wrong parent, a second service, a Worker with the wrong exe name or not x86, a runtime downgrade, and a .db in Server.
    • Real upgrades in both layouts: dashboard.pfx and appsettings*.json byte-identical, binaries not carried, backups present, no secret in any output, notes or fingerprint file.
    • Env changed during start: abort, values not printed, automatic rollback. Env entry removed mid-deploy: the new build is never started.
    • STOP_PENDING (only this install killed), health never ready leading to automatic rollback, manual rollback of a #229-style .bak.20261112T211945 set, and refusal of a mixed-stamp rollback.
  • Invoke-UpgradeMutations.ps1: all 12 mutations killed. These include the original #230 defects (ZB-only names; only appsettings*.json carried), the fail-closed guard removed, the env check made a no-op or names-only, redaction removed, STOP_PENDING killing by any name, downgrade and mixed rollback allowed, and .bak. sets not found.

windev dry run against the real install (ssh windev, powershell -NoProfile -File, Windows PowerShell 5.1)

-DryRun (and -WhatIf) with -ServerSource/-WorkerSource pointing at a scratch copy of the live 447070a5 build, -ExpectedSha 447070a5: EXIT 0.

[info]   gateway processes on the box (images ZB.MOM.WW.MxGateway.Server|MxGateway.Server / ZB.MOM.WW.MxGateway.Worker|MxGateway.Worker): ZB.MOM.WW.MxGateway.Server#9148<-6652 [C:\publish\mxaccessgw\Server-20261112-447070a5\ZB.MOM.WW.MxGateway.Server.exe], ZB.MOM.WW.MxGateway.Worker#14700<-9148 [...Worker\ZB.MOM.WW.MxGateway.Worker.exe], ZB.MOM.WW.MxGateway.Worker#22108<-9148 [...]
[info]   gateway Server #9148 is the service's own (child of service host #6652)
[info]   no other service launches ZB.MOM.WW.MxGateway.Server.exe
[info]   NSSM env block fingerprint (names + value sha256 prefixes, no values): 10 entries, block sha256 ccb572d01b46: ...
[info]   bundled .NET runtime: live 10.0.826.23019, staged 10.0.826.23019
[info]   Server: carry from live -> appsettings.Development.json, appsettings.json; live-only binaries dropped -> none
== Preflight OK == ... == DONE ==

Negative control on the same box: a copy with the image names and path match removed found none and aborted with service is RUNNING but no gateway Server process was found ... refusing. (EXIT 1).

The real run also caught a second bug the fakes had hidden: an empty Win32_Service result came back as $null and read as "check unavailable". It is fixed and now covered by a test.

windev afterwards: MxAccessGw SERVICE_RUNNING, same gateway PID 9148, :5130/health/ready 200. No backup dirs, lock file or store copies were created.

The runbook (docs/runbooks/DeployUpgradeNssmService.md) is updated: script behaviour, test status, both naming schemes in traps 1 and 8 and in section 6, the runtime-pack note in section 4, and the confirmed vd03/windev layouts in section 9. The vd03 record now points QA at the fixed script.

Not covered: no real (non-dry) upgrade has run on a box yet. The first one should be QA (#229), starting with -DryRun on each box.

Fixed on `main` in **`4890a3d1`** (`fix(deploy): Upgrade-MxGatewayService.ps1 safe on the wonder layout, guards fail closed`). ### What changed (`scripts/deploy/Upgrade-MxGatewayService.ps1`) 1. **Process guards (item 1).** Image names come from the installed exes (the leaf of NSSM `Application` and of `MxGateway__Worker__ExecutablePath`), plus both known schemes, `ZB.MOM.WW.MxGateway.*` and `MxGateway.*`. Processes are also matched by exact path through `Win32_Process`, which gives path and parent PID. - **Fail closed:** if the service is RUNNING, there must be exactly one gateway Server, at the `Application` path, and it must be a child of the NSSM service host PID. Otherwise the script aborts ("the process guards cannot see this gateway, so they would pass vacuously; refusing"). - The pre-start OrphanWorkerTerminator guard and the after-stop check now use the same matcher across the whole box. - The STOP_PENDING kill only touches this install's processes: those whose path is under the Server/Worker dir, or that carry the installed image name when the path can't be read. 2. **Staged checks (item 2).** The staged dirs must contain the **installed** exe names. The staged Worker must be x86. 3. **Carry-over (item 3).** `appsettings*.json` and certificates/keys (`*.pfx`, `*.p12`, `*.pem`, `*.key`, `*.crt`, `*.cer`, so vd03's `dashboard.pfx`) always keep the live copy. Every other live-only top-level file that isn't a binary is carried too. Carried files are hash-verified and keep any explicit ACLs. Live-only binaries stay in the `.bak`, live-only subdirectories are listed as a warning, and preflight prints the carry plan. 4. **Runtime pack (item 4).** If the staged `mscordaccore_amd64_amd64_<ver>.dll` is older than the live one, the script refuses. `-AllowRuntimeDowngrade` overrides this. 5. **Env block, never written.** It is read from the service's registry `AppEnvironmentExtra` and fingerprinted: entry names, a 12-hex SHA-256 prefix per value, and a whole-block hash. The fingerprint is compared after the stop, after the swap, before and after the start, and after the stability window. Any added, removed, changed or reordered entry aborts and triggers the automatic rollback. Values are never printed, and sensitive values are redacted from all output and the notes file. 6. **Other gaps, found by comparing with the #229 adapted scripts.** - **Master key:** the old preflight required `ZB_SECRETS_MASTER_KEY`, which **would also have blocked vd03** (`Secrets__MasterKey__Source=File`). A file key is now accepted if it is outside the swap, and it is backed up. A key that isn't in the env block at all (windev) is a warning. - **No `.db` files** may sit under Server/Worker. - **Backup dir:** `_deploy-backup-<stamp>` is restricted to SYSTEM + Administrators *before* anything is written, and nothing is written if `icacls` fails. It holds the raw env block, the service `.reg` export and the fingerprint. - **Store backups** are hash-checked. - **Failed backup:** the unchanged build is restarted. - **Verify** also requires the same gateway PID through the stability window. - **Rollback (runbook section 8)** accepts both `.bak-<stamp>` and #229's `.bak.<yyyyMMddTHHmmss>` sets. It refuses a Server backup without a Worker backup of the same stamp, and checks the env block against the saved fingerprint. - **Automatic rollback** no longer fails its own verify on the old SHA. 7. `-DryRun` / `-WhatIf` print every mutating step. **`-WhatIf` was broken on Windows PowerShell 5.1**: `Get-FileHash` returns nothing under `$WhatIfPreference`, so the plan aborted. The windev run caught this and it is fixed. 8. `Install-MxGatewayService.ps1`: its process guard now matches both schemes. ### Tests `scripts/deploy/tests/`. The repo has no Pester, so these are a plain harness plus a mutation runner. - `Test-UpgradeMxGatewayService.ps1`: **140 assertions, all passing on pwsh/macOS and on Windows PowerShell 5.1 (windev).** It runs against a fake box in both layouts: vd03 (`MxGateway.*.exe`, `dashboard.pfx`, file master key) and windev (`ZB.MOM.WW.*`, dated Server dir). `nssm`, the process table, the registry env block, health and the clock are faked. Directories are really renamed and copied. - It covers dry runs in both schemes (nothing changes, live processes seen, carry plan correct), the fail-closed cases (no gateway visible; unknown name and path), path-only matching, a second gateway of the *other* scheme, a stray Worker while stopped, a wrong parent, a second service, a Worker with the wrong exe name or not x86, a runtime downgrade, and a `.db` in Server. - Real upgrades in both layouts: `dashboard.pfx` and `appsettings*.json` byte-identical, binaries not carried, backups present, no secret in any output, notes or fingerprint file. - Env changed during start: abort, values not printed, automatic rollback. Env entry removed mid-deploy: the new build is never started. - STOP_PENDING (only this install killed), health never ready leading to automatic rollback, manual rollback of a #229-style `.bak.20261112T211945` set, and refusal of a mixed-stamp rollback. - `Invoke-UpgradeMutations.ps1`: **all 12 mutations killed.** These include the original #230 defects (ZB-only names; only `appsettings*.json` carried), the fail-closed guard removed, the env check made a no-op or names-only, redaction removed, STOP_PENDING killing by any name, downgrade and mixed rollback allowed, and `.bak.` sets not found. ### windev dry run against the real install (`ssh windev`, `powershell -NoProfile -File`, Windows PowerShell 5.1) `-DryRun` (and `-WhatIf`) with `-ServerSource`/`-WorkerSource` pointing at a scratch copy of the live `447070a5` build, `-ExpectedSha 447070a5`: **EXIT 0**. ``` [info] gateway processes on the box (images ZB.MOM.WW.MxGateway.Server|MxGateway.Server / ZB.MOM.WW.MxGateway.Worker|MxGateway.Worker): ZB.MOM.WW.MxGateway.Server#9148<-6652 [C:\publish\mxaccessgw\Server-20261112-447070a5\ZB.MOM.WW.MxGateway.Server.exe], ZB.MOM.WW.MxGateway.Worker#14700<-9148 [...Worker\ZB.MOM.WW.MxGateway.Worker.exe], ZB.MOM.WW.MxGateway.Worker#22108<-9148 [...] [info] gateway Server #9148 is the service's own (child of service host #6652) [info] no other service launches ZB.MOM.WW.MxGateway.Server.exe [info] NSSM env block fingerprint (names + value sha256 prefixes, no values): 10 entries, block sha256 ccb572d01b46: ... [info] bundled .NET runtime: live 10.0.826.23019, staged 10.0.826.23019 [info] Server: carry from live -> appsettings.Development.json, appsettings.json; live-only binaries dropped -> none == Preflight OK == ... == DONE == ``` **Negative control on the same box:** a copy with the image names and path match removed found `none` and aborted with `service is RUNNING but no gateway Server process was found ... refusing.` (EXIT 1). The real run also caught a second bug the fakes had hidden: an empty `Win32_Service` result came back as `$null` and read as "check unavailable". It is fixed and now covered by a test. **windev afterwards:** `MxAccessGw` SERVICE_RUNNING, same gateway PID 9148, `:5130/health/ready` 200. No backup dirs, lock file or store copies were created. The runbook (`docs/runbooks/DeployUpgradeNssmService.md`) is updated: script behaviour, test status, both naming schemes in traps 1 and 8 and in section 6, the runtime-pack note in section 4, and the confirmed vd03/windev layouts in section 9. The vd03 record now points QA at the fixed script. Not covered: no real (non-dry) upgrade has run on a box yet. The first one should be QA (#229), starting with `-DryRun` on each box.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/mxaccessgw#230