fix(deploy): Upgrade-MxGatewayService.ps1 safe on the wonder layout, guards fail closed (#230)
The process guards matched only ZB.MOM.WW.MxGateway.*; the wonder boxes run MxGateway.Server.exe / MxGateway.Worker.exe, so the one-gateway guard, the pre-start OrphanWorkerTerminator guard and the STOP_PENDING kill passed vacuously. - Image names come from the installed exes (NSSM Application, Worker env path) plus both known schemes; processes are also matched by exact path (Win32_Process). - One-gateway guard fails closed: a RUNNING service whose gateway is not visible, at the Application path, as the service host's child, is an abort. - STOP_PENDING kill is scoped to this install's processes. - Staged checks use the installed exe names; staged Worker must be x86; a staged self-contained runtime older than the live one is refused (-AllowRuntimeDowngrade). - Carry-over: appsettings*.json and certificates/keys (dashboard.pfx) always keep the live copy; other live-only non-binary top-level files are carried; hash-verified. - Env block read from the registry and fingerprinted (names + SHA-256 prefixes, never values) after stop, swap, start and the stability window; any change aborts. Sensitive values are redacted from all output and notes. - File-based master key (vd03) accepted and backed up; a missing env master key (windev) is a warning. Backup dir is ACL-restricted before anything is written. - Rollback accepts .bak-<stamp> and .bak.<yyyyMMddTHHmmss> sets and refuses a mixed Server/Worker stamp; auto-rollback no longer fails on the old SHA. - -WhatIf now works on Windows PowerShell 5.1 (Get-FileHash returned nothing under WhatIfPreference); an empty Win32_Service result no longer reads as "unavailable". - Install-MxGatewayService.ps1 process guard matches both schemes. - Dry-run harness scripts/deploy/tests (fake box in both layouts, 140 assertions, passes on pwsh/macOS and Windows PowerShell 5.1) plus a mutation runner (12/12 killed). - Runbook updated. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
@@ -34,7 +34,9 @@ The deploy used `preflight/swap/rollback` scripts adapted from the proven `2b135
|
||||
(`E:\psrem-stage\mxgw-229-76a71bc8\{common,pre,swap,rollback}229.ps1`). They match both exe names,
|
||||
carry `appsettings.json` (hash-checked), `appsettings.Development.json` and `dashboard.pfx` forward, never write NSSM
|
||||
`Application`/`AppDirectory`/`AppEnvironmentExtra`, and keep backups in the protected stage folder. The script gaps are filed
|
||||
as #230.
|
||||
as #230. **Fixed in #230:** the repo script now matches both exe naming schemes and fails closed, carries `dashboard.pfx`,
|
||||
accepts the file-based master key, fingerprints the env block, refuses a runtime downgrade, and can roll back these
|
||||
`.bak.20261112T211945` sets. Use it for QA.
|
||||
|
||||
## Timeline (box-local, EST)
|
||||
|
||||
|
||||
@@ -19,12 +19,25 @@
|
||||
|
||||
## Scripts (mxaccessgw#220)
|
||||
|
||||
> **UNTESTED ON A REAL BOX until the next deploy.** The scripts were only exercised on a Mac with
|
||||
> `pwsh` in `-DryRun` mode against a fake directory layout and a fake `nssm`
|
||||
> (parsing, preflight refusals, the one-gateway guard, the step sequence). Service stop/start,
|
||||
> STOP_PENDING handling, the health/SHA verification, `sc.exe failure`, the rollback swaps and the
|
||||
> NSSM UTF-16 output handling have not run against real Windows/NSSM. Run the first real use with
|
||||
> `-DryRun` on the box, read the plan, and have section 8 ready by hand.
|
||||
> **Test status (mxaccessgw#230, 2026-11-12).** The upgrade script has run in `-DryRun` against the real
|
||||
> windev install (Windows PowerShell 5.1, live service, live processes): the guards found the running
|
||||
> gateway and its Workers, and a copy with the image names removed failed closed. It has **not yet run a real
|
||||
> upgrade on any box**: the first real use (QA, #229) should start with `-DryRun` on the box, read the plan, and
|
||||
> have section 8 ready by hand. Stop/start, STOP_PENDING, carry-over, the env-block checks and both rollbacks are
|
||||
> covered by the harness in `scripts/deploy/tests/` against a fake box (temp directories really renamed and copied,
|
||||
> fake `nssm`/process table/registry/clock), in both naming schemes:
|
||||
> `pwsh -NoProfile -File scripts/deploy/tests/Test-UpgradeMxGatewayService.ps1` (assertions) and
|
||||
> `pwsh -NoProfile -File scripts/deploy/tests/Invoke-UpgradeMutations.ps1` (re-introduces each guarded defect
|
||||
> and requires the harness to fail). The harness also passes under Windows PowerShell 5.1 on windev. The windev
|
||||
> run caught two defects the Mac harness could not: an empty `Win32_Service` result read as "check unavailable",
|
||||
> and Windows PowerShell 5.1's `Get-FileHash` returning nothing under `-WhatIf`, which aborted every `-WhatIf` plan.
|
||||
> Both are fixed. `Install-MxGatewayService.ps1` is still untested on a real box.
|
||||
>
|
||||
> **Before #230 the upgrade script was not safe on the wonder boxes**: its process guards matched only
|
||||
> `ZB.MOM.WW.MxGateway.*`, but vd03/vq03/vq04 run `MxGateway.Server.exe` / `MxGateway.Worker.exe`, so the
|
||||
> one-gateway guard, the pre-start OrphanWorkerTerminator guard and the STOP_PENDING kill passed vacuously;
|
||||
> `Server\dashboard.pfx` was dropped; and preflight refused a file-based master key. Do not use a copy of the
|
||||
> script older than the #230 commit.
|
||||
|
||||
| Script | Where it runs | Does |
|
||||
|---|---|---|
|
||||
@@ -38,30 +51,65 @@ existence) still run, every mutating action is printed instead of executed.
|
||||
|
||||
- Parameters: `-ServerSource -WorkerSource -ServiceName MxAccessGw -NssmPath -ServerDir -WorkerDir
|
||||
-ExpectedSha -Label -HealthBaseUrl http://localhost:5130 -Rollback -BackupStamp -RestoreStores
|
||||
-NoAutoRollback -TakeNewAppSettings -SkipFreeSpaceCheck -MinFreeGB -StopTimeoutSeconds
|
||||
-NoAutoRollback -TakeNewAppSettings -AllowRuntimeDowngrade -SkipFreeSpaceCheck -MinFreeGB -StopTimeoutSeconds
|
||||
-StableSeconds -ReadyTimeoutSeconds -DryRun`. `ServerDir` defaults to NSSM `AppDirectory`, `WorkerDir`
|
||||
to the parent of `MxGateway__Worker__ExecutablePath` read from the env block.
|
||||
- Preflight refuses on: missing service, empty env block, no `ZB_SECRETS_MASTER_KEY`, a store path
|
||||
that is relative or inside the Server/Worker dir, staged Server/Worker built from different SHAs (or not
|
||||
matching `-ExpectedSha`), too little disk, **more than one gateway Server process, a gateway process
|
||||
while the service is STOPPED, or another service launching the same exe** (trap 1). It also asserts no
|
||||
Server/Worker process exists immediately before every `nssm start`.
|
||||
- Stop waits `-StopTimeoutSeconds`; on STOP_PENDING (trap 8) it kills the Server then Worker processes by
|
||||
image name (safe because preflight proved a single gateway), then the service host as a last resort.
|
||||
- **Process matching (trap 1).** Gateway processes are recognised by image name **and** by path. The names
|
||||
are those of the installed exes (the leaf of NSSM `Application` and of `MxGateway__Worker__ExecutablePath`)
|
||||
plus both naming schemes in the field: `ZB.MOM.WW.MxGateway.Server`/`.Worker` (windev) and
|
||||
`MxGateway.Server`/`.Worker` (wonder boxes), so a stray gateway of the other scheme is caught too. The
|
||||
process table comes from `Win32_Process` (path + parent PID).
|
||||
- **The one-gateway guard fails closed.** With the service RUNNING it requires exactly one gateway Server, at
|
||||
the `Application` path, whose parent is the NSSM service host PID. A RUNNING service whose gateway it cannot
|
||||
find is an **abort**, not a pass (that is how the pre-#230 guard went blind). It also refuses: more than one
|
||||
gateway Server; any gateway/Worker process while the service is STOPPED; a gateway while the service is in
|
||||
another state; another service launching the same exe. It asserts no Server/Worker process exists right
|
||||
before every `nssm start`.
|
||||
- Preflight also refuses on: missing service, empty env block, a store path (or `Secrets__MasterKey__FilePath`)
|
||||
that is relative or inside the Server/Worker dir, a `*.db`/`-wal`/`-shm` inside the Server/Worker dir, a
|
||||
staged dir that does not contain the **installed** exe names (`Split-Path -Leaf` of `Application` and of the
|
||||
Worker path), a staged Worker that is not x86, staged Server/Worker from different SHAs (or not matching
|
||||
`-ExpectedSha`), too little disk, and a staged **self-contained runtime older than the live one** (compares
|
||||
`mscordaccore_amd64_amd64_<ver>.dll`; `-AllowRuntimeDowngrade` overrides; see section 4). A master key that
|
||||
is not in the env block (neither `ZB_SECRETS_MASTER_KEY` nor `Secrets__MasterKey__FilePath`, as on windev)
|
||||
is a warning, since the upgrade does not change where it comes from.
|
||||
- **Env block (trap 4): read, fingerprinted, never written.** The block is read from the service's registry
|
||||
`Parameters\AppEnvironmentExtra` (what NSSM uses; `nssm get` is the fallback). The fingerprint is the entry
|
||||
names plus a 12-hex SHA-256 prefix of each value, and the whole-block hash; values are never printed and
|
||||
sensitive values are redacted from all output and notes. It is compared after the stop, after the swap,
|
||||
before and after the start, and after the stability window; any added, removed or changed entry (or a
|
||||
reorder) aborts and triggers the automatic rollback. On vd03 the pepper lives **only** in this block (#171),
|
||||
so this check is what proves it survived.
|
||||
- Stop waits `-StopTimeoutSeconds`; on STOP_PENDING (trap 8) it kills **this install's** Server, then its
|
||||
Workers (path under the Server/Worker dir, or the installed image name when the path is unreadable; never by
|
||||
a generic name), then the service host as a last resort. It then waits for every gateway/Worker process on
|
||||
the box to be gone.
|
||||
- Backup goes to `<install root>\_deploy-backup-<stamp>[-label]\`, restricted to SYSTEM + Administrators
|
||||
(inheritance off) **before** anything is written; if `icacls` fails nothing is written. It holds
|
||||
`AppEnvironmentExtra.txt` (the raw block: secrets), the service registry key export, `env-fingerprint.txt`
|
||||
(names + hash prefixes only) and a copy of a file-based master key. The stores are only **copied** (`.db`,
|
||||
`-wal`, `-shm` -> `*.bak-<stamp>`, hash-checked), never moved or deleted.
|
||||
- Swap is **rename-in-place** (`Server` -> `Server.bak-<stamp>[-label]`, same for `Worker`), so NSSM
|
||||
`Application`/`AppDirectory`/`ExecutablePath` never change. Live `appsettings*.json` are copied back
|
||||
byte-identical. The stores are only **copied** (`.db`, `-wal`, `-shm` -> `*.bak-<stamp>`), never moved
|
||||
or deleted. The script **never runs `nssm set AppEnvironmentExtra`**; it saves the block (secrets
|
||||
included, ACL-restricted via `icacls`) to `_env-AppEnvironmentExtra-<stamp>.txt` and compares the entry count
|
||||
after start. Notes (names only, no values) go to `_DEPLOY-NOTES-<stamp>.txt` beside the install.
|
||||
- Verify: `/health/ready` = 200, `/healthz`, `ProductVersion` SHA of the deployed Server exe against
|
||||
`-ExpectedSha` (still subject to the version-stamp caveat in trap 7), service stays RUNNING for
|
||||
`-StableSeconds`, env count unchanged. Health with zero sessions does not prove the Worker path: do the
|
||||
`Application`/`AppDirectory`/`ExecutablePath` never change. **Carried forward** from the live dir
|
||||
(hash-verified, explicit ACLs copied): `appsettings*.json` (unless `-TakeNewAppSettings`) and certificates/keys
|
||||
(`*.pfx`, `*.p12`, `*.pem`, `*.key`, `*.crt`, `*.cer`, e.g. vd03's `Server\dashboard.pfx`) always as the live
|
||||
copy, plus every other live-only top-level file that is not a binary. Live-only binaries
|
||||
(`.dll`/`.exe`/`.pdb`) are leftovers of the old build and stay in the `.bak`; live-only sub-directories are
|
||||
listed as a warning. `data\` (the wonder store dir) sits beside `Server\`, outside the swap, and is not touched.
|
||||
Preflight prints the carry plan.
|
||||
- Verify: `/health/ready` = 200, `/healthz` = 200, `ProductVersion` SHA of the deployed Server exe against
|
||||
`-ExpectedSha` (still subject to the version-stamp caveat in trap 7), exactly one gateway at the
|
||||
`Application` path that is the service host's child, the same PID and SERVICE_RUNNING through
|
||||
`-StableSeconds`, env fingerprint unchanged. Health with zero sessions does not prove the Worker path: do the
|
||||
section 7.3 API-key session read yourself.
|
||||
- A failed start or verify triggers an automatic rollback to the just-made backups (disable with
|
||||
`-NoAutoRollback`). Manual rollback: `-Rollback [-BackupStamp <stamp>] [-RestoreStores]`; the failed
|
||||
directories are kept as `*.failed-<stamp>`. `-RestoreStores` restores the db triplet from the matching
|
||||
`*.bak-<stamp>` (stale wal/shm are moved aside, nothing is deleted); use it only after a schema bump.
|
||||
`-NoAutoRollback`); a failed backup restarts the unchanged build. Manual rollback:
|
||||
`-Rollback [-BackupStamp <stamp>] [-RestoreStores]`. It accepts both backup spellings, `Server.bak-<stamp>`
|
||||
(this script) and `Server.bak.<yyyyMMddTHHmmss>` (the adapted scripts used for #229 on vd03), and refuses
|
||||
a Server backup without a Worker backup of the same stamp. It compares the env block to the saved
|
||||
`env-fingerprint.txt` (section 8 step 4). The failed directories are kept as `*.failed-<stamp>`.
|
||||
`-RestoreStores` restores the db triplet from the matching `*.bak-<stamp>` (stale wal/shm are moved aside,
|
||||
nothing is deleted); use it only after a schema bump.
|
||||
|
||||
**Install script**: `-ServerDir -WorkerDir [-ServiceName -NssmPath -DataDir C:\ProgramData\MxGateway
|
||||
-LogDir -EnvFile -Env -ObjectName -ServiceAccountPassword -EnvironmentName -RestartDelayMs -ThrottleMs
|
||||
@@ -135,7 +183,7 @@ days. The SQLite DB must also never be opened with a host-side `sqlite3` while t
|
||||
|
||||
1. **`OrphanWorkerTerminator` kills live workers if a second gateway starts.**
|
||||
On startup the Server enumerates processes named after the configured Worker exe
|
||||
(`ZB.MOM.WW.MxGateway.Worker`) and kills every one whose path equals the configured
|
||||
(`ZB.MOM.WW.MxGateway.Worker` on windev, `MxGateway.Worker` on the wonder boxes) and kills every one whose path equals the configured
|
||||
`ExecutablePath` - or whose path it cannot read, which is the normal case for an x86 process seen
|
||||
from an x64 gateway, so it falls back to **image-name match only**
|
||||
(`src/ZB.MOM.WW.MxGateway.Server/Workers/OrphanWorkerTerminator.cs`). It skips only its own PID's
|
||||
@@ -174,7 +222,8 @@ days. The SQLite DB must also never be opened with a host-side `sqlite3` while t
|
||||
8. **Stuck `STOP_PENDING`.** Stopping the service can hang in `STOP_PENDING` (workers must exit within
|
||||
`Worker:ShutdownTimeoutSeconds`, default 10). Observed on the QA boxes per deploy notes
|
||||
**[UNVERIFIED from this repo]**; if it hangs, wait, then kill the Server process by PID. Do not start the new
|
||||
build until `Get-Process` shows no `ZB.MOM.WW.MxGateway.Server` and no `ZB.MOM.WW.MxGateway.Worker`.
|
||||
build until no gateway Server and no Worker process is left. Check **both** naming schemes:
|
||||
`ZB.MOM.WW.MxGateway.Server`/`.Worker` (windev) and `MxGateway.Server`/`.Worker` (wonder boxes).
|
||||
|
||||
## 3. Preflight (no changes yet)
|
||||
|
||||
@@ -230,6 +279,14 @@ the host **[UNVERIFIED]**. Note: the Worker is net48 and must be built where the
|
||||
pack and MSBuild are present (windev; `scadaproj` notes say Build Tools 2022 + net48 reference
|
||||
assemblies verified 2026-09-10 **[UNVERIFIED from this repo]**).
|
||||
|
||||
**Self-contained runtime.** A self-contained Server carries its own .NET runtime, so the SDK that builds it
|
||||
decides the runtime on the box. On 2026-11-12 the windev SDK (10.0.204) produced runtime 10.0.8 while vd03 ran
|
||||
10.0.14, which would have been a downgrade; the Mac SDK (10.0.403) produced 10.0.14. Build where the SDK carries
|
||||
the newest runtime. The upgrade script refuses a downgrade (it compares `mscordaccore_amd64_amd64_<ver>.dll`
|
||||
between the live and staged dirs). The wonder boxes use the `MxGateway.Server.exe` / `MxGateway.Worker.exe`
|
||||
names: stage a byte copy of the apphost under that name and rename the Worker apphost and its `.config`
|
||||
(`docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md`).
|
||||
|
||||
For the wonder boxes the staged output is copied over SMB (UNC copy is always allowed) into a
|
||||
staging directory such as `E:\psrem-stage\<app>-<sha>\` per `scadaproj/deployment.md` §7.1
|
||||
(**[UNVERIFIED]** path convention); the on-box steps below are then the only remote-exec part.
|
||||
@@ -258,7 +315,7 @@ the other (`deployment.md` §7.1 records the 2026-09-10 round as "rolling, one s
|
||||
1. **Stop and confirm gone.**
|
||||
```powershell
|
||||
nssm stop MxAccessGw # or Stop-Service MxAccessGw -Force
|
||||
Get-Process ZB.MOM.WW.MxGateway.Server, ZB.MOM.WW.MxGateway.Worker -ErrorAction SilentlyContinue
|
||||
Get-Process ZB.MOM.WW.MxGateway.Server, ZB.MOM.WW.MxGateway.Worker, MxGateway.Server, MxGateway.Worker -ErrorAction SilentlyContinue
|
||||
```
|
||||
Both lists must be empty before continuing (traps 1 and 8). If a Worker is left over, the next
|
||||
start will kill it as an orphan (by design) - but it signals an unclean stop worth noting.
|
||||
@@ -327,9 +384,11 @@ window, API keys no longer authenticate, or a Worker session cannot be opened.
|
||||
|
||||
## 9. Open gaps (not verifiable from this repo)
|
||||
|
||||
- Exact install paths on the wonder boxes (`E:\ApiInstall\MxAccessGw\...`), the staging convention and
|
||||
the `_DEPLOY-NOTES` file format - taken from `scadaproj/deployment.md`, whose live content could not
|
||||
be cross-checked here.
|
||||
- Install layout on the wonder boxes, confirmed on vd03 (2026-11-12, #229/#230): `E:\ApiInstall\MxGateway\Server\MxGateway.Server.exe`
|
||||
(NSSM `Application`), `Worker\MxGateway.Worker.exe` (`MxGateway__Worker__ExecutablePath`), stores and `kek.key` in
|
||||
`E:\ApiInstall\MxGateway\data\`, `Server\dashboard.pfx`, master key `Secrets__MasterKey__Source=File`. windev:
|
||||
`C:\publish\mxaccessgw\Server-<date>-<sha>\ZB.MOM.WW.MxGateway.Server.exe` and `C:\publish\mxaccessgw\Worker\`, stores in
|
||||
`C:\ProgramData\MxGateway\`. vq03/vq04 are assumed to match vd03; the dry run shows it.
|
||||
- Whether the Server on each box is framework-dependent or self-contained (a box without .NET 10
|
||||
needs self-contained; the 2026-10-19 windev plan says framework-dependent).
|
||||
- The `STOP_PENDING` behaviour and the NSSM service account (`LocalSystem` per `deployment.md`).
|
||||
|
||||
Reference in New Issue
Block a user