From 477fa5599876edbe3ce31e73410e6319c677aabb Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 9 Nov 2026 15:10:26 -0500 Subject: [PATCH] feat(deploy): on-box NSSM upgrade/rollback and new-box install scripts, runbook updated (mxaccessgw#220) Add scripts/deploy/Upgrade-MxGatewayService.ps1 (preflight, stop with STOP_PENDING handling, backup, Server+Worker swap, start, /health/ready + SHA verify, auto/manual rollback, one-gateway guard, never touches the NSSM env block) and Install-MxGatewayService.ps1 (service install, restart/recovery settings, env block). Both support -DryRun/-WhatIf. Dry-run validated on macOS pwsh against a fake layout; untested on a real box. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV --- docs/runbooks/DeployUpgradeNssmService.md | 81 +++- scripts/deploy/Install-MxGatewayService.ps1 | 202 ++++++++ scripts/deploy/Upgrade-MxGatewayService.ps1 | 490 ++++++++++++++++++++ 3 files changed, 767 insertions(+), 6 deletions(-) create mode 100644 scripts/deploy/Install-MxGatewayService.ps1 create mode 100644 scripts/deploy/Upgrade-MxGatewayService.ps1 diff --git a/docs/runbooks/DeployUpgradeNssmService.md b/docs/runbooks/DeployUpgradeNssmService.md index b14f3f81..445d889e 100644 --- a/docs/runbooks/DeployUpgradeNssmService.md +++ b/docs/runbooks/DeployUpgradeNssmService.md @@ -1,7 +1,7 @@ # Deploy / Upgrade Runbook: NSSM-wrapped Gateway Service (Server + x86 Worker) -> Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). There is **no deploy script** in -> this repo; this runbook is the procedure. It is assembled only from what the repo contains +> Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). Since mxaccessgw#220 the procedure +> is also scripted (see "Scripts" below); this runbook remains the spec the scripts implement. It is assembled only from what the repo contains > (`docs/GatewayConfiguration.md`, `docs/Authentication.md`, `docs/Diagnostics.md`, > `docs/runbooks/*`, `docs/plans/2026-08-07-*`, `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`, > `src/.../appsettings.json`, `scripts/ci/`) cross-checked against `scadaproj/deployment.md` §5 and §7.1. @@ -17,6 +17,72 @@ > and direct SQL. Confirm the current rule before starting; this repo's note is dated 2026-09-11 and > the umbrella index has since relaxed it to deploy-only **[UNVERIFIED from this repo]**. +## Scripts (mxaccessgw#220) + +> **UNTESTED ON A REAL BOX until the next deploy.** The scripts were only exercised on a Mac with +> `pwsh` in `-DryRun` mode against a fake directory layout and a fake `nssm` +> (parsing, preflight refusals, the one-gateway guard, the step sequence). Service stop/start, +> STOP_PENDING handling, the health/SHA verification, `sc.exe failure`, the rollback swaps and the +> NSSM UTF-16 output handling have not run against real Windows/NSSM. Run the first real use with +> `-DryRun` on the box, read the plan, and have section 8 ready by hand. + +| Script | Where it runs | Does | +|---|---|---| +| `scripts/deploy/Upgrade-MxGatewayService.ps1` | **On the Windows box**, Windows PowerShell 5.1 | Sections 3, 5, 6, 7, 8: preflight, stop, backup, swap Server + Worker, start, verify, rollback | +| `scripts/deploy/Install-MxGatewayService.ps1` | On a **new** box | NSSM service install: recovery/restart, stop grace, log rotation, env block | + +Both take `-DryRun` (or `-WhatIf`): read-only probes (`nssm get/status`, process list, disk, file +existence) still run, every mutating action is printed instead of executed. + +**Upgrade script** (`-ServerSource`, `-WorkerSource` are the staged outputs from section 4, built from one SHA): + +- Parameters: `-ServerSource -WorkerSource -ServiceName MxAccessGw -NssmPath -ServerDir -WorkerDir + -ExpectedSha -Label -HealthBaseUrl http://localhost:5130 -Rollback -BackupStamp -RestoreStores + -NoAutoRollback -TakeNewAppSettings -SkipFreeSpaceCheck -MinFreeGB -StopTimeoutSeconds + -StableSeconds -ReadyTimeoutSeconds -DryRun`. `ServerDir` defaults to NSSM `AppDirectory`, `WorkerDir` + to the parent of `MxGateway__Worker__ExecutablePath` read from the env block. +- Preflight refuses on: missing service, empty env block, no `ZB_SECRETS_MASTER_KEY`, a store path + that is relative or inside the Server/Worker dir, staged Server/Worker built from different SHAs (or not + matching `-ExpectedSha`), too little disk, **more than one gateway Server process, a gateway process + while the service is STOPPED, or another service launching the same exe** (trap 1). It also asserts no + Server/Worker process exists immediately before every `nssm start`. +- Stop waits `-StopTimeoutSeconds`; on STOP_PENDING (trap 8) it kills the Server then Worker processes by + image name (safe because preflight proved a single gateway), then the service host as a last resort. +- Swap is **rename-in-place** (`Server` -> `Server.bak-[-label]`, same for `Worker`), so NSSM + `Application`/`AppDirectory`/`ExecutablePath` never change. Live `appsettings*.json` are copied back + byte-identical. The stores are only **copied** (`.db`, `-wal`, `-shm` -> `*.bak-`), never moved + or deleted. The script **never runs `nssm set AppEnvironmentExtra`**; it saves the block (secrets + included, ACL-restricted via `icacls`) to `_env-AppEnvironmentExtra-.txt` and compares the entry count + after start. Notes (names only, no values) go to `_DEPLOY-NOTES-.txt` beside the install. +- Verify: `/health/ready` = 200, `/healthz`, `ProductVersion` SHA of the deployed Server exe against + `-ExpectedSha` (still subject to the version-stamp caveat in trap 7), service stays RUNNING for + `-StableSeconds`, env count unchanged. Health with zero sessions does not prove the Worker path: do the + section 7.3 API-key session read yourself. +- A failed start or verify triggers an automatic rollback to the just-made backups (disable with + `-NoAutoRollback`). Manual rollback: `-Rollback [-BackupStamp ] [-RestoreStores]`; the failed + directories are kept as `*.failed-`. `-RestoreStores` restores the db triplet from the matching + `*.bak-` (stale wal/shm are moved aside, nothing is deleted); use it only after a schema bump. + +**Install script**: `-ServerDir -WorkerDir [-ServiceName -NssmPath -DataDir C:\ProgramData\MxGateway +-LogDir -EnvFile -Env -ObjectName -ServiceAccountPassword -EnvironmentName -RestartDelayMs -ThrottleMs +-StopGraceMs -RotateBytes -Start -DryRun]`. It refuses if the service exists or any gateway process is +running, requires the KEK and a `*Pepper*` env entry, requires store paths absolute and outside the content +root, and sets the env block in **one** `nssm set AppEnvironmentExtra` on a service that has none. Defaults +it applies: auto start, `AppExit Default Restart`, restart delay 5 s, throttle 10 s, console stop grace 20 s +(covers `Worker:ShutdownTimeoutSeconds` = 10), stdout/stderr to `\logs` with rotation, +`sc failure ... restart/5000/restart/5000/restart/30000`, reset 1 day. Secret values are masked in output. +The service is left stopped unless `-Start` is given. + +**Driving from the Mac (winrm skill, deployment-only).** Remote execution on the wonder boxes is allowed +only for the on-box steps of a deploy the owner asked for. The intended flow, none of which was run for #220: + +1. Build both artifacts on windev from one SHA (section 4); `smb push` them to a staging dir on the box + (UNC/SMB copies are always allowed), together with the two scripts. +2. `winrm exec` the script with `-DryRun` first, read the printed plan, then run it for real: + `powershell -NoProfile -ExecutionPolicy Bypass -File \Upgrade-MxGatewayService.ps1 -ServerSource \Server -WorkerSource \Worker -ExpectedSha -Label