# Deploy / Upgrade Runbook: NSSM-wrapped Gateway Service (Server + x86 Worker) > Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). Since mxaccessgw#220 the procedure > is also scripted (see "Scripts" below); this runbook remains the spec the scripts implement. It is assembled only from what the repo contains > (`docs/GatewayConfiguration.md`, `docs/Authentication.md`, `docs/Diagnostics.md`, > `docs/runbooks/*`, `docs/plans/2026-08-07-*`, `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`, > `src/.../appsettings.json`, `scripts/ci/`) cross-checked against `scadaproj/deployment.md` §5 and §7.1. > > **Verification legend.** Statements are either backed by a cited repo file, or tagged > **[UNVERIFIED]** (comes from deploy notes / other repos / memory and could not be confirmed from > this repo). Treat every **[UNVERIFIED]** item as "confirm on the box before relying on it". > > **Scope of remote execution.** On the wonder boxes (`wonder-app-vd03` DEV, `wonder-app-vq03`/`vq04` > QA) remote command execution is permitted for the on-box steps of a deploy the owner has asked for, > and nothing else (`CLAUDE.md` "Wonder box remote access" note; the umbrella `scadaproj` index > carries the current rule). No ad-hoc diagnostics over remote exec: use SMB file reads, HTTP probes > and direct SQL. Confirm the current rule before starting; this repo's note is dated 2026-09-11 and > the umbrella index has since relaxed it to deploy-only **[UNVERIFIED from this repo]**. ## Scripts (mxaccessgw#220) > **Test status (mxaccessgw#230, 2026-11-12).** The upgrade script has run in `-DryRun` against the real > windev install (Windows PowerShell 5.1, live service, live processes): the guards found the running > gateway and its Workers, and a copy with the image names removed failed closed. It has **not yet run a real > upgrade on any box**: the first real use (QA, #229) should start with `-DryRun` on the box, read the plan, and > have section 8 ready by hand. Stop/start, STOP_PENDING, carry-over, the env-block checks and both rollbacks are > covered by the harness in `scripts/deploy/tests/` against a fake box (temp directories really renamed and copied, > fake `nssm`/process table/registry/clock), in both naming schemes: > `pwsh -NoProfile -File scripts/deploy/tests/Test-UpgradeMxGatewayService.ps1` (assertions) and > `pwsh -NoProfile -File scripts/deploy/tests/Invoke-UpgradeMutations.ps1` (re-introduces each guarded defect > and requires the harness to fail). The harness also passes under Windows PowerShell 5.1 on windev. The windev > run caught two defects the Mac harness could not: an empty `Win32_Service` result read as "check unavailable", > and Windows PowerShell 5.1's `Get-FileHash` returning nothing under `-WhatIf`, which aborted every `-WhatIf` plan. > Both are fixed. `Install-MxGatewayService.ps1` is still untested on a real box. > > **Before #230 the upgrade script was not safe on the wonder boxes**: its process guards matched only > `ZB.MOM.WW.MxGateway.*`, but vd03/vq03/vq04 run `MxGateway.Server.exe` / `MxGateway.Worker.exe`, so the > one-gateway guard, the pre-start OrphanWorkerTerminator guard and the STOP_PENDING kill passed vacuously; > `Server\dashboard.pfx` was dropped; and preflight refused a file-based master key. Do not use a copy of the > script older than the #230 commit. | Script | Where it runs | Does | |---|---|---| | `scripts/deploy/Upgrade-MxGatewayService.ps1` | **On the Windows box**, Windows PowerShell 5.1 | Sections 3, 5, 6, 7, 8: preflight, stop, backup, swap Server + Worker, start, verify, rollback | | `scripts/deploy/Install-MxGatewayService.ps1` | On a **new** box | NSSM service install: recovery/restart, stop grace, log rotation, env block | Both take `-DryRun` (or `-WhatIf`): read-only probes (`nssm get/status`, process list, disk, file existence) still run, every mutating action is printed instead of executed. **Upgrade script** (`-ServerSource`, `-WorkerSource` are the staged outputs from section 4, built from one SHA): - Parameters: `-ServerSource -WorkerSource -ServiceName MxAccessGw -NssmPath -ServerDir -WorkerDir -ExpectedSha -Label -HealthBaseUrl http://localhost:5130 -Rollback -BackupStamp -RestoreStores -NoAutoRollback -TakeNewAppSettings -AllowRuntimeDowngrade -SkipFreeSpaceCheck -MinFreeGB -StopTimeoutSeconds -StableSeconds -ReadyTimeoutSeconds -DryRun`. `ServerDir` defaults to NSSM `AppDirectory`, `WorkerDir` to the parent of `MxGateway__Worker__ExecutablePath` read from the env block. - **Process matching (trap 1).** Gateway processes are recognised by image name **and** by path. The names are those of the installed exes (the leaf of NSSM `Application` and of `MxGateway__Worker__ExecutablePath`) plus both naming schemes in the field: `ZB.MOM.WW.MxGateway.Server`/`.Worker` (windev) and `MxGateway.Server`/`.Worker` (wonder boxes), so a stray gateway of the other scheme is caught too. The process table comes from `Win32_Process` (path + parent PID). - **The one-gateway guard fails closed.** With the service RUNNING it requires exactly one gateway Server, at the `Application` path, whose parent is the NSSM service host PID. A RUNNING service whose gateway it cannot find is an **abort**, not a pass (that is how the pre-#230 guard went blind). It also refuses: more than one gateway Server; any gateway/Worker process while the service is STOPPED; a gateway while the service is in another state; another service launching the same exe. It asserts no Server/Worker process exists right before every `nssm start`. - Preflight also refuses on: missing service, empty env block, a store path (or `Secrets__MasterKey__FilePath`) that is relative or inside the Server/Worker dir, a `*.db`/`-wal`/`-shm` inside the Server/Worker dir, a staged dir that does not contain the **installed** exe names (`Split-Path -Leaf` of `Application` and of the Worker path), a staged Worker that is not x86, staged Server/Worker from different SHAs (or not matching `-ExpectedSha`), too little disk, and a staged **self-contained runtime older than the live one** (compares `mscordaccore_amd64_amd64_.dll`; `-AllowRuntimeDowngrade` overrides; see section 4). A master key that is not in the env block (neither `ZB_SECRETS_MASTER_KEY` nor `Secrets__MasterKey__FilePath`, as on windev) is a warning, since the upgrade does not change where it comes from. - **Env block (trap 4): read, fingerprinted, never written.** The block is read from the service's registry `Parameters\AppEnvironmentExtra` (what NSSM uses; `nssm get` is the fallback). The fingerprint is the entry names plus a 12-hex SHA-256 prefix of each value, and the whole-block hash; values are never printed and sensitive values are redacted from all output and notes. It is compared after the stop, after the swap, before and after the start, and after the stability window; any added, removed or changed entry (or a reorder) aborts and triggers the automatic rollback. On vd03 the pepper lives **only** in this block (#171), so this check is what proves it survived. - Stop waits `-StopTimeoutSeconds`; on STOP_PENDING (trap 8) it kills **this install's** Server, then its Workers (path under the Server/Worker dir, or the installed image name when the path is unreadable; never by a generic name), then the service host as a last resort. It then waits for every gateway/Worker process on the box to be gone. - Backup goes to `\_deploy-backup-[-label]\`, restricted to SYSTEM + Administrators (inheritance off) **before** anything is written; if `icacls` fails nothing is written. It holds `AppEnvironmentExtra.txt` (the raw block: secrets), the service registry key export, `env-fingerprint.txt` (names + hash prefixes only) and a copy of a file-based master key. The stores are only **copied** (`.db`, `-wal`, `-shm` -> `*.bak-`, hash-checked), never moved or deleted. - Swap is **rename-in-place** (`Server` -> `Server.bak-[-label]`, same for `Worker`), so NSSM `Application`/`AppDirectory`/`ExecutablePath` never change. **Carried forward** from the live dir (hash-verified, explicit ACLs copied): `appsettings*.json` (unless `-TakeNewAppSettings`) and certificates/keys (`*.pfx`, `*.p12`, `*.pem`, `*.key`, `*.crt`, `*.cer`, e.g. vd03's `Server\dashboard.pfx`) always as the live copy, plus every other live-only top-level file that is not a binary. Live-only binaries (`.dll`/`.exe`/`.pdb`) are leftovers of the old build and stay in the `.bak`; live-only sub-directories are listed as a warning. `data\` (the wonder store dir) sits beside `Server\`, outside the swap, and is not touched. Preflight prints the carry plan. - Verify: `/health/ready` = 200, `/healthz` = 200, `ProductVersion` SHA of the deployed Server exe against `-ExpectedSha` (still subject to the version-stamp caveat in trap 7), exactly one gateway at the `Application` path that is the service host's child, the same PID and SERVICE_RUNNING through `-StableSeconds`, env fingerprint unchanged. Health with zero sessions does not prove the Worker path: do the section 7.3 API-key session read yourself. - A failed start or verify triggers an automatic rollback to the just-made backups (disable with `-NoAutoRollback`); a failed backup restarts the unchanged build. Manual rollback: `-Rollback [-BackupStamp ] [-RestoreStores]`. It accepts both backup spellings, `Server.bak-` (this script) and `Server.bak.` (the adapted scripts used for #229 on vd03), and refuses a Server backup without a Worker backup of the same stamp. It compares the env block to the saved `env-fingerprint.txt` (section 8 step 4). The failed directories are kept as `*.failed-`. `-RestoreStores` restores the db triplet from the matching `*.bak-` (stale wal/shm are moved aside, nothing is deleted); use it only after a schema bump. **Install script**: `-ServerDir -WorkerDir [-ServiceName -NssmPath -DataDir C:\ProgramData\MxGateway -LogDir -EnvFile -Env -ObjectName -ServiceAccountPassword -EnvironmentName -RestartDelayMs -ThrottleMs -StopGraceMs -RotateBytes -Start -DryRun]`. It refuses if the service exists or any gateway process is running, requires the KEK and a `*Pepper*` env entry, requires store paths absolute and outside the content root, and sets the env block in **one** `nssm set AppEnvironmentExtra` on a service that has none. Defaults it applies: auto start, `AppExit Default Restart`, restart delay 5 s, throttle 10 s, console stop grace 20 s (covers `Worker:ShutdownTimeoutSeconds` = 10), stdout/stderr to `\logs` with rotation, `sc failure ... restart/5000/restart/5000/restart/30000`, reset 1 day. Secret values are masked in output. The service is left stopped unless `-Start` is given. **Driving from the Mac (winrm skill, deployment-only).** Remote execution on the wonder boxes is allowed only for the on-box steps of a deploy the owner asked for. The intended flow, none of which was run for #220: 1. Build both artifacts on windev from one SHA (section 4); `smb push` them to a staging dir on the box (UNC/SMB copies are always allowed), together with the two scripts. 2. `winrm exec` the script with `-DryRun` first, read the printed plan, then run it for real: `powershell -NoProfile -ExecutionPolicy Bypass -File \Upgrade-MxGatewayService.ps1 -ServerSource \Server -WorkerSource \Worker -ExpectedSha -Label