feat(deploy): on-box NSSM upgrade/rollback and new-box install scripts, runbook updated (mxaccessgw#220)
Add scripts/deploy/Upgrade-MxGatewayService.ps1 (preflight, stop with STOP_PENDING handling, backup, Server+Worker swap, start, /health/ready + SHA verify, auto/manual rollback, one-gateway guard, never touches the NSSM env block) and Install-MxGatewayService.ps1 (service install, restart/recovery settings, env block). Both support -DryRun/-WhatIf. Dry-run validated on macOS pwsh against a fake layout; untested on a real box. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
# Deploy / Upgrade Runbook: NSSM-wrapped Gateway Service (Server + x86 Worker)
|
||||
|
||||
> Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). There is **no deploy script** in
|
||||
> this repo; this runbook is the procedure. It is assembled only from what the repo contains
|
||||
> Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). Since mxaccessgw#220 the procedure
|
||||
> is also scripted (see "Scripts" below); this runbook remains the spec the scripts implement. It is assembled only from what the repo contains
|
||||
> (`docs/GatewayConfiguration.md`, `docs/Authentication.md`, `docs/Diagnostics.md`,
|
||||
> `docs/runbooks/*`, `docs/plans/2026-08-07-*`, `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`,
|
||||
> `src/.../appsettings.json`, `scripts/ci/`) cross-checked against `scadaproj/deployment.md` §5 and §7.1.
|
||||
@@ -17,6 +17,72 @@
|
||||
> and direct SQL. Confirm the current rule before starting; this repo's note is dated 2026-09-11 and
|
||||
> the umbrella index has since relaxed it to deploy-only **[UNVERIFIED from this repo]**.
|
||||
|
||||
## Scripts (mxaccessgw#220)
|
||||
|
||||
> **UNTESTED ON A REAL BOX until the next deploy.** The scripts were only exercised on a Mac with
|
||||
> `pwsh` in `-DryRun` mode against a fake directory layout and a fake `nssm`
|
||||
> (parsing, preflight refusals, the one-gateway guard, the step sequence). Service stop/start,
|
||||
> STOP_PENDING handling, the health/SHA verification, `sc.exe failure`, the rollback swaps and the
|
||||
> NSSM UTF-16 output handling have not run against real Windows/NSSM. Run the first real use with
|
||||
> `-DryRun` on the box, read the plan, and have section 8 ready by hand.
|
||||
|
||||
| Script | Where it runs | Does |
|
||||
|---|---|---|
|
||||
| `scripts/deploy/Upgrade-MxGatewayService.ps1` | **On the Windows box**, Windows PowerShell 5.1 | Sections 3, 5, 6, 7, 8: preflight, stop, backup, swap Server + Worker, start, verify, rollback |
|
||||
| `scripts/deploy/Install-MxGatewayService.ps1` | On a **new** box | NSSM service install: recovery/restart, stop grace, log rotation, env block |
|
||||
|
||||
Both take `-DryRun` (or `-WhatIf`): read-only probes (`nssm get/status`, process list, disk, file
|
||||
existence) still run, every mutating action is printed instead of executed.
|
||||
|
||||
**Upgrade script** (`-ServerSource`, `-WorkerSource` are the staged outputs from section 4, built from one SHA):
|
||||
|
||||
- Parameters: `-ServerSource -WorkerSource -ServiceName MxAccessGw -NssmPath -ServerDir -WorkerDir
|
||||
-ExpectedSha -Label -HealthBaseUrl http://localhost:5130 -Rollback -BackupStamp -RestoreStores
|
||||
-NoAutoRollback -TakeNewAppSettings -SkipFreeSpaceCheck -MinFreeGB -StopTimeoutSeconds
|
||||
-StableSeconds -ReadyTimeoutSeconds -DryRun`. `ServerDir` defaults to NSSM `AppDirectory`, `WorkerDir`
|
||||
to the parent of `MxGateway__Worker__ExecutablePath` read from the env block.
|
||||
- Preflight refuses on: missing service, empty env block, no `ZB_SECRETS_MASTER_KEY`, a store path
|
||||
that is relative or inside the Server/Worker dir, staged Server/Worker built from different SHAs (or not
|
||||
matching `-ExpectedSha`), too little disk, **more than one gateway Server process, a gateway process
|
||||
while the service is STOPPED, or another service launching the same exe** (trap 1). It also asserts no
|
||||
Server/Worker process exists immediately before every `nssm start`.
|
||||
- Stop waits `-StopTimeoutSeconds`; on STOP_PENDING (trap 8) it kills the Server then Worker processes by
|
||||
image name (safe because preflight proved a single gateway), then the service host as a last resort.
|
||||
- Swap is **rename-in-place** (`Server` -> `Server.bak-<stamp>[-label]`, same for `Worker`), so NSSM
|
||||
`Application`/`AppDirectory`/`ExecutablePath` never change. Live `appsettings*.json` are copied back
|
||||
byte-identical. The stores are only **copied** (`.db`, `-wal`, `-shm` -> `*.bak-<stamp>`), never moved
|
||||
or deleted. The script **never runs `nssm set AppEnvironmentExtra`**; it saves the block (secrets
|
||||
included, ACL-restricted via `icacls`) to `_env-AppEnvironmentExtra-<stamp>.txt` and compares the entry count
|
||||
after start. Notes (names only, no values) go to `_DEPLOY-NOTES-<stamp>.txt` beside the install.
|
||||
- Verify: `/health/ready` = 200, `/healthz`, `ProductVersion` SHA of the deployed Server exe against
|
||||
`-ExpectedSha` (still subject to the version-stamp caveat in trap 7), service stays RUNNING for
|
||||
`-StableSeconds`, env count unchanged. Health with zero sessions does not prove the Worker path: do the
|
||||
section 7.3 API-key session read yourself.
|
||||
- A failed start or verify triggers an automatic rollback to the just-made backups (disable with
|
||||
`-NoAutoRollback`). Manual rollback: `-Rollback [-BackupStamp <stamp>] [-RestoreStores]`; the failed
|
||||
directories are kept as `*.failed-<stamp>`. `-RestoreStores` restores the db triplet from the matching
|
||||
`*.bak-<stamp>` (stale wal/shm are moved aside, nothing is deleted); use it only after a schema bump.
|
||||
|
||||
**Install script**: `-ServerDir -WorkerDir [-ServiceName -NssmPath -DataDir C:\ProgramData\MxGateway
|
||||
-LogDir -EnvFile -Env -ObjectName -ServiceAccountPassword -EnvironmentName -RestartDelayMs -ThrottleMs
|
||||
-StopGraceMs -RotateBytes -Start -DryRun]`. It refuses if the service exists or any gateway process is
|
||||
running, requires the KEK and a `*Pepper*` env entry, requires store paths absolute and outside the content
|
||||
root, and sets the env block in **one** `nssm set AppEnvironmentExtra` on a service that has none. Defaults
|
||||
it applies: auto start, `AppExit Default Restart`, restart delay 5 s, throttle 10 s, console stop grace 20 s
|
||||
(covers `Worker:ShutdownTimeoutSeconds` = 10), stdout/stderr to `<DataDir>\logs` with rotation,
|
||||
`sc failure ... restart/5000/restart/5000/restart/30000`, reset 1 day. Secret values are masked in output.
|
||||
The service is left stopped unless `-Start` is given.
|
||||
|
||||
**Driving from the Mac (winrm skill, deployment-only).** Remote execution on the wonder boxes is allowed
|
||||
only for the on-box steps of a deploy the owner asked for. The intended flow, none of which was run for #220:
|
||||
|
||||
1. Build both artifacts on windev from one SHA (section 4); `smb push` them to a staging dir on the box
|
||||
(UNC/SMB copies are always allowed), together with the two scripts.
|
||||
2. `winrm exec` the script with `-DryRun` first, read the printed plan, then run it for real:
|
||||
`powershell -NoProfile -ExecutionPolicy Bypass -File <stage>\Upgrade-MxGatewayService.ps1 -ServerSource <stage>\Server -WorkerSource <stage>\Worker -ExpectedSha <sha> -Label <label>`.
|
||||
One box at a time; the exit code is 0 on success, 1 on any abort or failed run.
|
||||
3. Do the section 7.3 Worker-path check from the Mac over gRPC (no remote exec needed), then write the deploy note.
|
||||
|
||||
## 0. What is being deployed
|
||||
|
||||
Two artifacts, deployed together because they share a wire protocol (`WorkerProtocolVersion`,
|
||||
@@ -267,7 +333,10 @@ window, API keys no longer authenticate, or a Worker session cannot be opened.
|
||||
- Whether the Server on each box is framework-dependent or self-contained (a box without .NET 10
|
||||
needs self-contained; the 2026-10-19 windev plan says framework-dependent).
|
||||
- The `STOP_PENDING` behaviour and the NSSM service account (`LocalSystem` per `deployment.md`).
|
||||
- NSSM install script: **none exists in the repo** - installing a *new* box (service creation,
|
||||
`AppExit`, recovery, log redirection settings) is not covered here; the only reference is
|
||||
`deployment.md`'s statement that VQ04 was cloned from VQ03's NSSM `AppEnvironmentExtra` config.
|
||||
- Automating this runbook as a script was the other half of the original finding; not done here.
|
||||
- New-box install and the upgrade are now scripted (see "Scripts"), but **untested on a real box**; the
|
||||
recovery/stop-grace/log-rotation defaults in `Install-MxGatewayService.ps1` are chosen from the repo's
|
||||
own settings (`Worker:ShutdownTimeoutSeconds`), not copied from an existing box. Compare against the
|
||||
`nssm dump` of a known-good box (vd03/vq03) before relying on them.
|
||||
- The Worker output folder and build-vs-publish question (section 4) is still **[UNVERIFIED]**: the upgrade
|
||||
script copies whatever the staged `-WorkerSource` directory contains over `Worker`, so stage exactly what
|
||||
the existing `Worker` folder holds. Confirm with a file listing of the live `Worker` dir on the next deploy.
|
||||
|
||||
Reference in New Issue
Block a user