Files
mxaccessgw/docs/runbooks/DeployUpgradeNssmService.md
T
Joseph Doherty 04a40f2c1a
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m50s
ci / java (push) Failing after 4m55s
ci / portable (push) Failing after 11m17s
docs(deploy): vd03 API-key pepper moved into the service environment (#171)
Record of the 2026-11-12 config-only change on wonder-app-vd03: MxGateway__ApiKeyPepper
added to the MxAccessGw AppEnvironmentExtra block (same value, sha256 prefix match),
removed from Server\appsettings.json; authenticated round-trip with a pre-change key,
ScadaBridge MxGateway link unchanged. Runbook and configuration doc now say where the
pepper lives on each wonder box.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 19:49:42 -05:00

343 lines
26 KiB
Markdown

# Deploy / Upgrade Runbook: NSSM-wrapped Gateway Service (Server + x86 Worker)
> Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). Since mxaccessgw#220 the procedure
> is also scripted (see "Scripts" below); this runbook remains the spec the scripts implement. It is assembled only from what the repo contains
> (`docs/GatewayConfiguration.md`, `docs/Authentication.md`, `docs/Diagnostics.md`,
> `docs/runbooks/*`, `docs/plans/2026-08-07-*`, `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`,
> `src/.../appsettings.json`, `scripts/ci/`) cross-checked against `scadaproj/deployment.md` §5 and §7.1.
>
> **Verification legend.** Statements are either backed by a cited repo file, or tagged
> **[UNVERIFIED]** (comes from deploy notes / other repos / memory and could not be confirmed from
> this repo). Treat every **[UNVERIFIED]** item as "confirm on the box before relying on it".
>
> **Scope of remote execution.** On the wonder boxes (`wonder-app-vd03` DEV, `wonder-app-vq03`/`vq04`
> QA) remote command execution is permitted for the on-box steps of a deploy the owner has asked for,
> and nothing else (`CLAUDE.md` "Wonder box remote access" note; the umbrella `scadaproj` index
> carries the current rule). No ad-hoc diagnostics over remote exec: use SMB file reads, HTTP probes
> and direct SQL. Confirm the current rule before starting; this repo's note is dated 2026-09-11 and
> the umbrella index has since relaxed it to deploy-only **[UNVERIFIED from this repo]**.
## Scripts (mxaccessgw#220)
> **UNTESTED ON A REAL BOX until the next deploy.** The scripts were only exercised on a Mac with
> `pwsh` in `-DryRun` mode against a fake directory layout and a fake `nssm`
> (parsing, preflight refusals, the one-gateway guard, the step sequence). Service stop/start,
> STOP_PENDING handling, the health/SHA verification, `sc.exe failure`, the rollback swaps and the
> NSSM UTF-16 output handling have not run against real Windows/NSSM. Run the first real use with
> `-DryRun` on the box, read the plan, and have section 8 ready by hand.
| Script | Where it runs | Does |
|---|---|---|
| `scripts/deploy/Upgrade-MxGatewayService.ps1` | **On the Windows box**, Windows PowerShell 5.1 | Sections 3, 5, 6, 7, 8: preflight, stop, backup, swap Server + Worker, start, verify, rollback |
| `scripts/deploy/Install-MxGatewayService.ps1` | On a **new** box | NSSM service install: recovery/restart, stop grace, log rotation, env block |
Both take `-DryRun` (or `-WhatIf`): read-only probes (`nssm get/status`, process list, disk, file
existence) still run, every mutating action is printed instead of executed.
**Upgrade script** (`-ServerSource`, `-WorkerSource` are the staged outputs from section 4, built from one SHA):
- Parameters: `-ServerSource -WorkerSource -ServiceName MxAccessGw -NssmPath -ServerDir -WorkerDir
-ExpectedSha -Label -HealthBaseUrl http://localhost:5130 -Rollback -BackupStamp -RestoreStores
-NoAutoRollback -TakeNewAppSettings -SkipFreeSpaceCheck -MinFreeGB -StopTimeoutSeconds
-StableSeconds -ReadyTimeoutSeconds -DryRun`. `ServerDir` defaults to NSSM `AppDirectory`, `WorkerDir`
to the parent of `MxGateway__Worker__ExecutablePath` read from the env block.
- Preflight refuses on: missing service, empty env block, no `ZB_SECRETS_MASTER_KEY`, a store path
that is relative or inside the Server/Worker dir, staged Server/Worker built from different SHAs (or not
matching `-ExpectedSha`), too little disk, **more than one gateway Server process, a gateway process
while the service is STOPPED, or another service launching the same exe** (trap 1). It also asserts no
Server/Worker process exists immediately before every `nssm start`.
- Stop waits `-StopTimeoutSeconds`; on STOP_PENDING (trap 8) it kills the Server then Worker processes by
image name (safe because preflight proved a single gateway), then the service host as a last resort.
- Swap is **rename-in-place** (`Server` -> `Server.bak-<stamp>[-label]`, same for `Worker`), so NSSM
`Application`/`AppDirectory`/`ExecutablePath` never change. Live `appsettings*.json` are copied back
byte-identical. The stores are only **copied** (`.db`, `-wal`, `-shm` -> `*.bak-<stamp>`), never moved
or deleted. The script **never runs `nssm set AppEnvironmentExtra`**; it saves the block (secrets
included, ACL-restricted via `icacls`) to `_env-AppEnvironmentExtra-<stamp>.txt` and compares the entry count
after start. Notes (names only, no values) go to `_DEPLOY-NOTES-<stamp>.txt` beside the install.
- Verify: `/health/ready` = 200, `/healthz`, `ProductVersion` SHA of the deployed Server exe against
`-ExpectedSha` (still subject to the version-stamp caveat in trap 7), service stays RUNNING for
`-StableSeconds`, env count unchanged. Health with zero sessions does not prove the Worker path: do the
section 7.3 API-key session read yourself.
- A failed start or verify triggers an automatic rollback to the just-made backups (disable with
`-NoAutoRollback`). Manual rollback: `-Rollback [-BackupStamp <stamp>] [-RestoreStores]`; the failed
directories are kept as `*.failed-<stamp>`. `-RestoreStores` restores the db triplet from the matching
`*.bak-<stamp>` (stale wal/shm are moved aside, nothing is deleted); use it only after a schema bump.
**Install script**: `-ServerDir -WorkerDir [-ServiceName -NssmPath -DataDir C:\ProgramData\MxGateway
-LogDir -EnvFile -Env -ObjectName -ServiceAccountPassword -EnvironmentName -RestartDelayMs -ThrottleMs
-StopGraceMs -RotateBytes -Start -DryRun]`. It refuses if the service exists or any gateway process is
running, requires the KEK and a `*Pepper*` env entry, requires store paths absolute and outside the content
root, and sets the env block in **one** `nssm set AppEnvironmentExtra` on a service that has none. Defaults
it applies: auto start, `AppExit Default Restart`, restart delay 5 s, throttle 10 s, console stop grace 20 s
(covers `Worker:ShutdownTimeoutSeconds` = 10), stdout/stderr to `<DataDir>\logs` with rotation,
`sc failure ... restart/5000/restart/5000/restart/30000`, reset 1 day. Secret values are masked in output.
The service is left stopped unless `-Start` is given.
**Driving from the Mac (winrm skill, deployment-only).** Remote execution on the wonder boxes is allowed
only for the on-box steps of a deploy the owner asked for. The intended flow, none of which was run for #220:
1. Build both artifacts on windev from one SHA (section 4); `smb push` them to a staging dir on the box
(UNC/SMB copies are always allowed), together with the two scripts.
2. `winrm exec` the script with `-DryRun` first, read the printed plan, then run it for real:
`powershell -NoProfile -ExecutionPolicy Bypass -File <stage>\Upgrade-MxGatewayService.ps1 -ServerSource <stage>\Server -WorkerSource <stage>\Worker -ExpectedSha <sha> -Label <label>`.
One box at a time; the exit code is 0 on success, 1 on any abort or failed run.
3. Do the section 7.3 Worker-path check from the Mac over gRPC (no remote exec needed), then write the deploy note.
## 0. What is being deployed
Two artifacts, deployed together because they share a wire protocol (`WorkerProtocolVersion`,
`appsettings.json` `MxGateway:Protocol`):
| Artifact | Build | Runs as |
|---|---|---|
| **Server** (x64 .NET 10, gRPC `:5120` h2c + Blazor dashboard `:5130`) | `dotnet publish src/ZB.MOM.WW.MxGateway.Server -c Release` (framework-dependent in the windev flow; self-contained is the alternative if the box lacks a .NET 10 runtime **[UNVERIFIED]**) | The NSSM service `MxAccessGw` |
| **Worker** (x86 .NET Framework 4.8, one process per gateway session, owns the MXAccess COM STA) | `dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86` (`CLAUDE.md`) - **Windows only; builds on windev**, not on macOS/Linux (`scripts/ci/README.md`) | Child of the Server, launched from `MxGateway:Worker:ExecutablePath`. **Not a service of its own.** |
The Server finds the Worker through `MxGateway:Worker:ExecutablePath`
(`docs/GatewayConfiguration.md`, default is a repo-relative `bin\x86\Release` path, so deployed hosts
override it via the NSSM env var `MxGateway__Worker__ExecutablePath`).
Service name `MxAccessGw` and the windev layout (`C:\publish\mxaccessgw\Server-YYYYMMDD`,
`C:\publish\mxaccessgw\Worker`, build clone `C:\build\mxaccessgw-ci`) are from
`docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md` Task "Deploy" and
`docs/runbooks/IdentifyingADeployedBuild.md`. The wonder boxes use `E:\ApiInstall\...` per
`scadaproj/deployment.md` §7.1; the exact `MxAccessGw` subfolder names there are **[UNVERIFIED]**.
Configuration is delivered as NSSM `AppEnvironmentExtra` environment variables
(`Kestrel__Endpoints__*`, `MxGateway__Authentication__SqlitePath`, `MxGateway__Worker__ExecutablePath`,
`ASPNETCORE_ENVIRONMENT`/`DOTNET_ENVIRONMENT`, `ZB_SECRETS_MASTER_KEY`, the API-key pepper,
`MxGateway__Ldap__ServiceAccountPassword`), not by editing `appsettings.json`
(`docs/GatewayConfiguration.md`, "Kestrel", "Secrets Master Key").
## 1. State that MUST survive the swap
The upgrade renames/replaces the application directory. Anything stored inside it is lost. The
validator therefore rejects stores inside the content root (see the 2026-08-09 incident below).
| State | Setting | Default location (Windows) | Notes |
|---|---|---|---|
| API-key store (SQLite) | `MxGateway:Authentication:SqlitePath` (NSSM: `MxGateway__Authentication__SqlitePath`) | `C:\ProgramData\MxGateway\gateway-auth.db` | Must be rooted; validator also rejects a path inside the content root. Back up the `-wal`/`-shm` siblings with it. |
| API-key **pepper** | secret `MxGateway:ApiKeyPepper` (`PepperSecretName`); lives outside the DB, in NSSM env / Secrets store | n/a | Without the same pepper every issued `mxgw_...` token fails verification. Never in the DB (`docs/Authentication.md`). Back it up in the org vault. On the wonder boxes it is the `MxGateway__ApiKeyPepper` env entry: QA vq03/vq04 since 2026-08-14, DEV vd03 since 2026-11-12 (moved out of `Server\appsettings.json`, mxaccessgw#171, `docs/deploy/2026-11-12-vd03-pepper-to-service-env.md`). |
| Secrets store (SQLite) | `Secrets:SqlitePath` (NSSM: `Secrets__SqlitePath`) | `C:\ProgramData\MxGateway\mxgateway-secrets.db` | **Must be absolute AND outside the content root**; fail-closed with no opt-out (`docs/GatewayConfiguration.md` L349; `scadaproj` Secrets notes). |
| Secrets master key (KEK) | `ZB_SECRETS_MASTER_KEY` in NSSM `AppEnvironmentExtra` | n/a | Losing it makes the store unrecoverable; no `RewrapAll` yet (`docs/GatewayConfiguration.md`, "Rotation / DR"). Back it up in the vault, not just on the box. |
| Galaxy snapshot cache | `MxGateway:Galaxy:SnapshotCachePath` | `C:\ProgramData\MxGateway\galaxy-snapshot.json` | Cache only; rejected inside content root so it survives deploys. Losing it is harmless. |
| TLS self-signed cert | `Tls:SelfSignedCertPath` | `C:\ProgramData\MxGateway\certs\gateway-selfsigned.pfx` | If lost, clients pinned to the old cert see a changed cert. |
| NSSM env block | `nssm get MxAccessGw AppEnvironmentExtra` | NSSM registry config | File copies do not touch it, but `nssm set AppEnvironmentExtra` **replaces the whole block**. |
| Local `appsettings*.json` overrides in the deploy dir | n/a | deploy dir | Copy the live ones aside and restore them after the swap (the windev plan copies `appsettings.json` / `appsettings.Development.json` from the current dir). |
**Incident that motivates the rule (2026-08-09, per `docs/GatewayConfiguration.md` L127):** the
API-key store lived inside the application directory; the upgrade renamed that directory to
`Server.bak.*`, silently starting an empty store, and no gRPC client could authenticate for two
days. The SQLite DB must also never be opened with a host-side `sqlite3` while the service runs
(copy the file triplet and query the copy; cross-repo lesson, **[UNVERIFIED from this repo]**).
## 2. Traps (read before touching anything)
1. **`OrphanWorkerTerminator` kills live workers if a second gateway starts.**
On startup the Server enumerates processes named after the configured Worker exe
(`ZB.MOM.WW.MxGateway.Worker`) and kills every one whose path equals the configured
`ExecutablePath` - or whose path it cannot read, which is the normal case for an x86 process seen
from an x64 gateway, so it falls back to **image-name match only**
(`src/ZB.MOM.WW.MxGateway.Server/Workers/OrphanWorkerTerminator.cs`). It skips only its own PID's
children logic (`currentProcessId`), not another gateway's workers. Consequence: starting a
second gateway process on the same box (a manual `dotnet ...Server.dll` or `Server.exe` while the
service is up, a second NSSM service, a "test" start of the new build before stopping the old one,
or a rollback started while the new one still runs) **terminates every live session's Worker** and
all MXAccess subscriptions with it. Rule: **exactly one gateway process per box at any time**;
stop the service and confirm the process has exited before starting anything else. Never smoke-test
the new build by launching it beside the running service. (On QA, vq03 and vq04 are independent
boxes with independent gateways; this trap is per-box, not per-pair.)
2. **A gateway restart does not reattach workers** (`OrphanWorkerTerminator` summary). Plan for every
client session to drop and for clients to reopen sessions. Session-resilience reattach is opt-in
design only (`docs/plans/2026-06-15-session-resilience-design.md`); do not assume it is on.
3. **Secrets store path.** `Secrets:SqlitePath` must be absolute and outside the content root or the
host fails closed at startup. A store left at an in-tree path is exactly what the swap removes.
Also: the `secret` CLI and the gateway must use the same `Secrets__SqlitePath`, or the CLI seeds a
different store and the gateway then fails closed as if the secret were never set
(`docs/GatewayConfiguration.md` L381-395).
4. **`nssm set ... AppEnvironmentExtra` replaces the whole block.** `nssm get` first, save the output
(redact `Password=`/key values in anything you paste), then re-include every entry
(`docs/GatewayConfiguration.md` L370-374; `docs/runbooks/SEC-36-ldap-credential-rotation.md`).
The `nssm set MxAccessGw Application`/`AppDirectory` commands used for repointing do **not** touch
the env block.
5. **Worker bitness.** The Worker must be built x86 (`-p:Platform=x86`) and `RequiredArchitecture`
stays `X86`. The Server (x64) cannot load MXAccess COM. Build on windev (`ssh windev` flow,
`scripts/ci/windev-worker-ci.ps1`), not on a Mac. Deploying a Worker from a different commit than
the Server risks a protocol mismatch; build both from the same SHA.
6. **Environment name gates security hard-stops** (`docs/GatewayConfiguration.md` L317-335):
`Dashboard:DisableLogin` and plaintext LDAP are refused only when the environment name is exactly
`Production`. Do not change `ASPNETCORE_ENVIRONMENT`/`DOTNET_ENVIRONMENT` during an upgrade; a
host that boots under a different label silently loses those guards, and a host that was
permissive may refuse to boot if you "fix" the label.
7. **Version stamp.** Do not trust `InformationalVersion` for builds from 2026-07-09 to 2026-08-10
(`docs/runbooks/IdentifyingADeployedBuild.md`). Record the SHA yourself in the deploy notes.
8. **Stuck `STOP_PENDING`.** Stopping the service can hang in `STOP_PENDING` (workers must exit within
`Worker:ShutdownTimeoutSeconds`, default 10). Observed on the QA boxes per deploy notes
**[UNVERIFIED from this repo]**; if it hangs, wait, then kill the Server process by PID. Do not start the new
build until `Get-Process` shows no `ZB.MOM.WW.MxGateway.Server` and no `ZB.MOM.WW.MxGateway.Worker`.
## 3. Preflight (no changes yet)
1. Decide the target SHA on `origin/main`; confirm CI is green for it (`windows-x86` job runs the
Worker build + `Worker.Tests` on windev; `nightly-windev` runs the live tier,
`scripts/ci/README.md`).
2. Read the current state of the service (read-only):
```text
nssm get MxAccessGw Application
nssm get MxAccessGw AppDirectory
nssm get MxAccessGw AppParameters
nssm get MxAccessGw AppEnvironmentExtra # save to the deploy notes; redact secrets
```
Note the entry count and names of the env block; verify the same count after the start.
3. Confirm the effective store paths from the env block: `MxGateway__Authentication__SqlitePath`,
`Secrets__SqlitePath`, `MxGateway__Worker__ExecutablePath`. If either store path is missing from
the env block it is on the code default (`C:\ProgramData\MxGateway\...`), which is fine; if it is
under the application directory, **stop and move it first** (see section 1).
4. Confirm the pepper and `ZB_SECRETS_MASTER_KEY` are present in the env block (names only).
5. Read the dashboard **Settings** page (`/settings` on the dashboard port, `:5130`) and save the
effective config; it now includes the heartbeat check interval, stuck ceiling, event-channel
full-mode timeout and replay byte cap. Compare after the upgrade to catch config drift.
6. Check the box is quiet: dashboard sessions list, or `GET /health/active` (the `mxaccess-sessions`
check reports `total/ready/faulted/starting/closing`). Agree a window with clients (OtOpcUa
`GalaxyDriver`, ScadaBridge MxGateway adapter) - their sessions will drop.
7. Check the new build's required config: diff the repo `src/ZB.MOM.WW.MxGateway.Server/appsettings.json`
and the changelog for new mandatory options, and `docs/GatewayConfiguration.md` for validator
changes (the validator can refuse to boot on a bad value, which is a rollback trigger).
8. Free disk for a full copy of the deploy dir and the stores (see backup).
9. Check auth-store schema compatibility: the Server migrates the API-key DB on start
(`Authentication:RunMigrationsOnStartup: true`) via the `ZB.MOM.WW.Auth` package. If the new build
bumps the Auth package, the schema can migrate forward and **the old binary may not read it back**
(see the 2026-08-07 follow-up plan's `AuthStoreMigrationException` check). Treat the DB backup as
the rollback for a schema bump.
## 4. Build (on windev; no deploy yet)
windev flow from `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`:
```powershell
git -C C:\build\mxaccessgw-ci fetch origin
git -C C:\build\mxaccessgw-ci worktree add C:\build\mxgw-deploy <sha> # detached worktree, host checkout untouched
cd C:\build\mxgw-deploy
dotnet publish src\ZB.MOM.WW.MxGateway.Server -c Release -o C:\publish\mxaccessgw\Server-YYYYMMDD
dotnet build src\ZB.MOM.WW.MxGateway.Worker\ZB.MOM.WW.MxGateway.Worker.csproj -c Release -p:Platform=x86
# stage the Worker output (bin\x86\Release) into a temp dir, e.g. C:\publish\mxaccessgw\Worker-new
```
Remove the worktree afterwards (`git worktree remove`); an absent `C:\build\mxgw-deploy` is the
normal end state (`IdentifyingADeployedBuild.md`). Whether the Worker is built with `dotnet build`
or `dotnet publish`, and its exact output folder, should be matched to the existing `Worker` folder on
the host **[UNVERIFIED]**. Note: the Worker is net48 and must be built where the net48 targeting
pack and MSBuild are present (windev; `scadaproj` notes say Build Tools 2022 + net48 reference
assemblies verified 2026-09-10 **[UNVERIFIED from this repo]**).
For the wonder boxes the staged output is copied over SMB (UNC copy is always allowed) into a
staging directory such as `E:\psrem-stage\<app>-<sha>\` per `scadaproj/deployment.md` §7.1
(**[UNVERIFIED]** path convention); the on-box steps below are then the only remote-exec part.
## 5. Backup
With the service **still running** copy nothing live; do these after stopping (section 6 step 1), or
copy the stores with the service stopped. Back up:
1. The current deploy directory (`Server-YYYYMMDD` or `Server`) to a sibling `Server.bak.<timestamp>`
(the convention `IdentifyingADeployedBuild.md` reads as a deploy history chain), and the current
`Worker` directory to `Worker.bak-<timestamp>[-label]`.
2. `gateway-auth.db` plus `gateway-auth.db-wal` / `-shm` if present, and `mxgateway-secrets.db` (plus
its `-wal`/`-shm`), to `*.bak-<label>` next to the originals. **Never delete or move the live DB.**
3. The saved `AppEnvironmentExtra` text and the `nssm get Application/AppDirectory` values (for rollback
repointing).
4. Off-box: the KEK and pepper in the org vault (not on the box only).
5. Record the old and new SHA, timestamps and backup paths in a `_DEPLOY-NOTES-<timestamp>.txt`
next to the install (the convention seen on wonder per `deployment.md` §7.1, **[UNVERIFIED]**).
## 6. Stop / swap / start
Do this on **one box at a time**; QA vq03 and vq04 are independent gateways, so roll them one after
the other (`deployment.md` §7.1 records the 2026-09-10 round as "rolling, one stop per box").
1. **Stop and confirm gone.**
```powershell
nssm stop MxAccessGw # or Stop-Service MxAccessGw -Force
Get-Process ZB.MOM.WW.MxGateway.Server, ZB.MOM.WW.MxGateway.Worker -ErrorAction SilentlyContinue
```
Both lists must be empty before continuing (traps 1 and 8). If a Worker is left over, the next
start will kill it as an orphan (by design) - but it signals an unclean stop worth noting.
2. **Backup** (section 5) now that nothing holds the stores open.
3. **Swap the Server.** Preferred: deploy to a *new dated directory* and repoint NSSM, which leaves the
old directory intact as the rollback target:
```powershell
Copy-Item <old>\appsettings.json, <old>\appsettings.Development.json <new>\ # preserve live file config
nssm set MxAccessGw Application <new>\ZB.MOM.WW.MxGateway.Server.exe
nssm set MxAccessGw AppDirectory <new>
```
(`2026-10-19-ackmsg-as-operator.md`; `IdentifyingADeployedBuild.md` "Deployment-side naming".)
Alternative used on the wonder boxes: rename `app` to `app.bak.<timestamp>` and copy the new files
into `app` (`deployment.md` §7.1; **[UNVERIFIED]**). Either way, restore any host-specific
`appsettings*.json` byte-identical; do not edit the NSSM env block unless the release notes require it.
4. **Swap the Worker.** Rename `Worker` to `Worker.bak-<timestamp>`, copy the new x86 Worker build in
its place, and make sure `MxGateway__Worker__ExecutablePath` points at it
(the path is matched case-insensitively by the orphan terminator, so keep one canonical spelling).
5. **Verify the env block is untouched**: `nssm get MxAccessGw AppEnvironmentExtra` entry count equals the
preflight count.
6. **Start** (`nssm start MxAccessGw`). Exactly once; do not retry-start concurrently with anything else.
## 7. Post-deploy verification
All of this can be done over HTTP/gRPC from another machine or from the box without further remote
exec beyond the deploy window.
1. Service `RUNNING` and stable for at least 60 s (no restart loop); the Application event log has a
clean host start and **no** new `AuthStoreMigrationException` / `Hosting failed to start` after the
start time (`2026-08-07-followups-windev-ldapfixtures-runners.md` Task 2).
2. Health: `GET http://<host>:5130/healthz` (live), `/health/ready` (`auth-store` check),
`/health/active` (`mxaccess-sessions`). These endpoints are anonymous by design
(`GatewayApplication.cs`, `docs/Diagnostics.md`). Zero sessions is Healthy, so green does not prove
the Worker/COM path works.
3. Prove the **Worker** path, since an idle health probe does not: open a gRPC session with an existing
API key (any of the clients, or the `mxa` CLI) and do a read of a known tag. This also proves the
API-key store and pepper survived: a `mxgw_...` token issued before the upgrade must authenticate.
Negative control: a wrong token must be rejected (`UNAUTHENTICATED`).
4. Dashboard: login works (or the localhost anonymous policy applies), `/settings` matches the
preflight capture, the version/Worker protocol line is right.
5. Identify the deployed build by behaviour, not the version stamp where it is unreliable
(`IdentifyingADeployedBuild.md`): a plain `Write` returning `statuses[0]` proves a Worker at or
past `53f69cd`.
6. Check the Server log for `Terminated orphan worker process` - expected only if the previous stop was
unclean.
7. Check downstream clients reconnected (OtOpcUa `GalaxyDriver`; ScadaBridge MxGateway adapter) and, on
a pair of QA boxes, repeat sections 5-7 for the second box.
8. Write the deploy note: SHA, backup paths, env-entry count, evidence for each check above.
## 8. Rollback
Triggers: service will not stay `RUNNING`, host fails validator at start, health red beyond the grace
window, API keys no longer authenticate, or a Worker session cannot be opened.
1. `nssm stop MxAccessGw`; confirm no Server/Worker process (trap 1: **never start the old build while
the new one is still up**).
2. Repoint: `nssm set MxAccessGw Application <old>\...Server.exe` and `AppDirectory <old>`, or rename
`app.bak.<ts>` back to `app`, and restore `Worker.bak-<ts>` over `Worker`.
3. If the API-key DB was migrated (schema bump, trap in section 3 step 9) or damaged, restore
`gateway-auth.db` from the backup (restore the `-wal`/`-shm` together or none). Otherwise leave the
live store alone: the env-block stores are outside the swap and need no restore.
4. `nssm get AppEnvironmentExtra` entry count equals the preflight count; if you changed it, restore the
saved block (all entries).
5. `nssm start MxAccessGw`; repeat section 7 steps 1-3.
6. Report: what failed, the log excerpt, backup paths used.
## 9. Open gaps (not verifiable from this repo)
- Exact install paths on the wonder boxes (`E:\ApiInstall\MxAccessGw\...`), the staging convention and
the `_DEPLOY-NOTES` file format - taken from `scadaproj/deployment.md`, whose live content could not
be cross-checked here.
- Whether the Server on each box is framework-dependent or self-contained (a box without .NET 10
needs self-contained; the 2026-10-19 windev plan says framework-dependent).
- The `STOP_PENDING` behaviour and the NSSM service account (`LocalSystem` per `deployment.md`).
- New-box install and the upgrade are now scripted (see "Scripts"), but **untested on a real box**; the
recovery/stop-grace/log-rotation defaults in `Install-MxGatewayService.ps1` are chosen from the repo's
own settings (`Worker:ShutdownTimeoutSeconds`), not copied from an existing box. Compare against the
`nssm dump` of a known-good box (vd03/vq03) before relying on them.
- The Worker output folder and build-vs-publish question (section 4) is still **[UNVERIFIED]**: the upgrade
script copies whatever the staged `-WorkerSource` directory contains over `Worker`, so stage exactly what
the existing `Worker` folder holds. Confirm with a file listing of the live `Worker` dir on the next deploy.