# Deploy / Upgrade Runbook: NSSM-wrapped Gateway Service (Server + x86 Worker) > Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). There is **no deploy script** in > this repo; this runbook is the procedure. It is assembled only from what the repo contains > (`docs/GatewayConfiguration.md`, `docs/Authentication.md`, `docs/Diagnostics.md`, > `docs/runbooks/*`, `docs/plans/2026-08-07-*`, `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`, > `src/.../appsettings.json`, `scripts/ci/`) cross-checked against `scadaproj/deployment.md` §5 and §7.1. > > **Verification legend.** Statements are either backed by a cited repo file, or tagged > **[UNVERIFIED]** (comes from deploy notes / other repos / memory and could not be confirmed from > this repo). Treat every **[UNVERIFIED]** item as "confirm on the box before relying on it". > > **Scope of remote execution.** On the wonder boxes (`wonder-app-vd03` DEV, `wonder-app-vq03`/`vq04` > QA) remote command execution is permitted for the on-box steps of a deploy the owner has asked for, > and nothing else (`CLAUDE.md` "Wonder box remote access" note; the umbrella `scadaproj` index > carries the current rule). No ad-hoc diagnostics over remote exec: use SMB file reads, HTTP probes > and direct SQL. Confirm the current rule before starting; this repo's note is dated 2026-09-11 and > the umbrella index has since relaxed it to deploy-only **[UNVERIFIED from this repo]**. ## 0. What is being deployed Two artifacts, deployed together because they share a wire protocol (`WorkerProtocolVersion`, `appsettings.json` `MxGateway:Protocol`): | Artifact | Build | Runs as | |---|---|---| | **Server** (x64 .NET 10, gRPC `:5120` h2c + Blazor dashboard `:5130`) | `dotnet publish src/ZB.MOM.WW.MxGateway.Server -c Release` (framework-dependent in the windev flow; self-contained is the alternative if the box lacks a .NET 10 runtime **[UNVERIFIED]**) | The NSSM service `MxAccessGw` | | **Worker** (x86 .NET Framework 4.8, one process per gateway session, owns the MXAccess COM STA) | `dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86` (`CLAUDE.md`) - **Windows only; builds on windev**, not on macOS/Linux (`scripts/ci/README.md`) | Child of the Server, launched from `MxGateway:Worker:ExecutablePath`. **Not a service of its own.** | The Server finds the Worker through `MxGateway:Worker:ExecutablePath` (`docs/GatewayConfiguration.md`, default is a repo-relative `bin\x86\Release` path, so deployed hosts override it via the NSSM env var `MxGateway__Worker__ExecutablePath`). Service name `MxAccessGw` and the windev layout (`C:\publish\mxaccessgw\Server-YYYYMMDD`, `C:\publish\mxaccessgw\Worker`, build clone `C:\build\mxaccessgw-ci`) are from `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md` Task "Deploy" and `docs/runbooks/IdentifyingADeployedBuild.md`. The wonder boxes use `E:\ApiInstall\...` per `scadaproj/deployment.md` §7.1; the exact `MxAccessGw` subfolder names there are **[UNVERIFIED]**. Configuration is delivered as NSSM `AppEnvironmentExtra` environment variables (`Kestrel__Endpoints__*`, `MxGateway__Authentication__SqlitePath`, `MxGateway__Worker__ExecutablePath`, `ASPNETCORE_ENVIRONMENT`/`DOTNET_ENVIRONMENT`, `ZB_SECRETS_MASTER_KEY`, the API-key pepper, `MxGateway__Ldap__ServiceAccountPassword`), not by editing `appsettings.json` (`docs/GatewayConfiguration.md`, "Kestrel", "Secrets Master Key"). ## 1. State that MUST survive the swap The upgrade renames/replaces the application directory. Anything stored inside it is lost. The validator therefore rejects stores inside the content root (see the 2026-08-09 incident below). | State | Setting | Default location (Windows) | Notes | |---|---|---|---| | API-key store (SQLite) | `MxGateway:Authentication:SqlitePath` (NSSM: `MxGateway__Authentication__SqlitePath`) | `C:\ProgramData\MxGateway\gateway-auth.db` | Must be rooted; validator also rejects a path inside the content root. Back up the `-wal`/`-shm` siblings with it. | | API-key **pepper** | secret `MxGateway:ApiKeyPepper` (`PepperSecretName`); lives outside the DB, in NSSM env / Secrets store | n/a | Without the same pepper every issued `mxgw_...` token fails verification. Never in the DB (`docs/Authentication.md`). Back it up in the org vault. | | Secrets store (SQLite) | `Secrets:SqlitePath` (NSSM: `Secrets__SqlitePath`) | `C:\ProgramData\MxGateway\mxgateway-secrets.db` | **Must be absolute AND outside the content root**; fail-closed with no opt-out (`docs/GatewayConfiguration.md` L349; `scadaproj` Secrets notes). | | Secrets master key (KEK) | `ZB_SECRETS_MASTER_KEY` in NSSM `AppEnvironmentExtra` | n/a | Losing it makes the store unrecoverable; no `RewrapAll` yet (`docs/GatewayConfiguration.md`, "Rotation / DR"). Back it up in the vault, not just on the box. | | Galaxy snapshot cache | `MxGateway:Galaxy:SnapshotCachePath` | `C:\ProgramData\MxGateway\galaxy-snapshot.json` | Cache only; rejected inside content root so it survives deploys. Losing it is harmless. | | TLS self-signed cert | `Tls:SelfSignedCertPath` | `C:\ProgramData\MxGateway\certs\gateway-selfsigned.pfx` | If lost, clients pinned to the old cert see a changed cert. | | NSSM env block | `nssm get MxAccessGw AppEnvironmentExtra` | NSSM registry config | File copies do not touch it, but `nssm set AppEnvironmentExtra` **replaces the whole block**. | | Local `appsettings*.json` overrides in the deploy dir | n/a | deploy dir | Copy the live ones aside and restore them after the swap (the windev plan copies `appsettings.json` / `appsettings.Development.json` from the current dir). | **Incident that motivates the rule (2026-08-09, per `docs/GatewayConfiguration.md` L127):** the API-key store lived inside the application directory; the upgrade renamed that directory to `Server.bak.*`, silently starting an empty store, and no gRPC client could authenticate for two days. The SQLite DB must also never be opened with a host-side `sqlite3` while the service runs (copy the file triplet and query the copy; cross-repo lesson, **[UNVERIFIED from this repo]**). ## 2. Traps (read before touching anything) 1. **`OrphanWorkerTerminator` kills live workers if a second gateway starts.** On startup the Server enumerates processes named after the configured Worker exe (`ZB.MOM.WW.MxGateway.Worker`) and kills every one whose path equals the configured `ExecutablePath` - or whose path it cannot read, which is the normal case for an x86 process seen from an x64 gateway, so it falls back to **image-name match only** (`src/ZB.MOM.WW.MxGateway.Server/Workers/OrphanWorkerTerminator.cs`). It skips only its own PID's children logic (`currentProcessId`), not another gateway's workers. Consequence: starting a second gateway process on the same box (a manual `dotnet ...Server.dll` or `Server.exe` while the service is up, a second NSSM service, a "test" start of the new build before stopping the old one, or a rollback started while the new one still runs) **terminates every live session's Worker** and all MXAccess subscriptions with it. Rule: **exactly one gateway process per box at any time**; stop the service and confirm the process has exited before starting anything else. Never smoke-test the new build by launching it beside the running service. (On QA, vq03 and vq04 are independent boxes with independent gateways; this trap is per-box, not per-pair.) 2. **A gateway restart does not reattach workers** (`OrphanWorkerTerminator` summary). Plan for every client session to drop and for clients to reopen sessions. Session-resilience reattach is opt-in design only (`docs/plans/2026-06-15-session-resilience-design.md`); do not assume it is on. 3. **Secrets store path.** `Secrets:SqlitePath` must be absolute and outside the content root or the host fails closed at startup. A store left at an in-tree path is exactly what the swap removes. Also: the `secret` CLI and the gateway must use the same `Secrets__SqlitePath`, or the CLI seeds a different store and the gateway then fails closed as if the secret were never set (`docs/GatewayConfiguration.md` L381-395). 4. **`nssm set ... AppEnvironmentExtra` replaces the whole block.** `nssm get` first, save the output (redact `Password=`/key values in anything you paste), then re-include every entry (`docs/GatewayConfiguration.md` L370-374; `docs/runbooks/SEC-36-ldap-credential-rotation.md`). The `nssm set MxAccessGw Application`/`AppDirectory` commands used for repointing do **not** touch the env block. 5. **Worker bitness.** The Worker must be built x86 (`-p:Platform=x86`) and `RequiredArchitecture` stays `X86`. The Server (x64) cannot load MXAccess COM. Build on windev (`ssh windev` flow, `scripts/ci/windev-worker-ci.ps1`), not on a Mac. Deploying a Worker from a different commit than the Server risks a protocol mismatch; build both from the same SHA. 6. **Environment name gates security hard-stops** (`docs/GatewayConfiguration.md` L317-335): `Dashboard:DisableLogin` and plaintext LDAP are refused only when the environment name is exactly `Production`. Do not change `ASPNETCORE_ENVIRONMENT`/`DOTNET_ENVIRONMENT` during an upgrade; a host that boots under a different label silently loses those guards, and a host that was permissive may refuse to boot if you "fix" the label. 7. **Version stamp.** Do not trust `InformationalVersion` for builds from 2026-07-09 to 2026-08-10 (`docs/runbooks/IdentifyingADeployedBuild.md`). Record the SHA yourself in the deploy notes. 8. **Stuck `STOP_PENDING`.** Stopping the service can hang in `STOP_PENDING` (workers must exit within `Worker:ShutdownTimeoutSeconds`, default 10). Observed on the QA boxes per deploy notes **[UNVERIFIED from this repo]**; if it hangs, wait, then kill the Server process by PID. Do not start the new build until `Get-Process` shows no `ZB.MOM.WW.MxGateway.Server` and no `ZB.MOM.WW.MxGateway.Worker`. ## 3. Preflight (no changes yet) 1. Decide the target SHA on `origin/main`; confirm CI is green for it (`windows-x86` job runs the Worker build + `Worker.Tests` on windev; `nightly-windev` runs the live tier, `scripts/ci/README.md`). 2. Read the current state of the service (read-only): ```text nssm get MxAccessGw Application nssm get MxAccessGw AppDirectory nssm get MxAccessGw AppParameters nssm get MxAccessGw AppEnvironmentExtra # save to the deploy notes; redact secrets ``` Note the entry count and names of the env block; verify the same count after the start. 3. Confirm the effective store paths from the env block: `MxGateway__Authentication__SqlitePath`, `Secrets__SqlitePath`, `MxGateway__Worker__ExecutablePath`. If either store path is missing from the env block it is on the code default (`C:\ProgramData\MxGateway\...`), which is fine; if it is under the application directory, **stop and move it first** (see section 1). 4. Confirm the pepper and `ZB_SECRETS_MASTER_KEY` are present in the env block (names only). 5. Read the dashboard **Settings** page (`/settings` on the dashboard port, `:5130`) and save the effective config; it now includes the heartbeat check interval, stuck ceiling, event-channel full-mode timeout and replay byte cap. Compare after the upgrade to catch config drift. 6. Check the box is quiet: dashboard sessions list, or `GET /health/active` (the `mxaccess-sessions` check reports `total/ready/faulted/starting/closing`). Agree a window with clients (OtOpcUa `GalaxyDriver`, ScadaBridge MxGateway adapter) - their sessions will drop. 7. Check the new build's required config: diff the repo `src/ZB.MOM.WW.MxGateway.Server/appsettings.json` and the changelog for new mandatory options, and `docs/GatewayConfiguration.md` for validator changes (the validator can refuse to boot on a bad value, which is a rollback trigger). 8. Free disk for a full copy of the deploy dir and the stores (see backup). 9. Check auth-store schema compatibility: the Server migrates the API-key DB on start (`Authentication:RunMigrationsOnStartup: true`) via the `ZB.MOM.WW.Auth` package. If the new build bumps the Auth package, the schema can migrate forward and **the old binary may not read it back** (see the 2026-08-07 follow-up plan's `AuthStoreMigrationException` check). Treat the DB backup as the rollback for a schema bump. ## 4. Build (on windev; no deploy yet) windev flow from `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`: ```powershell git -C C:\build\mxaccessgw-ci fetch origin git -C C:\build\mxaccessgw-ci worktree add C:\build\mxgw-deploy # detached worktree, host checkout untouched cd C:\build\mxgw-deploy dotnet publish src\ZB.MOM.WW.MxGateway.Server -c Release -o C:\publish\mxaccessgw\Server-YYYYMMDD dotnet build src\ZB.MOM.WW.MxGateway.Worker\ZB.MOM.WW.MxGateway.Worker.csproj -c Release -p:Platform=x86 # stage the Worker output (bin\x86\Release) into a temp dir, e.g. C:\publish\mxaccessgw\Worker-new ``` Remove the worktree afterwards (`git worktree remove`); an absent `C:\build\mxgw-deploy` is the normal end state (`IdentifyingADeployedBuild.md`). Whether the Worker is built with `dotnet build` or `dotnet publish`, and its exact output folder, should be matched to the existing `Worker` folder on the host **[UNVERIFIED]**. Note: the Worker is net48 and must be built where the net48 targeting pack and MSBuild are present (windev; `scadaproj` notes say Build Tools 2022 + net48 reference assemblies verified 2026-09-10 **[UNVERIFIED from this repo]**). For the wonder boxes the staged output is copied over SMB (UNC copy is always allowed) into a staging directory such as `E:\psrem-stage\-\` per `scadaproj/deployment.md` §7.1 (**[UNVERIFIED]** path convention); the on-box steps below are then the only remote-exec part. ## 5. Backup With the service **still running** copy nothing live; do these after stopping (section 6 step 1), or copy the stores with the service stopped. Back up: 1. The current deploy directory (`Server-YYYYMMDD` or `Server`) to a sibling `Server.bak.` (the convention `IdentifyingADeployedBuild.md` reads as a deploy history chain), and the current `Worker` directory to `Worker.bak-[-label]`. 2. `gateway-auth.db` plus `gateway-auth.db-wal` / `-shm` if present, and `mxgateway-secrets.db` (plus its `-wal`/`-shm`), to `*.bak-