diff --git a/docs/runbooks/DeployUpgradeNssmService.md b/docs/runbooks/DeployUpgradeNssmService.md new file mode 100644 index 00000000..b14f3f81 --- /dev/null +++ b/docs/runbooks/DeployUpgradeNssmService.md @@ -0,0 +1,273 @@ +# Deploy / Upgrade Runbook: NSSM-wrapped Gateway Service (Server + x86 Worker) + +> Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). There is **no deploy script** in +> this repo; this runbook is the procedure. It is assembled only from what the repo contains +> (`docs/GatewayConfiguration.md`, `docs/Authentication.md`, `docs/Diagnostics.md`, +> `docs/runbooks/*`, `docs/plans/2026-08-07-*`, `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`, +> `src/.../appsettings.json`, `scripts/ci/`) cross-checked against `scadaproj/deployment.md` §5 and §7.1. +> +> **Verification legend.** Statements are either backed by a cited repo file, or tagged +> **[UNVERIFIED]** (comes from deploy notes / other repos / memory and could not be confirmed from +> this repo). Treat every **[UNVERIFIED]** item as "confirm on the box before relying on it". +> +> **Scope of remote execution.** On the wonder boxes (`wonder-app-vd03` DEV, `wonder-app-vq03`/`vq04` +> QA) remote command execution is permitted for the on-box steps of a deploy the owner has asked for, +> and nothing else (`CLAUDE.md` "Wonder box remote access" note; the umbrella `scadaproj` index +> carries the current rule). No ad-hoc diagnostics over remote exec: use SMB file reads, HTTP probes +> and direct SQL. Confirm the current rule before starting; this repo's note is dated 2026-09-11 and +> the umbrella index has since relaxed it to deploy-only **[UNVERIFIED from this repo]**. + +## 0. What is being deployed + +Two artifacts, deployed together because they share a wire protocol (`WorkerProtocolVersion`, +`appsettings.json` `MxGateway:Protocol`): + +| Artifact | Build | Runs as | +|---|---|---| +| **Server** (x64 .NET 10, gRPC `:5120` h2c + Blazor dashboard `:5130`) | `dotnet publish src/ZB.MOM.WW.MxGateway.Server -c Release` (framework-dependent in the windev flow; self-contained is the alternative if the box lacks a .NET 10 runtime **[UNVERIFIED]**) | The NSSM service `MxAccessGw` | +| **Worker** (x86 .NET Framework 4.8, one process per gateway session, owns the MXAccess COM STA) | `dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86` (`CLAUDE.md`) - **Windows only; builds on windev**, not on macOS/Linux (`scripts/ci/README.md`) | Child of the Server, launched from `MxGateway:Worker:ExecutablePath`. **Not a service of its own.** | + +The Server finds the Worker through `MxGateway:Worker:ExecutablePath` +(`docs/GatewayConfiguration.md`, default is a repo-relative `bin\x86\Release` path, so deployed hosts +override it via the NSSM env var `MxGateway__Worker__ExecutablePath`). + +Service name `MxAccessGw` and the windev layout (`C:\publish\mxaccessgw\Server-YYYYMMDD`, +`C:\publish\mxaccessgw\Worker`, build clone `C:\build\mxaccessgw-ci`) are from +`docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md` Task "Deploy" and +`docs/runbooks/IdentifyingADeployedBuild.md`. The wonder boxes use `E:\ApiInstall\...` per +`scadaproj/deployment.md` §7.1; the exact `MxAccessGw` subfolder names there are **[UNVERIFIED]**. + +Configuration is delivered as NSSM `AppEnvironmentExtra` environment variables +(`Kestrel__Endpoints__*`, `MxGateway__Authentication__SqlitePath`, `MxGateway__Worker__ExecutablePath`, +`ASPNETCORE_ENVIRONMENT`/`DOTNET_ENVIRONMENT`, `ZB_SECRETS_MASTER_KEY`, the API-key pepper, +`MxGateway__Ldap__ServiceAccountPassword`), not by editing `appsettings.json` +(`docs/GatewayConfiguration.md`, "Kestrel", "Secrets Master Key"). + +## 1. State that MUST survive the swap + +The upgrade renames/replaces the application directory. Anything stored inside it is lost. The +validator therefore rejects stores inside the content root (see the 2026-08-09 incident below). + +| State | Setting | Default location (Windows) | Notes | +|---|---|---|---| +| API-key store (SQLite) | `MxGateway:Authentication:SqlitePath` (NSSM: `MxGateway__Authentication__SqlitePath`) | `C:\ProgramData\MxGateway\gateway-auth.db` | Must be rooted; validator also rejects a path inside the content root. Back up the `-wal`/`-shm` siblings with it. | +| API-key **pepper** | secret `MxGateway:ApiKeyPepper` (`PepperSecretName`); lives outside the DB, in NSSM env / Secrets store | n/a | Without the same pepper every issued `mxgw_...` token fails verification. Never in the DB (`docs/Authentication.md`). Back it up in the org vault. | +| Secrets store (SQLite) | `Secrets:SqlitePath` (NSSM: `Secrets__SqlitePath`) | `C:\ProgramData\MxGateway\mxgateway-secrets.db` | **Must be absolute AND outside the content root**; fail-closed with no opt-out (`docs/GatewayConfiguration.md` L349; `scadaproj` Secrets notes). | +| Secrets master key (KEK) | `ZB_SECRETS_MASTER_KEY` in NSSM `AppEnvironmentExtra` | n/a | Losing it makes the store unrecoverable; no `RewrapAll` yet (`docs/GatewayConfiguration.md`, "Rotation / DR"). Back it up in the vault, not just on the box. | +| Galaxy snapshot cache | `MxGateway:Galaxy:SnapshotCachePath` | `C:\ProgramData\MxGateway\galaxy-snapshot.json` | Cache only; rejected inside content root so it survives deploys. Losing it is harmless. | +| TLS self-signed cert | `Tls:SelfSignedCertPath` | `C:\ProgramData\MxGateway\certs\gateway-selfsigned.pfx` | If lost, clients pinned to the old cert see a changed cert. | +| NSSM env block | `nssm get MxAccessGw AppEnvironmentExtra` | NSSM registry config | File copies do not touch it, but `nssm set AppEnvironmentExtra` **replaces the whole block**. | +| Local `appsettings*.json` overrides in the deploy dir | n/a | deploy dir | Copy the live ones aside and restore them after the swap (the windev plan copies `appsettings.json` / `appsettings.Development.json` from the current dir). | + +**Incident that motivates the rule (2026-08-09, per `docs/GatewayConfiguration.md` L127):** the +API-key store lived inside the application directory; the upgrade renamed that directory to +`Server.bak.*`, silently starting an empty store, and no gRPC client could authenticate for two +days. The SQLite DB must also never be opened with a host-side `sqlite3` while the service runs +(copy the file triplet and query the copy; cross-repo lesson, **[UNVERIFIED from this repo]**). + +## 2. Traps (read before touching anything) + +1. **`OrphanWorkerTerminator` kills live workers if a second gateway starts.** + On startup the Server enumerates processes named after the configured Worker exe + (`ZB.MOM.WW.MxGateway.Worker`) and kills every one whose path equals the configured + `ExecutablePath` - or whose path it cannot read, which is the normal case for an x86 process seen + from an x64 gateway, so it falls back to **image-name match only** + (`src/ZB.MOM.WW.MxGateway.Server/Workers/OrphanWorkerTerminator.cs`). It skips only its own PID's + children logic (`currentProcessId`), not another gateway's workers. Consequence: starting a + second gateway process on the same box (a manual `dotnet ...Server.dll` or `Server.exe` while the + service is up, a second NSSM service, a "test" start of the new build before stopping the old one, + or a rollback started while the new one still runs) **terminates every live session's Worker** and + all MXAccess subscriptions with it. Rule: **exactly one gateway process per box at any time**; + stop the service and confirm the process has exited before starting anything else. Never smoke-test + the new build by launching it beside the running service. (On QA, vq03 and vq04 are independent + boxes with independent gateways; this trap is per-box, not per-pair.) +2. **A gateway restart does not reattach workers** (`OrphanWorkerTerminator` summary). Plan for every + client session to drop and for clients to reopen sessions. Session-resilience reattach is opt-in + design only (`docs/plans/2026-06-15-session-resilience-design.md`); do not assume it is on. +3. **Secrets store path.** `Secrets:SqlitePath` must be absolute and outside the content root or the + host fails closed at startup. A store left at an in-tree path is exactly what the swap removes. + Also: the `secret` CLI and the gateway must use the same `Secrets__SqlitePath`, or the CLI seeds a + different store and the gateway then fails closed as if the secret were never set + (`docs/GatewayConfiguration.md` L381-395). +4. **`nssm set ... AppEnvironmentExtra` replaces the whole block.** `nssm get` first, save the output + (redact `Password=`/key values in anything you paste), then re-include every entry + (`docs/GatewayConfiguration.md` L370-374; `docs/runbooks/SEC-36-ldap-credential-rotation.md`). + The `nssm set MxAccessGw Application`/`AppDirectory` commands used for repointing do **not** touch + the env block. +5. **Worker bitness.** The Worker must be built x86 (`-p:Platform=x86`) and `RequiredArchitecture` + stays `X86`. The Server (x64) cannot load MXAccess COM. Build on windev (`ssh windev` flow, + `scripts/ci/windev-worker-ci.ps1`), not on a Mac. Deploying a Worker from a different commit than + the Server risks a protocol mismatch; build both from the same SHA. +6. **Environment name gates security hard-stops** (`docs/GatewayConfiguration.md` L317-335): + `Dashboard:DisableLogin` and plaintext LDAP are refused only when the environment name is exactly + `Production`. Do not change `ASPNETCORE_ENVIRONMENT`/`DOTNET_ENVIRONMENT` during an upgrade; a + host that boots under a different label silently loses those guards, and a host that was + permissive may refuse to boot if you "fix" the label. +7. **Version stamp.** Do not trust `InformationalVersion` for builds from 2026-07-09 to 2026-08-10 + (`docs/runbooks/IdentifyingADeployedBuild.md`). Record the SHA yourself in the deploy notes. +8. **Stuck `STOP_PENDING`.** Stopping the service can hang in `STOP_PENDING` (workers must exit within + `Worker:ShutdownTimeoutSeconds`, default 10). Observed on the QA boxes per deploy notes + **[UNVERIFIED from this repo]**; if it hangs, wait, then kill the Server process by PID. Do not start the new + build until `Get-Process` shows no `ZB.MOM.WW.MxGateway.Server` and no `ZB.MOM.WW.MxGateway.Worker`. + +## 3. Preflight (no changes yet) + +1. Decide the target SHA on `origin/main`; confirm CI is green for it (`windows-x86` job runs the + Worker build + `Worker.Tests` on windev; `nightly-windev` runs the live tier, + `scripts/ci/README.md`). +2. Read the current state of the service (read-only): + ```text + nssm get MxAccessGw Application + nssm get MxAccessGw AppDirectory + nssm get MxAccessGw AppParameters + nssm get MxAccessGw AppEnvironmentExtra # save to the deploy notes; redact secrets + ``` + Note the entry count and names of the env block; verify the same count after the start. +3. Confirm the effective store paths from the env block: `MxGateway__Authentication__SqlitePath`, + `Secrets__SqlitePath`, `MxGateway__Worker__ExecutablePath`. If either store path is missing from + the env block it is on the code default (`C:\ProgramData\MxGateway\...`), which is fine; if it is + under the application directory, **stop and move it first** (see section 1). +4. Confirm the pepper and `ZB_SECRETS_MASTER_KEY` are present in the env block (names only). +5. Read the dashboard **Settings** page (`/settings` on the dashboard port, `:5130`) and save the + effective config; it now includes the heartbeat check interval, stuck ceiling, event-channel + full-mode timeout and replay byte cap. Compare after the upgrade to catch config drift. +6. Check the box is quiet: dashboard sessions list, or `GET /health/active` (the `mxaccess-sessions` + check reports `total/ready/faulted/starting/closing`). Agree a window with clients (OtOpcUa + `GalaxyDriver`, ScadaBridge MxGateway adapter) - their sessions will drop. +7. Check the new build's required config: diff the repo `src/ZB.MOM.WW.MxGateway.Server/appsettings.json` + and the changelog for new mandatory options, and `docs/GatewayConfiguration.md` for validator + changes (the validator can refuse to boot on a bad value, which is a rollback trigger). +8. Free disk for a full copy of the deploy dir and the stores (see backup). +9. Check auth-store schema compatibility: the Server migrates the API-key DB on start + (`Authentication:RunMigrationsOnStartup: true`) via the `ZB.MOM.WW.Auth` package. If the new build + bumps the Auth package, the schema can migrate forward and **the old binary may not read it back** + (see the 2026-08-07 follow-up plan's `AuthStoreMigrationException` check). Treat the DB backup as + the rollback for a schema bump. + +## 4. Build (on windev; no deploy yet) + +windev flow from `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`: + +```powershell +git -C C:\build\mxaccessgw-ci fetch origin +git -C C:\build\mxaccessgw-ci worktree add C:\build\mxgw-deploy # detached worktree, host checkout untouched +cd C:\build\mxgw-deploy +dotnet publish src\ZB.MOM.WW.MxGateway.Server -c Release -o C:\publish\mxaccessgw\Server-YYYYMMDD +dotnet build src\ZB.MOM.WW.MxGateway.Worker\ZB.MOM.WW.MxGateway.Worker.csproj -c Release -p:Platform=x86 +# stage the Worker output (bin\x86\Release) into a temp dir, e.g. C:\publish\mxaccessgw\Worker-new +``` + +Remove the worktree afterwards (`git worktree remove`); an absent `C:\build\mxgw-deploy` is the +normal end state (`IdentifyingADeployedBuild.md`). Whether the Worker is built with `dotnet build` +or `dotnet publish`, and its exact output folder, should be matched to the existing `Worker` folder on +the host **[UNVERIFIED]**. Note: the Worker is net48 and must be built where the net48 targeting +pack and MSBuild are present (windev; `scadaproj` notes say Build Tools 2022 + net48 reference +assemblies verified 2026-09-10 **[UNVERIFIED from this repo]**). + +For the wonder boxes the staged output is copied over SMB (UNC copy is always allowed) into a +staging directory such as `E:\psrem-stage\-\` per `scadaproj/deployment.md` §7.1 +(**[UNVERIFIED]** path convention); the on-box steps below are then the only remote-exec part. + +## 5. Backup + +With the service **still running** copy nothing live; do these after stopping (section 6 step 1), or +copy the stores with the service stopped. Back up: + +1. The current deploy directory (`Server-YYYYMMDD` or `Server`) to a sibling `Server.bak.` + (the convention `IdentifyingADeployedBuild.md` reads as a deploy history chain), and the current + `Worker` directory to `Worker.bak-[-label]`. +2. `gateway-auth.db` plus `gateway-auth.db-wal` / `-shm` if present, and `mxgateway-secrets.db` (plus + its `-wal`/`-shm`), to `*.bak-