2f6e480a41
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
274 lines
20 KiB
Markdown
274 lines
20 KiB
Markdown
# Deploy / Upgrade Runbook: NSSM-wrapped Gateway Service (Server + x86 Worker)
|
|
|
|
> Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). There is **no deploy script** in
|
|
> this repo; this runbook is the procedure. It is assembled only from what the repo contains
|
|
> (`docs/GatewayConfiguration.md`, `docs/Authentication.md`, `docs/Diagnostics.md`,
|
|
> `docs/runbooks/*`, `docs/plans/2026-08-07-*`, `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`,
|
|
> `src/.../appsettings.json`, `scripts/ci/`) cross-checked against `scadaproj/deployment.md` §5 and §7.1.
|
|
>
|
|
> **Verification legend.** Statements are either backed by a cited repo file, or tagged
|
|
> **[UNVERIFIED]** (comes from deploy notes / other repos / memory and could not be confirmed from
|
|
> this repo). Treat every **[UNVERIFIED]** item as "confirm on the box before relying on it".
|
|
>
|
|
> **Scope of remote execution.** On the wonder boxes (`wonder-app-vd03` DEV, `wonder-app-vq03`/`vq04`
|
|
> QA) remote command execution is permitted for the on-box steps of a deploy the owner has asked for,
|
|
> and nothing else (`CLAUDE.md` "Wonder box remote access" note; the umbrella `scadaproj` index
|
|
> carries the current rule). No ad-hoc diagnostics over remote exec: use SMB file reads, HTTP probes
|
|
> and direct SQL. Confirm the current rule before starting; this repo's note is dated 2026-09-11 and
|
|
> the umbrella index has since relaxed it to deploy-only **[UNVERIFIED from this repo]**.
|
|
|
|
## 0. What is being deployed
|
|
|
|
Two artifacts, deployed together because they share a wire protocol (`WorkerProtocolVersion`,
|
|
`appsettings.json` `MxGateway:Protocol`):
|
|
|
|
| Artifact | Build | Runs as |
|
|
|---|---|---|
|
|
| **Server** (x64 .NET 10, gRPC `:5120` h2c + Blazor dashboard `:5130`) | `dotnet publish src/ZB.MOM.WW.MxGateway.Server -c Release` (framework-dependent in the windev flow; self-contained is the alternative if the box lacks a .NET 10 runtime **[UNVERIFIED]**) | The NSSM service `MxAccessGw` |
|
|
| **Worker** (x86 .NET Framework 4.8, one process per gateway session, owns the MXAccess COM STA) | `dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86` (`CLAUDE.md`) - **Windows only; builds on windev**, not on macOS/Linux (`scripts/ci/README.md`) | Child of the Server, launched from `MxGateway:Worker:ExecutablePath`. **Not a service of its own.** |
|
|
|
|
The Server finds the Worker through `MxGateway:Worker:ExecutablePath`
|
|
(`docs/GatewayConfiguration.md`, default is a repo-relative `bin\x86\Release` path, so deployed hosts
|
|
override it via the NSSM env var `MxGateway__Worker__ExecutablePath`).
|
|
|
|
Service name `MxAccessGw` and the windev layout (`C:\publish\mxaccessgw\Server-YYYYMMDD`,
|
|
`C:\publish\mxaccessgw\Worker`, build clone `C:\build\mxaccessgw-ci`) are from
|
|
`docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md` Task "Deploy" and
|
|
`docs/runbooks/IdentifyingADeployedBuild.md`. The wonder boxes use `E:\ApiInstall\...` per
|
|
`scadaproj/deployment.md` §7.1; the exact `MxAccessGw` subfolder names there are **[UNVERIFIED]**.
|
|
|
|
Configuration is delivered as NSSM `AppEnvironmentExtra` environment variables
|
|
(`Kestrel__Endpoints__*`, `MxGateway__Authentication__SqlitePath`, `MxGateway__Worker__ExecutablePath`,
|
|
`ASPNETCORE_ENVIRONMENT`/`DOTNET_ENVIRONMENT`, `ZB_SECRETS_MASTER_KEY`, the API-key pepper,
|
|
`MxGateway__Ldap__ServiceAccountPassword`), not by editing `appsettings.json`
|
|
(`docs/GatewayConfiguration.md`, "Kestrel", "Secrets Master Key").
|
|
|
|
## 1. State that MUST survive the swap
|
|
|
|
The upgrade renames/replaces the application directory. Anything stored inside it is lost. The
|
|
validator therefore rejects stores inside the content root (see the 2026-08-09 incident below).
|
|
|
|
| State | Setting | Default location (Windows) | Notes |
|
|
|---|---|---|---|
|
|
| API-key store (SQLite) | `MxGateway:Authentication:SqlitePath` (NSSM: `MxGateway__Authentication__SqlitePath`) | `C:\ProgramData\MxGateway\gateway-auth.db` | Must be rooted; validator also rejects a path inside the content root. Back up the `-wal`/`-shm` siblings with it. |
|
|
| API-key **pepper** | secret `MxGateway:ApiKeyPepper` (`PepperSecretName`); lives outside the DB, in NSSM env / Secrets store | n/a | Without the same pepper every issued `mxgw_...` token fails verification. Never in the DB (`docs/Authentication.md`). Back it up in the org vault. |
|
|
| Secrets store (SQLite) | `Secrets:SqlitePath` (NSSM: `Secrets__SqlitePath`) | `C:\ProgramData\MxGateway\mxgateway-secrets.db` | **Must be absolute AND outside the content root**; fail-closed with no opt-out (`docs/GatewayConfiguration.md` L349; `scadaproj` Secrets notes). |
|
|
| Secrets master key (KEK) | `ZB_SECRETS_MASTER_KEY` in NSSM `AppEnvironmentExtra` | n/a | Losing it makes the store unrecoverable; no `RewrapAll` yet (`docs/GatewayConfiguration.md`, "Rotation / DR"). Back it up in the vault, not just on the box. |
|
|
| Galaxy snapshot cache | `MxGateway:Galaxy:SnapshotCachePath` | `C:\ProgramData\MxGateway\galaxy-snapshot.json` | Cache only; rejected inside content root so it survives deploys. Losing it is harmless. |
|
|
| TLS self-signed cert | `Tls:SelfSignedCertPath` | `C:\ProgramData\MxGateway\certs\gateway-selfsigned.pfx` | If lost, clients pinned to the old cert see a changed cert. |
|
|
| NSSM env block | `nssm get MxAccessGw AppEnvironmentExtra` | NSSM registry config | File copies do not touch it, but `nssm set AppEnvironmentExtra` **replaces the whole block**. |
|
|
| Local `appsettings*.json` overrides in the deploy dir | n/a | deploy dir | Copy the live ones aside and restore them after the swap (the windev plan copies `appsettings.json` / `appsettings.Development.json` from the current dir). |
|
|
|
|
**Incident that motivates the rule (2026-08-09, per `docs/GatewayConfiguration.md` L127):** the
|
|
API-key store lived inside the application directory; the upgrade renamed that directory to
|
|
`Server.bak.*`, silently starting an empty store, and no gRPC client could authenticate for two
|
|
days. The SQLite DB must also never be opened with a host-side `sqlite3` while the service runs
|
|
(copy the file triplet and query the copy; cross-repo lesson, **[UNVERIFIED from this repo]**).
|
|
|
|
## 2. Traps (read before touching anything)
|
|
|
|
1. **`OrphanWorkerTerminator` kills live workers if a second gateway starts.**
|
|
On startup the Server enumerates processes named after the configured Worker exe
|
|
(`ZB.MOM.WW.MxGateway.Worker`) and kills every one whose path equals the configured
|
|
`ExecutablePath` - or whose path it cannot read, which is the normal case for an x86 process seen
|
|
from an x64 gateway, so it falls back to **image-name match only**
|
|
(`src/ZB.MOM.WW.MxGateway.Server/Workers/OrphanWorkerTerminator.cs`). It skips only its own PID's
|
|
children logic (`currentProcessId`), not another gateway's workers. Consequence: starting a
|
|
second gateway process on the same box (a manual `dotnet ...Server.dll` or `Server.exe` while the
|
|
service is up, a second NSSM service, a "test" start of the new build before stopping the old one,
|
|
or a rollback started while the new one still runs) **terminates every live session's Worker** and
|
|
all MXAccess subscriptions with it. Rule: **exactly one gateway process per box at any time**;
|
|
stop the service and confirm the process has exited before starting anything else. Never smoke-test
|
|
the new build by launching it beside the running service. (On QA, vq03 and vq04 are independent
|
|
boxes with independent gateways; this trap is per-box, not per-pair.)
|
|
2. **A gateway restart does not reattach workers** (`OrphanWorkerTerminator` summary). Plan for every
|
|
client session to drop and for clients to reopen sessions. Session-resilience reattach is opt-in
|
|
design only (`docs/plans/2026-06-15-session-resilience-design.md`); do not assume it is on.
|
|
3. **Secrets store path.** `Secrets:SqlitePath` must be absolute and outside the content root or the
|
|
host fails closed at startup. A store left at an in-tree path is exactly what the swap removes.
|
|
Also: the `secret` CLI and the gateway must use the same `Secrets__SqlitePath`, or the CLI seeds a
|
|
different store and the gateway then fails closed as if the secret were never set
|
|
(`docs/GatewayConfiguration.md` L381-395).
|
|
4. **`nssm set ... AppEnvironmentExtra` replaces the whole block.** `nssm get` first, save the output
|
|
(redact `Password=`/key values in anything you paste), then re-include every entry
|
|
(`docs/GatewayConfiguration.md` L370-374; `docs/runbooks/SEC-36-ldap-credential-rotation.md`).
|
|
The `nssm set MxAccessGw Application`/`AppDirectory` commands used for repointing do **not** touch
|
|
the env block.
|
|
5. **Worker bitness.** The Worker must be built x86 (`-p:Platform=x86`) and `RequiredArchitecture`
|
|
stays `X86`. The Server (x64) cannot load MXAccess COM. Build on windev (`ssh windev` flow,
|
|
`scripts/ci/windev-worker-ci.ps1`), not on a Mac. Deploying a Worker from a different commit than
|
|
the Server risks a protocol mismatch; build both from the same SHA.
|
|
6. **Environment name gates security hard-stops** (`docs/GatewayConfiguration.md` L317-335):
|
|
`Dashboard:DisableLogin` and plaintext LDAP are refused only when the environment name is exactly
|
|
`Production`. Do not change `ASPNETCORE_ENVIRONMENT`/`DOTNET_ENVIRONMENT` during an upgrade; a
|
|
host that boots under a different label silently loses those guards, and a host that was
|
|
permissive may refuse to boot if you "fix" the label.
|
|
7. **Version stamp.** Do not trust `InformationalVersion` for builds from 2026-07-09 to 2026-08-10
|
|
(`docs/runbooks/IdentifyingADeployedBuild.md`). Record the SHA yourself in the deploy notes.
|
|
8. **Stuck `STOP_PENDING`.** Stopping the service can hang in `STOP_PENDING` (workers must exit within
|
|
`Worker:ShutdownTimeoutSeconds`, default 10). Observed on the QA boxes per deploy notes
|
|
**[UNVERIFIED from this repo]**; if it hangs, wait, then kill the Server process by PID. Do not start the new
|
|
build until `Get-Process` shows no `ZB.MOM.WW.MxGateway.Server` and no `ZB.MOM.WW.MxGateway.Worker`.
|
|
|
|
## 3. Preflight (no changes yet)
|
|
|
|
1. Decide the target SHA on `origin/main`; confirm CI is green for it (`windows-x86` job runs the
|
|
Worker build + `Worker.Tests` on windev; `nightly-windev` runs the live tier,
|
|
`scripts/ci/README.md`).
|
|
2. Read the current state of the service (read-only):
|
|
```text
|
|
nssm get MxAccessGw Application
|
|
nssm get MxAccessGw AppDirectory
|
|
nssm get MxAccessGw AppParameters
|
|
nssm get MxAccessGw AppEnvironmentExtra # save to the deploy notes; redact secrets
|
|
```
|
|
Note the entry count and names of the env block; verify the same count after the start.
|
|
3. Confirm the effective store paths from the env block: `MxGateway__Authentication__SqlitePath`,
|
|
`Secrets__SqlitePath`, `MxGateway__Worker__ExecutablePath`. If either store path is missing from
|
|
the env block it is on the code default (`C:\ProgramData\MxGateway\...`), which is fine; if it is
|
|
under the application directory, **stop and move it first** (see section 1).
|
|
4. Confirm the pepper and `ZB_SECRETS_MASTER_KEY` are present in the env block (names only).
|
|
5. Read the dashboard **Settings** page (`/settings` on the dashboard port, `:5130`) and save the
|
|
effective config; it now includes the heartbeat check interval, stuck ceiling, event-channel
|
|
full-mode timeout and replay byte cap. Compare after the upgrade to catch config drift.
|
|
6. Check the box is quiet: dashboard sessions list, or `GET /health/active` (the `mxaccess-sessions`
|
|
check reports `total/ready/faulted/starting/closing`). Agree a window with clients (OtOpcUa
|
|
`GalaxyDriver`, ScadaBridge MxGateway adapter) - their sessions will drop.
|
|
7. Check the new build's required config: diff the repo `src/ZB.MOM.WW.MxGateway.Server/appsettings.json`
|
|
and the changelog for new mandatory options, and `docs/GatewayConfiguration.md` for validator
|
|
changes (the validator can refuse to boot on a bad value, which is a rollback trigger).
|
|
8. Free disk for a full copy of the deploy dir and the stores (see backup).
|
|
9. Check auth-store schema compatibility: the Server migrates the API-key DB on start
|
|
(`Authentication:RunMigrationsOnStartup: true`) via the `ZB.MOM.WW.Auth` package. If the new build
|
|
bumps the Auth package, the schema can migrate forward and **the old binary may not read it back**
|
|
(see the 2026-08-07 follow-up plan's `AuthStoreMigrationException` check). Treat the DB backup as
|
|
the rollback for a schema bump.
|
|
|
|
## 4. Build (on windev; no deploy yet)
|
|
|
|
windev flow from `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`:
|
|
|
|
```powershell
|
|
git -C C:\build\mxaccessgw-ci fetch origin
|
|
git -C C:\build\mxaccessgw-ci worktree add C:\build\mxgw-deploy <sha> # detached worktree, host checkout untouched
|
|
cd C:\build\mxgw-deploy
|
|
dotnet publish src\ZB.MOM.WW.MxGateway.Server -c Release -o C:\publish\mxaccessgw\Server-YYYYMMDD
|
|
dotnet build src\ZB.MOM.WW.MxGateway.Worker\ZB.MOM.WW.MxGateway.Worker.csproj -c Release -p:Platform=x86
|
|
# stage the Worker output (bin\x86\Release) into a temp dir, e.g. C:\publish\mxaccessgw\Worker-new
|
|
```
|
|
|
|
Remove the worktree afterwards (`git worktree remove`); an absent `C:\build\mxgw-deploy` is the
|
|
normal end state (`IdentifyingADeployedBuild.md`). Whether the Worker is built with `dotnet build`
|
|
or `dotnet publish`, and its exact output folder, should be matched to the existing `Worker` folder on
|
|
the host **[UNVERIFIED]**. Note: the Worker is net48 and must be built where the net48 targeting
|
|
pack and MSBuild are present (windev; `scadaproj` notes say Build Tools 2022 + net48 reference
|
|
assemblies verified 2026-09-10 **[UNVERIFIED from this repo]**).
|
|
|
|
For the wonder boxes the staged output is copied over SMB (UNC copy is always allowed) into a
|
|
staging directory such as `E:\psrem-stage\<app>-<sha>\` per `scadaproj/deployment.md` §7.1
|
|
(**[UNVERIFIED]** path convention); the on-box steps below are then the only remote-exec part.
|
|
|
|
## 5. Backup
|
|
|
|
With the service **still running** copy nothing live; do these after stopping (section 6 step 1), or
|
|
copy the stores with the service stopped. Back up:
|
|
|
|
1. The current deploy directory (`Server-YYYYMMDD` or `Server`) to a sibling `Server.bak.<timestamp>`
|
|
(the convention `IdentifyingADeployedBuild.md` reads as a deploy history chain), and the current
|
|
`Worker` directory to `Worker.bak-<timestamp>[-label]`.
|
|
2. `gateway-auth.db` plus `gateway-auth.db-wal` / `-shm` if present, and `mxgateway-secrets.db` (plus
|
|
its `-wal`/`-shm`), to `*.bak-<label>` next to the originals. **Never delete or move the live DB.**
|
|
3. The saved `AppEnvironmentExtra` text and the `nssm get Application/AppDirectory` values (for rollback
|
|
repointing).
|
|
4. Off-box: the KEK and pepper in the org vault (not on the box only).
|
|
5. Record the old and new SHA, timestamps and backup paths in a `_DEPLOY-NOTES-<timestamp>.txt`
|
|
next to the install (the convention seen on wonder per `deployment.md` §7.1, **[UNVERIFIED]**).
|
|
|
|
## 6. Stop / swap / start
|
|
|
|
Do this on **one box at a time**; QA vq03 and vq04 are independent gateways, so roll them one after
|
|
the other (`deployment.md` §7.1 records the 2026-09-10 round as "rolling, one stop per box").
|
|
|
|
1. **Stop and confirm gone.**
|
|
```powershell
|
|
nssm stop MxAccessGw # or Stop-Service MxAccessGw -Force
|
|
Get-Process ZB.MOM.WW.MxGateway.Server, ZB.MOM.WW.MxGateway.Worker -ErrorAction SilentlyContinue
|
|
```
|
|
Both lists must be empty before continuing (traps 1 and 8). If a Worker is left over, the next
|
|
start will kill it as an orphan (by design) - but it signals an unclean stop worth noting.
|
|
2. **Backup** (section 5) now that nothing holds the stores open.
|
|
3. **Swap the Server.** Preferred: deploy to a *new dated directory* and repoint NSSM, which leaves the
|
|
old directory intact as the rollback target:
|
|
```powershell
|
|
Copy-Item <old>\appsettings.json, <old>\appsettings.Development.json <new>\ # preserve live file config
|
|
nssm set MxAccessGw Application <new>\ZB.MOM.WW.MxGateway.Server.exe
|
|
nssm set MxAccessGw AppDirectory <new>
|
|
```
|
|
(`2026-10-19-ackmsg-as-operator.md`; `IdentifyingADeployedBuild.md` "Deployment-side naming".)
|
|
Alternative used on the wonder boxes: rename `app` to `app.bak.<timestamp>` and copy the new files
|
|
into `app` (`deployment.md` §7.1; **[UNVERIFIED]**). Either way, restore any host-specific
|
|
`appsettings*.json` byte-identical; do not edit the NSSM env block unless the release notes require it.
|
|
4. **Swap the Worker.** Rename `Worker` to `Worker.bak-<timestamp>`, copy the new x86 Worker build in
|
|
its place, and make sure `MxGateway__Worker__ExecutablePath` points at it
|
|
(the path is matched case-insensitively by the orphan terminator, so keep one canonical spelling).
|
|
5. **Verify the env block is untouched**: `nssm get MxAccessGw AppEnvironmentExtra` entry count equals the
|
|
preflight count.
|
|
6. **Start** (`nssm start MxAccessGw`). Exactly once; do not retry-start concurrently with anything else.
|
|
|
|
## 7. Post-deploy verification
|
|
|
|
All of this can be done over HTTP/gRPC from another machine or from the box without further remote
|
|
exec beyond the deploy window.
|
|
|
|
1. Service `RUNNING` and stable for at least 60 s (no restart loop); the Application event log has a
|
|
clean host start and **no** new `AuthStoreMigrationException` / `Hosting failed to start` after the
|
|
start time (`2026-08-07-followups-windev-ldapfixtures-runners.md` Task 2).
|
|
2. Health: `GET http://<host>:5130/healthz` (live), `/health/ready` (`auth-store` check),
|
|
`/health/active` (`mxaccess-sessions`). These endpoints are anonymous by design
|
|
(`GatewayApplication.cs`, `docs/Diagnostics.md`). Zero sessions is Healthy, so green does not prove
|
|
the Worker/COM path works.
|
|
3. Prove the **Worker** path, since an idle health probe does not: open a gRPC session with an existing
|
|
API key (any of the clients, or the `mxa` CLI) and do a read of a known tag. This also proves the
|
|
API-key store and pepper survived: a `mxgw_...` token issued before the upgrade must authenticate.
|
|
Negative control: a wrong token must be rejected (`UNAUTHENTICATED`).
|
|
4. Dashboard: login works (or the localhost anonymous policy applies), `/settings` matches the
|
|
preflight capture, the version/Worker protocol line is right.
|
|
5. Identify the deployed build by behaviour, not the version stamp where it is unreliable
|
|
(`IdentifyingADeployedBuild.md`): a plain `Write` returning `statuses[0]` proves a Worker at or
|
|
past `53f69cd`.
|
|
6. Check the Server log for `Terminated orphan worker process` - expected only if the previous stop was
|
|
unclean.
|
|
7. Check downstream clients reconnected (OtOpcUa `GalaxyDriver`; ScadaBridge MxGateway adapter) and, on
|
|
a pair of QA boxes, repeat sections 5-7 for the second box.
|
|
8. Write the deploy note: SHA, backup paths, env-entry count, evidence for each check above.
|
|
|
|
## 8. Rollback
|
|
|
|
Triggers: service will not stay `RUNNING`, host fails validator at start, health red beyond the grace
|
|
window, API keys no longer authenticate, or a Worker session cannot be opened.
|
|
|
|
1. `nssm stop MxAccessGw`; confirm no Server/Worker process (trap 1: **never start the old build while
|
|
the new one is still up**).
|
|
2. Repoint: `nssm set MxAccessGw Application <old>\...Server.exe` and `AppDirectory <old>`, or rename
|
|
`app.bak.<ts>` back to `app`, and restore `Worker.bak-<ts>` over `Worker`.
|
|
3. If the API-key DB was migrated (schema bump, trap in section 3 step 9) or damaged, restore
|
|
`gateway-auth.db` from the backup (restore the `-wal`/`-shm` together or none). Otherwise leave the
|
|
live store alone: the env-block stores are outside the swap and need no restore.
|
|
4. `nssm get AppEnvironmentExtra` entry count equals the preflight count; if you changed it, restore the
|
|
saved block (all entries).
|
|
5. `nssm start MxAccessGw`; repeat section 7 steps 1-3.
|
|
6. Report: what failed, the log excerpt, backup paths used.
|
|
|
|
## 9. Open gaps (not verifiable from this repo)
|
|
|
|
- Exact install paths on the wonder boxes (`E:\ApiInstall\MxAccessGw\...`), the staging convention and
|
|
the `_DEPLOY-NOTES` file format - taken from `scadaproj/deployment.md`, whose live content could not
|
|
be cross-checked here.
|
|
- Whether the Server on each box is framework-dependent or self-contained (a box without .NET 10
|
|
needs self-contained; the 2026-10-19 windev plan says framework-dependent).
|
|
- The `STOP_PENDING` behaviour and the NSSM service account (`LocalSystem` per `deployment.md`).
|
|
- NSSM install script: **none exists in the repo** - installing a *new* box (service creation,
|
|
`AppExit`, recovery, log redirection settings) is not covered here; the only reference is
|
|
`deployment.md`'s statement that VQ04 was cloned from VQ03's NSSM `AppEnvironmentExtra` config.
|
|
- Automating this runbook as a script was the other half of the original finding; not done here.
|