Files
mxaccessgw/docs/runbooks/DeployUpgradeNssmService.md
T
Joseph Doherty e74918ccfb
ci / portable (push) Failing after 10m10s
ci / java (push) Failing after 20m12s
ci / windows-x86 (push) Failing after 22m40s
ci: drop the nightly-windev schedule; run the deep windev tier on demand
The scheduled nightly-windev job failed most nights for host reasons and
filed a Gitea issue each time. Remove the cron trigger and the job, and add
a workflow_dispatch trigger whose windev_mode input (test/build/live) drives
the windows-x86 job, so the live tier runs on demand - from the Actions page
or via scripts/ci/run-windev-ci.sh live - the same way clients are packed and
published on demand. Docs updated to match.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-15 09:42:13 -05:00

403 lines
32 KiB
Markdown

# Deploy / Upgrade Runbook: NSSM-wrapped Gateway Service (Server + x86 Worker)
> Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). Since mxaccessgw#220 the procedure
> is also scripted (see "Scripts" below); this runbook remains the spec the scripts implement. It is assembled only from what the repo contains
> (`docs/GatewayConfiguration.md`, `docs/Authentication.md`, `docs/Diagnostics.md`,
> `docs/runbooks/*`, `docs/plans/2026-08-07-*`, `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`,
> `src/.../appsettings.json`, `scripts/ci/`) cross-checked against `scadaproj/deployment.md` §5 and §7.1.
>
> **Verification legend.** Statements are either backed by a cited repo file, or tagged
> **[UNVERIFIED]** (comes from deploy notes / other repos / memory and could not be confirmed from
> this repo). Treat every **[UNVERIFIED]** item as "confirm on the box before relying on it".
>
> **Scope of remote execution.** On the wonder boxes (`wonder-app-vd03` DEV, `wonder-app-vq03`/`vq04`
> QA) remote command execution is permitted for the on-box steps of a deploy the owner has asked for,
> and nothing else (`CLAUDE.md` "Wonder box remote access" note; the umbrella `scadaproj` index
> carries the current rule). No ad-hoc diagnostics over remote exec: use SMB file reads, HTTP probes
> and direct SQL. Confirm the current rule before starting; this repo's note is dated 2026-09-11 and
> the umbrella index has since relaxed it to deploy-only **[UNVERIFIED from this repo]**.
## Scripts (mxaccessgw#220)
> **Test status (mxaccessgw#230, 2026-11-12).** The upgrade script has run in `-DryRun` against the real
> windev install (Windows PowerShell 5.1, live service, live processes): the guards found the running
> gateway and its Workers, and a copy with the image names removed failed closed. It has **not yet run a real
> upgrade on any box**: the first real use (QA, #229) should start with `-DryRun` on the box, read the plan, and
> have section 8 ready by hand. Stop/start, STOP_PENDING, carry-over, the env-block checks and both rollbacks are
> covered by the harness in `scripts/deploy/tests/` against a fake box (temp directories really renamed and copied,
> fake `nssm`/process table/registry/clock), in both naming schemes:
> `pwsh -NoProfile -File scripts/deploy/tests/Test-UpgradeMxGatewayService.ps1` (assertions) and
> `pwsh -NoProfile -File scripts/deploy/tests/Invoke-UpgradeMutations.ps1` (re-introduces each guarded defect
> and requires the harness to fail). The harness also passes under Windows PowerShell 5.1 on windev. The windev
> run caught two defects the Mac harness could not: an empty `Win32_Service` result read as "check unavailable",
> and Windows PowerShell 5.1's `Get-FileHash` returning nothing under `-WhatIf`, which aborted every `-WhatIf` plan.
> Both are fixed. `Install-MxGatewayService.ps1` is still untested on a real box.
>
> **Before #230 the upgrade script was not safe on the wonder boxes**: its process guards matched only
> `ZB.MOM.WW.MxGateway.*`, but vd03/vq03/vq04 run `MxGateway.Server.exe` / `MxGateway.Worker.exe`, so the
> one-gateway guard, the pre-start OrphanWorkerTerminator guard and the STOP_PENDING kill passed vacuously;
> `Server\dashboard.pfx` was dropped; and preflight refused a file-based master key. Do not use a copy of the
> script older than the #230 commit.
| Script | Where it runs | Does |
|---|---|---|
| `scripts/deploy/Upgrade-MxGatewayService.ps1` | **On the Windows box**, Windows PowerShell 5.1 | Sections 3, 5, 6, 7, 8: preflight, stop, backup, swap Server + Worker, start, verify, rollback |
| `scripts/deploy/Install-MxGatewayService.ps1` | On a **new** box | NSSM service install: recovery/restart, stop grace, log rotation, env block |
Both take `-DryRun` (or `-WhatIf`): read-only probes (`nssm get/status`, process list, disk, file
existence) still run, every mutating action is printed instead of executed.
**Upgrade script** (`-ServerSource`, `-WorkerSource` are the staged outputs from section 4, built from one SHA):
- Parameters: `-ServerSource -WorkerSource -ServiceName MxAccessGw -NssmPath -ServerDir -WorkerDir
-ExpectedSha -Label -HealthBaseUrl http://localhost:5130 -Rollback -BackupStamp -RestoreStores
-NoAutoRollback -TakeNewAppSettings -AllowRuntimeDowngrade -SkipFreeSpaceCheck -MinFreeGB -StopTimeoutSeconds
-StableSeconds -ReadyTimeoutSeconds -DryRun`. `ServerDir` defaults to NSSM `AppDirectory`, `WorkerDir`
to the parent of `MxGateway__Worker__ExecutablePath` read from the env block.
- **Process matching (trap 1).** Gateway processes are recognised by image name **and** by path. The names
are those of the installed exes (the leaf of NSSM `Application` and of `MxGateway__Worker__ExecutablePath`)
plus both naming schemes in the field: `ZB.MOM.WW.MxGateway.Server`/`.Worker` (windev) and
`MxGateway.Server`/`.Worker` (wonder boxes), so a stray gateway of the other scheme is caught too. The
process table comes from `Win32_Process` (path + parent PID).
- **The one-gateway guard fails closed.** With the service RUNNING it requires exactly one gateway Server, at
the `Application` path, whose parent is the NSSM service host PID. A RUNNING service whose gateway it cannot
find is an **abort**, not a pass (that is how the pre-#230 guard went blind). It also refuses: more than one
gateway Server; any gateway/Worker process while the service is STOPPED; a gateway while the service is in
another state; another service launching the same exe. It asserts no Server/Worker process exists right
before every `nssm start`.
- Preflight also refuses on: missing service, empty env block, a store path (or `Secrets__MasterKey__FilePath`)
that is relative or inside the Server/Worker dir, a `*.db`/`-wal`/`-shm` inside the Server/Worker dir, a
staged dir that does not contain the **installed** exe names (`Split-Path -Leaf` of `Application` and of the
Worker path), a staged Worker that is not x86, staged Server/Worker from different SHAs (or not matching
`-ExpectedSha`), too little disk, and a staged **self-contained runtime older than the live one** (compares
`mscordaccore_amd64_amd64_<ver>.dll`; `-AllowRuntimeDowngrade` overrides; see section 4). A master key that
is not in the env block (neither `ZB_SECRETS_MASTER_KEY` nor `Secrets__MasterKey__FilePath`, as on windev)
is a warning, since the upgrade does not change where it comes from.
- **Env block (trap 4): read, fingerprinted, never written.** The block is read from the service's registry
`Parameters\AppEnvironmentExtra` (what NSSM uses; `nssm get` is the fallback). The fingerprint is the entry
names plus a 12-hex SHA-256 prefix of each value, and the whole-block hash; values are never printed and
sensitive values are redacted from all output and notes. It is compared after the stop, after the swap,
before and after the start, and after the stability window; any added, removed or changed entry (or a
reorder) aborts and triggers the automatic rollback. On vd03 the pepper lives **only** in this block (#171),
so this check is what proves it survived.
- Stop waits `-StopTimeoutSeconds`; on STOP_PENDING (trap 8) it kills **this install's** Server, then its
Workers (path under the Server/Worker dir, or the installed image name when the path is unreadable; never by
a generic name), then the service host as a last resort. It then waits for every gateway/Worker process on
the box to be gone.
- Backup goes to `<install root>\_deploy-backup-<stamp>[-label]\`, restricted to SYSTEM + Administrators
(inheritance off) **before** anything is written; if `icacls` fails nothing is written. It holds
`AppEnvironmentExtra.txt` (the raw block: secrets), the service registry key export, `env-fingerprint.txt`
(names + hash prefixes only) and a copy of a file-based master key. The stores are only **copied** (`.db`,
`-wal`, `-shm` -> `*.bak-<stamp>`, hash-checked), never moved or deleted.
- Swap is **rename-in-place** (`Server` -> `Server.bak-<stamp>[-label]`, same for `Worker`), so NSSM
`Application`/`AppDirectory`/`ExecutablePath` never change. **Carried forward** from the live dir
(hash-verified, explicit ACLs copied): `appsettings*.json` (unless `-TakeNewAppSettings`) and certificates/keys
(`*.pfx`, `*.p12`, `*.pem`, `*.key`, `*.crt`, `*.cer`, e.g. vd03's `Server\dashboard.pfx`) always as the live
copy, plus every other live-only top-level file that is not a binary. Live-only binaries
(`.dll`/`.exe`/`.pdb`) are leftovers of the old build and stay in the `.bak`; live-only sub-directories are
listed as a warning. `data\` (the wonder store dir) sits beside `Server\`, outside the swap, and is not touched.
Preflight prints the carry plan.
- Verify: `/health/ready` = 200, `/healthz` = 200, `ProductVersion` SHA of the deployed Server exe against
`-ExpectedSha` (still subject to the version-stamp caveat in trap 7), exactly one gateway at the
`Application` path that is the service host's child, the same PID and SERVICE_RUNNING through
`-StableSeconds`, env fingerprint unchanged. Health with zero sessions does not prove the Worker path: do the
section 7.3 API-key session read yourself.
- A failed start or verify triggers an automatic rollback to the just-made backups (disable with
`-NoAutoRollback`); a failed backup restarts the unchanged build. Manual rollback:
`-Rollback [-BackupStamp <stamp>] [-RestoreStores]`. It accepts both backup spellings, `Server.bak-<stamp>`
(this script) and `Server.bak.<yyyyMMddTHHmmss>` (the adapted scripts used for #229 on vd03), and refuses
a Server backup without a Worker backup of the same stamp. It compares the env block to the saved
`env-fingerprint.txt` (section 8 step 4). The failed directories are kept as `*.failed-<stamp>`.
`-RestoreStores` restores the db triplet from the matching `*.bak-<stamp>` (stale wal/shm are moved aside,
nothing is deleted); use it only after a schema bump.
**Install script**: `-ServerDir -WorkerDir [-ServiceName -NssmPath -DataDir C:\ProgramData\MxGateway
-LogDir -EnvFile -Env -ObjectName -ServiceAccountPassword -EnvironmentName -RestartDelayMs -ThrottleMs
-StopGraceMs -RotateBytes -Start -DryRun]`. It refuses if the service exists or any gateway process is
running, requires the KEK and a `*Pepper*` env entry, requires store paths absolute and outside the content
root, and sets the env block in **one** `nssm set AppEnvironmentExtra` on a service that has none. Defaults
it applies: auto start, `AppExit Default Restart`, restart delay 5 s, throttle 10 s, console stop grace 20 s
(covers `Worker:ShutdownTimeoutSeconds` = 10), stdout/stderr to `<DataDir>\logs` with rotation,
`sc failure ... restart/5000/restart/5000/restart/30000`, reset 1 day. Secret values are masked in output.
The service is left stopped unless `-Start` is given.
**Driving from the Mac (winrm skill, deployment-only).** Remote execution on the wonder boxes is allowed
only for the on-box steps of a deploy the owner asked for. The intended flow, none of which was run for #220:
1. Build both artifacts on windev from one SHA (section 4); `smb push` them to a staging dir on the box
(UNC/SMB copies are always allowed), together with the two scripts.
2. `winrm exec` the script with `-DryRun` first, read the printed plan, then run it for real:
`powershell -NoProfile -ExecutionPolicy Bypass -File <stage>\Upgrade-MxGatewayService.ps1 -ServerSource <stage>\Server -WorkerSource <stage>\Worker -ExpectedSha <sha> -Label <label>`.
One box at a time; the exit code is 0 on success, 1 on any abort or failed run.
3. Do the section 7.3 Worker-path check from the Mac over gRPC (no remote exec needed), then write the deploy note.
## 0. What is being deployed
Two artifacts, deployed together because they share a wire protocol (`WorkerProtocolVersion`,
`appsettings.json` `MxGateway:Protocol`):
| Artifact | Build | Runs as |
|---|---|---|
| **Server** (x64 .NET 10, gRPC `:5120` h2c + Blazor dashboard `:5130`) | `dotnet publish src/ZB.MOM.WW.MxGateway.Server -c Release` (framework-dependent in the windev flow; self-contained is the alternative if the box lacks a .NET 10 runtime **[UNVERIFIED]**) | The NSSM service `MxAccessGw` |
| **Worker** (x86 .NET Framework 4.8, one process per gateway session, owns the MXAccess COM STA) | `dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86` (`CLAUDE.md`) - **Windows only; builds on windev**, not on macOS/Linux (`scripts/ci/README.md`) | Child of the Server, launched from `MxGateway:Worker:ExecutablePath`. **Not a service of its own.** |
The Server finds the Worker through `MxGateway:Worker:ExecutablePath`
(`docs/GatewayConfiguration.md`, default is a repo-relative `bin\x86\Release` path, so deployed hosts
override it via the NSSM env var `MxGateway__Worker__ExecutablePath`).
Service name `MxAccessGw` and the windev layout (`C:\publish\mxaccessgw\Server-YYYYMMDD`,
`C:\publish\mxaccessgw\Worker`, build clone `C:\build\mxaccessgw-ci`) are from
`docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md` Task "Deploy" and
`docs/runbooks/IdentifyingADeployedBuild.md`. The wonder boxes use `E:\ApiInstall\...` per
`scadaproj/deployment.md` §7.1; the exact `MxAccessGw` subfolder names there are **[UNVERIFIED]**.
Configuration is delivered as NSSM `AppEnvironmentExtra` environment variables
(`Kestrel__Endpoints__*`, `MxGateway__Authentication__SqlitePath`, `MxGateway__Worker__ExecutablePath`,
`ASPNETCORE_ENVIRONMENT`/`DOTNET_ENVIRONMENT`, `ZB_SECRETS_MASTER_KEY`, the API-key pepper,
`MxGateway__Ldap__ServiceAccountPassword`), not by editing `appsettings.json`
(`docs/GatewayConfiguration.md`, "Kestrel", "Secrets Master Key").
## 1. State that MUST survive the swap
The upgrade renames/replaces the application directory. Anything stored inside it is lost. The
validator therefore rejects stores inside the content root (see the 2026-08-09 incident below).
| State | Setting | Default location (Windows) | Notes |
|---|---|---|---|
| API-key store (SQLite) | `MxGateway:Authentication:SqlitePath` (NSSM: `MxGateway__Authentication__SqlitePath`) | `C:\ProgramData\MxGateway\gateway-auth.db` | Must be rooted; validator also rejects a path inside the content root. Back up the `-wal`/`-shm` siblings with it. |
| API-key **pepper** | secret `MxGateway:ApiKeyPepper` (`PepperSecretName`); lives outside the DB, in NSSM env / Secrets store | n/a | Without the same pepper every issued `mxgw_...` token fails verification. Never in the DB (`docs/Authentication.md`). Back it up in the org vault. On the wonder boxes it is the `MxGateway__ApiKeyPepper` env entry: QA vq03/vq04 since 2026-08-14, DEV vd03 since 2026-11-12 (moved out of `Server\appsettings.json`, mxaccessgw#171, `docs/deploy/2026-11-12-vd03-pepper-to-service-env.md`). |
| Secrets store (SQLite) | `Secrets:SqlitePath` (NSSM: `Secrets__SqlitePath`) | `C:\ProgramData\MxGateway\mxgateway-secrets.db` | **Must be absolute AND outside the content root**; fail-closed with no opt-out (`docs/GatewayConfiguration.md` L349; `scadaproj` Secrets notes). |
| Secrets master key (KEK) | `ZB_SECRETS_MASTER_KEY` in NSSM `AppEnvironmentExtra` | n/a | Losing it makes the store unrecoverable; no `RewrapAll` yet (`docs/GatewayConfiguration.md`, "Rotation / DR"). Back it up in the vault, not just on the box. |
| Galaxy snapshot cache | `MxGateway:Galaxy:SnapshotCachePath` | `C:\ProgramData\MxGateway\galaxy-snapshot.json` | Cache only; rejected inside content root so it survives deploys. Losing it is harmless. |
| TLS self-signed cert | `Tls:SelfSignedCertPath` | `C:\ProgramData\MxGateway\certs\gateway-selfsigned.pfx` | If lost, clients pinned to the old cert see a changed cert. |
| NSSM env block | `nssm get MxAccessGw AppEnvironmentExtra` | NSSM registry config | File copies do not touch it, but `nssm set AppEnvironmentExtra` **replaces the whole block**. |
| Local `appsettings*.json` overrides in the deploy dir | n/a | deploy dir | Copy the live ones aside and restore them after the swap (the windev plan copies `appsettings.json` / `appsettings.Development.json` from the current dir). |
**Incident that motivates the rule (2026-08-09, per `docs/GatewayConfiguration.md` L127):** the
API-key store lived inside the application directory; the upgrade renamed that directory to
`Server.bak.*`, silently starting an empty store, and no gRPC client could authenticate for two
days. The SQLite DB must also never be opened with a host-side `sqlite3` while the service runs
(copy the file triplet and query the copy; cross-repo lesson, **[UNVERIFIED from this repo]**).
## 2. Traps (read before touching anything)
1. **`OrphanWorkerTerminator` kills live workers if a second gateway starts.**
On startup the Server enumerates processes named after the configured Worker exe
(`ZB.MOM.WW.MxGateway.Worker` on windev, `MxGateway.Worker` on the wonder boxes) and kills every one whose path equals the configured
`ExecutablePath` - or whose path it cannot read, which is the normal case for an x86 process seen
from an x64 gateway, so it falls back to **image-name match only**
(`src/ZB.MOM.WW.MxGateway.Server/Workers/OrphanWorkerTerminator.cs`). It skips only its own PID's
children logic (`currentProcessId`), not another gateway's workers. Consequence: starting a
second gateway process on the same box (a manual `dotnet ...Server.dll` or `Server.exe` while the
service is up, a second NSSM service, a "test" start of the new build before stopping the old one,
or a rollback started while the new one still runs) **terminates every live session's Worker** and
all MXAccess subscriptions with it. Rule: **exactly one gateway process per box at any time**;
stop the service and confirm the process has exited before starting anything else. Never smoke-test
the new build by launching it beside the running service. (On QA, vq03 and vq04 are independent
boxes with independent gateways; this trap is per-box, not per-pair.)
2. **A gateway restart does not reattach workers** (`OrphanWorkerTerminator` summary). Plan for every
client session to drop and for clients to reopen sessions. Session-resilience reattach is opt-in
design only (`docs/plans/2026-06-15-session-resilience-design.md`); do not assume it is on.
3. **Secrets store path.** `Secrets:SqlitePath` must be absolute and outside the content root or the
host fails closed at startup. A store left at an in-tree path is exactly what the swap removes.
Also: the `secret` CLI and the gateway must use the same `Secrets__SqlitePath`, or the CLI seeds a
different store and the gateway then fails closed as if the secret were never set
(`docs/GatewayConfiguration.md` L381-395).
4. **`nssm set ... AppEnvironmentExtra` replaces the whole block.** `nssm get` first, save the output
(redact `Password=`/key values in anything you paste), then re-include every entry
(`docs/GatewayConfiguration.md` L370-374; `docs/runbooks/SEC-36-ldap-credential-rotation.md`).
The `nssm set MxAccessGw Application`/`AppDirectory` commands used for repointing do **not** touch
the env block.
5. **Worker bitness.** The Worker must be built x86 (`-p:Platform=x86`) and `RequiredArchitecture`
stays `X86`. The Server (x64) cannot load MXAccess COM. Build on windev (`ssh windev` flow,
`scripts/ci/windev-worker-ci.ps1`), not on a Mac. Deploying a Worker from a different commit than
the Server risks a protocol mismatch; build both from the same SHA.
6. **Environment name gates security hard-stops** (`docs/GatewayConfiguration.md` L317-335):
`Dashboard:DisableLogin` and plaintext LDAP are refused only when the environment name is exactly
`Production`. Do not change `ASPNETCORE_ENVIRONMENT`/`DOTNET_ENVIRONMENT` during an upgrade; a
host that boots under a different label silently loses those guards, and a host that was
permissive may refuse to boot if you "fix" the label.
7. **Version stamp.** Do not trust `InformationalVersion` for builds from 2026-07-09 to 2026-08-10
(`docs/runbooks/IdentifyingADeployedBuild.md`). Record the SHA yourself in the deploy notes.
8. **Stuck `STOP_PENDING`.** Stopping the service can hang in `STOP_PENDING` (workers must exit within
`Worker:ShutdownTimeoutSeconds`, default 10). Observed on the QA boxes per deploy notes
**[UNVERIFIED from this repo]**; if it hangs, wait, then kill the Server process by PID. Do not start the new
build until no gateway Server and no Worker process is left. Check **both** naming schemes:
`ZB.MOM.WW.MxGateway.Server`/`.Worker` (windev) and `MxGateway.Server`/`.Worker` (wonder boxes).
## 3. Preflight (no changes yet)
1. Decide the target SHA on `origin/main`; confirm CI is green for it (`windows-x86` job runs the
Worker build + `Worker.Tests` on windev; run the live tier on demand with
`CI_SHA=<sha> scripts/ci/run-windev-ci.sh live` or a `windev_mode: live` dispatch,
`scripts/ci/README.md`).
2. Read the current state of the service (read-only):
```text
nssm get MxAccessGw Application
nssm get MxAccessGw AppDirectory
nssm get MxAccessGw AppParameters
nssm get MxAccessGw AppEnvironmentExtra # save to the deploy notes; redact secrets
```
Note the entry count and names of the env block; verify the same count after the start.
3. Confirm the effective store paths from the env block: `MxGateway__Authentication__SqlitePath`,
`Secrets__SqlitePath`, `MxGateway__Worker__ExecutablePath`. If either store path is missing from
the env block it is on the code default (`C:\ProgramData\MxGateway\...`), which is fine; if it is
under the application directory, **stop and move it first** (see section 1).
4. Confirm the pepper and `ZB_SECRETS_MASTER_KEY` are present in the env block (names only).
5. Read the dashboard **Settings** page (`/settings` on the dashboard port, `:5130`) and save the
effective config; it now includes the heartbeat check interval, stuck ceiling, event-channel
full-mode timeout and replay byte cap. Compare after the upgrade to catch config drift.
6. Check the box is quiet: dashboard sessions list, or `GET /health/active` (the `mxaccess-sessions`
check reports `total/ready/faulted/starting/closing`). Agree a window with clients (OtOpcUa
`GalaxyDriver`, ScadaBridge MxGateway adapter) - their sessions will drop.
7. Check the new build's required config: diff the repo `src/ZB.MOM.WW.MxGateway.Server/appsettings.json`
and the changelog for new mandatory options, and `docs/GatewayConfiguration.md` for validator
changes (the validator can refuse to boot on a bad value, which is a rollback trigger).
8. Free disk for a full copy of the deploy dir and the stores (see backup).
9. Check auth-store schema compatibility: the Server migrates the API-key DB on start
(`Authentication:RunMigrationsOnStartup: true`) via the `ZB.MOM.WW.Auth` package. If the new build
bumps the Auth package, the schema can migrate forward and **the old binary may not read it back**
(see the 2026-08-07 follow-up plan's `AuthStoreMigrationException` check). Treat the DB backup as
the rollback for a schema bump.
## 4. Build (on windev; no deploy yet)
windev flow from `docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md`:
```powershell
git -C C:\build\mxaccessgw-ci fetch origin
git -C C:\build\mxaccessgw-ci worktree add C:\build\mxgw-deploy <sha> # detached worktree, host checkout untouched
cd C:\build\mxgw-deploy
dotnet publish src\ZB.MOM.WW.MxGateway.Server -c Release -o C:\publish\mxaccessgw\Server-YYYYMMDD
dotnet build src\ZB.MOM.WW.MxGateway.Worker\ZB.MOM.WW.MxGateway.Worker.csproj -c Release -p:Platform=x86
# stage the Worker output (bin\x86\Release) into a temp dir, e.g. C:\publish\mxaccessgw\Worker-new
```
Remove the worktree afterwards (`git worktree remove`); an absent `C:\build\mxgw-deploy` is the
normal end state (`IdentifyingADeployedBuild.md`). Whether the Worker is built with `dotnet build`
or `dotnet publish`, and its exact output folder, should be matched to the existing `Worker` folder on
the host **[UNVERIFIED]**. Note: the Worker is net48 and must be built where the net48 targeting
pack and MSBuild are present (windev; `scadaproj` notes say Build Tools 2022 + net48 reference
assemblies verified 2026-09-10 **[UNVERIFIED from this repo]**).
**Self-contained runtime.** A self-contained Server carries its own .NET runtime, so the SDK that builds it
decides the runtime on the box. On 2026-11-12 the windev SDK (10.0.204) produced runtime 10.0.8 while vd03 ran
10.0.14, which would have been a downgrade; the Mac SDK (10.0.403) produced 10.0.14. Build where the SDK carries
the newest runtime. The upgrade script refuses a downgrade (it compares `mscordaccore_amd64_amd64_<ver>.dll`
between the live and staged dirs). The wonder boxes use the `MxGateway.Server.exe` / `MxGateway.Worker.exe`
names: stage a byte copy of the apphost under that name and rename the Worker apphost and its `.config`
(`docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md`).
For the wonder boxes the staged output is copied over SMB (UNC copy is always allowed) into a
staging directory such as `E:\psrem-stage\<app>-<sha>\` per `scadaproj/deployment.md` §7.1
(**[UNVERIFIED]** path convention); the on-box steps below are then the only remote-exec part.
## 5. Backup
With the service **still running** copy nothing live; do these after stopping (section 6 step 1), or
copy the stores with the service stopped. Back up:
1. The current deploy directory (`Server-YYYYMMDD` or `Server`) to a sibling `Server.bak.<timestamp>`
(the convention `IdentifyingADeployedBuild.md` reads as a deploy history chain), and the current
`Worker` directory to `Worker.bak-<timestamp>[-label]`.
2. `gateway-auth.db` plus `gateway-auth.db-wal` / `-shm` if present, and `mxgateway-secrets.db` (plus
its `-wal`/`-shm`), to `*.bak-<label>` next to the originals. **Never delete or move the live DB.**
3. The saved `AppEnvironmentExtra` text and the `nssm get Application/AppDirectory` values (for rollback
repointing).
4. Off-box: the KEK and pepper in the org vault (not on the box only).
5. Record the old and new SHA, timestamps and backup paths in a `_DEPLOY-NOTES-<timestamp>.txt`
next to the install (the convention seen on wonder per `deployment.md` §7.1, **[UNVERIFIED]**).
## 6. Stop / swap / start
Do this on **one box at a time**; QA vq03 and vq04 are independent gateways, so roll them one after
the other (`deployment.md` §7.1 records the 2026-09-10 round as "rolling, one stop per box").
1. **Stop and confirm gone.**
```powershell
nssm stop MxAccessGw # or Stop-Service MxAccessGw -Force
Get-Process ZB.MOM.WW.MxGateway.Server, ZB.MOM.WW.MxGateway.Worker, MxGateway.Server, MxGateway.Worker -ErrorAction SilentlyContinue
```
Both lists must be empty before continuing (traps 1 and 8). If a Worker is left over, the next
start will kill it as an orphan (by design) - but it signals an unclean stop worth noting.
2. **Backup** (section 5) now that nothing holds the stores open.
3. **Swap the Server.** Preferred: deploy to a *new dated directory* and repoint NSSM, which leaves the
old directory intact as the rollback target:
```powershell
Copy-Item <old>\appsettings.json, <old>\appsettings.Development.json <new>\ # preserve live file config
nssm set MxAccessGw Application <new>\ZB.MOM.WW.MxGateway.Server.exe
nssm set MxAccessGw AppDirectory <new>
```
(`2026-10-19-ackmsg-as-operator.md`; `IdentifyingADeployedBuild.md` "Deployment-side naming".)
Alternative used on the wonder boxes: rename `app` to `app.bak.<timestamp>` and copy the new files
into `app` (`deployment.md` §7.1; **[UNVERIFIED]**). Either way, restore any host-specific
`appsettings*.json` byte-identical; do not edit the NSSM env block unless the release notes require it.
4. **Swap the Worker.** Rename `Worker` to `Worker.bak-<timestamp>`, copy the new x86 Worker build in
its place, and make sure `MxGateway__Worker__ExecutablePath` points at it
(the path is matched case-insensitively by the orphan terminator, so keep one canonical spelling).
5. **Verify the env block is untouched**: `nssm get MxAccessGw AppEnvironmentExtra` entry count equals the
preflight count.
6. **Start** (`nssm start MxAccessGw`). Exactly once; do not retry-start concurrently with anything else.
## 7. Post-deploy verification
All of this can be done over HTTP/gRPC from another machine or from the box without further remote
exec beyond the deploy window.
1. Service `RUNNING` and stable for at least 60 s (no restart loop); the Application event log has a
clean host start and **no** new `AuthStoreMigrationException` / `Hosting failed to start` after the
start time (`2026-08-07-followups-windev-ldapfixtures-runners.md` Task 2).
2. Health: `GET http://<host>:5130/healthz` (live), `/health/ready` (`auth-store` check),
`/health/active` (`mxaccess-sessions`). These endpoints are anonymous by design
(`GatewayApplication.cs`, `docs/Diagnostics.md`). Zero sessions is Healthy, so green does not prove
the Worker/COM path works.
3. Prove the **Worker** path, since an idle health probe does not: open a gRPC session with an existing
API key (any of the clients, or the `mxa` CLI) and do a read of a known tag. This also proves the
API-key store and pepper survived: a `mxgw_...` token issued before the upgrade must authenticate.
Negative control: a wrong token must be rejected (`UNAUTHENTICATED`).
4. Dashboard: login works (or the localhost anonymous policy applies), `/settings` matches the
preflight capture, the version/Worker protocol line is right.
5. Identify the deployed build by behaviour, not the version stamp where it is unreliable
(`IdentifyingADeployedBuild.md`): a plain `Write` returning `statuses[0]` proves a Worker at or
past `53f69cd`.
6. Check the Server log for `Terminated orphan worker process` - expected only if the previous stop was
unclean.
7. Check downstream clients reconnected (OtOpcUa `GalaxyDriver`; ScadaBridge MxGateway adapter) and, on
a pair of QA boxes, repeat sections 5-7 for the second box.
8. Write the deploy note: SHA, backup paths, env-entry count, evidence for each check above.
## 8. Rollback
Triggers: service will not stay `RUNNING`, host fails validator at start, health red beyond the grace
window, API keys no longer authenticate, or a Worker session cannot be opened.
1. `nssm stop MxAccessGw`; confirm no Server/Worker process (trap 1: **never start the old build while
the new one is still up**).
2. Repoint: `nssm set MxAccessGw Application <old>\...Server.exe` and `AppDirectory <old>`, or rename
`app.bak.<ts>` back to `app`, and restore `Worker.bak-<ts>` over `Worker`.
3. If the API-key DB was migrated (schema bump, trap in section 3 step 9) or damaged, restore
`gateway-auth.db` from the backup (restore the `-wal`/`-shm` together or none). Otherwise leave the
live store alone: the env-block stores are outside the swap and need no restore.
4. `nssm get AppEnvironmentExtra` entry count equals the preflight count; if you changed it, restore the
saved block (all entries).
5. `nssm start MxAccessGw`; repeat section 7 steps 1-3.
6. Report: what failed, the log excerpt, backup paths used.
## 9. Open gaps (not verifiable from this repo)
- Install layout on the wonder boxes, confirmed on vd03 (2026-11-12, #229/#230): `E:\ApiInstall\MxGateway\Server\MxGateway.Server.exe`
(NSSM `Application`), `Worker\MxGateway.Worker.exe` (`MxGateway__Worker__ExecutablePath`), stores and `kek.key` in
`E:\ApiInstall\MxGateway\data\`, `Server\dashboard.pfx`, master key `Secrets__MasterKey__Source=File`. windev:
`C:\publish\mxaccessgw\Server-<date>-<sha>\ZB.MOM.WW.MxGateway.Server.exe` and `C:\publish\mxaccessgw\Worker\`, stores in
`C:\ProgramData\MxGateway\`. vq03/vq04 are assumed to match vd03; the dry run shows it.
- Whether the Server on each box is framework-dependent or self-contained (a box without .NET 10
needs self-contained; the 2026-10-19 windev plan says framework-dependent).
- The `STOP_PENDING` behaviour and the NSSM service account (`LocalSystem` per `deployment.md`).
- New-box install and the upgrade are now scripted (see "Scripts"), but **untested on a real box**; the
recovery/stop-grace/log-rotation defaults in `Install-MxGatewayService.ps1` are chosen from the repo's
own settings (`Worker:ShutdownTimeoutSeconds`), not copied from an existing box. Compare against the
`nssm dump` of a known-good box (vd03/vq03) before relying on them.
- The Worker output folder and build-vs-publish question (section 4) is still **[UNVERIFIED]**: the upgrade
script copies whatever the staged `-WorkerSource` directory contains over `Worker`, so stage exactly what
the existing `Worker` folder holds. Confirm with a file listing of the live `Worker` dir on the next deploy.