Files
mxaccessgw/docs/runbooks/DeployUpgradeNssmService.md
T
Joseph Doherty 477fa55998
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m26s
ci / java (push) Successful in 2m12s
ci / portable (push) Failing after 4m11s
feat(deploy): on-box NSSM upgrade/rollback and new-box install scripts, runbook updated (mxaccessgw#220)
Add scripts/deploy/Upgrade-MxGatewayService.ps1 (preflight, stop with STOP_PENDING handling,
backup, Server+Worker swap, start, /health/ready + SHA verify, auto/manual rollback,
one-gateway guard, never touches the NSSM env block) and Install-MxGatewayService.ps1
(service install, restart/recovery settings, env block). Both support -DryRun/-WhatIf.
Dry-run validated on macOS pwsh against a fake layout; untested on a real box.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 15:10:26 -05:00

26 KiB

Deploy / Upgrade Runbook: NSSM-wrapped Gateway Service (Server + x86 Worker)

Written for mxaccessgw#207 (archreview R21-O31 / cycle-1 TST-10). Since mxaccessgw#220 the procedure is also scripted (see "Scripts" below); this runbook remains the spec the scripts implement. It is assembled only from what the repo contains (docs/GatewayConfiguration.md, docs/Authentication.md, docs/Diagnostics.md, docs/runbooks/*, docs/plans/2026-08-07-*, docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md, src/.../appsettings.json, scripts/ci/) cross-checked against scadaproj/deployment.md §5 and §7.1.

Verification legend. Statements are either backed by a cited repo file, or tagged [UNVERIFIED] (comes from deploy notes / other repos / memory and could not be confirmed from this repo). Treat every [UNVERIFIED] item as "confirm on the box before relying on it".

Scope of remote execution. On the wonder boxes (wonder-app-vd03 DEV, wonder-app-vq03/vq04 QA) remote command execution is permitted for the on-box steps of a deploy the owner has asked for, and nothing else (CLAUDE.md "Wonder box remote access" note; the umbrella scadaproj index carries the current rule). No ad-hoc diagnostics over remote exec: use SMB file reads, HTTP probes and direct SQL. Confirm the current rule before starting; this repo's note is dated 2026-09-11 and the umbrella index has since relaxed it to deploy-only [UNVERIFIED from this repo].

Scripts (mxaccessgw#220)

UNTESTED ON A REAL BOX until the next deploy. The scripts were only exercised on a Mac with pwsh in -DryRun mode against a fake directory layout and a fake nssm (parsing, preflight refusals, the one-gateway guard, the step sequence). Service stop/start, STOP_PENDING handling, the health/SHA verification, sc.exe failure, the rollback swaps and the NSSM UTF-16 output handling have not run against real Windows/NSSM. Run the first real use with -DryRun on the box, read the plan, and have section 8 ready by hand.

Script Where it runs Does
scripts/deploy/Upgrade-MxGatewayService.ps1 On the Windows box, Windows PowerShell 5.1 Sections 3, 5, 6, 7, 8: preflight, stop, backup, swap Server + Worker, start, verify, rollback
scripts/deploy/Install-MxGatewayService.ps1 On a new box NSSM service install: recovery/restart, stop grace, log rotation, env block

Both take -DryRun (or -WhatIf): read-only probes (nssm get/status, process list, disk, file existence) still run, every mutating action is printed instead of executed.

Upgrade script (-ServerSource, -WorkerSource are the staged outputs from section 4, built from one SHA):

  • Parameters: -ServerSource -WorkerSource -ServiceName MxAccessGw -NssmPath -ServerDir -WorkerDir -ExpectedSha -Label -HealthBaseUrl http://localhost:5130 -Rollback -BackupStamp -RestoreStores -NoAutoRollback -TakeNewAppSettings -SkipFreeSpaceCheck -MinFreeGB -StopTimeoutSeconds -StableSeconds -ReadyTimeoutSeconds -DryRun. ServerDir defaults to NSSM AppDirectory, WorkerDir to the parent of MxGateway__Worker__ExecutablePath read from the env block.
  • Preflight refuses on: missing service, empty env block, no ZB_SECRETS_MASTER_KEY, a store path that is relative or inside the Server/Worker dir, staged Server/Worker built from different SHAs (or not matching -ExpectedSha), too little disk, more than one gateway Server process, a gateway process while the service is STOPPED, or another service launching the same exe (trap 1). It also asserts no Server/Worker process exists immediately before every nssm start.
  • Stop waits -StopTimeoutSeconds; on STOP_PENDING (trap 8) it kills the Server then Worker processes by image name (safe because preflight proved a single gateway), then the service host as a last resort.
  • Swap is rename-in-place (Server -> Server.bak-<stamp>[-label], same for Worker), so NSSM Application/AppDirectory/ExecutablePath never change. Live appsettings*.json are copied back byte-identical. The stores are only copied (.db, -wal, -shm -> *.bak-<stamp>), never moved or deleted. The script never runs nssm set AppEnvironmentExtra; it saves the block (secrets included, ACL-restricted via icacls) to _env-AppEnvironmentExtra-<stamp>.txt and compares the entry count after start. Notes (names only, no values) go to _DEPLOY-NOTES-<stamp>.txt beside the install.
  • Verify: /health/ready = 200, /healthz, ProductVersion SHA of the deployed Server exe against -ExpectedSha (still subject to the version-stamp caveat in trap 7), service stays RUNNING for -StableSeconds, env count unchanged. Health with zero sessions does not prove the Worker path: do the section 7.3 API-key session read yourself.
  • A failed start or verify triggers an automatic rollback to the just-made backups (disable with -NoAutoRollback). Manual rollback: -Rollback [-BackupStamp <stamp>] [-RestoreStores]; the failed directories are kept as *.failed-<stamp>. -RestoreStores restores the db triplet from the matching *.bak-<stamp> (stale wal/shm are moved aside, nothing is deleted); use it only after a schema bump.

Install script: -ServerDir -WorkerDir [-ServiceName -NssmPath -DataDir C:\ProgramData\MxGateway -LogDir -EnvFile -Env -ObjectName -ServiceAccountPassword -EnvironmentName -RestartDelayMs -ThrottleMs -StopGraceMs -RotateBytes -Start -DryRun]. It refuses if the service exists or any gateway process is running, requires the KEK and a *Pepper* env entry, requires store paths absolute and outside the content root, and sets the env block in one nssm set AppEnvironmentExtra on a service that has none. Defaults it applies: auto start, AppExit Default Restart, restart delay 5 s, throttle 10 s, console stop grace 20 s (covers Worker:ShutdownTimeoutSeconds = 10), stdout/stderr to <DataDir>\logs with rotation, sc failure ... restart/5000/restart/5000/restart/30000, reset 1 day. Secret values are masked in output. The service is left stopped unless -Start is given.

Driving from the Mac (winrm skill, deployment-only). Remote execution on the wonder boxes is allowed only for the on-box steps of a deploy the owner asked for. The intended flow, none of which was run for #220:

  1. Build both artifacts on windev from one SHA (section 4); smb push them to a staging dir on the box (UNC/SMB copies are always allowed), together with the two scripts.
  2. winrm exec the script with -DryRun first, read the printed plan, then run it for real: powershell -NoProfile -ExecutionPolicy Bypass -File <stage>\Upgrade-MxGatewayService.ps1 -ServerSource <stage>\Server -WorkerSource <stage>\Worker -ExpectedSha <sha> -Label <label>. One box at a time; the exit code is 0 on success, 1 on any abort or failed run.
  3. Do the section 7.3 Worker-path check from the Mac over gRPC (no remote exec needed), then write the deploy note.

0. What is being deployed

Two artifacts, deployed together because they share a wire protocol (WorkerProtocolVersion, appsettings.json MxGateway:Protocol):

Artifact Build Runs as
Server (x64 .NET 10, gRPC :5120 h2c + Blazor dashboard :5130) dotnet publish src/ZB.MOM.WW.MxGateway.Server -c Release (framework-dependent in the windev flow; self-contained is the alternative if the box lacks a .NET 10 runtime [UNVERIFIED]) The NSSM service MxAccessGw
Worker (x86 .NET Framework 4.8, one process per gateway session, owns the MXAccess COM STA) dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86 (CLAUDE.md) - Windows only; builds on windev, not on macOS/Linux (scripts/ci/README.md) Child of the Server, launched from MxGateway:Worker:ExecutablePath. Not a service of its own.

The Server finds the Worker through MxGateway:Worker:ExecutablePath (docs/GatewayConfiguration.md, default is a repo-relative bin\x86\Release path, so deployed hosts override it via the NSSM env var MxGateway__Worker__ExecutablePath).

Service name MxAccessGw and the windev layout (C:\publish\mxaccessgw\Server-YYYYMMDD, C:\publish\mxaccessgw\Worker, build clone C:\build\mxaccessgw-ci) are from docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md Task "Deploy" and docs/runbooks/IdentifyingADeployedBuild.md. The wonder boxes use E:\ApiInstall\... per scadaproj/deployment.md §7.1; the exact MxAccessGw subfolder names there are [UNVERIFIED].

Configuration is delivered as NSSM AppEnvironmentExtra environment variables (Kestrel__Endpoints__*, MxGateway__Authentication__SqlitePath, MxGateway__Worker__ExecutablePath, ASPNETCORE_ENVIRONMENT/DOTNET_ENVIRONMENT, ZB_SECRETS_MASTER_KEY, the API-key pepper, MxGateway__Ldap__ServiceAccountPassword), not by editing appsettings.json (docs/GatewayConfiguration.md, "Kestrel", "Secrets Master Key").

1. State that MUST survive the swap

The upgrade renames/replaces the application directory. Anything stored inside it is lost. The validator therefore rejects stores inside the content root (see the 2026-08-09 incident below).

State Setting Default location (Windows) Notes
API-key store (SQLite) MxGateway:Authentication:SqlitePath (NSSM: MxGateway__Authentication__SqlitePath) C:\ProgramData\MxGateway\gateway-auth.db Must be rooted; validator also rejects a path inside the content root. Back up the -wal/-shm siblings with it.
API-key pepper secret MxGateway:ApiKeyPepper (PepperSecretName); lives outside the DB, in NSSM env / Secrets store n/a Without the same pepper every issued mxgw_... token fails verification. Never in the DB (docs/Authentication.md). Back it up in the org vault.
Secrets store (SQLite) Secrets:SqlitePath (NSSM: Secrets__SqlitePath) C:\ProgramData\MxGateway\mxgateway-secrets.db Must be absolute AND outside the content root; fail-closed with no opt-out (docs/GatewayConfiguration.md L349; scadaproj Secrets notes).
Secrets master key (KEK) ZB_SECRETS_MASTER_KEY in NSSM AppEnvironmentExtra n/a Losing it makes the store unrecoverable; no RewrapAll yet (docs/GatewayConfiguration.md, "Rotation / DR"). Back it up in the vault, not just on the box.
Galaxy snapshot cache MxGateway:Galaxy:SnapshotCachePath C:\ProgramData\MxGateway\galaxy-snapshot.json Cache only; rejected inside content root so it survives deploys. Losing it is harmless.
TLS self-signed cert Tls:SelfSignedCertPath C:\ProgramData\MxGateway\certs\gateway-selfsigned.pfx If lost, clients pinned to the old cert see a changed cert.
NSSM env block nssm get MxAccessGw AppEnvironmentExtra NSSM registry config File copies do not touch it, but nssm set AppEnvironmentExtra replaces the whole block.
Local appsettings*.json overrides in the deploy dir n/a deploy dir Copy the live ones aside and restore them after the swap (the windev plan copies appsettings.json / appsettings.Development.json from the current dir).

Incident that motivates the rule (2026-08-09, per docs/GatewayConfiguration.md L127): the API-key store lived inside the application directory; the upgrade renamed that directory to Server.bak.*, silently starting an empty store, and no gRPC client could authenticate for two days. The SQLite DB must also never be opened with a host-side sqlite3 while the service runs (copy the file triplet and query the copy; cross-repo lesson, [UNVERIFIED from this repo]).

2. Traps (read before touching anything)

  1. OrphanWorkerTerminator kills live workers if a second gateway starts. On startup the Server enumerates processes named after the configured Worker exe (ZB.MOM.WW.MxGateway.Worker) and kills every one whose path equals the configured ExecutablePath - or whose path it cannot read, which is the normal case for an x86 process seen from an x64 gateway, so it falls back to image-name match only (src/ZB.MOM.WW.MxGateway.Server/Workers/OrphanWorkerTerminator.cs). It skips only its own PID's children logic (currentProcessId), not another gateway's workers. Consequence: starting a second gateway process on the same box (a manual dotnet ...Server.dll or Server.exe while the service is up, a second NSSM service, a "test" start of the new build before stopping the old one, or a rollback started while the new one still runs) terminates every live session's Worker and all MXAccess subscriptions with it. Rule: exactly one gateway process per box at any time; stop the service and confirm the process has exited before starting anything else. Never smoke-test the new build by launching it beside the running service. (On QA, vq03 and vq04 are independent boxes with independent gateways; this trap is per-box, not per-pair.)
  2. A gateway restart does not reattach workers (OrphanWorkerTerminator summary). Plan for every client session to drop and for clients to reopen sessions. Session-resilience reattach is opt-in design only (docs/plans/2026-06-15-session-resilience-design.md); do not assume it is on.
  3. Secrets store path. Secrets:SqlitePath must be absolute and outside the content root or the host fails closed at startup. A store left at an in-tree path is exactly what the swap removes. Also: the secret CLI and the gateway must use the same Secrets__SqlitePath, or the CLI seeds a different store and the gateway then fails closed as if the secret were never set (docs/GatewayConfiguration.md L381-395).
  4. nssm set ... AppEnvironmentExtra replaces the whole block. nssm get first, save the output (redact Password=/key values in anything you paste), then re-include every entry (docs/GatewayConfiguration.md L370-374; docs/runbooks/SEC-36-ldap-credential-rotation.md). The nssm set MxAccessGw Application/AppDirectory commands used for repointing do not touch the env block.
  5. Worker bitness. The Worker must be built x86 (-p:Platform=x86) and RequiredArchitecture stays X86. The Server (x64) cannot load MXAccess COM. Build on windev (ssh windev flow, scripts/ci/windev-worker-ci.ps1), not on a Mac. Deploying a Worker from a different commit than the Server risks a protocol mismatch; build both from the same SHA.
  6. Environment name gates security hard-stops (docs/GatewayConfiguration.md L317-335): Dashboard:DisableLogin and plaintext LDAP are refused only when the environment name is exactly Production. Do not change ASPNETCORE_ENVIRONMENT/DOTNET_ENVIRONMENT during an upgrade; a host that boots under a different label silently loses those guards, and a host that was permissive may refuse to boot if you "fix" the label.
  7. Version stamp. Do not trust InformationalVersion for builds from 2026-07-09 to 2026-08-10 (docs/runbooks/IdentifyingADeployedBuild.md). Record the SHA yourself in the deploy notes.
  8. Stuck STOP_PENDING. Stopping the service can hang in STOP_PENDING (workers must exit within Worker:ShutdownTimeoutSeconds, default 10). Observed on the QA boxes per deploy notes [UNVERIFIED from this repo]; if it hangs, wait, then kill the Server process by PID. Do not start the new build until Get-Process shows no ZB.MOM.WW.MxGateway.Server and no ZB.MOM.WW.MxGateway.Worker.

3. Preflight (no changes yet)

  1. Decide the target SHA on origin/main; confirm CI is green for it (windows-x86 job runs the Worker build + Worker.Tests on windev; nightly-windev runs the live tier, scripts/ci/README.md).
  2. Read the current state of the service (read-only):
    nssm get MxAccessGw Application
    nssm get MxAccessGw AppDirectory
    nssm get MxAccessGw AppParameters
    nssm get MxAccessGw AppEnvironmentExtra        # save to the deploy notes; redact secrets
    
    Note the entry count and names of the env block; verify the same count after the start.
  3. Confirm the effective store paths from the env block: MxGateway__Authentication__SqlitePath, Secrets__SqlitePath, MxGateway__Worker__ExecutablePath. If either store path is missing from the env block it is on the code default (C:\ProgramData\MxGateway\...), which is fine; if it is under the application directory, stop and move it first (see section 1).
  4. Confirm the pepper and ZB_SECRETS_MASTER_KEY are present in the env block (names only).
  5. Read the dashboard Settings page (/settings on the dashboard port, :5130) and save the effective config; it now includes the heartbeat check interval, stuck ceiling, event-channel full-mode timeout and replay byte cap. Compare after the upgrade to catch config drift.
  6. Check the box is quiet: dashboard sessions list, or GET /health/active (the mxaccess-sessions check reports total/ready/faulted/starting/closing). Agree a window with clients (OtOpcUa GalaxyDriver, ScadaBridge MxGateway adapter) - their sessions will drop.
  7. Check the new build's required config: diff the repo src/ZB.MOM.WW.MxGateway.Server/appsettings.json and the changelog for new mandatory options, and docs/GatewayConfiguration.md for validator changes (the validator can refuse to boot on a bad value, which is a rollback trigger).
  8. Free disk for a full copy of the deploy dir and the stores (see backup).
  9. Check auth-store schema compatibility: the Server migrates the API-key DB on start (Authentication:RunMigrationsOnStartup: true) via the ZB.MOM.WW.Auth package. If the new build bumps the Auth package, the schema can migrate forward and the old binary may not read it back (see the 2026-08-07 follow-up plan's AuthStoreMigrationException check). Treat the DB backup as the rollback for a schema bump.

4. Build (on windev; no deploy yet)

windev flow from docs/superpowers/plans/2026-10-19-ackmsg-as-operator.md:

git -C C:\build\mxaccessgw-ci fetch origin
git -C C:\build\mxaccessgw-ci worktree add C:\build\mxgw-deploy <sha>      # detached worktree, host checkout untouched
cd C:\build\mxgw-deploy
dotnet publish src\ZB.MOM.WW.MxGateway.Server -c Release -o C:\publish\mxaccessgw\Server-YYYYMMDD
dotnet build   src\ZB.MOM.WW.MxGateway.Worker\ZB.MOM.WW.MxGateway.Worker.csproj -c Release -p:Platform=x86
# stage the Worker output (bin\x86\Release) into a temp dir, e.g. C:\publish\mxaccessgw\Worker-new

Remove the worktree afterwards (git worktree remove); an absent C:\build\mxgw-deploy is the normal end state (IdentifyingADeployedBuild.md). Whether the Worker is built with dotnet build or dotnet publish, and its exact output folder, should be matched to the existing Worker folder on the host [UNVERIFIED]. Note: the Worker is net48 and must be built where the net48 targeting pack and MSBuild are present (windev; scadaproj notes say Build Tools 2022 + net48 reference assemblies verified 2026-09-10 [UNVERIFIED from this repo]).

For the wonder boxes the staged output is copied over SMB (UNC copy is always allowed) into a staging directory such as E:\psrem-stage\<app>-<sha>\ per scadaproj/deployment.md §7.1 ([UNVERIFIED] path convention); the on-box steps below are then the only remote-exec part.

5. Backup

With the service still running copy nothing live; do these after stopping (section 6 step 1), or copy the stores with the service stopped. Back up:

  1. The current deploy directory (Server-YYYYMMDD or Server) to a sibling Server.bak.<timestamp> (the convention IdentifyingADeployedBuild.md reads as a deploy history chain), and the current Worker directory to Worker.bak-<timestamp>[-label].
  2. gateway-auth.db plus gateway-auth.db-wal / -shm if present, and mxgateway-secrets.db (plus its -wal/-shm), to *.bak-<label> next to the originals. Never delete or move the live DB.
  3. The saved AppEnvironmentExtra text and the nssm get Application/AppDirectory values (for rollback repointing).
  4. Off-box: the KEK and pepper in the org vault (not on the box only).
  5. Record the old and new SHA, timestamps and backup paths in a _DEPLOY-NOTES-<timestamp>.txt next to the install (the convention seen on wonder per deployment.md §7.1, [UNVERIFIED]).

6. Stop / swap / start

Do this on one box at a time; QA vq03 and vq04 are independent gateways, so roll them one after the other (deployment.md §7.1 records the 2026-09-10 round as "rolling, one stop per box").

  1. Stop and confirm gone.
    nssm stop MxAccessGw           # or Stop-Service MxAccessGw -Force
    Get-Process ZB.MOM.WW.MxGateway.Server, ZB.MOM.WW.MxGateway.Worker -ErrorAction SilentlyContinue
    
    Both lists must be empty before continuing (traps 1 and 8). If a Worker is left over, the next start will kill it as an orphan (by design) - but it signals an unclean stop worth noting.
  2. Backup (section 5) now that nothing holds the stores open.
  3. Swap the Server. Preferred: deploy to a new dated directory and repoint NSSM, which leaves the old directory intact as the rollback target:
    Copy-Item <old>\appsettings.json, <old>\appsettings.Development.json <new>\   # preserve live file config
    nssm set MxAccessGw Application   <new>\ZB.MOM.WW.MxGateway.Server.exe
    nssm set MxAccessGw AppDirectory  <new>
    
    (2026-10-19-ackmsg-as-operator.md; IdentifyingADeployedBuild.md "Deployment-side naming".) Alternative used on the wonder boxes: rename app to app.bak.<timestamp> and copy the new files into app (deployment.md §7.1; [UNVERIFIED]). Either way, restore any host-specific appsettings*.json byte-identical; do not edit the NSSM env block unless the release notes require it.
  4. Swap the Worker. Rename Worker to Worker.bak-<timestamp>, copy the new x86 Worker build in its place, and make sure MxGateway__Worker__ExecutablePath points at it (the path is matched case-insensitively by the orphan terminator, so keep one canonical spelling).
  5. Verify the env block is untouched: nssm get MxAccessGw AppEnvironmentExtra entry count equals the preflight count.
  6. Start (nssm start MxAccessGw). Exactly once; do not retry-start concurrently with anything else.

7. Post-deploy verification

All of this can be done over HTTP/gRPC from another machine or from the box without further remote exec beyond the deploy window.

  1. Service RUNNING and stable for at least 60 s (no restart loop); the Application event log has a clean host start and no new AuthStoreMigrationException / Hosting failed to start after the start time (2026-08-07-followups-windev-ldapfixtures-runners.md Task 2).
  2. Health: GET http://<host>:5130/healthz (live), /health/ready (auth-store check), /health/active (mxaccess-sessions). These endpoints are anonymous by design (GatewayApplication.cs, docs/Diagnostics.md). Zero sessions is Healthy, so green does not prove the Worker/COM path works.
  3. Prove the Worker path, since an idle health probe does not: open a gRPC session with an existing API key (any of the clients, or the mxa CLI) and do a read of a known tag. This also proves the API-key store and pepper survived: a mxgw_... token issued before the upgrade must authenticate. Negative control: a wrong token must be rejected (UNAUTHENTICATED).
  4. Dashboard: login works (or the localhost anonymous policy applies), /settings matches the preflight capture, the version/Worker protocol line is right.
  5. Identify the deployed build by behaviour, not the version stamp where it is unreliable (IdentifyingADeployedBuild.md): a plain Write returning statuses[0] proves a Worker at or past 53f69cd.
  6. Check the Server log for Terminated orphan worker process - expected only if the previous stop was unclean.
  7. Check downstream clients reconnected (OtOpcUa GalaxyDriver; ScadaBridge MxGateway adapter) and, on a pair of QA boxes, repeat sections 5-7 for the second box.
  8. Write the deploy note: SHA, backup paths, env-entry count, evidence for each check above.

8. Rollback

Triggers: service will not stay RUNNING, host fails validator at start, health red beyond the grace window, API keys no longer authenticate, or a Worker session cannot be opened.

  1. nssm stop MxAccessGw; confirm no Server/Worker process (trap 1: never start the old build while the new one is still up).
  2. Repoint: nssm set MxAccessGw Application <old>\...Server.exe and AppDirectory <old>, or rename app.bak.<ts> back to app, and restore Worker.bak-<ts> over Worker.
  3. If the API-key DB was migrated (schema bump, trap in section 3 step 9) or damaged, restore gateway-auth.db from the backup (restore the -wal/-shm together or none). Otherwise leave the live store alone: the env-block stores are outside the swap and need no restore.
  4. nssm get AppEnvironmentExtra entry count equals the preflight count; if you changed it, restore the saved block (all entries).
  5. nssm start MxAccessGw; repeat section 7 steps 1-3.
  6. Report: what failed, the log excerpt, backup paths used.

9. Open gaps (not verifiable from this repo)

  • Exact install paths on the wonder boxes (E:\ApiInstall\MxAccessGw\...), the staging convention and the _DEPLOY-NOTES file format - taken from scadaproj/deployment.md, whose live content could not be cross-checked here.
  • Whether the Server on each box is framework-dependent or self-contained (a box without .NET 10 needs self-contained; the 2026-10-19 windev plan says framework-dependent).
  • The STOP_PENDING behaviour and the NSSM service account (LocalSystem per deployment.md).
  • New-box install and the upgrade are now scripted (see "Scripts"), but untested on a real box; the recovery/stop-grace/log-rotation defaults in Install-MxGatewayService.ps1 are chosen from the repo's own settings (Worker:ShutdownTimeoutSeconds), not copied from an existing box. Compare against the nssm dump of a known-good box (vd03/vq03) before relying on them.
  • The Worker output folder and build-vs-publish question (section 4) is still [UNVERIFIED]: the upgrade script copies whatever the staged -WorkerSource directory contains over Worker, so stage exactly what the existing Worker folder holds. Confirm with a file listing of the live Worker dir on the next deploy.