Files
mxaccessgw/docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md
T
Joseph Doherty 4890a3d16e
ci / windows-x86 (push) Successful in 2m41s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Failing after 12m34s
ci / java (push) Failing after 14m50s
fix(deploy): Upgrade-MxGatewayService.ps1 safe on the wonder layout, guards fail closed (#230)
The process guards matched only ZB.MOM.WW.MxGateway.*; the wonder boxes run
MxGateway.Server.exe / MxGateway.Worker.exe, so the one-gateway guard, the
pre-start OrphanWorkerTerminator guard and the STOP_PENDING kill passed vacuously.

- Image names come from the installed exes (NSSM Application, Worker env path) plus
  both known schemes; processes are also matched by exact path (Win32_Process).
- One-gateway guard fails closed: a RUNNING service whose gateway is not visible, at
  the Application path, as the service host's child, is an abort.
- STOP_PENDING kill is scoped to this install's processes.
- Staged checks use the installed exe names; staged Worker must be x86; a staged
  self-contained runtime older than the live one is refused (-AllowRuntimeDowngrade).
- Carry-over: appsettings*.json and certificates/keys (dashboard.pfx) always keep the
  live copy; other live-only non-binary top-level files are carried; hash-verified.
- Env block read from the registry and fingerprinted (names + SHA-256 prefixes, never
  values) after stop, swap, start and the stability window; any change aborts.
  Sensitive values are redacted from all output and notes.
- File-based master key (vd03) accepted and backed up; a missing env master key
  (windev) is a warning. Backup dir is ACL-restricted before anything is written.
- Rollback accepts .bak-<stamp> and .bak.<yyyyMMddTHHmmss> sets and refuses a
  mixed Server/Worker stamp; auto-rollback no longer fails on the old SHA.
- -WhatIf now works on Windows PowerShell 5.1 (Get-FileHash returned nothing under
  WhatIfPreference); an empty Win32_Service result no longer reads as "unavailable".
- Install-MxGatewayService.ps1 process guard matches both schemes.
- Dry-run harness scripts/deploy/tests (fake box in both layouts, 140 assertions,
  passes on pwsh/macOS and Windows PowerShell 5.1) plus a mutation runner (12/12 killed).
- Runbook updated.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 22:23:19 -05:00

7.4 KiB

2026-11-12: wonder-app-vd03 (DEV): 2b13564e -> 76a71bc8 (Worker time fixes #226 + #228, mxaccessgw#229)

Server and x86 Worker deployed together from origin/main 76a71bc8 (76a71bc82d7ed41bad307d866257cb46951d9d2f). It ships #226 (892bdc1a, wnwrap alarm times are UTC), #228 (9ef6fc84, the Write2 time argument is sent as box-local time) and #227 (additive, opt-in emit_stream_attached). There are no config, package or schema changes since 2b13564e. Procedure: 2026-11-12-pending-worker-time-fixes-226-228.md. QA is not done yet (#229 stays open; gated by #170).

Build and package

Part Where How
Source Mac git archive 76a71bc8 (tar sha256 9f3bdf19...de04f24), the same tar for both parts
Worker windev, scratch C:\build\_tmp-mxgw-229-76a71bc8 dotnet build ...Worker.csproj -c Release -p:Platform=x86 -p:SourceRevisionId=76a71bc8; Worker.Tests x86: 783 passed, 0 failed, 12 skipped. Apphost + .config renamed to MxGateway.Worker.exe(.config); the 19-file set is name-identical to the live one, and the .exe.config is byte-identical. PE x86.
Server Mac (SDK 10.0.403) dotnet publish ...Server.csproj -c Release -r win-x64 --self-contained true -p:SourceRevisionId=76a71bc8 (453 files) + MxGateway.Server.exe as a byte copy of the apphost (the NSSM Application)

The package is mxgw-76a71bc8.zip (sha256 B2771EEC...F211A302) with MANIFEST.sha256 covering all 473 files. The hash was checked at every hop (windev -> Mac -> vd03 SMB push, plus an SMB pull round trip). ProductVersion is 0.1.2+76a71bc8 on both exes.

A windev-built Server was rejected. The windev SDK (10.0.204) publishes with runtime pack 10.0.8, but live was on 10.0.14, so it would have downgraded the self-contained runtime. The preflight showed this as a differing mscordaccore_amd64_amd64_10.0.826.*.dll. Build the self-contained Server where the SDK carries the newest runtime (the Mac here, as for 2b13564e), or update the windev SDK.

Why not Upgrade-MxGatewayService.ps1

The repo script has not been run on a real box. Reading it against vd03 found two blockers:

  • Its one-gateway and process-gone guards match only the ZB.MOM.WW.MxGateway.Server / .Worker process names. vd03 runs MxGateway.Server.exe / MxGateway.Worker.exe, so every OrphanWorkerTerminator guard and the STOP_PENDING kill would pass vacuously. The staged-Worker check also requires ZB.MOM.WW.MxGateway.Worker.exe.
  • It carries only appsettings*.json forward. vd03's Server\ also holds dashboard.pfx.

The deploy used preflight/swap/rollback scripts adapted from the proven 2b13564e set (E:\psrem-stage\mxgw-229-76a71bc8\{common,pre,swap,rollback}229.ps1). They match both exe names, carry appsettings.json (hash-checked), appsettings.Development.json and dashboard.pfx forward, never write NSSM Application/AppDirectory/AppEnvironmentExtra, and keep backups in the protected stage folder. The script gaps are filed as #230. Fixed in #230: the repo script now matches both exe naming schemes and fails closed, carries dashboard.pfx, accepts the file-based master key, fingerprints the env block, refuses a runtime downgrade, and can roll back these .bak.20261112T211945 sets. Use it for QA.

Timeline (box-local, EST)

  • 21:19:20 preflight PASS (read-only). Zip and manifest OK; live 0.1.2+2b13564e Server and Worker; exactly one gateway process; env 7 entries; MxGateway__ApiKeyPepper sha256 prefix 0d01abcc1fe5; Server\appsettings.json sha256 464ACE1B... with no pepper; E: 179.9 GB free.
  • 21:19:48 backup: Server\appsettings*.json, dashboard.pfx and the service .reg copied to the backup folder.
  • 21:19:50 stop (Stop-Service -NoWait). Stopped in 2 s with no STOP_PENDING and 0 gateway/worker processes. data\ (6 files) copied, hash-verified.
  • 21:19:52 rename Server/Worker -> *.bak.20261112T211945. New files copied: Server 454, Worker 19. appsettings.json restored (hash match), appsettings.Development.json and dashboard.pfx carried forward (hash match).
  • 21:19:59 /health/ready 200, 4 s after start. Downtime (stop -> ready) about 9 s. After 30 s: same gateway PID (17972), one gateway process on the box, /health/active 2 of 2 sessions ready, /healthz 200.

Verification

Check Result
Build stamp Server and Worker ProductVersion 0.1.2+76a71bc8
Health (from the Mac) /healthz 200, /health/ready 200 (auth-store, worker-executable, galaxy-sql), /health/active 200 (2 of 2 sessions)
NSSM env block 7 entries, names identical, block hash 3f283e9a1d57 before, mid-swap and after; pepper prefix 0d01abcc1fe5 unchanged. Never written.
Server\appsettings.json 464ACE1B... before and after (the #171 no-pepper version, not the install-root master)
Existing key authenticates a key minted before the stop opened a session, read MESReceiver_025.MoveInBatchID (0, Good) and closed. The Worker path is proven.
Wrong key MxGatewayAuthenticationException (Unauthenticated)
#228 LiveGatewayWrite2Tests with MXGATEWAY_LIVE_GATEWAY_ASSERT_WRITE2_TIMESTAMP=1 on MESReceiver_025.MoveInBatchID: 2/2 passed. KeepsSentTimestamp: sent 2026-11-13T02:20:34.000Z, stored 02:20:34.000Z (on 2b13564e it was stored +5 h). The value was restored to 0 both times.
#226 QueryActiveAlarms: Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert last transition 2026-07-28T20:02:46.830Z (before: 2026-07-29T01:02:46.830Z), which matches AVEVA history. ...TableAlarms_008.CommunicationAlarm moved 2026-11-10T16:44:38.420Z -> 11:44:38.420Z (-5 h, EST).
ScadaBridge vd03 site site-a report 02:21:07Z: MxGateway Connected, 810/810 resolved, good 398 / bad 412, identical to the pre-deploy report. Z28061/Z28062 debug snapshots identical.
OtOpcUa vd03 :9000/health/ready and /health/active 200 (no Galaxy driver on vd03)
Folder ACLs New Server\/Worker\ inherit from E:\ApiInstall\MxGateway and carry the inherited NT SERVICE\OtOpcUaScriptWorker deny (lmxopcua#891). The old folders also had a redundant explicit BUILTIN\Administrators FullControl that the parent already grants, so the SDDL differs but nothing is loosened.

Test key: issue229-write2-test was minted on the vd03 dashboard. Scopes: session:open, session:close, invoke:read, invoke:write, events:read; read and write globs MESReceiver_025.*. The secret only lived in a mode-0600 scratch file and was never printed. Afterwards the key was revoked and then deleted, a probe with it got Unauthenticated, the scratch file was deleted, and the key list is back to the original 6.

Backup and rollback

  • Previous install: E:\ApiInstall\MxGateway\Server.bak.20261112T211945 + Worker.bak.20261112T211945 (2b13564e).
  • E:\psrem-stage\mxgw-229-76a71bc8\backup-20261112T211945\ (stage folder ACL: SYSTEM + Administrators only, inheritance off): data\ (auth and secrets store triplets, kek.key, dashboard.pfx), Server-appsettings*.json, Server-dashboard.pfx, MxAccessGw-servicekey-20261112T211945.reg.
  • Notes: E:\ApiInstall\MxGateway\_DEPLOY-NOTES-20261112T211945-issue229.txt (names and hash prefixes only).
  • Rollback: powershell -NoProfile -ExecutionPolicy Bypass -File E:\psrem-stage\mxgw-229-76a71bc8\rollback229.ps1 -Stamp 20261112T211945 (add -RestoreData only if the stores are damaged). There is no store or schema change, so rollback restores the old (wrong) times and nothing else.