Files
mxaccessgw/docs/deploy/2026-11-12-qa-vq03-vq04-76a71bc8.md
T
Joseph Doherty 2cc0d57541
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m29s
ci / java (push) Failing after 9m20s
ci / portable (push) Failing after 16m20s
docs(deploy): QA vq03 + vq04 on 76a71bc8, #228 verified live on both (#229, #170)
First real run of the #230-fixed Upgrade-MxGatewayService.ps1: rolling, ~8-9 s each,
env block untouched. #228 KeepsSentTimestamp passes on both (old build +5 h as the
negative control); #226 not observable on QA (empty alarm snapshot, no transitions).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:48:17 -05:00

11 KiB
Raw Blame History

2026-11-12: QA wonder-app-vq03 + vq04: 4f931e4 -> 76a71bc8 (mxaccessgw#229 QA leg, #170)

Server and x86 Worker on both QA gateways moved from 4f931e4 (deployed 2026-09-10) to origin/main 76a71bc8, the build DEV vd03 has run since earlier the same evening (2026-11-12-vd03-76a71bc8-worker-time-fixes.md). The jump is 170 commits. Besides #226/#228/#227 it brings the 2026-10-21 review remediation and the operator-authenticated alarm ack (488649c, #170). There is no SQLite DDL change in the range (checked with git diff 4f931e4 76a71bc8), so a rollback needs no store restore. Owner-approved; rolling, vq03 first and then vq04. First real run of the fixed scripts/deploy/Upgrade-MxGatewayService.ps1 (4890a3d1, #230).

Prerequisites (checked before the first stop)

Check Result
#170: OtOpcUa nodes that ack Galaxy alarms through this gateway None on QA. An anonymous OPC UA browse of both QA OtOpcUa endpoints (opc.tcp://10.220.158.2:4840 and .3:4840, OtOpcUa CLI from the Mac) shows only the script-runner-canary Calculation driver, with no Galaxy driver. This matches the OtOpcUa QA record (qa-upgrade-b8b4c18f-to-fea7de0a.md: 1 driver, the canary). The Mac SQL login cannot open OtOpcUaConfig, so the read-only check went over OPC UA instead of SQL.
ScadaBridge QA (b3021e7c) MxGateway adapter vs the new ack contract It never acknowledges. git grep AcknowledgeAlarm at b3021e7c and at main finds no call site, and NativeAlarmActor is documented read-only. It pins Client/Contracts 0.2.1, the same pin as DEV ScadaBridge 424db155, which has run against 76a71bc8 on vd03 since the DEV deploy (810/810 resolved, unchanged). The data connection points only at http://10.220.158.2:5120 (vq03), with no failover endpoint.
Artifacts The DEV-deployed package, pulled from vd03's stage E:\psrem-stage\mxgw-229-76a71bc8\ over SMB: mxgw-76a71bc8.zip sha256 B2771EEC2B6406A7AA84D7A125176F0C0F9A97138F82808AF519A693F211A302, the same as DEV's BUILD-INFO.txt. All 473 files match MANIFEST.sha256 on the Mac and again on each box after the SMB push. MxGateway.Server.exe and ZB.MOM.WW.MxGateway.Server.exe are byte-identical (sha256 7d7974f0...6ebedc). MxGateway.Worker.exe sha256 is 844ab6ad...38f1871, PE32 x86. Every exe carries ProductVersion 0.1.2+76a71bc8. The bundled runtime is 10.0.14 (mscordaccore_amd64_amd64_10.0.1426.52326) against 10.0.5 live, so this is an upgrade and not a downgrade.

Procedure

Staged on each box at E:\psrem-stage\mxgw-229-qa-76a71bc8\:

  • the zip and its .sha256, plus BUILD-INFO.txt;
  • Upgrade-MxGatewayService.ps1 from 4890a3d1 (sha256 9c721bca...06efb3);
  • stage229qa.ps1, which checks the zip hash, expands it to expanded\, verifies the manifest and reads the stamps;
  • dry229qa.ps1 and real229qa.ps1, thin wrappers that pass -NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe -ExpectedSha 76a71bc8 -Label issue229;
  • rollback229qa.ps1.

Dry run on both boxes first. The plan matched #230's checklist:

  • ZB.MOM.WW.MxGateway.Server#<pid> is the child of the NSSM service host. QA's NSSM Application is app\ZB.MOM.WW.MxGateway.Server.exe and the Worker is worker\MxGateway.Worker.exe.
  • The carry plan covers appsettings.json and appsettings.Development.json. QA has no dashboard.pfx.
  • The env fingerprint has 33 names. vq03 and vq04 differ only in MxGateway__Galaxy__ConnectionString, because vq04 points at vq03's SQL. MxGateway__ApiKeyPepper sha256 prefix 88188ab11b50 is the same on both boxes.
  • The stores (data\gateway-auth.db triplet, data\secrets.db) and the KEK (C:\ProgramData\ZB\kek\mxgateway.kek) are outside the swap.
  • Expected warnings: ASPNETCORE_ENVIRONMENT is not Production (QA runs Staging, unchanged), and the live-only app\logs\ directory stays in the .bak. New logs go to MxAccessGateway\logs\ anyway.
vq03 vq04
Script stamp (box-local EST) 20261112-233019 20261112-233532
Stop clean within the 30 s wait, no STOP_PENDING, 0 gateway/Worker processes left same
Store backup taken with the service fully stopped (data\*.bak-<stamp>-issue229, hash-verified) same
Env block 33 entries, block sha256 4d8a12b11735 at preflight, after the stop, after the swap, before and after the start, and after the 60 s window. Never written. same, block e731b618765c
Downtime about 8 s. Old process Application is shutting down at 23:30:21; new process Now listening on :5130/:5120 at 23:30:29. about 9 s. A Mac-side 1 s /health/ready poll failed 8 times, from 04:35:36Z to 04:35:44Z.
Script verify /health/ready and /healthz 200. Deployed SHA 76a71bc8. Exactly one gateway, a child of the service host. Same PID for 60 s. same
Folder ACL new app\/worker\ inherit from E:\ApiInstall\MxAccessGateway, which carries the NT SERVICE\OtOpcUaScriptWorker deny (lmxopcua#891). The script reported no SDDL difference. same
Startup log 0 WRN/ERR. Alarm subscribe ... watchList=641; Gateway alarm monitor active on \\wonder-app-vq03\Galaxy!NAM same

install-service.bat was not used. No second gateway instance was started, and no admin apikey verb was run.

Verification (from the Mac unless noted)

Check vq03 vq04
/healthz, /health/ready (auth-store, worker-executable, galaxy-sql), /health/active 200 / 200 / 200, 2 of 2 sessions 200 / 200 / 200, 1 of 1 sessions
Existing key (otopcua-galaxy-qa, token file C:\ProgramData\ZB\qa-build\token-otopcua-galaxy-qa.txt) galaxy-test-connection {ok:true}; open-session OK, with a new Worker and gateway protocol 3; close-session CLOSED same
Negative control: bogus key Missing or invalid API key. same
#228 (LiveGatewayWrite2Tests, ASSERT_WRITE2_TIMESTAMP=1, item MESReceiver_025.MoveInBatchID) Before the deploy, on 4f931e4: KeepsSentTimestamp FAILED, sent 04:26:07Z, stored 09:26:07Z (+5 h; this is the negative control). After: 2/2 passed, sent = stored 04:32:02.000Z 2/2 passed, sent = stored 04:38:27.000Z
ScadaBridge QA site site-a (active vq03) MxGateway Connected, 2196/2196 resolved, good 2196 / bad 0 at 04:32:35Z, about 2 min after the restart. Same as the pre-deploy report. Z28061 and Z28062 snapshots 244/244 Good. unaffected (SB does not use vq04); 2196 Good at 04:39:35Z
OtOpcUa QA ServiceLevel vq03 240 / vq04 250 before, during and after (vq04 is Primary since the OTel round) same
Overview :5320 12/12 instances Up; both MxAccessGateway cards Active/Up same

The QA #228 item. QA's Galaxy (\\wonder-app-vq03\Galaxy) uses the same object names as DEV. MESReceiver_025 is contained by Left_002, which is contained by Z28062 (gobject 6201), so it is Z28062's left-side MES receiver (ScadaBridge Z28062.LeftMESReceiver.MoveInBatchID). Its value was 0 and had been unchanged since 2026-10-27. No ScadaBridge script triggers on it: every script on CvdReactor and LeftMESReceiver is call-triggered. Nothing was written on Z28061. Each run wrote 212212 and restored 0 through a plain Write. There were three runs (pre-vq03, post-vq03, post-vq04), with 2 writes and 2 restores each. If the attribute is historized, the pre-deploy run left one point stamped +5 h (09:26:07Z) for about 30 s, the same side effect DEV had.

Temporary test keys. qa229-write2-test was minted on each box's dashboard (LDAP login with the box GLAuth test user).

  • Scopes: session:open, session:close, invoke:read, invoke:write, events:read. Read and write globs: MESReceiver_025.*.
  • The vq03 key was minted before its deploy, and the vq04 key before its deploy too, so each key also proves the store survived the swap.
  • The secret was kept only in mode-0600 scratch files and never printed.
  • Afterwards each key was revoked and then deleted. A probe with each deleted key got Missing or invalid API key. The scratch files were removed, and each box's key list is back to the original scadabridge-qa and otopcua-galaxy-qa.

#226 (wnwrap alarm times are UTC): could not be observed on QA. The QA gateway's alarm feed (provider ALARM_PROVIDER_MODE_ALARMMGR) returned an empty snapshot before and after the deploy. QueryActiveAlarms returned 0, and the Historian (Runtime.dbo.v_AlarmEventHistory2 on wonder-sql-vq03) recorded no alarm transition in the capture windows: a stream on vq03 from 04:25Z to the restart, and a 15-minute stream after it. The post-deploy stream on vq03 ran from 04:32Z to 04:47:30Z (provider ALARMMGR since 04:30:31Z): empty snapshot, 0 transitions. The Historian has 0 alarm rows for 03:48Z–04:48Z. The fix is the same Worker binary as DEV, where it was checked live (RuntimeAlert 2026-07-28T20:02:46.830Z = AVEVA history). The empty QA snapshot is not new: it was already empty on 4f931e4. Earlier the same day the Historian still showed recent UNACK_ALM rows, for example Z28061.HeartbeatTimeoutAlarm at 02:04:35Z. This was not diagnosed here, since it is out of scope for a deploy.

#170 (operator-authenticated ack). This is now deployed on DEV and QA. No consumer on either tier acks Galaxy alarms through the gateway: OtOpcUa has no Galaxy driver on vd03 or QA, and ScadaBridge never acks. So the Historian ACK_ALM-with-the-real-operator check has no live caller yet. It belongs to the first Galaxy driver authored against these gateways (lmxopcua dc0a24f2+ plus an AlarmAck fallback account or per-key identity; see #170's comments).

Backup and rollback

Each box has the following under E:\ApiInstall\MxAccessGateway\:

  • previous build (4f931e4): app.bak-<stamp>-issue229 and worker.bak-<stamp>-issue229;
  • _deploy-backup-<stamp>-issue229\, ACL SYSTEM + Administrators only, inheritance off. It holds AppEnvironmentExtra.txt, the service .reg export, env-fingerprint.txt and a copy of the KEK;
  • store copies: data\gateway-auth.db{,-wal,-shm}.bak-<stamp>-issue229 and data\secrets.db.bak-<stamp>-issue229;
  • notes: _DEPLOY-NOTES-<stamp>.txt, which holds names and hash prefixes only.

Stamps: vq03 20261112-233019, vq04 20261112-233532.

To roll back, run this on the box, elevated: powershell -NoProfile -ExecutionPolicy Bypass -File E:\psrem-stage\mxgw-229-qa-76a71bc8\rollback229qa.ps1 -Stamp <stamp>

  • Add -RestoreStores only if the stores are damaged; there is no schema change.
  • The rollback was dry-run on both boxes after the deploy. It resolves app.bak-/worker.bak-<stamp>-issue229, and the env block matches the saved pre-upgrade fingerprint.
  • Rollback brings back the old (wrong) alarm and Write2 times, and the pre-#170 ack.