First real run of the #230-fixed Upgrade-MxGatewayService.ps1: rolling, ~8-9 s each, env block untouched. #228 KeepsSentTimestamp passes on both (old build +5 h as the negative control); #226 not observable on QA (empty alarm snapshot, no transitions). Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
11 KiB
2026-11-12: QA wonder-app-vq03 + vq04: 4f931e4 -> 76a71bc8 (mxaccessgw#229 QA leg, #170)
Server and x86 Worker on both QA gateways moved from 4f931e4 (deployed 2026-09-10) to origin/main 76a71bc8,
the build DEV vd03 has run since earlier the same evening (2026-11-12-vd03-76a71bc8-worker-time-fixes.md). The jump is
170 commits. Besides #226/#228/#227 it brings the 2026-10-21 review remediation and the operator-authenticated alarm ack
(488649c, #170). There is no SQLite DDL change in the range (checked with git diff 4f931e4 76a71bc8), so a rollback
needs no store restore. Owner-approved; rolling, vq03 first and then vq04. First real run of the fixed
scripts/deploy/Upgrade-MxGatewayService.ps1 (4890a3d1, #230).
Prerequisites (checked before the first stop)
| Check | Result |
|---|---|
| #170: OtOpcUa nodes that ack Galaxy alarms through this gateway | None on QA. An anonymous OPC UA browse of both QA OtOpcUa endpoints (opc.tcp://10.220.158.2:4840 and .3:4840, OtOpcUa CLI from the Mac) shows only the script-runner-canary Calculation driver, with no Galaxy driver. This matches the OtOpcUa QA record (qa-upgrade-b8b4c18f-to-fea7de0a.md: 1 driver, the canary). The Mac SQL login cannot open OtOpcUaConfig, so the read-only check went over OPC UA instead of SQL. |
ScadaBridge QA (b3021e7c) MxGateway adapter vs the new ack contract |
It never acknowledges. git grep AcknowledgeAlarm at b3021e7c and at main finds no call site, and NativeAlarmActor is documented read-only. It pins Client/Contracts 0.2.1, the same pin as DEV ScadaBridge 424db155, which has run against 76a71bc8 on vd03 since the DEV deploy (810/810 resolved, unchanged). The data connection points only at http://10.220.158.2:5120 (vq03), with no failover endpoint. |
| Artifacts | The DEV-deployed package, pulled from vd03's stage E:\psrem-stage\mxgw-229-76a71bc8\ over SMB: mxgw-76a71bc8.zip sha256 B2771EEC2B6406A7AA84D7A125176F0C0F9A97138F82808AF519A693F211A302, the same as DEV's BUILD-INFO.txt. All 473 files match MANIFEST.sha256 on the Mac and again on each box after the SMB push. MxGateway.Server.exe and ZB.MOM.WW.MxGateway.Server.exe are byte-identical (sha256 7d7974f0...6ebedc). MxGateway.Worker.exe sha256 is 844ab6ad...38f1871, PE32 x86. Every exe carries ProductVersion 0.1.2+76a71bc8. The bundled runtime is 10.0.14 (mscordaccore_amd64_amd64_10.0.1426.52326) against 10.0.5 live, so this is an upgrade and not a downgrade. |
Procedure
Staged on each box at E:\psrem-stage\mxgw-229-qa-76a71bc8\:
- the zip and its
.sha256, plusBUILD-INFO.txt; Upgrade-MxGatewayService.ps1from4890a3d1(sha2569c721bca...06efb3);stage229qa.ps1, which checks the zip hash, expands it toexpanded\, verifies the manifest and reads the stamps;dry229qa.ps1andreal229qa.ps1, thin wrappers that pass-NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe -ExpectedSha 76a71bc8 -Label issue229;rollback229qa.ps1.
Dry run on both boxes first. The plan matched #230's checklist:
ZB.MOM.WW.MxGateway.Server#<pid>is the child of the NSSM service host. QA's NSSMApplicationisapp\ZB.MOM.WW.MxGateway.Server.exeand the Worker isworker\MxGateway.Worker.exe.- The carry plan covers
appsettings.jsonandappsettings.Development.json. QA has nodashboard.pfx. - The env fingerprint has 33 names. vq03 and vq04 differ only in
MxGateway__Galaxy__ConnectionString, because vq04 points at vq03's SQL.MxGateway__ApiKeyPeppersha256 prefix88188ab11b50is the same on both boxes. - The stores (
data\gateway-auth.dbtriplet,data\secrets.db) and the KEK (C:\ProgramData\ZB\kek\mxgateway.kek) are outside the swap. - Expected warnings:
ASPNETCORE_ENVIRONMENTis notProduction(QA runsStaging, unchanged), and the live-onlyapp\logs\directory stays in the.bak. New logs go toMxAccessGateway\logs\anyway.
| vq03 | vq04 | |
|---|---|---|
| Script stamp (box-local EST) | 20261112-233019 |
20261112-233532 |
| Stop | clean within the 30 s wait, no STOP_PENDING, 0 gateway/Worker processes left | same |
| Store backup | taken with the service fully stopped (data\*.bak-<stamp>-issue229, hash-verified) |
same |
| Env block | 33 entries, block sha256 4d8a12b11735 at preflight, after the stop, after the swap, before and after the start, and after the 60 s window. Never written. |
same, block e731b618765c |
| Downtime | about 8 s. Old process Application is shutting down at 23:30:21; new process Now listening on :5130/:5120 at 23:30:29. |
about 9 s. A Mac-side 1 s /health/ready poll failed 8 times, from 04:35:36Z to 04:35:44Z. |
| Script verify | /health/ready and /healthz 200. Deployed SHA 76a71bc8. Exactly one gateway, a child of the service host. Same PID for 60 s. |
same |
| Folder ACL | new app\/worker\ inherit from E:\ApiInstall\MxAccessGateway, which carries the NT SERVICE\OtOpcUaScriptWorker deny (lmxopcua#891). The script reported no SDDL difference. |
same |
| Startup log | 0 WRN/ERR. Alarm subscribe ... watchList=641; Gateway alarm monitor active on \\wonder-app-vq03\Galaxy!NAM |
same |
install-service.bat was not used. No second gateway instance was started, and no admin apikey verb was run.
Verification (from the Mac unless noted)
| Check | vq03 | vq04 |
|---|---|---|
/healthz, /health/ready (auth-store, worker-executable, galaxy-sql), /health/active |
200 / 200 / 200, 2 of 2 sessions | 200 / 200 / 200, 1 of 1 sessions |
Existing key (otopcua-galaxy-qa, token file C:\ProgramData\ZB\qa-build\token-otopcua-galaxy-qa.txt) |
galaxy-test-connection {ok:true}; open-session OK, with a new Worker and gateway protocol 3; close-session CLOSED |
same |
| Negative control: bogus key | Missing or invalid API key. |
same |
#228 (LiveGatewayWrite2Tests, ASSERT_WRITE2_TIMESTAMP=1, item MESReceiver_025.MoveInBatchID) |
Before the deploy, on 4f931e4: KeepsSentTimestamp FAILED, sent 04:26:07Z, stored 09:26:07Z (+5 h; this is the negative control). After: 2/2 passed, sent = stored 04:32:02.000Z |
2/2 passed, sent = stored 04:38:27.000Z |
ScadaBridge QA site site-a (active vq03) |
MxGateway Connected, 2196/2196 resolved, good 2196 / bad 0 at 04:32:35Z, about 2 min after the restart. Same as the pre-deploy report. Z28061 and Z28062 snapshots 244/244 Good. |
unaffected (SB does not use vq04); 2196 Good at 04:39:35Z |
| OtOpcUa QA ServiceLevel | vq03 240 / vq04 250 before, during and after (vq04 is Primary since the OTel round) | same |
Overview :5320 |
12/12 instances Up; both MxAccessGateway cards Active/Up | same |
The QA #228 item. QA's Galaxy (\\wonder-app-vq03\Galaxy) uses the same object names as DEV. MESReceiver_025 is
contained by Left_002, which is contained by Z28062 (gobject 6201), so it is Z28062's left-side MES receiver
(ScadaBridge Z28062.LeftMESReceiver.MoveInBatchID). Its value was 0 and had been unchanged since 2026-10-27. No
ScadaBridge script triggers on it: every script on CvdReactor and LeftMESReceiver is call-triggered. Nothing was
written on Z28061. Each run wrote 212212 and restored 0 through a plain Write. There were three runs (pre-vq03,
post-vq03, post-vq04), with 2 writes and 2 restores each. If the attribute is historized, the pre-deploy run left one
point stamped +5 h (09:26:07Z) for about 30 s, the same side effect DEV had.
Temporary test keys. qa229-write2-test was minted on each box's dashboard (LDAP login with the box GLAuth test user).
- Scopes:
session:open,session:close,invoke:read,invoke:write,events:read. Read and write globs:MESReceiver_025.*. - The vq03 key was minted before its deploy, and the vq04 key before its deploy too, so each key also proves the store survived the swap.
- The secret was kept only in mode-0600 scratch files and never printed.
- Afterwards each key was revoked and then deleted. A probe with each deleted key got
Missing or invalid API key. The scratch files were removed, and each box's key list is back to the originalscadabridge-qaandotopcua-galaxy-qa.
#226 (wnwrap alarm times are UTC): could not be observed on QA. The QA gateway's alarm feed (provider
ALARM_PROVIDER_MODE_ALARMMGR) returned an empty snapshot before and after the deploy. QueryActiveAlarms returned 0,
and the Historian (Runtime.dbo.v_AlarmEventHistory2 on wonder-sql-vq03) recorded no alarm transition in the capture
windows: a stream on vq03 from 04:25Z to the restart, and a 15-minute stream after it. The post-deploy stream on vq03 ran from 04:32Z to 04:47:30Z (provider ALARMMGR since 04:30:31Z): empty snapshot, 0 transitions. The Historian has 0 alarm rows for 03:48Z–04:48Z.
The fix is the same Worker binary as DEV, where it was checked live (RuntimeAlert 2026-07-28T20:02:46.830Z = AVEVA history).
The empty QA snapshot is not new: it was already empty on 4f931e4. Earlier the same day the Historian still showed recent
UNACK_ALM rows, for example Z28061.HeartbeatTimeoutAlarm at 02:04:35Z. This was not diagnosed here, since it is out of
scope for a deploy.
#170 (operator-authenticated ack). This is now deployed on DEV and QA. No consumer on either tier acks Galaxy alarms
through the gateway: OtOpcUa has no Galaxy driver on vd03 or QA, and ScadaBridge never acks. So the Historian
ACK_ALM-with-the-real-operator check has no live caller yet. It belongs to the first Galaxy driver authored against
these gateways (lmxopcua dc0a24f2+ plus an AlarmAck fallback account or per-key identity; see #170's comments).
Backup and rollback
Each box has the following under E:\ApiInstall\MxAccessGateway\:
- previous build (
4f931e4):app.bak-<stamp>-issue229andworker.bak-<stamp>-issue229; _deploy-backup-<stamp>-issue229\, ACL SYSTEM + Administrators only, inheritance off. It holdsAppEnvironmentExtra.txt, the service.regexport,env-fingerprint.txtand a copy of the KEK;- store copies:
data\gateway-auth.db{,-wal,-shm}.bak-<stamp>-issue229anddata\secrets.db.bak-<stamp>-issue229; - notes:
_DEPLOY-NOTES-<stamp>.txt, which holds names and hash prefixes only.
Stamps: vq03 20261112-233019, vq04 20261112-233532.
To roll back, run this on the box, elevated:
powershell -NoProfile -ExecutionPolicy Bypass -File E:\psrem-stage\mxgw-229-qa-76a71bc8\rollback229qa.ps1 -Stamp <stamp>
- Add
-RestoreStoresonly if the stores are damaged; there is no schema change. - The rollback was dry-run on both boxes after the deploy. It resolves
app.bak-/worker.bak-<stamp>-issue229, and the env block matches the saved pre-upgrade fingerprint. - Rollback brings back the old (wrong) alarm and Write2 times, and the pre-#170 ack.