# 2026-11-12: QA wonder-app-vq03 + vq04: `4f931e4` -> `76a71bc8` (mxaccessgw#229 QA leg, #170) Server and x86 Worker on both QA gateways moved from `4f931e4` (deployed 2026-09-10) to `origin/main` `76a71bc8`, the build DEV vd03 has run since earlier the same evening (`2026-11-12-vd03-76a71bc8-worker-time-fixes.md`). The jump is 170 commits. Besides #226/#228/#227 it brings the 2026-10-21 review remediation and the operator-authenticated alarm ack (`488649c`, #170). There is no SQLite DDL change in the range (checked with `git diff 4f931e4 76a71bc8`), so a rollback needs no store restore. Owner-approved; rolling, vq03 first and then vq04. **First real run of the fixed `scripts/deploy/Upgrade-MxGatewayService.ps1` (`4890a3d1`, #230).** ## Prerequisites (checked before the first stop) | Check | Result | |---|---| | #170: OtOpcUa nodes that ack Galaxy alarms through this gateway | **None on QA.** An anonymous OPC UA browse of both QA OtOpcUa endpoints (`opc.tcp://10.220.158.2:4840` and `.3:4840`, OtOpcUa CLI from the Mac) shows only the `script-runner-canary` Calculation driver, with no Galaxy driver. This matches the OtOpcUa QA record (`qa-upgrade-b8b4c18f-to-fea7de0a.md`: 1 driver, the canary). The Mac SQL login cannot open `OtOpcUaConfig`, so the read-only check went over OPC UA instead of SQL. | | ScadaBridge QA (`b3021e7c`) MxGateway adapter vs the new ack contract | It never acknowledges. `git grep AcknowledgeAlarm` at `b3021e7c` and at `main` finds no call site, and `NativeAlarmActor` is documented read-only. It pins Client/Contracts `0.2.1`, the same pin as DEV ScadaBridge `424db155`, which has run against `76a71bc8` on vd03 since the DEV deploy (810/810 resolved, unchanged). The data connection points only at `http://10.220.158.2:5120` (vq03), with no failover endpoint. | | Artifacts | The **DEV-deployed package**, pulled from vd03's stage `E:\psrem-stage\mxgw-229-76a71bc8\` over SMB: `mxgw-76a71bc8.zip` sha256 `B2771EEC2B6406A7AA84D7A125176F0C0F9A97138F82808AF519A693F211A302`, the same as DEV's `BUILD-INFO.txt`. All 473 files match `MANIFEST.sha256` on the Mac and again on each box after the SMB push. `MxGateway.Server.exe` and `ZB.MOM.WW.MxGateway.Server.exe` are byte-identical (sha256 `7d7974f0...6ebedc`). `MxGateway.Worker.exe` sha256 is `844ab6ad...38f1871`, PE32 x86. Every exe carries ProductVersion `0.1.2+76a71bc8`. The bundled runtime is 10.0.14 (`mscordaccore_amd64_amd64_10.0.1426.52326`) against 10.0.5 live, so this is an upgrade and not a downgrade. | ## Procedure Staged on each box at `E:\psrem-stage\mxgw-229-qa-76a71bc8\`: - the zip and its `.sha256`, plus `BUILD-INFO.txt`; - `Upgrade-MxGatewayService.ps1` from `4890a3d1` (sha256 `9c721bca...06efb3`); - `stage229qa.ps1`, which checks the zip hash, expands it to `expanded\`, verifies the manifest and reads the stamps; - `dry229qa.ps1` and `real229qa.ps1`, thin wrappers that pass `-NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe -ExpectedSha 76a71bc8 -Label issue229`; - `rollback229qa.ps1`. **Dry run on both boxes first.** The plan matched #230's checklist: - `ZB.MOM.WW.MxGateway.Server#` is the child of the NSSM service host. QA's NSSM `Application` is `app\ZB.MOM.WW.MxGateway.Server.exe` and the Worker is `worker\MxGateway.Worker.exe`. - The carry plan covers `appsettings.json` and `appsettings.Development.json`. QA has no `dashboard.pfx`. - The env fingerprint has 33 names. vq03 and vq04 differ only in `MxGateway__Galaxy__ConnectionString`, because vq04 points at vq03's SQL. `MxGateway__ApiKeyPepper` sha256 prefix `88188ab11b50` is the same on both boxes. - The stores (`data\gateway-auth.db` triplet, `data\secrets.db`) and the KEK (`C:\ProgramData\ZB\kek\mxgateway.kek`) are outside the swap. - Expected warnings: `ASPNETCORE_ENVIRONMENT` is not `Production` (QA runs `Staging`, unchanged), and the live-only `app\logs\` directory stays in the `.bak`. New logs go to `MxAccessGateway\logs\` anyway. | | vq03 | vq04 | |---|---|---| | Script stamp (box-local EST) | `20261112-233019` | `20261112-233532` | | Stop | clean within the 30 s wait, no STOP_PENDING, 0 gateway/Worker processes left | same | | Store backup | taken with the service fully stopped (`data\*.bak--issue229`, hash-verified) | same | | Env block | 33 entries, block sha256 `4d8a12b11735` at preflight, after the stop, after the swap, before and after the start, and after the 60 s window. Never written. | same, block `e731b618765c` | | Downtime | **about 8 s.** Old process `Application is shutting down` at 23:30:21; new process `Now listening` on :5130/:5120 at 23:30:29. | **about 9 s.** A Mac-side 1 s `/health/ready` poll failed 8 times, from 04:35:36Z to 04:35:44Z. | | Script verify | `/health/ready` and `/healthz` 200. Deployed SHA `76a71bc8`. Exactly one gateway, a child of the service host. Same PID for 60 s. | same | | Folder ACL | new `app\`/`worker\` inherit from `E:\ApiInstall\MxAccessGateway`, which carries the `NT SERVICE\OtOpcUaScriptWorker` deny (lmxopcua#891). The script reported no SDDL difference. | same | | Startup log | 0 WRN/ERR. `Alarm subscribe ... watchList=641`; `Gateway alarm monitor active on \\wonder-app-vq03\Galaxy!NAM` | same | `install-service.bat` was not used. No second gateway instance was started, and no admin `apikey` verb was run. ## Verification (from the Mac unless noted) | Check | vq03 | vq04 | |---|---|---| | `/healthz`, `/health/ready` (auth-store, worker-executable, galaxy-sql), `/health/active` | 200 / 200 / 200, 2 of 2 sessions | 200 / 200 / 200, 1 of 1 sessions | | Existing key (`otopcua-galaxy-qa`, token file `C:\ProgramData\ZB\qa-build\token-otopcua-galaxy-qa.txt`) | `galaxy-test-connection` `{ok:true}`; `open-session` OK, with a new Worker and gateway protocol 3; `close-session` CLOSED | same | | Negative control: bogus key | `Missing or invalid API key.` | same | | **#228** (`LiveGatewayWrite2Tests`, `ASSERT_WRITE2_TIMESTAMP=1`, item **`MESReceiver_025.MoveInBatchID`**) | **Before** the deploy, on `4f931e4`: `KeepsSentTimestamp` **FAILED**, sent `04:26:07Z`, stored `09:26:07Z` (+5 h; this is the negative control). **After: 2/2 passed**, sent = stored `04:32:02.000Z` | **2/2 passed**, sent = stored `04:38:27.000Z` | | ScadaBridge QA site `site-a` (active vq03) | MxGateway `Connected`, 2196/2196 resolved, good 2196 / bad 0 at 04:32:35Z, about 2 min after the restart. Same as the pre-deploy report. Z28061 and Z28062 snapshots 244/244 Good. | unaffected (SB does not use vq04); 2196 Good at 04:39:35Z | | OtOpcUa QA ServiceLevel | vq03 240 / vq04 250 before, during and after (vq04 is Primary since the OTel round) | same | | Overview `:5320` | 12/12 instances Up; both MxAccessGateway cards Active/Up | same | **The QA #228 item.** QA's Galaxy (`\\wonder-app-vq03\Galaxy`) uses the same object names as DEV. `MESReceiver_025` is contained by `Left_002`, which is contained by **`Z28062`** (gobject 6201), so it is Z28062's left-side MES receiver (ScadaBridge `Z28062.LeftMESReceiver.MoveInBatchID`). Its value was `0` and had been unchanged since 2026-10-27. No ScadaBridge script triggers on it: every script on `CvdReactor` and `LeftMESReceiver` is call-triggered. Nothing was written on Z28061. Each run wrote `212212` and restored `0` through a plain `Write`. There were three runs (pre-vq03, post-vq03, post-vq04), with 2 writes and 2 restores each. If the attribute is historized, the pre-deploy run left one point stamped +5 h (`09:26:07Z`) for about 30 s, the same side effect DEV had. **Temporary test keys.** `qa229-write2-test` was minted on each box's dashboard (LDAP login with the box GLAuth test user). - Scopes: `session:open`, `session:close`, `invoke:read`, `invoke:write`, `events:read`. Read and write globs: `MESReceiver_025.*`. - The vq03 key was minted **before** its deploy, and the vq04 key before its deploy too, so each key also proves the store survived the swap. - The secret was kept only in mode-0600 scratch files and never printed. - Afterwards each key was revoked and then deleted. A probe with each deleted key got `Missing or invalid API key`. The scratch files were removed, and each box's key list is back to the original `scadabridge-qa` and `otopcua-galaxy-qa`. **#226 (wnwrap alarm times are UTC): could not be observed on QA.** The QA gateway's alarm feed (provider `ALARM_PROVIDER_MODE_ALARMMGR`) returned an **empty snapshot** before and after the deploy. `QueryActiveAlarms` returned 0, and the Historian (`Runtime.dbo.v_AlarmEventHistory2` on wonder-sql-vq03) recorded **no alarm transition** in the capture windows: a stream on vq03 from 04:25Z to the restart, and a 15-minute stream after it. The post-deploy stream on vq03 ran from 04:32Z to 04:47:30Z (provider ALARMMGR since 04:30:31Z): empty snapshot, 0 transitions. The Historian has 0 alarm rows for 03:48Z–04:48Z. The fix is the same Worker binary as DEV, where it was checked live (`RuntimeAlert` `2026-07-28T20:02:46.830Z` = AVEVA history). The empty QA snapshot is not new: it was already empty on `4f931e4`. Earlier the same day the Historian still showed recent `UNACK_ALM` rows, for example `Z28061.HeartbeatTimeoutAlarm` at 02:04:35Z. This was not diagnosed here, since it is out of scope for a deploy. **#170 (operator-authenticated ack).** This is now deployed on DEV and QA. No consumer on either tier acks Galaxy alarms through the gateway: OtOpcUa has no Galaxy driver on vd03 or QA, and ScadaBridge never acks. So the Historian `ACK_ALM`-with-the-real-operator check has no live caller yet. It belongs to the first Galaxy driver authored against these gateways (lmxopcua `dc0a24f2`+ plus an `AlarmAck` fallback account or per-key identity; see #170's comments). ## Backup and rollback Each box has the following under `E:\ApiInstall\MxAccessGateway\`: - previous build (`4f931e4`): `app.bak--issue229` and `worker.bak--issue229`; - `_deploy-backup--issue229\`, ACL SYSTEM + Administrators only, inheritance off. It holds `AppEnvironmentExtra.txt`, the service `.reg` export, `env-fingerprint.txt` and a copy of the KEK; - store copies: `data\gateway-auth.db{,-wal,-shm}.bak--issue229` and `data\secrets.db.bak--issue229`; - notes: `_DEPLOY-NOTES-.txt`, which holds names and hash prefixes only. Stamps: vq03 `20261112-233019`, vq04 `20261112-233532`. To roll back, run this on the box, elevated: `powershell -NoProfile -ExecutionPolicy Bypass -File E:\psrem-stage\mxgw-229-qa-76a71bc8\rollback229qa.ps1 -Stamp ` - Add `-RestoreStores` only if the stores are damaged; there is no schema change. - The rollback was dry-run on both boxes after the deploy. It resolves `app.bak-`/`worker.bak--issue229`, and the env block matches the saved pre-upgrade fingerprint. - Rollback brings back the old (wrong) alarm and Write2 times, and the pre-#170 ack.