docs(deploy): QA vq03 + vq04 on 76a71bc8, #228 verified live on both (#229, #170)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m29s
ci / java (push) Failing after 9m20s
ci / portable (push) Failing after 16m20s

First real run of the #230-fixed Upgrade-MxGatewayService.ps1: rolling, ~8-9 s each,
env block untouched. #228 KeepsSentTimestamp passes on both (old build +5 h as the
negative control); #226 not observable on QA (empty alarm snapshot, no transitions).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 23:48:17 -05:00
parent 9e4be25807
commit 2cc0d57541
3 changed files with 103 additions and 2 deletions
@@ -1,6 +1,6 @@
# Pending: Worker time-conversion fixes #226 + #228 (DEV first, then QA) # Pending: Worker time-conversion fixes #226 + #228 (DEV first, then QA)
**Status: DEV deployed 2026-11-12 (`76a71bc8`, both checks passed, record `2026-11-12-vd03-76a71bc8-worker-time-fixes.md`); QA NOT deployed (#229, gated by #170).** This is the deploy note for the next gateway deploy. Ship both fixes in one **Status: DEV deployed 2026-11-12 (`76a71bc8`, both checks passed, record `2026-11-12-vd03-76a71bc8-worker-time-fixes.md`); QA vq03 + vq04 deployed the same night (`76a71bc8`, #228 passed on both, record `2026-11-12-qa-vq03-vq04-76a71bc8.md`).** This is the deploy note for the next gateway deploy. Ship both fixes in one
deploy: they are both in the x86 Worker, both change timestamps consumers see, and both are on `main`. deploy: they are both in the x86 Worker, both change timestamps consumers see, and both are on `main`.
| Fix | Commit | Behaviour change on the box | | Fix | Commit | Behaviour change on the box |
@@ -0,0 +1,101 @@
# 2026-11-12: QA wonder-app-vq03 + vq04: `4f931e4` -> `76a71bc8` (mxaccessgw#229 QA leg, #170)
Server and x86 Worker on both QA gateways moved from `4f931e4` (deployed 2026-09-10) to `origin/main` `76a71bc8`,
the build DEV vd03 has run since earlier the same evening (`2026-11-12-vd03-76a71bc8-worker-time-fixes.md`). The jump is
170 commits. Besides #226/#228/#227 it brings the 2026-10-21 review remediation and the operator-authenticated alarm ack
(`488649c`, #170). There is no SQLite DDL change in the range (checked with `git diff 4f931e4 76a71bc8`), so a rollback
needs no store restore. Owner-approved; rolling, vq03 first and then vq04. **First real run of the fixed
`scripts/deploy/Upgrade-MxGatewayService.ps1` (`4890a3d1`, #230).**
## Prerequisites (checked before the first stop)
| Check | Result |
|---|---|
| #170: OtOpcUa nodes that ack Galaxy alarms through this gateway | **None on QA.** An anonymous OPC UA browse of both QA OtOpcUa endpoints (`opc.tcp://10.220.158.2:4840` and `.3:4840`, OtOpcUa CLI from the Mac) shows only the `script-runner-canary` Calculation driver, with no Galaxy driver. This matches the OtOpcUa QA record (`qa-upgrade-b8b4c18f-to-fea7de0a.md`: 1 driver, the canary). The Mac SQL login cannot open `OtOpcUaConfig`, so the read-only check went over OPC UA instead of SQL. |
| ScadaBridge QA (`b3021e7c`) MxGateway adapter vs the new ack contract | It never acknowledges. `git grep AcknowledgeAlarm` at `b3021e7c` and at `main` finds no call site, and `NativeAlarmActor` is documented read-only. It pins Client/Contracts `0.2.1`, the same pin as DEV ScadaBridge `424db155`, which has run against `76a71bc8` on vd03 since the DEV deploy (810/810 resolved, unchanged). The data connection points only at `http://10.220.158.2:5120` (vq03), with no failover endpoint. |
| Artifacts | The **DEV-deployed package**, pulled from vd03's stage `E:\psrem-stage\mxgw-229-76a71bc8\` over SMB: `mxgw-76a71bc8.zip` sha256 `B2771EEC2B6406A7AA84D7A125176F0C0F9A97138F82808AF519A693F211A302`, the same as DEV's `BUILD-INFO.txt`. All 473 files match `MANIFEST.sha256` on the Mac and again on each box after the SMB push. `MxGateway.Server.exe` and `ZB.MOM.WW.MxGateway.Server.exe` are byte-identical (sha256 `7d7974f0...6ebedc`). `MxGateway.Worker.exe` sha256 is `844ab6ad...38f1871`, PE32 x86. Every exe carries ProductVersion `0.1.2+76a71bc8`. The bundled runtime is 10.0.14 (`mscordaccore_amd64_amd64_10.0.1426.52326`) against 10.0.5 live, so this is an upgrade and not a downgrade. |
## Procedure
Staged on each box at `E:\psrem-stage\mxgw-229-qa-76a71bc8\`:
- the zip and its `.sha256`, plus `BUILD-INFO.txt`;
- `Upgrade-MxGatewayService.ps1` from `4890a3d1` (sha256 `9c721bca...06efb3`);
- `stage229qa.ps1`, which checks the zip hash, expands it to `expanded\`, verifies the manifest and reads the stamps;
- `dry229qa.ps1` and `real229qa.ps1`, thin wrappers that pass `-NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe -ExpectedSha 76a71bc8 -Label issue229`;
- `rollback229qa.ps1`.
**Dry run on both boxes first.** The plan matched #230's checklist:
- `ZB.MOM.WW.MxGateway.Server#<pid>` is the child of the NSSM service host. QA's NSSM `Application` is `app\ZB.MOM.WW.MxGateway.Server.exe` and the Worker is `worker\MxGateway.Worker.exe`.
- The carry plan covers `appsettings.json` and `appsettings.Development.json`. QA has no `dashboard.pfx`.
- The env fingerprint has 33 names. vq03 and vq04 differ only in `MxGateway__Galaxy__ConnectionString`, because vq04 points at vq03's SQL. `MxGateway__ApiKeyPepper` sha256 prefix `88188ab11b50` is the same on both boxes.
- The stores (`data\gateway-auth.db` triplet, `data\secrets.db`) and the KEK (`C:\ProgramData\ZB\kek\mxgateway.kek`) are outside the swap.
- Expected warnings: `ASPNETCORE_ENVIRONMENT` is not `Production` (QA runs `Staging`, unchanged), and the live-only `app\logs\` directory stays in the `.bak`. New logs go to `MxAccessGateway\logs\` anyway.
| | vq03 | vq04 |
|---|---|---|
| Script stamp (box-local EST) | `20261112-233019` | `20261112-233532` |
| Stop | clean within the 30 s wait, no STOP_PENDING, 0 gateway/Worker processes left | same |
| Store backup | taken with the service fully stopped (`data\*.bak-<stamp>-issue229`, hash-verified) | same |
| Env block | 33 entries, block sha256 `4d8a12b11735` at preflight, after the stop, after the swap, before and after the start, and after the 60 s window. Never written. | same, block `e731b618765c` |
| Downtime | **about 8 s.** Old process `Application is shutting down` at 23:30:21; new process `Now listening` on :5130/:5120 at 23:30:29. | **about 9 s.** A Mac-side 1 s `/health/ready` poll failed 8 times, from 04:35:36Z to 04:35:44Z. |
| Script verify | `/health/ready` and `/healthz` 200. Deployed SHA `76a71bc8`. Exactly one gateway, a child of the service host. Same PID for 60 s. | same |
| Folder ACL | new `app\`/`worker\` inherit from `E:\ApiInstall\MxAccessGateway`, which carries the `NT SERVICE\OtOpcUaScriptWorker` deny (lmxopcua#891). The script reported no SDDL difference. | same |
| Startup log | 0 WRN/ERR. `Alarm subscribe ... watchList=641`; `Gateway alarm monitor active on \\wonder-app-vq03\Galaxy!NAM` | same |
`install-service.bat` was not used. No second gateway instance was started, and no admin `apikey` verb was run.
## Verification (from the Mac unless noted)
| Check | vq03 | vq04 |
|---|---|---|
| `/healthz`, `/health/ready` (auth-store, worker-executable, galaxy-sql), `/health/active` | 200 / 200 / 200, 2 of 2 sessions | 200 / 200 / 200, 1 of 1 sessions |
| Existing key (`otopcua-galaxy-qa`, token file `C:\ProgramData\ZB\qa-build\token-otopcua-galaxy-qa.txt`) | `galaxy-test-connection` `{ok:true}`; `open-session` OK, with a new Worker and gateway protocol 3; `close-session` CLOSED | same |
| Negative control: bogus key | `Missing or invalid API key.` | same |
| **#228** (`LiveGatewayWrite2Tests`, `ASSERT_WRITE2_TIMESTAMP=1`, item **`MESReceiver_025.MoveInBatchID`**) | **Before** the deploy, on `4f931e4`: `KeepsSentTimestamp` **FAILED**, sent `04:26:07Z`, stored `09:26:07Z` (+5 h; this is the negative control). **After: 2/2 passed**, sent = stored `04:32:02.000Z` | **2/2 passed**, sent = stored `04:38:27.000Z` |
| ScadaBridge QA site `site-a` (active vq03) | MxGateway `Connected`, 2196/2196 resolved, good 2196 / bad 0 at 04:32:35Z, about 2 min after the restart. Same as the pre-deploy report. Z28061 and Z28062 snapshots 244/244 Good. | unaffected (SB does not use vq04); 2196 Good at 04:39:35Z |
| OtOpcUa QA ServiceLevel | vq03 240 / vq04 250 before, during and after (vq04 is Primary since the OTel round) | same |
| Overview `:5320` | 12/12 instances Up; both MxAccessGateway cards Active/Up | same |
**The QA #228 item.** QA's Galaxy (`\\wonder-app-vq03\Galaxy`) uses the same object names as DEV. `MESReceiver_025` is
contained by `Left_002`, which is contained by **`Z28062`** (gobject 6201), so it is Z28062's left-side MES receiver
(ScadaBridge `Z28062.LeftMESReceiver.MoveInBatchID`). Its value was `0` and had been unchanged since 2026-10-27. No
ScadaBridge script triggers on it: every script on `CvdReactor` and `LeftMESReceiver` is call-triggered. Nothing was
written on Z28061. Each run wrote `212212` and restored `0` through a plain `Write`. There were three runs (pre-vq03,
post-vq03, post-vq04), with 2 writes and 2 restores each. If the attribute is historized, the pre-deploy run left one
point stamped +5 h (`09:26:07Z`) for about 30 s, the same side effect DEV had.
**Temporary test keys.** `qa229-write2-test` was minted on each box's dashboard (LDAP login with the box GLAuth test user).
- Scopes: `session:open`, `session:close`, `invoke:read`, `invoke:write`, `events:read`. Read and write globs: `MESReceiver_025.*`.
- The vq03 key was minted **before** its deploy, and the vq04 key before its deploy too, so each key also proves the store survived the swap.
- The secret was kept only in mode-0600 scratch files and never printed.
- Afterwards each key was revoked and then deleted. A probe with each deleted key got `Missing or invalid API key`. The scratch files were removed, and each box's key list is back to the original `scadabridge-qa` and `otopcua-galaxy-qa`.
**#226 (wnwrap alarm times are UTC): could not be observed on QA.** The QA gateway's alarm feed (provider
`ALARM_PROVIDER_MODE_ALARMMGR`) returned an **empty snapshot** before and after the deploy. `QueryActiveAlarms` returned 0,
and the Historian (`Runtime.dbo.v_AlarmEventHistory2` on wonder-sql-vq03) recorded **no alarm transition** in the capture
windows: a stream on vq03 from 04:25Z to the restart, and a 15-minute stream after it. The post-deploy stream on vq03 ran from 04:32Z to 04:47:30Z (provider ALARMMGR since 04:30:31Z): empty snapshot, 0 transitions. The Historian has 0 alarm rows for 03:48Z–04:48Z.
The fix is the same Worker binary as DEV, where it was checked live (`RuntimeAlert` `2026-07-28T20:02:46.830Z` = AVEVA history).
The empty QA snapshot is not new: it was already empty on `4f931e4`. Earlier the same day the Historian still showed recent
`UNACK_ALM` rows, for example `Z28061.HeartbeatTimeoutAlarm` at 02:04:35Z. This was not diagnosed here, since it is out of
scope for a deploy.
**#170 (operator-authenticated ack).** This is now deployed on DEV and QA. No consumer on either tier acks Galaxy alarms
through the gateway: OtOpcUa has no Galaxy driver on vd03 or QA, and ScadaBridge never acks. So the Historian
`ACK_ALM`-with-the-real-operator check has no live caller yet. It belongs to the first Galaxy driver authored against
these gateways (lmxopcua `dc0a24f2`+ plus an `AlarmAck` fallback account or per-key identity; see #170's comments).
## Backup and rollback
Each box has the following under `E:\ApiInstall\MxAccessGateway\`:
- previous build (`4f931e4`): `app.bak-<stamp>-issue229` and `worker.bak-<stamp>-issue229`;
- `_deploy-backup-<stamp>-issue229\`, ACL SYSTEM + Administrators only, inheritance off. It holds `AppEnvironmentExtra.txt`, the service `.reg` export, `env-fingerprint.txt` and a copy of the KEK;
- store copies: `data\gateway-auth.db{,-wal,-shm}.bak-<stamp>-issue229` and `data\secrets.db.bak-<stamp>-issue229`;
- notes: `_DEPLOY-NOTES-<stamp>.txt`, which holds names and hash prefixes only.
Stamps: vq03 `20261112-233019`, vq04 `20261112-233532`.
To roll back, run this on the box, elevated:
`powershell -NoProfile -ExecutionPolicy Bypass -File E:\psrem-stage\mxgw-229-qa-76a71bc8\rollback229qa.ps1 -Stamp <stamp>`
- Add `-RestoreStores` only if the stores are damaged; there is no schema change.
- The rollback was dry-run on both boxes after the deploy. It resolves `app.bak-`/`worker.bak-<stamp>-issue229`, and the env block matches the saved pre-upgrade fingerprint.
- Rollback brings back the old (wrong) alarm and Write2 times, and the pre-#170 ack.
@@ -3,7 +3,7 @@
Server and x86 Worker deployed together from `origin/main` `76a71bc8` (`76a71bc82d7ed41bad307d866257cb46951d9d2f`). Server and x86 Worker deployed together from `origin/main` `76a71bc8` (`76a71bc82d7ed41bad307d866257cb46951d9d2f`).
It ships #226 (`892bdc1a`, wnwrap alarm times are UTC), #228 (`9ef6fc84`, the `Write2` time argument is sent as It ships #226 (`892bdc1a`, wnwrap alarm times are UTC), #228 (`9ef6fc84`, the `Write2` time argument is sent as
box-local time) and #227 (additive, opt-in `emit_stream_attached`). There are no config, package or schema changes since `2b13564e`. box-local time) and #227 (additive, opt-in `emit_stream_attached`). There are no config, package or schema changes since `2b13564e`.
Procedure: `2026-11-12-pending-worker-time-fixes-226-228.md`. QA is not done yet (#229 stays open; gated by #170). Procedure: `2026-11-12-pending-worker-time-fixes-226-228.md`. QA followed the same night: `2026-11-12-qa-vq03-vq04-76a71bc8.md`.
## Build and package ## Build and package