Files
mxaccessgw/docs/deploy/2026-11-12-qa-vq03-vq04-76a71bc8.md
T
Joseph Doherty 2cc0d57541
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 2m29s
ci / java (push) Failing after 9m20s
ci / portable (push) Failing after 16m20s
docs(deploy): QA vq03 + vq04 on 76a71bc8, #228 verified live on both (#229, #170)
First real run of the #230-fixed Upgrade-MxGatewayService.ps1: rolling, ~8-9 s each,
env block untouched. #228 KeepsSentTimestamp passes on both (old build +5 h as the
negative control); #226 not observable on QA (empty alarm snapshot, no transitions).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:48:17 -05:00

102 lines
11 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 2026-11-12: QA wonder-app-vq03 + vq04: `4f931e4` -> `76a71bc8` (mxaccessgw#229 QA leg, #170)
Server and x86 Worker on both QA gateways moved from `4f931e4` (deployed 2026-09-10) to `origin/main` `76a71bc8`,
the build DEV vd03 has run since earlier the same evening (`2026-11-12-vd03-76a71bc8-worker-time-fixes.md`). The jump is
170 commits. Besides #226/#228/#227 it brings the 2026-10-21 review remediation and the operator-authenticated alarm ack
(`488649c`, #170). There is no SQLite DDL change in the range (checked with `git diff 4f931e4 76a71bc8`), so a rollback
needs no store restore. Owner-approved; rolling, vq03 first and then vq04. **First real run of the fixed
`scripts/deploy/Upgrade-MxGatewayService.ps1` (`4890a3d1`, #230).**
## Prerequisites (checked before the first stop)
| Check | Result |
|---|---|
| #170: OtOpcUa nodes that ack Galaxy alarms through this gateway | **None on QA.** An anonymous OPC UA browse of both QA OtOpcUa endpoints (`opc.tcp://10.220.158.2:4840` and `.3:4840`, OtOpcUa CLI from the Mac) shows only the `script-runner-canary` Calculation driver, with no Galaxy driver. This matches the OtOpcUa QA record (`qa-upgrade-b8b4c18f-to-fea7de0a.md`: 1 driver, the canary). The Mac SQL login cannot open `OtOpcUaConfig`, so the read-only check went over OPC UA instead of SQL. |
| ScadaBridge QA (`b3021e7c`) MxGateway adapter vs the new ack contract | It never acknowledges. `git grep AcknowledgeAlarm` at `b3021e7c` and at `main` finds no call site, and `NativeAlarmActor` is documented read-only. It pins Client/Contracts `0.2.1`, the same pin as DEV ScadaBridge `424db155`, which has run against `76a71bc8` on vd03 since the DEV deploy (810/810 resolved, unchanged). The data connection points only at `http://10.220.158.2:5120` (vq03), with no failover endpoint. |
| Artifacts | The **DEV-deployed package**, pulled from vd03's stage `E:\psrem-stage\mxgw-229-76a71bc8\` over SMB: `mxgw-76a71bc8.zip` sha256 `B2771EEC2B6406A7AA84D7A125176F0C0F9A97138F82808AF519A693F211A302`, the same as DEV's `BUILD-INFO.txt`. All 473 files match `MANIFEST.sha256` on the Mac and again on each box after the SMB push. `MxGateway.Server.exe` and `ZB.MOM.WW.MxGateway.Server.exe` are byte-identical (sha256 `7d7974f0...6ebedc`). `MxGateway.Worker.exe` sha256 is `844ab6ad...38f1871`, PE32 x86. Every exe carries ProductVersion `0.1.2+76a71bc8`. The bundled runtime is 10.0.14 (`mscordaccore_amd64_amd64_10.0.1426.52326`) against 10.0.5 live, so this is an upgrade and not a downgrade. |
## Procedure
Staged on each box at `E:\psrem-stage\mxgw-229-qa-76a71bc8\`:
- the zip and its `.sha256`, plus `BUILD-INFO.txt`;
- `Upgrade-MxGatewayService.ps1` from `4890a3d1` (sha256 `9c721bca...06efb3`);
- `stage229qa.ps1`, which checks the zip hash, expands it to `expanded\`, verifies the manifest and reads the stamps;
- `dry229qa.ps1` and `real229qa.ps1`, thin wrappers that pass `-NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe -ExpectedSha 76a71bc8 -Label issue229`;
- `rollback229qa.ps1`.
**Dry run on both boxes first.** The plan matched #230's checklist:
- `ZB.MOM.WW.MxGateway.Server#<pid>` is the child of the NSSM service host. QA's NSSM `Application` is `app\ZB.MOM.WW.MxGateway.Server.exe` and the Worker is `worker\MxGateway.Worker.exe`.
- The carry plan covers `appsettings.json` and `appsettings.Development.json`. QA has no `dashboard.pfx`.
- The env fingerprint has 33 names. vq03 and vq04 differ only in `MxGateway__Galaxy__ConnectionString`, because vq04 points at vq03's SQL. `MxGateway__ApiKeyPepper` sha256 prefix `88188ab11b50` is the same on both boxes.
- The stores (`data\gateway-auth.db` triplet, `data\secrets.db`) and the KEK (`C:\ProgramData\ZB\kek\mxgateway.kek`) are outside the swap.
- Expected warnings: `ASPNETCORE_ENVIRONMENT` is not `Production` (QA runs `Staging`, unchanged), and the live-only `app\logs\` directory stays in the `.bak`. New logs go to `MxAccessGateway\logs\` anyway.
| | vq03 | vq04 |
|---|---|---|
| Script stamp (box-local EST) | `20261112-233019` | `20261112-233532` |
| Stop | clean within the 30 s wait, no STOP_PENDING, 0 gateway/Worker processes left | same |
| Store backup | taken with the service fully stopped (`data\*.bak-<stamp>-issue229`, hash-verified) | same |
| Env block | 33 entries, block sha256 `4d8a12b11735` at preflight, after the stop, after the swap, before and after the start, and after the 60 s window. Never written. | same, block `e731b618765c` |
| Downtime | **about 8 s.** Old process `Application is shutting down` at 23:30:21; new process `Now listening` on :5130/:5120 at 23:30:29. | **about 9 s.** A Mac-side 1 s `/health/ready` poll failed 8 times, from 04:35:36Z to 04:35:44Z. |
| Script verify | `/health/ready` and `/healthz` 200. Deployed SHA `76a71bc8`. Exactly one gateway, a child of the service host. Same PID for 60 s. | same |
| Folder ACL | new `app\`/`worker\` inherit from `E:\ApiInstall\MxAccessGateway`, which carries the `NT SERVICE\OtOpcUaScriptWorker` deny (lmxopcua#891). The script reported no SDDL difference. | same |
| Startup log | 0 WRN/ERR. `Alarm subscribe ... watchList=641`; `Gateway alarm monitor active on \\wonder-app-vq03\Galaxy!NAM` | same |
`install-service.bat` was not used. No second gateway instance was started, and no admin `apikey` verb was run.
## Verification (from the Mac unless noted)
| Check | vq03 | vq04 |
|---|---|---|
| `/healthz`, `/health/ready` (auth-store, worker-executable, galaxy-sql), `/health/active` | 200 / 200 / 200, 2 of 2 sessions | 200 / 200 / 200, 1 of 1 sessions |
| Existing key (`otopcua-galaxy-qa`, token file `C:\ProgramData\ZB\qa-build\token-otopcua-galaxy-qa.txt`) | `galaxy-test-connection` `{ok:true}`; `open-session` OK, with a new Worker and gateway protocol 3; `close-session` CLOSED | same |
| Negative control: bogus key | `Missing or invalid API key.` | same |
| **#228** (`LiveGatewayWrite2Tests`, `ASSERT_WRITE2_TIMESTAMP=1`, item **`MESReceiver_025.MoveInBatchID`**) | **Before** the deploy, on `4f931e4`: `KeepsSentTimestamp` **FAILED**, sent `04:26:07Z`, stored `09:26:07Z` (+5 h; this is the negative control). **After: 2/2 passed**, sent = stored `04:32:02.000Z` | **2/2 passed**, sent = stored `04:38:27.000Z` |
| ScadaBridge QA site `site-a` (active vq03) | MxGateway `Connected`, 2196/2196 resolved, good 2196 / bad 0 at 04:32:35Z, about 2 min after the restart. Same as the pre-deploy report. Z28061 and Z28062 snapshots 244/244 Good. | unaffected (SB does not use vq04); 2196 Good at 04:39:35Z |
| OtOpcUa QA ServiceLevel | vq03 240 / vq04 250 before, during and after (vq04 is Primary since the OTel round) | same |
| Overview `:5320` | 12/12 instances Up; both MxAccessGateway cards Active/Up | same |
**The QA #228 item.** QA's Galaxy (`\\wonder-app-vq03\Galaxy`) uses the same object names as DEV. `MESReceiver_025` is
contained by `Left_002`, which is contained by **`Z28062`** (gobject 6201), so it is Z28062's left-side MES receiver
(ScadaBridge `Z28062.LeftMESReceiver.MoveInBatchID`). Its value was `0` and had been unchanged since 2026-10-27. No
ScadaBridge script triggers on it: every script on `CvdReactor` and `LeftMESReceiver` is call-triggered. Nothing was
written on Z28061. Each run wrote `212212` and restored `0` through a plain `Write`. There were three runs (pre-vq03,
post-vq03, post-vq04), with 2 writes and 2 restores each. If the attribute is historized, the pre-deploy run left one
point stamped +5 h (`09:26:07Z`) for about 30 s, the same side effect DEV had.
**Temporary test keys.** `qa229-write2-test` was minted on each box's dashboard (LDAP login with the box GLAuth test user).
- Scopes: `session:open`, `session:close`, `invoke:read`, `invoke:write`, `events:read`. Read and write globs: `MESReceiver_025.*`.
- The vq03 key was minted **before** its deploy, and the vq04 key before its deploy too, so each key also proves the store survived the swap.
- The secret was kept only in mode-0600 scratch files and never printed.
- Afterwards each key was revoked and then deleted. A probe with each deleted key got `Missing or invalid API key`. The scratch files were removed, and each box's key list is back to the original `scadabridge-qa` and `otopcua-galaxy-qa`.
**#226 (wnwrap alarm times are UTC): could not be observed on QA.** The QA gateway's alarm feed (provider
`ALARM_PROVIDER_MODE_ALARMMGR`) returned an **empty snapshot** before and after the deploy. `QueryActiveAlarms` returned 0,
and the Historian (`Runtime.dbo.v_AlarmEventHistory2` on wonder-sql-vq03) recorded **no alarm transition** in the capture
windows: a stream on vq03 from 04:25Z to the restart, and a 15-minute stream after it. The post-deploy stream on vq03 ran from 04:32Z to 04:47:30Z (provider ALARMMGR since 04:30:31Z): empty snapshot, 0 transitions. The Historian has 0 alarm rows for 03:48Z–04:48Z.
The fix is the same Worker binary as DEV, where it was checked live (`RuntimeAlert` `2026-07-28T20:02:46.830Z` = AVEVA history).
The empty QA snapshot is not new: it was already empty on `4f931e4`. Earlier the same day the Historian still showed recent
`UNACK_ALM` rows, for example `Z28061.HeartbeatTimeoutAlarm` at 02:04:35Z. This was not diagnosed here, since it is out of
scope for a deploy.
**#170 (operator-authenticated ack).** This is now deployed on DEV and QA. No consumer on either tier acks Galaxy alarms
through the gateway: OtOpcUa has no Galaxy driver on vd03 or QA, and ScadaBridge never acks. So the Historian
`ACK_ALM`-with-the-real-operator check has no live caller yet. It belongs to the first Galaxy driver authored against
these gateways (lmxopcua `dc0a24f2`+ plus an `AlarmAck` fallback account or per-key identity; see #170's comments).
## Backup and rollback
Each box has the following under `E:\ApiInstall\MxAccessGateway\`:
- previous build (`4f931e4`): `app.bak-<stamp>-issue229` and `worker.bak-<stamp>-issue229`;
- `_deploy-backup-<stamp>-issue229\`, ACL SYSTEM + Administrators only, inheritance off. It holds `AppEnvironmentExtra.txt`, the service `.reg` export, `env-fingerprint.txt` and a copy of the KEK;
- store copies: `data\gateway-auth.db{,-wal,-shm}.bak-<stamp>-issue229` and `data\secrets.db.bak-<stamp>-issue229`;
- notes: `_DEPLOY-NOTES-<stamp>.txt`, which holds names and hash prefixes only.
Stamps: vq03 `20261112-233019`, vq04 `20261112-233532`.
To roll back, run this on the box, elevated:
`powershell -NoProfile -ExecutionPolicy Bypass -File E:\psrem-stage\mxgw-229-qa-76a71bc8\rollback229qa.ps1 -Stamp <stamp>`
- Add `-RestoreStores` only if the stores are damaged; there is no schema change.
- The rollback was dry-run on both boxes after the deploy. It resolves `app.bak-`/`worker.bak-<stamp>-issue229`, and the env block matches the saved pre-upgrade fingerprint.
- Rollback brings back the old (wrong) alarm and Write2 times, and the pre-#170 ack.