2cc0d57541
First real run of the #230-fixed Upgrade-MxGatewayService.ps1: rolling, ~8-9 s each, env block untouched. #228 KeepsSentTimestamp passes on both (old build +5 h as the negative control); #226 not observable on QA (empty alarm snapshot, no transitions). Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
102 lines
11 KiB
Markdown
102 lines
11 KiB
Markdown
# 2026-11-12: QA wonder-app-vq03 + vq04: `4f931e4` -> `76a71bc8` (mxaccessgw#229 QA leg, #170)
|
||
|
||
Server and x86 Worker on both QA gateways moved from `4f931e4` (deployed 2026-09-10) to `origin/main` `76a71bc8`,
|
||
the build DEV vd03 has run since earlier the same evening (`2026-11-12-vd03-76a71bc8-worker-time-fixes.md`). The jump is
|
||
170 commits. Besides #226/#228/#227 it brings the 2026-10-21 review remediation and the operator-authenticated alarm ack
|
||
(`488649c`, #170). There is no SQLite DDL change in the range (checked with `git diff 4f931e4 76a71bc8`), so a rollback
|
||
needs no store restore. Owner-approved; rolling, vq03 first and then vq04. **First real run of the fixed
|
||
`scripts/deploy/Upgrade-MxGatewayService.ps1` (`4890a3d1`, #230).**
|
||
|
||
## Prerequisites (checked before the first stop)
|
||
|
||
| Check | Result |
|
||
|---|---|
|
||
| #170: OtOpcUa nodes that ack Galaxy alarms through this gateway | **None on QA.** An anonymous OPC UA browse of both QA OtOpcUa endpoints (`opc.tcp://10.220.158.2:4840` and `.3:4840`, OtOpcUa CLI from the Mac) shows only the `script-runner-canary` Calculation driver, with no Galaxy driver. This matches the OtOpcUa QA record (`qa-upgrade-b8b4c18f-to-fea7de0a.md`: 1 driver, the canary). The Mac SQL login cannot open `OtOpcUaConfig`, so the read-only check went over OPC UA instead of SQL. |
|
||
| ScadaBridge QA (`b3021e7c`) MxGateway adapter vs the new ack contract | It never acknowledges. `git grep AcknowledgeAlarm` at `b3021e7c` and at `main` finds no call site, and `NativeAlarmActor` is documented read-only. It pins Client/Contracts `0.2.1`, the same pin as DEV ScadaBridge `424db155`, which has run against `76a71bc8` on vd03 since the DEV deploy (810/810 resolved, unchanged). The data connection points only at `http://10.220.158.2:5120` (vq03), with no failover endpoint. |
|
||
| Artifacts | The **DEV-deployed package**, pulled from vd03's stage `E:\psrem-stage\mxgw-229-76a71bc8\` over SMB: `mxgw-76a71bc8.zip` sha256 `B2771EEC2B6406A7AA84D7A125176F0C0F9A97138F82808AF519A693F211A302`, the same as DEV's `BUILD-INFO.txt`. All 473 files match `MANIFEST.sha256` on the Mac and again on each box after the SMB push. `MxGateway.Server.exe` and `ZB.MOM.WW.MxGateway.Server.exe` are byte-identical (sha256 `7d7974f0...6ebedc`). `MxGateway.Worker.exe` sha256 is `844ab6ad...38f1871`, PE32 x86. Every exe carries ProductVersion `0.1.2+76a71bc8`. The bundled runtime is 10.0.14 (`mscordaccore_amd64_amd64_10.0.1426.52326`) against 10.0.5 live, so this is an upgrade and not a downgrade. |
|
||
|
||
## Procedure
|
||
|
||
Staged on each box at `E:\psrem-stage\mxgw-229-qa-76a71bc8\`:
|
||
- the zip and its `.sha256`, plus `BUILD-INFO.txt`;
|
||
- `Upgrade-MxGatewayService.ps1` from `4890a3d1` (sha256 `9c721bca...06efb3`);
|
||
- `stage229qa.ps1`, which checks the zip hash, expands it to `expanded\`, verifies the manifest and reads the stamps;
|
||
- `dry229qa.ps1` and `real229qa.ps1`, thin wrappers that pass `-NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe -ExpectedSha 76a71bc8 -Label issue229`;
|
||
- `rollback229qa.ps1`.
|
||
|
||
**Dry run on both boxes first.** The plan matched #230's checklist:
|
||
- `ZB.MOM.WW.MxGateway.Server#<pid>` is the child of the NSSM service host. QA's NSSM `Application` is `app\ZB.MOM.WW.MxGateway.Server.exe` and the Worker is `worker\MxGateway.Worker.exe`.
|
||
- The carry plan covers `appsettings.json` and `appsettings.Development.json`. QA has no `dashboard.pfx`.
|
||
- The env fingerprint has 33 names. vq03 and vq04 differ only in `MxGateway__Galaxy__ConnectionString`, because vq04 points at vq03's SQL. `MxGateway__ApiKeyPepper` sha256 prefix `88188ab11b50` is the same on both boxes.
|
||
- The stores (`data\gateway-auth.db` triplet, `data\secrets.db`) and the KEK (`C:\ProgramData\ZB\kek\mxgateway.kek`) are outside the swap.
|
||
- Expected warnings: `ASPNETCORE_ENVIRONMENT` is not `Production` (QA runs `Staging`, unchanged), and the live-only `app\logs\` directory stays in the `.bak`. New logs go to `MxAccessGateway\logs\` anyway.
|
||
|
||
| | vq03 | vq04 |
|
||
|---|---|---|
|
||
| Script stamp (box-local EST) | `20261112-233019` | `20261112-233532` |
|
||
| Stop | clean within the 30 s wait, no STOP_PENDING, 0 gateway/Worker processes left | same |
|
||
| Store backup | taken with the service fully stopped (`data\*.bak-<stamp>-issue229`, hash-verified) | same |
|
||
| Env block | 33 entries, block sha256 `4d8a12b11735` at preflight, after the stop, after the swap, before and after the start, and after the 60 s window. Never written. | same, block `e731b618765c` |
|
||
| Downtime | **about 8 s.** Old process `Application is shutting down` at 23:30:21; new process `Now listening` on :5130/:5120 at 23:30:29. | **about 9 s.** A Mac-side 1 s `/health/ready` poll failed 8 times, from 04:35:36Z to 04:35:44Z. |
|
||
| Script verify | `/health/ready` and `/healthz` 200. Deployed SHA `76a71bc8`. Exactly one gateway, a child of the service host. Same PID for 60 s. | same |
|
||
| Folder ACL | new `app\`/`worker\` inherit from `E:\ApiInstall\MxAccessGateway`, which carries the `NT SERVICE\OtOpcUaScriptWorker` deny (lmxopcua#891). The script reported no SDDL difference. | same |
|
||
| Startup log | 0 WRN/ERR. `Alarm subscribe ... watchList=641`; `Gateway alarm monitor active on \\wonder-app-vq03\Galaxy!NAM` | same |
|
||
|
||
`install-service.bat` was not used. No second gateway instance was started, and no admin `apikey` verb was run.
|
||
|
||
## Verification (from the Mac unless noted)
|
||
|
||
| Check | vq03 | vq04 |
|
||
|---|---|---|
|
||
| `/healthz`, `/health/ready` (auth-store, worker-executable, galaxy-sql), `/health/active` | 200 / 200 / 200, 2 of 2 sessions | 200 / 200 / 200, 1 of 1 sessions |
|
||
| Existing key (`otopcua-galaxy-qa`, token file `C:\ProgramData\ZB\qa-build\token-otopcua-galaxy-qa.txt`) | `galaxy-test-connection` `{ok:true}`; `open-session` OK, with a new Worker and gateway protocol 3; `close-session` CLOSED | same |
|
||
| Negative control: bogus key | `Missing or invalid API key.` | same |
|
||
| **#228** (`LiveGatewayWrite2Tests`, `ASSERT_WRITE2_TIMESTAMP=1`, item **`MESReceiver_025.MoveInBatchID`**) | **Before** the deploy, on `4f931e4`: `KeepsSentTimestamp` **FAILED**, sent `04:26:07Z`, stored `09:26:07Z` (+5 h; this is the negative control). **After: 2/2 passed**, sent = stored `04:32:02.000Z` | **2/2 passed**, sent = stored `04:38:27.000Z` |
|
||
| ScadaBridge QA site `site-a` (active vq03) | MxGateway `Connected`, 2196/2196 resolved, good 2196 / bad 0 at 04:32:35Z, about 2 min after the restart. Same as the pre-deploy report. Z28061 and Z28062 snapshots 244/244 Good. | unaffected (SB does not use vq04); 2196 Good at 04:39:35Z |
|
||
| OtOpcUa QA ServiceLevel | vq03 240 / vq04 250 before, during and after (vq04 is Primary since the OTel round) | same |
|
||
| Overview `:5320` | 12/12 instances Up; both MxAccessGateway cards Active/Up | same |
|
||
|
||
**The QA #228 item.** QA's Galaxy (`\\wonder-app-vq03\Galaxy`) uses the same object names as DEV. `MESReceiver_025` is
|
||
contained by `Left_002`, which is contained by **`Z28062`** (gobject 6201), so it is Z28062's left-side MES receiver
|
||
(ScadaBridge `Z28062.LeftMESReceiver.MoveInBatchID`). Its value was `0` and had been unchanged since 2026-10-27. No
|
||
ScadaBridge script triggers on it: every script on `CvdReactor` and `LeftMESReceiver` is call-triggered. Nothing was
|
||
written on Z28061. Each run wrote `212212` and restored `0` through a plain `Write`. There were three runs (pre-vq03,
|
||
post-vq03, post-vq04), with 2 writes and 2 restores each. If the attribute is historized, the pre-deploy run left one
|
||
point stamped +5 h (`09:26:07Z`) for about 30 s, the same side effect DEV had.
|
||
|
||
**Temporary test keys.** `qa229-write2-test` was minted on each box's dashboard (LDAP login with the box GLAuth test user).
|
||
- Scopes: `session:open`, `session:close`, `invoke:read`, `invoke:write`, `events:read`. Read and write globs: `MESReceiver_025.*`.
|
||
- The vq03 key was minted **before** its deploy, and the vq04 key before its deploy too, so each key also proves the store survived the swap.
|
||
- The secret was kept only in mode-0600 scratch files and never printed.
|
||
- Afterwards each key was revoked and then deleted. A probe with each deleted key got `Missing or invalid API key`. The scratch files were removed, and each box's key list is back to the original `scadabridge-qa` and `otopcua-galaxy-qa`.
|
||
|
||
**#226 (wnwrap alarm times are UTC): could not be observed on QA.** The QA gateway's alarm feed (provider
|
||
`ALARM_PROVIDER_MODE_ALARMMGR`) returned an **empty snapshot** before and after the deploy. `QueryActiveAlarms` returned 0,
|
||
and the Historian (`Runtime.dbo.v_AlarmEventHistory2` on wonder-sql-vq03) recorded **no alarm transition** in the capture
|
||
windows: a stream on vq03 from 04:25Z to the restart, and a 15-minute stream after it. The post-deploy stream on vq03 ran from 04:32Z to 04:47:30Z (provider ALARMMGR since 04:30:31Z): empty snapshot, 0 transitions. The Historian has 0 alarm rows for 03:48Z–04:48Z.
|
||
The fix is the same Worker binary as DEV, where it was checked live (`RuntimeAlert` `2026-07-28T20:02:46.830Z` = AVEVA history).
|
||
The empty QA snapshot is not new: it was already empty on `4f931e4`. Earlier the same day the Historian still showed recent
|
||
`UNACK_ALM` rows, for example `Z28061.HeartbeatTimeoutAlarm` at 02:04:35Z. This was not diagnosed here, since it is out of
|
||
scope for a deploy.
|
||
|
||
**#170 (operator-authenticated ack).** This is now deployed on DEV and QA. No consumer on either tier acks Galaxy alarms
|
||
through the gateway: OtOpcUa has no Galaxy driver on vd03 or QA, and ScadaBridge never acks. So the Historian
|
||
`ACK_ALM`-with-the-real-operator check has no live caller yet. It belongs to the first Galaxy driver authored against
|
||
these gateways (lmxopcua `dc0a24f2`+ plus an `AlarmAck` fallback account or per-key identity; see #170's comments).
|
||
|
||
## Backup and rollback
|
||
|
||
Each box has the following under `E:\ApiInstall\MxAccessGateway\`:
|
||
- previous build (`4f931e4`): `app.bak-<stamp>-issue229` and `worker.bak-<stamp>-issue229`;
|
||
- `_deploy-backup-<stamp>-issue229\`, ACL SYSTEM + Administrators only, inheritance off. It holds `AppEnvironmentExtra.txt`, the service `.reg` export, `env-fingerprint.txt` and a copy of the KEK;
|
||
- store copies: `data\gateway-auth.db{,-wal,-shm}.bak-<stamp>-issue229` and `data\secrets.db.bak-<stamp>-issue229`;
|
||
- notes: `_DEPLOY-NOTES-<stamp>.txt`, which holds names and hash prefixes only.
|
||
|
||
Stamps: vq03 `20261112-233019`, vq04 `20261112-233532`.
|
||
|
||
To roll back, run this on the box, elevated:
|
||
`powershell -NoProfile -ExecutionPolicy Bypass -File E:\psrem-stage\mxgw-229-qa-76a71bc8\rollback229qa.ps1 -Stamp <stamp>`
|
||
- Add `-RestoreStores` only if the stores are damaged; there is no schema change.
|
||
- The rollback was dry-run on both boxes after the deploy. It resolves `app.bak-`/`worker.bak-<stamp>-issue229`, and the env block matches the saved pre-upgrade fingerprint.
|
||
- Rollback brings back the old (wrong) alarm and Write2 times, and the pre-#170 ack.
|