diff --git a/docs/deploy/2026-11-12-pending-worker-time-fixes-226-228.md b/docs/deploy/2026-11-12-pending-worker-time-fixes-226-228.md index 62bfe5c8..695f6c88 100644 --- a/docs/deploy/2026-11-12-pending-worker-time-fixes-226-228.md +++ b/docs/deploy/2026-11-12-pending-worker-time-fixes-226-228.md @@ -1,6 +1,6 @@ # Pending: Worker time-conversion fixes #226 + #228 (DEV first, then QA) -**Status: DEV deployed 2026-11-12 (`76a71bc8`, both checks passed, record `2026-11-12-vd03-76a71bc8-worker-time-fixes.md`); QA NOT deployed (#229, gated by #170).** This is the deploy note for the next gateway deploy. Ship both fixes in one +**Status: DEV deployed 2026-11-12 (`76a71bc8`, both checks passed, record `2026-11-12-vd03-76a71bc8-worker-time-fixes.md`); QA vq03 + vq04 deployed the same night (`76a71bc8`, #228 passed on both, record `2026-11-12-qa-vq03-vq04-76a71bc8.md`).** This is the deploy note for the next gateway deploy. Ship both fixes in one deploy: they are both in the x86 Worker, both change timestamps consumers see, and both are on `main`. | Fix | Commit | Behaviour change on the box | diff --git a/docs/deploy/2026-11-12-qa-vq03-vq04-76a71bc8.md b/docs/deploy/2026-11-12-qa-vq03-vq04-76a71bc8.md new file mode 100644 index 00000000..ad60fdd0 --- /dev/null +++ b/docs/deploy/2026-11-12-qa-vq03-vq04-76a71bc8.md @@ -0,0 +1,101 @@ +# 2026-11-12: QA wonder-app-vq03 + vq04: `4f931e4` -> `76a71bc8` (mxaccessgw#229 QA leg, #170) + +Server and x86 Worker on both QA gateways moved from `4f931e4` (deployed 2026-09-10) to `origin/main` `76a71bc8`, +the build DEV vd03 has run since earlier the same evening (`2026-11-12-vd03-76a71bc8-worker-time-fixes.md`). The jump is +170 commits. Besides #226/#228/#227 it brings the 2026-10-21 review remediation and the operator-authenticated alarm ack +(`488649c`, #170). There is no SQLite DDL change in the range (checked with `git diff 4f931e4 76a71bc8`), so a rollback +needs no store restore. Owner-approved; rolling, vq03 first and then vq04. **First real run of the fixed +`scripts/deploy/Upgrade-MxGatewayService.ps1` (`4890a3d1`, #230).** + +## Prerequisites (checked before the first stop) + +| Check | Result | +|---|---| +| #170: OtOpcUa nodes that ack Galaxy alarms through this gateway | **None on QA.** An anonymous OPC UA browse of both QA OtOpcUa endpoints (`opc.tcp://10.220.158.2:4840` and `.3:4840`, OtOpcUa CLI from the Mac) shows only the `script-runner-canary` Calculation driver, with no Galaxy driver. This matches the OtOpcUa QA record (`qa-upgrade-b8b4c18f-to-fea7de0a.md`: 1 driver, the canary). The Mac SQL login cannot open `OtOpcUaConfig`, so the read-only check went over OPC UA instead of SQL. | +| ScadaBridge QA (`b3021e7c`) MxGateway adapter vs the new ack contract | It never acknowledges. `git grep AcknowledgeAlarm` at `b3021e7c` and at `main` finds no call site, and `NativeAlarmActor` is documented read-only. It pins Client/Contracts `0.2.1`, the same pin as DEV ScadaBridge `424db155`, which has run against `76a71bc8` on vd03 since the DEV deploy (810/810 resolved, unchanged). The data connection points only at `http://10.220.158.2:5120` (vq03), with no failover endpoint. | +| Artifacts | The **DEV-deployed package**, pulled from vd03's stage `E:\psrem-stage\mxgw-229-76a71bc8\` over SMB: `mxgw-76a71bc8.zip` sha256 `B2771EEC2B6406A7AA84D7A125176F0C0F9A97138F82808AF519A693F211A302`, the same as DEV's `BUILD-INFO.txt`. All 473 files match `MANIFEST.sha256` on the Mac and again on each box after the SMB push. `MxGateway.Server.exe` and `ZB.MOM.WW.MxGateway.Server.exe` are byte-identical (sha256 `7d7974f0...6ebedc`). `MxGateway.Worker.exe` sha256 is `844ab6ad...38f1871`, PE32 x86. Every exe carries ProductVersion `0.1.2+76a71bc8`. The bundled runtime is 10.0.14 (`mscordaccore_amd64_amd64_10.0.1426.52326`) against 10.0.5 live, so this is an upgrade and not a downgrade. | + +## Procedure + +Staged on each box at `E:\psrem-stage\mxgw-229-qa-76a71bc8\`: +- the zip and its `.sha256`, plus `BUILD-INFO.txt`; +- `Upgrade-MxGatewayService.ps1` from `4890a3d1` (sha256 `9c721bca...06efb3`); +- `stage229qa.ps1`, which checks the zip hash, expands it to `expanded\`, verifies the manifest and reads the stamps; +- `dry229qa.ps1` and `real229qa.ps1`, thin wrappers that pass `-NssmPath C:\ProgramData\ZB\qa-build\tools\nssm.exe -ExpectedSha 76a71bc8 -Label issue229`; +- `rollback229qa.ps1`. + +**Dry run on both boxes first.** The plan matched #230's checklist: +- `ZB.MOM.WW.MxGateway.Server#` is the child of the NSSM service host. QA's NSSM `Application` is `app\ZB.MOM.WW.MxGateway.Server.exe` and the Worker is `worker\MxGateway.Worker.exe`. +- The carry plan covers `appsettings.json` and `appsettings.Development.json`. QA has no `dashboard.pfx`. +- The env fingerprint has 33 names. vq03 and vq04 differ only in `MxGateway__Galaxy__ConnectionString`, because vq04 points at vq03's SQL. `MxGateway__ApiKeyPepper` sha256 prefix `88188ab11b50` is the same on both boxes. +- The stores (`data\gateway-auth.db` triplet, `data\secrets.db`) and the KEK (`C:\ProgramData\ZB\kek\mxgateway.kek`) are outside the swap. +- Expected warnings: `ASPNETCORE_ENVIRONMENT` is not `Production` (QA runs `Staging`, unchanged), and the live-only `app\logs\` directory stays in the `.bak`. New logs go to `MxAccessGateway\logs\` anyway. + +| | vq03 | vq04 | +|---|---|---| +| Script stamp (box-local EST) | `20261112-233019` | `20261112-233532` | +| Stop | clean within the 30 s wait, no STOP_PENDING, 0 gateway/Worker processes left | same | +| Store backup | taken with the service fully stopped (`data\*.bak--issue229`, hash-verified) | same | +| Env block | 33 entries, block sha256 `4d8a12b11735` at preflight, after the stop, after the swap, before and after the start, and after the 60 s window. Never written. | same, block `e731b618765c` | +| Downtime | **about 8 s.** Old process `Application is shutting down` at 23:30:21; new process `Now listening` on :5130/:5120 at 23:30:29. | **about 9 s.** A Mac-side 1 s `/health/ready` poll failed 8 times, from 04:35:36Z to 04:35:44Z. | +| Script verify | `/health/ready` and `/healthz` 200. Deployed SHA `76a71bc8`. Exactly one gateway, a child of the service host. Same PID for 60 s. | same | +| Folder ACL | new `app\`/`worker\` inherit from `E:\ApiInstall\MxAccessGateway`, which carries the `NT SERVICE\OtOpcUaScriptWorker` deny (lmxopcua#891). The script reported no SDDL difference. | same | +| Startup log | 0 WRN/ERR. `Alarm subscribe ... watchList=641`; `Gateway alarm monitor active on \\wonder-app-vq03\Galaxy!NAM` | same | + +`install-service.bat` was not used. No second gateway instance was started, and no admin `apikey` verb was run. + +## Verification (from the Mac unless noted) + +| Check | vq03 | vq04 | +|---|---|---| +| `/healthz`, `/health/ready` (auth-store, worker-executable, galaxy-sql), `/health/active` | 200 / 200 / 200, 2 of 2 sessions | 200 / 200 / 200, 1 of 1 sessions | +| Existing key (`otopcua-galaxy-qa`, token file `C:\ProgramData\ZB\qa-build\token-otopcua-galaxy-qa.txt`) | `galaxy-test-connection` `{ok:true}`; `open-session` OK, with a new Worker and gateway protocol 3; `close-session` CLOSED | same | +| Negative control: bogus key | `Missing or invalid API key.` | same | +| **#228** (`LiveGatewayWrite2Tests`, `ASSERT_WRITE2_TIMESTAMP=1`, item **`MESReceiver_025.MoveInBatchID`**) | **Before** the deploy, on `4f931e4`: `KeepsSentTimestamp` **FAILED**, sent `04:26:07Z`, stored `09:26:07Z` (+5 h; this is the negative control). **After: 2/2 passed**, sent = stored `04:32:02.000Z` | **2/2 passed**, sent = stored `04:38:27.000Z` | +| ScadaBridge QA site `site-a` (active vq03) | MxGateway `Connected`, 2196/2196 resolved, good 2196 / bad 0 at 04:32:35Z, about 2 min after the restart. Same as the pre-deploy report. Z28061 and Z28062 snapshots 244/244 Good. | unaffected (SB does not use vq04); 2196 Good at 04:39:35Z | +| OtOpcUa QA ServiceLevel | vq03 240 / vq04 250 before, during and after (vq04 is Primary since the OTel round) | same | +| Overview `:5320` | 12/12 instances Up; both MxAccessGateway cards Active/Up | same | + +**The QA #228 item.** QA's Galaxy (`\\wonder-app-vq03\Galaxy`) uses the same object names as DEV. `MESReceiver_025` is +contained by `Left_002`, which is contained by **`Z28062`** (gobject 6201), so it is Z28062's left-side MES receiver +(ScadaBridge `Z28062.LeftMESReceiver.MoveInBatchID`). Its value was `0` and had been unchanged since 2026-10-27. No +ScadaBridge script triggers on it: every script on `CvdReactor` and `LeftMESReceiver` is call-triggered. Nothing was +written on Z28061. Each run wrote `212212` and restored `0` through a plain `Write`. There were three runs (pre-vq03, +post-vq03, post-vq04), with 2 writes and 2 restores each. If the attribute is historized, the pre-deploy run left one +point stamped +5 h (`09:26:07Z`) for about 30 s, the same side effect DEV had. + +**Temporary test keys.** `qa229-write2-test` was minted on each box's dashboard (LDAP login with the box GLAuth test user). +- Scopes: `session:open`, `session:close`, `invoke:read`, `invoke:write`, `events:read`. Read and write globs: `MESReceiver_025.*`. +- The vq03 key was minted **before** its deploy, and the vq04 key before its deploy too, so each key also proves the store survived the swap. +- The secret was kept only in mode-0600 scratch files and never printed. +- Afterwards each key was revoked and then deleted. A probe with each deleted key got `Missing or invalid API key`. The scratch files were removed, and each box's key list is back to the original `scadabridge-qa` and `otopcua-galaxy-qa`. + +**#226 (wnwrap alarm times are UTC): could not be observed on QA.** The QA gateway's alarm feed (provider +`ALARM_PROVIDER_MODE_ALARMMGR`) returned an **empty snapshot** before and after the deploy. `QueryActiveAlarms` returned 0, +and the Historian (`Runtime.dbo.v_AlarmEventHistory2` on wonder-sql-vq03) recorded **no alarm transition** in the capture +windows: a stream on vq03 from 04:25Z to the restart, and a 15-minute stream after it. The post-deploy stream on vq03 ran from 04:32Z to 04:47:30Z (provider ALARMMGR since 04:30:31Z): empty snapshot, 0 transitions. The Historian has 0 alarm rows for 03:48Z–04:48Z. +The fix is the same Worker binary as DEV, where it was checked live (`RuntimeAlert` `2026-07-28T20:02:46.830Z` = AVEVA history). +The empty QA snapshot is not new: it was already empty on `4f931e4`. Earlier the same day the Historian still showed recent +`UNACK_ALM` rows, for example `Z28061.HeartbeatTimeoutAlarm` at 02:04:35Z. This was not diagnosed here, since it is out of +scope for a deploy. + +**#170 (operator-authenticated ack).** This is now deployed on DEV and QA. No consumer on either tier acks Galaxy alarms +through the gateway: OtOpcUa has no Galaxy driver on vd03 or QA, and ScadaBridge never acks. So the Historian +`ACK_ALM`-with-the-real-operator check has no live caller yet. It belongs to the first Galaxy driver authored against +these gateways (lmxopcua `dc0a24f2`+ plus an `AlarmAck` fallback account or per-key identity; see #170's comments). + +## Backup and rollback + +Each box has the following under `E:\ApiInstall\MxAccessGateway\`: +- previous build (`4f931e4`): `app.bak--issue229` and `worker.bak--issue229`; +- `_deploy-backup--issue229\`, ACL SYSTEM + Administrators only, inheritance off. It holds `AppEnvironmentExtra.txt`, the service `.reg` export, `env-fingerprint.txt` and a copy of the KEK; +- store copies: `data\gateway-auth.db{,-wal,-shm}.bak--issue229` and `data\secrets.db.bak--issue229`; +- notes: `_DEPLOY-NOTES-.txt`, which holds names and hash prefixes only. + +Stamps: vq03 `20261112-233019`, vq04 `20261112-233532`. + +To roll back, run this on the box, elevated: +`powershell -NoProfile -ExecutionPolicy Bypass -File E:\psrem-stage\mxgw-229-qa-76a71bc8\rollback229qa.ps1 -Stamp ` +- Add `-RestoreStores` only if the stores are damaged; there is no schema change. +- The rollback was dry-run on both boxes after the deploy. It resolves `app.bak-`/`worker.bak--issue229`, and the env block matches the saved pre-upgrade fingerprint. +- Rollback brings back the old (wrong) alarm and Write2 times, and the pre-#170 ack. diff --git a/docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md b/docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md index 6265c023..9c19eac4 100644 --- a/docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md +++ b/docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md @@ -3,7 +3,7 @@ Server and x86 Worker deployed together from `origin/main` `76a71bc8` (`76a71bc82d7ed41bad307d866257cb46951d9d2f`). It ships #226 (`892bdc1a`, wnwrap alarm times are UTC), #228 (`9ef6fc84`, the `Write2` time argument is sent as box-local time) and #227 (additive, opt-in `emit_stream_attached`). There are no config, package or schema changes since `2b13564e`. -Procedure: `2026-11-12-pending-worker-time-fixes-226-228.md`. QA is not done yet (#229 stays open; gated by #170). +Procedure: `2026-11-12-pending-worker-time-fixes-226-228.md`. QA followed the same night: `2026-11-12-qa-vq03-vq04-76a71bc8.md`. ## Build and package