Deploy Worker time fixes #226 + #228 to DEV, then QA #229

Closed
opened 2026-11-12 21:00:33 -05:00 by dohertj2 · 3 comments
Owner

Deploy the x86 Worker time-conversion fixes #226 (892bdc1a) and #228 (9ef6fc84) together: DEV wonder-app-vd03 first, then QA vq03/vq04. DEV and QA deployed 2026-11-12 (76a71bc8). The exact procedure is in docs/deploy/2026-11-12-pending-worker-time-fixes-226-228.md (76a71bc8).

  • Build Server + Worker from one main SHA at or after 9ef6fc84 on windev (Worker -p:Platform=x86), and deploy with scripts/deploy/Upgrade-MxGatewayService.ps1 per docs/runbooks/DeployUpgradeNssmService.md.
  • DEV (2b13564e now): the only extra change is #227, which is additive and opt-in. Post-deploy checks:
    • #228: LiveGatewayWrite2Tests with MXGATEWAY_LIVE_GATEWAY_ASSERT_WRITE2_TIMESTAMP=1 on MESReceiver_025.MoveInBatchID passes. Stored timestamp = sent timestamp.
    • #226: QueryActiveAlarms for Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert reads 2026-07-28T20:02:46.830Z.
  • QA (4f931e4 now): a main build also carries the operator-authenticated alarm ack, so #170's prerequisites apply first: lmxopcua dc0a24f2+ plus an AlarmAck fallback account or per-key identities on every QA OtOpcUa Galaxy driver node. Then deploy rolling, vq03 then vq04, and run the same checks.
  • Consumer impact: wnwrap alarm times move 4-5 h earlier, to correct UTC (#226). Timestamped writes now store the instant sent (#228). No consumer calls Write2 today.

Close this issue once both tiers are deployed and checked, and the deploy records are written.

Checklist

  • Build Server + x86 Worker from one main SHA (76a71bc8); Worker.Tests x86 783 passed / 0 failed / 12 skipped
  • DEV vd03 deployed 2b13564e -> 76a71bc8 (2026-11-12, about 9 s downtime, env block and pepper prefix unchanged)
  • DEV #228 check: LiveGatewayWrite2Tests with ASSERT_WRITE2_TIMESTAMP=1 passed (sent and stored both 02:20:34.000Z)
  • DEV #226 check: RuntimeAlert reads 2026-07-28T20:02:46.830Z
  • DEV deploy record (docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md, 31f7f9fc)
  • QA: #170 prerequisites on both QA OtOpcUa nodes (lmxopcua dc0a24f2+, plus an AlarmAck fallback account or per-key identities)
  • QA vq03 deployed and checked (#228 with a QA Z28062 attribute, #226 against the QA Historian)
  • QA vq04 deployed and checked
  • QA deploy record and scadaproj index row
  • Follow-up: #230 (the upgrade script's process guards miss the vd03 exe names, and it does not carry dashboard.pfx over)
Deploy the x86 Worker time-conversion fixes **#226** (`892bdc1a`) and **#228** (`9ef6fc84`) together: DEV `wonder-app-vd03` first, then QA `vq03`/`vq04`. **DEV and QA deployed 2026-11-12 (`76a71bc8`).** The exact procedure is in `docs/deploy/2026-11-12-pending-worker-time-fixes-226-228.md` (`76a71bc8`). - Build Server + Worker from one `main` SHA at or after `9ef6fc84` on windev (Worker `-p:Platform=x86`), and deploy with `scripts/deploy/Upgrade-MxGatewayService.ps1` per `docs/runbooks/DeployUpgradeNssmService.md`. - **DEV** (`2b13564e` now): the only extra change is #227, which is additive and opt-in. Post-deploy checks: - #228: `LiveGatewayWrite2Tests` with `MXGATEWAY_LIVE_GATEWAY_ASSERT_WRITE2_TIMESTAMP=1` on `MESReceiver_025.MoveInBatchID` passes. Stored timestamp = sent timestamp. - #226: `QueryActiveAlarms` for `Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert` reads `2026-07-28T20:02:46.830Z`. - **QA** (`4f931e4` now): a `main` build also carries the operator-authenticated alarm ack, so **#170's prerequisites apply first**: lmxopcua `dc0a24f2`+ plus an `AlarmAck` fallback account or per-key identities on every QA OtOpcUa Galaxy driver node. Then deploy rolling, vq03 then vq04, and run the same checks. - **Consumer impact:** wnwrap alarm times move 4-5 h earlier, to correct UTC (#226). Timestamped writes now store the instant sent (#228). No consumer calls `Write2` today. Close this issue once both tiers are deployed and checked, and the deploy records are written. ## Checklist - [x] Build Server + x86 Worker from one `main` SHA (`76a71bc8`); Worker.Tests x86 783 passed / 0 failed / 12 skipped - [x] DEV vd03 deployed `2b13564e` -> `76a71bc8` (2026-11-12, about 9 s downtime, env block and pepper prefix unchanged) - [x] DEV #228 check: `LiveGatewayWrite2Tests` with `ASSERT_WRITE2_TIMESTAMP=1` passed (sent and stored both `02:20:34.000Z`) - [x] DEV #226 check: `RuntimeAlert` reads `2026-07-28T20:02:46.830Z` - [x] DEV deploy record (`docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md`, `31f7f9fc`) - [x] QA: #170 prerequisites on both QA OtOpcUa nodes (lmxopcua `dc0a24f2`+, plus an `AlarmAck` fallback account or per-key identities) - [x] QA vq03 deployed and checked (#228 with a QA Z28062 attribute, #226 against the QA Historian) - [x] QA vq04 deployed and checked - [x] QA deploy record and scadaproj index row - Follow-up: #230 (the upgrade script's process guards miss the vd03 exe names, and it does not carry `dashboard.pfx` over)
dohertj2 added the area:workerpriority:p2type:deploy-gate labels 2026-11-12 21:00:33 -05:00
Author
Owner

DEV wonder-app-vd03 deployed, 2026-11-12: 2b13564e -> 76a71bc8. Both checks pass. Record: docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md (31f7f9fc).

Build. Built from one git archive of 76a71bc8.

  • Worker on windev: -c Release -p:Platform=x86. Worker.Tests x86: 783 passed, 0 failed, 12 skipped.
  • Server self-contained win-x64 on the Mac. A windev-published Server would have downgraded the bundled runtime from 10.0.14 to 10.0.8, so I rejected it.
  • Both exes are 0.1.2+76a71bc8. The zip was hash-checked at every hop.

Deploy. Stop, back up, rename-swap Server and Worker, start once. About 9 s of downtime (stop at 21:19:50 EST, ready at 21:19:59).

  • The stop took 2 s, with no STOP_PENDING. There was exactly one gateway process before and after, on the same PID for 30 s.
  • The NSSM env block was never written. It has 7 entries with identical names and the same block hash before, mid-swap and after. The MxGateway__ApiKeyPepper sha256 prefix is still 0d01abcc1fe5.
  • Server\appsettings.json was restored hash-identical (464ACE1B..., the #171 version without the pepper). install-service.bat and the install-root master were not used.
  • I did not use Upgrade-MxGatewayService.ps1. Its process guards only match ZB.MOM.WW.MxGateway.* names, but vd03 runs MxGateway.Server.exe and MxGateway.Worker.exe. It also drops dashboard.pfx. Filed as #230. I used adapted copies of the proven 2b13564e scripts instead.

Verification

Check Result
Health /healthz, /health/ready, /health/active all 200 (2 of 2 sessions after the restart)
Existing key A key minted before the stop opened a session, read MESReceiver_025.MoveInBatchID, and closed. A wrong key got Unauthenticated.
#228 LiveGatewayWrite2Tests with ..._ASSERT_WRITE2_TIMESTAMP=1 on MESReceiver_025.MoveInBatchID: 2/2 passed. KeepsSentTimestamp sent 02:20:34.000Z and stored 02:20:34.000Z. The value was restored to 0.
#226 QueryActiveAlarms: Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert reads 2026-07-28T20:02:46.830Z (before the deploy: 2026-07-29T01:02:46.830Z). CommunicationAlarm moved from 16:44:38.420Z to 11:44:38.420Z on 11-10, which is -5 h.
ScadaBridge vd03 MxGateway Connected, 810/810 tags, good 398 / bad 412. Identical to the pre-deploy report; the Z28061/Z28062 snapshots are also identical.
OtOpcUa vd03 /health/ready and /health/active 200

Test key. issue229-write2-test, with scopes session open/close, invoke read/write and events:read, and globs MESReceiver_025.*. The secret only lived in a mode-0600 scratch file and was never printed. Afterwards I revoked and deleted it, and a probe with it then got Unauthenticated. The scratch file is deleted, and the key list is back to the original 6.

Rollback.

  • Previous install: Server.bak.20261112T211945 + Worker.bak.20261112T211945.
  • Backups (data\ stores + kek, appsettings, dashboard.pfx, service .reg): E:\psrem-stage\mxgw-229-76a71bc8\backup-20261112T211945\, readable by SYSTEM and Administrators only.
  • Script: rollback229.ps1 -Stamp 20261112T211945 in the same stage folder.

Keeping this issue open for QA (vq03/vq04 still on 4f931e4). The #170 prerequisites still apply.

**DEV wonder-app-vd03 deployed, 2026-11-12: `2b13564e` -> `76a71bc8`. Both checks pass.** Record: `docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md` (`31f7f9fc`). **Build.** Built from one `git archive` of `76a71bc8`. - Worker on windev: `-c Release -p:Platform=x86`. **Worker.Tests x86: 783 passed, 0 failed, 12 skipped.** - Server self-contained win-x64 on the Mac. A windev-published Server would have downgraded the bundled runtime from 10.0.14 to 10.0.8, so I rejected it. - Both exes are `0.1.2+76a71bc8`. The zip was hash-checked at every hop. **Deploy.** Stop, back up, rename-swap Server and Worker, start once. **About 9 s of downtime** (stop at 21:19:50 EST, ready at 21:19:59). - The stop took 2 s, with no STOP_PENDING. There was exactly one gateway process before and after, on the same PID for 30 s. - The NSSM env block was never written. It has 7 entries with identical names and the same block hash before, mid-swap and after. The `MxGateway__ApiKeyPepper` sha256 prefix is still `0d01abcc1fe5`. - `Server\appsettings.json` was restored hash-identical (`464ACE1B...`, the #171 version without the pepper). `install-service.bat` and the install-root master were not used. - I did not use `Upgrade-MxGatewayService.ps1`. Its process guards only match `ZB.MOM.WW.MxGateway.*` names, but vd03 runs `MxGateway.Server.exe` and `MxGateway.Worker.exe`. It also drops `dashboard.pfx`. Filed as #230. I used adapted copies of the proven `2b13564e` scripts instead. **Verification** | Check | Result | |---|---| | Health | `/healthz`, `/health/ready`, `/health/active` all 200 (2 of 2 sessions after the restart) | | Existing key | A key minted before the stop opened a session, read `MESReceiver_025.MoveInBatchID`, and closed. A wrong key got Unauthenticated. | | **#228** | `LiveGatewayWrite2Tests` with `..._ASSERT_WRITE2_TIMESTAMP=1` on `MESReceiver_025.MoveInBatchID`: **2/2 passed**. `KeepsSentTimestamp` sent `02:20:34.000Z` and stored `02:20:34.000Z`. The value was restored to `0`. | | **#226** | `QueryActiveAlarms`: `Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert` reads **`2026-07-28T20:02:46.830Z`** (before the deploy: `2026-07-29T01:02:46.830Z`). `CommunicationAlarm` moved from `16:44:38.420Z` to `11:44:38.420Z` on 11-10, which is -5 h. | | ScadaBridge vd03 | MxGateway `Connected`, 810/810 tags, good 398 / bad 412. Identical to the pre-deploy report; the Z28061/Z28062 snapshots are also identical. | | OtOpcUa vd03 | `/health/ready` and `/health/active` 200 | **Test key.** `issue229-write2-test`, with scopes session open/close, invoke read/write and events:read, and globs `MESReceiver_025.*`. The secret only lived in a mode-0600 scratch file and was never printed. Afterwards I **revoked and deleted** it, and a probe with it then got Unauthenticated. The scratch file is deleted, and the key list is back to the original 6. **Rollback.** - Previous install: `Server.bak.20261112T211945` + `Worker.bak.20261112T211945`. - Backups (data\ stores + kek, appsettings, `dashboard.pfx`, service `.reg`): `E:\psrem-stage\mxgw-229-76a71bc8\backup-20261112T211945\`, readable by SYSTEM and Administrators only. - Script: `rollback229.ps1 -Stamp 20261112T211945` in the same stage folder. Keeping this issue open for **QA** (vq03/vq04 still on `4f931e4`). The #170 prerequisites still apply.
Author
Owner

For QA (vq03/vq04): use the fixed scripts/deploy/Upgrade-MxGatewayService.ps1 from main 4890a3d1 (#230, closed), not the pre-#230 copy.

The old copy's process guards were blind to MxGateway.Server.exe / MxGateway.Worker.exe. It also dropped Server\dashboard.pfx, and it would have refused a file-based master key.

The fixed script fails closed if it can't see the running gateway, carries dashboard.pfx, fingerprints the NSSM env block (pepper included) without ever writing it, refuses a runtime downgrade, and can roll back the .bak.<stamp> sets left on vd03.

Run it with -DryRun on each box first and read the plan. In particular, check that the gateway-process line shows MxGateway.Server#<pid> as the service host's child, that the carry plan includes dashboard.pfx, and that the env fingerprint names match what you expect. Then run it for real.

It has had a real dry run on windev but no real upgrade yet. Keep runbook section 8 at hand.

For QA (vq03/vq04): use the fixed `scripts/deploy/Upgrade-MxGatewayService.ps1` from `main` `4890a3d1` (#230, closed), not the pre-#230 copy. The old copy's process guards were blind to `MxGateway.Server.exe` / `MxGateway.Worker.exe`. It also dropped `Server\dashboard.pfx`, and it would have refused a file-based master key. The fixed script fails closed if it can't see the running gateway, carries `dashboard.pfx`, fingerprints the NSSM env block (pepper included) without ever writing it, refuses a runtime downgrade, and can roll back the `.bak.<stamp>` sets left on vd03. Run it with `-DryRun` on each box first and read the plan. In particular, check that the gateway-process line shows `MxGateway.Server#<pid>` as the service host's child, that the carry plan includes `dashboard.pfx`, and that the env fingerprint names match what you expect. Then run it for real. It has had a real dry run on windev but no real upgrade yet. Keep runbook section 8 at hand.
Author
Owner

QA wonder-app-vq03 + vq04 deployed, 2026-11-12: 4f931e4 -> 76a71bc8, rolling (vq03 first, then vq04). Record: docs/deploy/2026-11-12-qa-vq03-vq04-76a71bc8.md (2cc0d575).

Artifacts. I reused the DEV-deployed package, pulled from vd03's stage over SMB.

  • mxgw-76a71bc8.zip sha256 B2771EEC2B6406A7AA84D7A125176F0C0F9A97138F82808AF519A693F211A302.
  • All 473 files match MANIFEST.sha256 on the Mac and again on each box.
  • Server MxGateway.Server.exe = ZB.MOM.WW.MxGateway.Server.exe sha256 7d7974f0…6ebedc; Worker MxGateway.Worker.exe sha256 844ab6ad…38f1871, PE32 x86.
  • Every exe is 0.1.2+76a71bc8. The runtime goes from 10.0.5 to 10.0.14, an upgrade.

Prerequisites (#170).

  • QA OtOpcUa has no Galaxy driver. An anonymous browse of both nodes shows only the script-runner-canary Calculation driver, so no OtOpcUa node acks through this gateway.
  • ScadaBridge QA b3021e7c never calls AcknowledgeAlarm (NativeAlarmActor is read-only). It is on Client 0.2.1, the same pin as DEV ScadaBridge, which runs fine against 76a71bc8.

Procedure. I used the #230-fixed Upgrade-MxGatewayService.ps1 from 4890a3d1. This was its first real run.

  • On each box: a dry run, then the real run, then a rollback dry run.
  • The dry run showed ZB.MOM.WW.MxGateway.Server#<pid> as the service host's child and carried appsettings*.json. QA has no dashboard.pfx.
  • Env block: 33 names. vq03 and vq04 differ only in the Galaxy connection string. The pepper prefix is 88188ab11b50 on both.
  • The env block was identical at every checkpoint and was never written. install-service.bat was not used, and no second gateway was started.
  • Downtime: vq03 about 8 s, vq04 about 9 s.
Check vq03 vq04
Health /healthz, /health/ready, /health/active 200 (2/2 sessions) 200 (1/1)
Existing key otopcua-galaxy-qa galaxy-test-connection ok; open/close session ok (Worker path) same
Bogus key Missing or invalid API key. same
#228 on MESReceiver_025.MoveInBatchID (Z28062 Left, the same name as DEV) old build KeepsSentTimestamp FAILED: sent 04:26:07Z, stored 09:26:07Z (+5 h; negative control). New build: 2/2 passed, sent = stored 04:32:02.000Z 2/2 passed, sent = stored 04:38:27.000Z
#226 could not be observed on QA. The gateway alarm feed (ALARMMGR) snapshot is empty before and after the deploy, and the Historian logged no alarm transition in the capture windows. The post-deploy stream on vq03 ran from 04:32Z to 04:47:30Z (provider ALARMMGR since 04:30:31Z): empty snapshot, 0 transitions. The Historian has 0 alarm rows for 03:48Z–04:48Z. The fix is the same Worker binary DEV verified live. same
ScadaBridge QA site MxGateway Connected, 2196/2196 Good about 2 min after the restart; Z28061/Z28062 244/244 Good n/a (SB uses only vq03)
OtOpcUa QA ServiceLevel 240 (vq03) / 250 (vq04) throughout same
Overview :5320 12/12 Up same

Temporary keys qa229-write2-test, scoped to MESReceiver_025.*, were minted before each box's deploy and deleted afterwards. A probe with each deleted key is refused, and each box is back to its two original keys. Only Z28062 was written; each test wrote 212212 and restored 0.

Rollback (on the box): E:\psrem-stage\mxgw-229-qa-76a71bc8\rollback229qa.ps1 -Stamp 20261112-233019 on vq03, -Stamp 20261112-233532 on vq04. Backups are under E:\ApiInstall\MxAccessGateway\: app.bak-<stamp>-issue229, worker.bak-…, the protected _deploy-backup-…, and data\*.bak-….

Both tiers are deployed and checked, and the records are written, so I'm closing this issue. The empty QA alarm snapshot predates this deploy and was not investigated.

**QA wonder-app-vq03 + vq04 deployed, 2026-11-12: `4f931e4` -> `76a71bc8`, rolling (vq03 first, then vq04).** Record: `docs/deploy/2026-11-12-qa-vq03-vq04-76a71bc8.md` (`2cc0d575`). **Artifacts.** I reused the DEV-deployed package, pulled from vd03's stage over SMB. - `mxgw-76a71bc8.zip` sha256 `B2771EEC2B6406A7AA84D7A125176F0C0F9A97138F82808AF519A693F211A302`. - All 473 files match `MANIFEST.sha256` on the Mac and again on each box. - Server `MxGateway.Server.exe` = `ZB.MOM.WW.MxGateway.Server.exe` sha256 `7d7974f0…6ebedc`; Worker `MxGateway.Worker.exe` sha256 `844ab6ad…38f1871`, PE32 x86. - Every exe is `0.1.2+76a71bc8`. The runtime goes from 10.0.5 to 10.0.14, an upgrade. **Prerequisites (#170).** - QA OtOpcUa has no Galaxy driver. An anonymous browse of both nodes shows only the `script-runner-canary` Calculation driver, so no OtOpcUa node acks through this gateway. - ScadaBridge QA `b3021e7c` never calls `AcknowledgeAlarm` (`NativeAlarmActor` is read-only). It is on Client 0.2.1, the same pin as DEV ScadaBridge, which runs fine against `76a71bc8`. **Procedure.** I used the #230-fixed `Upgrade-MxGatewayService.ps1` from `4890a3d1`. This was its first real run. - On each box: a dry run, then the real run, then a rollback dry run. - The dry run showed `ZB.MOM.WW.MxGateway.Server#<pid>` as the service host's child and carried `appsettings*.json`. QA has no `dashboard.pfx`. - Env block: 33 names. vq03 and vq04 differ only in the Galaxy connection string. The pepper prefix is `88188ab11b50` on both. - The env block was identical at every checkpoint and was never written. `install-service.bat` was not used, and no second gateway was started. - **Downtime: vq03 about 8 s, vq04 about 9 s.** | Check | vq03 | vq04 | |---|---|---| | Health | `/healthz`, `/health/ready`, `/health/active` 200 (2/2 sessions) | 200 (1/1) | | Existing key `otopcua-galaxy-qa` | `galaxy-test-connection` ok; open/close session ok (Worker path) | same | | Bogus key | `Missing or invalid API key.` | same | | **#228** on `MESReceiver_025.MoveInBatchID` (Z28062 Left, the same name as DEV) | **old build `KeepsSentTimestamp` FAILED: sent `04:26:07Z`, stored `09:26:07Z` (+5 h; negative control). New build: 2/2 passed, sent = stored `04:32:02.000Z`** | 2/2 passed, sent = stored `04:38:27.000Z` | | **#226** | **could not be observed on QA.** The gateway alarm feed (ALARMMGR) snapshot is empty before and after the deploy, and the Historian logged no alarm transition in the capture windows. The post-deploy stream on vq03 ran from 04:32Z to 04:47:30Z (provider ALARMMGR since 04:30:31Z): empty snapshot, 0 transitions. The Historian has 0 alarm rows for 03:48Z–04:48Z. The fix is the same Worker binary DEV verified live. | same | | ScadaBridge QA site | MxGateway Connected, 2196/2196 Good about 2 min after the restart; Z28061/Z28062 244/244 Good | n/a (SB uses only vq03) | | OtOpcUa QA ServiceLevel | 240 (vq03) / 250 (vq04) throughout | same | | Overview `:5320` | 12/12 Up | same | Temporary keys `qa229-write2-test`, scoped to `MESReceiver_025.*`, were minted before each box's deploy and deleted afterwards. A probe with each deleted key is refused, and each box is back to its two original keys. Only Z28062 was written; each test wrote `212212` and restored `0`. **Rollback** (on the box): `E:\psrem-stage\mxgw-229-qa-76a71bc8\rollback229qa.ps1 -Stamp 20261112-233019` on vq03, `-Stamp 20261112-233532` on vq04. Backups are under `E:\ApiInstall\MxAccessGateway\`: `app.bak-<stamp>-issue229`, `worker.bak-…`, the protected `_deploy-backup-…`, and `data\*.bak-…`. Both tiers are deployed and checked, and the records are written, so I'm closing this issue. The empty QA alarm snapshot predates this deploy and was not investigated.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/mxaccessgw#229