From 31f7f9fc6e432aa710401c6fca10a4f8755d47a2 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 12 Nov 2026 21:23:40 -0500 Subject: [PATCH] docs(deploy): DEV vd03 on 76a71bc8, Worker time fixes #226 + #228 verified live (#229) Server and x86 Worker from 76a71bc8 on wonder-app-vd03, about 9 s downtime, env block and pepper prefix unchanged. Write2 KeepsSentTimestamp passes; RuntimeAlert reads 2026-07-28T20:02:46.830Z. QA still pending (#170). Upgrade-script gaps filed as #230. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi --- ...11-12-pending-worker-time-fixes-226-228.md | 2 +- ...6-11-12-vd03-76a71bc8-worker-time-fixes.md | 80 +++++++++++++++++++ 2 files changed, 81 insertions(+), 1 deletion(-) create mode 100644 docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md diff --git a/docs/deploy/2026-11-12-pending-worker-time-fixes-226-228.md b/docs/deploy/2026-11-12-pending-worker-time-fixes-226-228.md index b44601bc..62bfe5c8 100644 --- a/docs/deploy/2026-11-12-pending-worker-time-fixes-226-228.md +++ b/docs/deploy/2026-11-12-pending-worker-time-fixes-226-228.md @@ -1,6 +1,6 @@ # Pending: Worker time-conversion fixes #226 + #228 (DEV first, then QA) -**Status: NOT deployed.** This is the deploy note for the next gateway deploy. Ship both fixes in one +**Status: DEV deployed 2026-11-12 (`76a71bc8`, both checks passed, record `2026-11-12-vd03-76a71bc8-worker-time-fixes.md`); QA NOT deployed (#229, gated by #170).** This is the deploy note for the next gateway deploy. Ship both fixes in one deploy: they are both in the x86 Worker, both change timestamps consumers see, and both are on `main`. | Fix | Commit | Behaviour change on the box | diff --git a/docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md b/docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md new file mode 100644 index 00000000..f7b8dc8d --- /dev/null +++ b/docs/deploy/2026-11-12-vd03-76a71bc8-worker-time-fixes.md @@ -0,0 +1,80 @@ +# 2026-11-12: wonder-app-vd03 (DEV): `2b13564e` -> `76a71bc8` (Worker time fixes #226 + #228, mxaccessgw#229) + +Server and x86 Worker deployed together from `origin/main` `76a71bc8` (`76a71bc82d7ed41bad307d866257cb46951d9d2f`). +It ships #226 (`892bdc1a`, wnwrap alarm times are UTC), #228 (`9ef6fc84`, the `Write2` time argument is sent as +box-local time) and #227 (additive, opt-in `emit_stream_attached`). There are no config, package or schema changes since `2b13564e`. +Procedure: `2026-11-12-pending-worker-time-fixes-226-228.md`. QA is not done yet (#229 stays open; gated by #170). + +## Build and package + +| Part | Where | How | +|---|---|---| +| Source | Mac | `git archive 76a71bc8` (tar sha256 `9f3bdf19...de04f24`), the same tar for both parts | +| Worker | windev, scratch `C:\build\_tmp-mxgw-229-76a71bc8` | `dotnet build ...Worker.csproj -c Release -p:Platform=x86 -p:SourceRevisionId=76a71bc8`; **Worker.Tests x86: 783 passed, 0 failed, 12 skipped**. Apphost + `.config` renamed to `MxGateway.Worker.exe(.config)`; the 19-file set is name-identical to the live one, and the `.exe.config` is byte-identical. PE x86. | +| Server | Mac (SDK 10.0.403) | `dotnet publish ...Server.csproj -c Release -r win-x64 --self-contained true -p:SourceRevisionId=76a71bc8` (453 files) + `MxGateway.Server.exe` as a byte copy of the apphost (the NSSM `Application`) | + +The package is `mxgw-76a71bc8.zip` (sha256 `B2771EEC...F211A302`) with `MANIFEST.sha256` covering all 473 files. +The hash was checked at every hop (windev -> Mac -> vd03 SMB push, plus an SMB pull round trip). ProductVersion is +`0.1.2+76a71bc8` on both exes. + +**A windev-built Server was rejected.** The windev SDK (10.0.204) publishes with runtime pack **10.0.8**, but live +was on **10.0.14**, so it would have downgraded the self-contained runtime. The preflight showed this as a differing +`mscordaccore_amd64_amd64_10.0.826.*.dll`. Build the self-contained Server where the SDK carries the newest runtime (the Mac +here, as for `2b13564e`), or update the windev SDK. + +## Why not `Upgrade-MxGatewayService.ps1` + +The repo script has not been run on a real box. Reading it against vd03 found two blockers: +- Its one-gateway and process-gone guards match only the `ZB.MOM.WW.MxGateway.Server` / `.Worker` process names. + vd03 runs `MxGateway.Server.exe` / `MxGateway.Worker.exe`, so every OrphanWorkerTerminator guard and the + STOP_PENDING kill would pass vacuously. The staged-Worker check also requires `ZB.MOM.WW.MxGateway.Worker.exe`. +- It carries only `appsettings*.json` forward. vd03's `Server\` also holds `dashboard.pfx`. + +The deploy used `preflight/swap/rollback` scripts adapted from the proven `2b13564e` set +(`E:\psrem-stage\mxgw-229-76a71bc8\{common,pre,swap,rollback}229.ps1`). They match both exe names, +carry `appsettings.json` (hash-checked), `appsettings.Development.json` and `dashboard.pfx` forward, never write NSSM +`Application`/`AppDirectory`/`AppEnvironmentExtra`, and keep backups in the protected stage folder. The script gaps are filed +as #230. + +## Timeline (box-local, EST) + +- 21:19:20 preflight PASS (read-only). Zip and manifest OK; live `0.1.2+2b13564e` Server and Worker; exactly one gateway + process; env 7 entries; `MxGateway__ApiKeyPepper` sha256 prefix `0d01abcc1fe5`; `Server\appsettings.json` sha256 + `464ACE1B...` with no pepper; E: 179.9 GB free. +- 21:19:48 backup: `Server\appsettings*.json`, `dashboard.pfx` and the service `.reg` copied to the backup folder. +- 21:19:50 stop (`Stop-Service -NoWait`). Stopped in 2 s with no STOP_PENDING and 0 gateway/worker processes. `data\` (6 files) copied, hash-verified. +- 21:19:52 rename `Server`/`Worker` -> `*.bak.20261112T211945`. New files copied: Server 454, Worker 19. `appsettings.json` + restored (hash match), `appsettings.Development.json` and `dashboard.pfx` carried forward (hash match). +- 21:19:59 `/health/ready` 200, 4 s after start. **Downtime (stop -> ready) about 9 s.** After 30 s: same gateway PID + (17972), one gateway process on the box, `/health/active` 2 of 2 sessions ready, `/healthz` 200. + +## Verification + +| Check | Result | +|---|---| +| Build stamp | Server and Worker `ProductVersion 0.1.2+76a71bc8` | +| Health (from the Mac) | `/healthz` 200, `/health/ready` 200 (auth-store, worker-executable, galaxy-sql), `/health/active` 200 (2 of 2 sessions) | +| NSSM env block | 7 entries, names identical, block hash `3f283e9a1d57` before, mid-swap and after; pepper prefix `0d01abcc1fe5` unchanged. Never written. | +| `Server\appsettings.json` | `464ACE1B...` before and after (the #171 no-pepper version, not the install-root master) | +| Existing key authenticates | a key minted **before** the stop opened a session, read `MESReceiver_025.MoveInBatchID` (`0`, Good) and closed. The Worker path is proven. | +| Wrong key | `MxGatewayAuthenticationException` (Unauthenticated) | +| **#228** | `LiveGatewayWrite2Tests` with `MXGATEWAY_LIVE_GATEWAY_ASSERT_WRITE2_TIMESTAMP=1` on `MESReceiver_025.MoveInBatchID`: **2/2 passed**. `KeepsSentTimestamp`: sent `2026-11-13T02:20:34.000Z`, stored `02:20:34.000Z` (on `2b13564e` it was stored +5 h). The value was restored to `0` both times. | +| **#226** | `QueryActiveAlarms`: `Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert` last transition **`2026-07-28T20:02:46.830Z`** (before: `2026-07-29T01:02:46.830Z`), which matches AVEVA history. `...TableAlarms_008.CommunicationAlarm` moved `2026-11-10T16:44:38.420Z` -> `11:44:38.420Z` (-5 h, EST). | +| ScadaBridge vd03 | site `site-a` report 02:21:07Z: MxGateway `Connected`, 810/810 resolved, good 398 / bad 412, identical to the pre-deploy report. `Z28061`/`Z28062` debug snapshots identical. | +| OtOpcUa vd03 | `:9000/health/ready` and `/health/active` 200 (no Galaxy driver on vd03) | +| Folder ACLs | New `Server\`/`Worker\` inherit from `E:\ApiInstall\MxGateway` and carry the inherited `NT SERVICE\OtOpcUaScriptWorker` deny (lmxopcua#891). The old folders also had a redundant explicit `BUILTIN\Administrators` FullControl that the parent already grants, so the SDDL differs but nothing is loosened. | + +**Test key:** `issue229-write2-test` was minted on the vd03 dashboard. Scopes: `session:open`, `session:close`, `invoke:read`, +`invoke:write`, `events:read`; read and write globs `MESReceiver_025.*`. The secret only lived in a mode-0600 scratch file and +was never printed. Afterwards the key was revoked and then deleted, a probe with it got Unauthenticated, the scratch file was +deleted, and the key list is back to the original 6. + +## Backup and rollback + +- Previous install: `E:\ApiInstall\MxGateway\Server.bak.20261112T211945` + `Worker.bak.20261112T211945` (`2b13564e`). +- `E:\psrem-stage\mxgw-229-76a71bc8\backup-20261112T211945\` (stage folder ACL: SYSTEM + Administrators only, inheritance off): + `data\` (auth and secrets store triplets, `kek.key`, `dashboard.pfx`), `Server-appsettings*.json`, `Server-dashboard.pfx`, + `MxAccessGw-servicekey-20261112T211945.reg`. +- Notes: `E:\ApiInstall\MxGateway\_DEPLOY-NOTES-20261112T211945-issue229.txt` (names and hash prefixes only). +- Rollback: `powershell -NoProfile -ExecutionPolicy Bypass -File E:\psrem-stage\mxgw-229-76a71bc8\rollback229.ps1 -Stamp 20261112T211945` + (add `-RestoreData` only if the stores are damaged). There is no store or schema change, so rollback restores the old (wrong) times and nothing else.