Files
mxaccessgw/docs/deploy/2026-11-22-prod-par-e74918cc.md
T
Joseph Doherty 8e0e232fd6
ci / windows-x86 (push) Successful in 2m6s
ci / java (push) Failing after 6m33s
ci / portable (push) Failing after 10m42s
deploy(prod): mxgw e74918cc on par-vp03/par-vp04 (site-par Connected; galaxy-sql waits on scadaproj#60)
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-22 07:45:06 -05:00

89 lines
5.9 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 2026-11-22: PROD install of MxAccessGateway `e74918cc` on wonder-par-vp03 / wonder-par-vp04 (site-par)
Owner: "keep checking par-vp04 every minute. when its ready then execute the par server updates (both)". This follows the
2026-11-20 app-box install ([`2026-11-20-prod-e74918cc.md`](2026-11-20-prod-e74918cc.md)); the build, stage pack and env
are the same. Driven from the Mac with the `winrm` CLI (profiles `parvp03`/`parvp04`, NAM\adm_dohertj2). No firewall changes, and no
ScadaBridge service was touched.
## Gate: WinPlatform / MXAccess on the PAR boxes
Checked over SMB only.
- **par-vp03** already had `LmxProxy.dll` at 12:0xZ: 318,496 bytes, sha256 `bd75a2fd…7ef15459`, the same file as vp03. Its
`ArchestrA\Framework\Bin` held the full platform runtime (102 entries).
- **par-vp04** started with 49 entries and no MXAccess. It was polled every minute. `LmxProxy.dll` appeared at about
12:19Z (101 entries), and the folder was stable at 102 entries from 12:21Z. The poller declared it ready at 12:23:45Z
with the same sha256 `bd75a2fde222…`.
## Stage fixes (by SMB, before anything ran)
- **PATCH 4 (`Install-MxGatewayService.ps1`, nssm status probe under `Continue`; mxaccessgw#234):** both PAR boxes still had
the PATCH 3 copy (`11a23e9f…`). The patched copy (`fb6f0f47…`) was pushed, the old one kept as
`Install-MxGatewayService.ps1.patch3`, and the `SHA256SUMS.txt` line updated on both boxes.
- **Value files:** `_values\mxgw-<box>\` on each box held three files, all from the Keychain, never printed and never on argv:
- `MxGateway__ApiKeyPepper.txt`: `zb-prod/mxgw-pepper`, the same as vp03/vp04;
- `MxGateway__Galaxy__ConnectionString.txt`:
`Server=wonder-app-vp03.zmr.zimmer.com;Database=ZB;Integrated Security=True;TrustServerCertificate=True;Encrypt=False;`;
- the LDAP seed file `nam-ldap-mxgw.txt`.
- **API-key store:** vp03's gateway could not be stopped for a clean copy, because site-app runs on it. So vp04's
`gateway-auth.db` (+wal/shm) was pulled live; it is vp03's store as copied on 2026-11-20. Checked on the Mac:
- consolidated (the WAL held 0 frames);
- `integrity_check` ok;
- one key, `scadabridge-prod`, scopes `events:read invoke:read invoke:write metadata:read session:close session:open`,
not revoked;
- single-file sha256 `3c5705d7…43bffe55`.
## Install (UTC)
| Step | par-vp03 | par-vp04 |
|---|---|---|
| dry | 12:06:28 DONE (LmxProxy `6000.0045.2657.1` / `BD75A2FDE222`, 38 env entries, Server and Worker `0.1.2+e74918cc`, one benign WorkerDir WARN); `dry\` removed | 12:23:54 DONE, same output; `dry\` removed |
| real | 12:24:00 DONE; KEK generated `AA55B221`; service `MxAccessGw` Stopped, Automatic, LocalSystem | 12:24:09 DONE; KEK `01F9EE28`; Stopped, Automatic, LocalSystem |
| seed | 12:24:21 `nam-ldap-svc-password` → `data\secrets.db`, `readback-match=True`, value file removed | 12:24:26 same |
| key store | 12:24:25 `data\gateway-auth.db` = `3C5705D78769` (no earlier store existed) | 12:24:30 same |
| start | 12:24:35; Running; listening on 5120 and 5130; stderr empty | 12:26:11; same |
| `/health/ready` | **503**: auth-store Healthy, worker-executable Healthy, **galaxy-sql Unhealthy** | **503**, same |
**galaxy-sql: expected, same cause as vp04 (scadaproj#60).** The log shows
`Login failed for user 'ZMR\WONDER-PAR-VP03$'` / `'ZMR\WONDER-PAR-VP04$'`, `Error Number:4060`. The machine accounts reach
vp03's SQL Server but have no access to `ZB`. The gateway retries every 30 s and turns ready without a restart once the DBA
grants it (T-SQL on #60). MXAccess sessions do not depend on it.
## Consumer: ScadaBridge site-par
The site-par node on par-vp03 logged `[MxGateway] Reconnected successfully on "Primary" endpoint` → `Entering Connected
state` at **12:24:56Z**, 20 s after par-vp03's gateway started. There was no ScadaBridge change; the copied store makes the
existing `scadabridge-prod` token valid. (A no-op `deploy artifacts --site-id 1` at 12:29Z applied nothing; it was not needed.)
## Unrelated event in the same window: app-box services restarted at 12:31Z
Every ScadaBridge and mxgw service on **wonder-app-vp03 and vp04** (centrals, site-app nodes, mxgw) received a process
termination ("CLR termination hook") at 12:31:03–12:31:43Z. They came back at about 12:34:30Z. The central pair re-formed
(vp03 active), and all health endpoints were 200 by 12:35:58Z. Nothing in this run touched those boxes (SMB reads only). The
pattern looks like an external restart or reboot of both app boxes. The cause was not investigated: that would need
remote diagnostics, which the deploy-only WinRM scope does not cover.
Separately, at 10:51Z vp03 stalled for about 47.6 s ("heartbeat delayed 47592 ms", which looks like a VM freeze). vp04
downed it under `leader-auto-down`, and vp03 restarted with exit 75 as designed.
After the restart, site-app's `MxGateway` connection was not rebuilt. This is by design: on boot the site builds only the
connections a deployed instance references, and PROD has no instances yet. `deploy artifacts --site-id 2` at 12:43:13Z
rebuilt it: **Connected at 12:43:17Z**.
## Clean-up
- `_values\mxgw-par*`, `_values\mxgw-store` and the seed file were deleted on both boxes (verified; only the empty `_values\seed` folder is left).
- The local value files and key-store copies on the Mac were deleted.
## Open
- **scadaproj#60:** `ZB` read access for `ZMR\WONDER-APP-VP04$`, `ZMR\WONDER-PAR-VP03$` and `ZMR\WONDER-PAR-VP04$`.
- **Alarms are off on all four gateways** until the owner picks an area.
- **mxaccessgw#234:** the repo copy of the install script still needs PATCH 4.
## Rollback
On each PAR box: `nssm stop MxAccessGw; nssm remove MxAccessGw confirm`, then delete `E:\ApiInstall\MxAccessGateway` and
`C:\ProgramData\ZB\kek\mxgateway.kek`. site-par's connection then goes back to retrying. To serve it from the app-box
gateways instead: `data-connection update --id 1` with `10.220.153.205:5120` / `10.220.153.210:5120`, then
`deploy artifacts --site-id 1`.