deploy(prod): mxgw e74918cc on par-vp03/par-vp04 (site-par Connected; galaxy-sql waits on scadaproj#60)
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
This commit is contained in:
@@ -0,0 +1,88 @@
|
|||||||
|
# 2026-11-22: PROD install of MxAccessGateway `e74918cc` on wonder-par-vp03 / wonder-par-vp04 (site-par)
|
||||||
|
|
||||||
|
Owner: "keep checking par-vp04 every minute. when its ready then execute the par server updates (both)". This follows the
|
||||||
|
2026-11-20 app-box install ([`2026-11-20-prod-e74918cc.md`](2026-11-20-prod-e74918cc.md)); the build, stage pack and env
|
||||||
|
are the same. Driven from the Mac with the `winrm` CLI (profiles `parvp03`/`parvp04`, NAM\adm_dohertj2). No firewall changes, and no
|
||||||
|
ScadaBridge service was touched.
|
||||||
|
|
||||||
|
## Gate: WinPlatform / MXAccess on the PAR boxes
|
||||||
|
|
||||||
|
Checked over SMB only.
|
||||||
|
- **par-vp03** already had `LmxProxy.dll` at 12:0xZ: 318,496 bytes, sha256 `bd75a2fd…7ef15459`, the same file as vp03. Its
|
||||||
|
`ArchestrA\Framework\Bin` held the full platform runtime (102 entries).
|
||||||
|
- **par-vp04** started with 49 entries and no MXAccess. It was polled every minute. `LmxProxy.dll` appeared at about
|
||||||
|
12:19Z (101 entries), and the folder was stable at 102 entries from 12:21Z. The poller declared it ready at 12:23:45Z
|
||||||
|
with the same sha256 `bd75a2fde222…`.
|
||||||
|
|
||||||
|
## Stage fixes (by SMB, before anything ran)
|
||||||
|
|
||||||
|
- **PATCH 4 (`Install-MxGatewayService.ps1`, nssm status probe under `Continue`; mxaccessgw#234):** both PAR boxes still had
|
||||||
|
the PATCH 3 copy (`11a23e9f…`). The patched copy (`fb6f0f47…`) was pushed, the old one kept as
|
||||||
|
`Install-MxGatewayService.ps1.patch3`, and the `SHA256SUMS.txt` line updated on both boxes.
|
||||||
|
- **Value files:** `_values\mxgw-<box>\` on each box held three files, all from the Keychain, never printed and never on argv:
|
||||||
|
- `MxGateway__ApiKeyPepper.txt`: `zb-prod/mxgw-pepper`, the same as vp03/vp04;
|
||||||
|
- `MxGateway__Galaxy__ConnectionString.txt`:
|
||||||
|
`Server=wonder-app-vp03.zmr.zimmer.com;Database=ZB;Integrated Security=True;TrustServerCertificate=True;Encrypt=False;`;
|
||||||
|
- the LDAP seed file `nam-ldap-mxgw.txt`.
|
||||||
|
- **API-key store:** vp03's gateway could not be stopped for a clean copy, because site-app runs on it. So vp04's
|
||||||
|
`gateway-auth.db` (+wal/shm) was pulled live; it is vp03's store as copied on 2026-11-20. Checked on the Mac:
|
||||||
|
- consolidated (the WAL held 0 frames);
|
||||||
|
- `integrity_check` ok;
|
||||||
|
- one key, `scadabridge-prod`, scopes `events:read invoke:read invoke:write metadata:read session:close session:open`,
|
||||||
|
not revoked;
|
||||||
|
- single-file sha256 `3c5705d7…43bffe55`.
|
||||||
|
|
||||||
|
## Install (UTC)
|
||||||
|
|
||||||
|
| Step | par-vp03 | par-vp04 |
|
||||||
|
|---|---|---|
|
||||||
|
| dry | 12:06:28 DONE (LmxProxy `6000.0045.2657.1` / `BD75A2FDE222`, 38 env entries, Server and Worker `0.1.2+e74918cc`, one benign WorkerDir WARN); `dry\` removed | 12:23:54 DONE, same output; `dry\` removed |
|
||||||
|
| real | 12:24:00 DONE; KEK generated `AA55B221`; service `MxAccessGw` Stopped, Automatic, LocalSystem | 12:24:09 DONE; KEK `01F9EE28`; Stopped, Automatic, LocalSystem |
|
||||||
|
| seed | 12:24:21 `nam-ldap-svc-password` → `data\secrets.db`, `readback-match=True`, value file removed | 12:24:26 same |
|
||||||
|
| key store | 12:24:25 `data\gateway-auth.db` = `3C5705D78769` (no earlier store existed) | 12:24:30 same |
|
||||||
|
| start | 12:24:35; Running; listening on 5120 and 5130; stderr empty | 12:26:11; same |
|
||||||
|
| `/health/ready` | **503**: auth-store Healthy, worker-executable Healthy, **galaxy-sql Unhealthy** | **503**, same |
|
||||||
|
|
||||||
|
**galaxy-sql: expected, same cause as vp04 (scadaproj#60).** The log shows
|
||||||
|
`Login failed for user 'ZMR\WONDER-PAR-VP03$'` / `'ZMR\WONDER-PAR-VP04$'`, `Error Number:4060`. The machine accounts reach
|
||||||
|
vp03's SQL Server but have no access to `ZB`. The gateway retries every 30 s and turns ready without a restart once the DBA
|
||||||
|
grants it (T-SQL on #60). MXAccess sessions do not depend on it.
|
||||||
|
|
||||||
|
## Consumer: ScadaBridge site-par
|
||||||
|
|
||||||
|
The site-par node on par-vp03 logged `[MxGateway] Reconnected successfully on "Primary" endpoint` → `Entering Connected
|
||||||
|
state` at **12:24:56Z**, 20 s after par-vp03's gateway started. There was no ScadaBridge change; the copied store makes the
|
||||||
|
existing `scadabridge-prod` token valid. (A no-op `deploy artifacts --site-id 1` at 12:29Z applied nothing; it was not needed.)
|
||||||
|
|
||||||
|
## Unrelated event in the same window: app-box services restarted at 12:31Z
|
||||||
|
|
||||||
|
Every ScadaBridge and mxgw service on **wonder-app-vp03 and vp04** (centrals, site-app nodes, mxgw) received a process
|
||||||
|
termination ("CLR termination hook") at 12:31:03–12:31:43Z. They came back at about 12:34:30Z. The central pair re-formed
|
||||||
|
(vp03 active), and all health endpoints were 200 by 12:35:58Z. Nothing in this run touched those boxes (SMB reads only). The
|
||||||
|
pattern looks like an external restart or reboot of both app boxes. The cause was not investigated: that would need
|
||||||
|
remote diagnostics, which the deploy-only WinRM scope does not cover.
|
||||||
|
|
||||||
|
Separately, at 10:51Z vp03 stalled for about 47.6 s ("heartbeat delayed 47592 ms", which looks like a VM freeze). vp04
|
||||||
|
downed it under `leader-auto-down`, and vp03 restarted with exit 75 as designed.
|
||||||
|
|
||||||
|
After the restart, site-app's `MxGateway` connection was not rebuilt. This is by design: on boot the site builds only the
|
||||||
|
connections a deployed instance references, and PROD has no instances yet. `deploy artifacts --site-id 2` at 12:43:13Z
|
||||||
|
rebuilt it: **Connected at 12:43:17Z**.
|
||||||
|
|
||||||
|
## Clean-up
|
||||||
|
|
||||||
|
- `_values\mxgw-par*`, `_values\mxgw-store` and the seed file were deleted on both boxes (verified; only the empty `_values\seed` folder is left).
|
||||||
|
- The local value files and key-store copies on the Mac were deleted.
|
||||||
|
|
||||||
|
## Open
|
||||||
|
|
||||||
|
- **scadaproj#60:** `ZB` read access for `ZMR\WONDER-APP-VP04$`, `ZMR\WONDER-PAR-VP03$` and `ZMR\WONDER-PAR-VP04$`.
|
||||||
|
- **Alarms are off on all four gateways** until the owner picks an area.
|
||||||
|
- **mxaccessgw#234:** the repo copy of the install script still needs PATCH 4.
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
On each PAR box: `nssm stop MxAccessGw; nssm remove MxAccessGw confirm`, then delete `E:\ApiInstall\MxAccessGateway` and
|
||||||
|
`C:\ProgramData\ZB\kek\mxgateway.kek`. site-par's connection then goes back to retrying. To serve it from the app-box
|
||||||
|
gateways instead: `data-connection update --id 1` with `10.220.153.205:5120` / `10.220.153.210:5120`, then
|
||||||
|
`deploy artifacts --site-id 1`.
|
||||||
Reference in New Issue
Block a user