Files
mxaccessgw/docs/deploy/2026-11-22-prod-par-e74918cc.md
T
Joseph Doherty 8e0e232fd6
ci / windows-x86 (push) Successful in 2m6s
ci / java (push) Failing after 6m33s
ci / portable (push) Failing after 10m42s
deploy(prod): mxgw e74918cc on par-vp03/par-vp04 (site-par Connected; galaxy-sql waits on scadaproj#60)
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-22 07:45:06 -05:00

5.9 KiB
Raw Blame History

2026-11-22: PROD install of MxAccessGateway e74918cc on wonder-par-vp03 / wonder-par-vp04 (site-par)

Owner: "keep checking par-vp04 every minute. when its ready then execute the par server updates (both)". This follows the 2026-11-20 app-box install (2026-11-20-prod-e74918cc.md); the build, stage pack and env are the same. Driven from the Mac with the winrm CLI (profiles parvp03/parvp04, NAM\adm_dohertj2). No firewall changes, and no ScadaBridge service was touched.

Gate: WinPlatform / MXAccess on the PAR boxes

Checked over SMB only.

  • par-vp03 already had LmxProxy.dll at 12:0xZ: 318,496 bytes, sha256 bd75a2fd…7ef15459, the same file as vp03. Its ArchestrA\Framework\Bin held the full platform runtime (102 entries).
  • par-vp04 started with 49 entries and no MXAccess. It was polled every minute. LmxProxy.dll appeared at about 12:19Z (101 entries), and the folder was stable at 102 entries from 12:21Z. The poller declared it ready at 12:23:45Z with the same sha256 bd75a2fde222….

Stage fixes (by SMB, before anything ran)

  • PATCH 4 (Install-MxGatewayService.ps1, nssm status probe under Continue; mxaccessgw#234): both PAR boxes still had the PATCH 3 copy (11a23e9f…). The patched copy (fb6f0f47…) was pushed, the old one kept as Install-MxGatewayService.ps1.patch3, and the SHA256SUMS.txt line updated on both boxes.
  • Value files: _values\mxgw-<box>\ on each box held three files, all from the Keychain, never printed and never on argv:
    • MxGateway__ApiKeyPepper.txt: zb-prod/mxgw-pepper, the same as vp03/vp04;
    • MxGateway__Galaxy__ConnectionString.txt: Server=wonder-app-vp03.zmr.zimmer.com;Database=ZB;Integrated Security=True;TrustServerCertificate=True;Encrypt=False;;
    • the LDAP seed file nam-ldap-mxgw.txt.
  • API-key store: vp03's gateway could not be stopped for a clean copy, because site-app runs on it. So vp04's gateway-auth.db (+wal/shm) was pulled live; it is vp03's store as copied on 2026-11-20. Checked on the Mac:
    • consolidated (the WAL held 0 frames);
    • integrity_check ok;
    • one key, scadabridge-prod, scopes events:read invoke:read invoke:write metadata:read session:close session:open, not revoked;
    • single-file sha256 3c5705d7…43bffe55.

Install (UTC)

Step par-vp03 par-vp04
dry 12:06:28 DONE (LmxProxy 6000.0045.2657.1 / BD75A2FDE222, 38 env entries, Server and Worker 0.1.2+e74918cc, one benign WorkerDir WARN); dry\ removed 12:23:54 DONE, same output; dry\ removed
real 12:24:00 DONE; KEK generated AA55B221; service MxAccessGw Stopped, Automatic, LocalSystem 12:24:09 DONE; KEK 01F9EE28; Stopped, Automatic, LocalSystem
seed 12:24:21 nam-ldap-svc-password → data\secrets.db, readback-match=True, value file removed 12:24:26 same
key store 12:24:25 data\gateway-auth.db = 3C5705D78769 (no earlier store existed) 12:24:30 same
start 12:24:35; Running; listening on 5120 and 5130; stderr empty 12:26:11; same
/health/ready 503: auth-store Healthy, worker-executable Healthy, galaxy-sql Unhealthy 503, same

galaxy-sql: expected, same cause as vp04 (scadaproj#60). The log shows Login failed for user 'ZMR\WONDER-PAR-VP03$' / 'ZMR\WONDER-PAR-VP04$', Error Number:4060. The machine accounts reach vp03's SQL Server but have no access to ZB. The gateway retries every 30 s and turns ready without a restart once the DBA grants it (T-SQL on #60). MXAccess sessions do not depend on it.

Consumer: ScadaBridge site-par

The site-par node on par-vp03 logged [MxGateway] Reconnected successfully on "Primary" endpoint → Entering Connected state at 12:24:56Z, 20 s after par-vp03's gateway started. There was no ScadaBridge change; the copied store makes the existing scadabridge-prod token valid. (A no-op deploy artifacts --site-id 1 at 12:29Z applied nothing; it was not needed.)

Unrelated event in the same window: app-box services restarted at 12:31Z

Every ScadaBridge and mxgw service on wonder-app-vp03 and vp04 (centrals, site-app nodes, mxgw) received a process termination ("CLR termination hook") at 12:31:03–12:31:43Z. They came back at about 12:34:30Z. The central pair re-formed (vp03 active), and all health endpoints were 200 by 12:35:58Z. Nothing in this run touched those boxes (SMB reads only). The pattern looks like an external restart or reboot of both app boxes. The cause was not investigated: that would need remote diagnostics, which the deploy-only WinRM scope does not cover.

Separately, at 10:51Z vp03 stalled for about 47.6 s ("heartbeat delayed 47592 ms", which looks like a VM freeze). vp04 downed it under leader-auto-down, and vp03 restarted with exit 75 as designed.

After the restart, site-app's MxGateway connection was not rebuilt. This is by design: on boot the site builds only the connections a deployed instance references, and PROD has no instances yet. deploy artifacts --site-id 2 at 12:43:13Z rebuilt it: Connected at 12:43:17Z.

Clean-up

  • _values\mxgw-par*, _values\mxgw-store and the seed file were deleted on both boxes (verified; only the empty _values\seed folder is left).
  • The local value files and key-store copies on the Mac were deleted.

Open

  • scadaproj#60: ZB read access for ZMR\WONDER-APP-VP04$, ZMR\WONDER-PAR-VP03$ and ZMR\WONDER-PAR-VP04$.
  • Alarms are off on all four gateways until the owner picks an area.
  • mxaccessgw#234: the repo copy of the install script still needs PATCH 4.

Rollback

On each PAR box: nssm stop MxAccessGw; nssm remove MxAccessGw confirm, then delete E:\ApiInstall\MxAccessGateway and C:\ProgramData\ZB\kek\mxgateway.kek. site-par's connection then goes back to retrying. To serve it from the app-box gateways instead: data-connection update --id 1 with 10.220.153.205:5120 / 10.220.153.210:5120, then deploy artifacts --site-id 1.