Files
mxaccessgw/docs/deploy/2026-11-20-prod-e74918cc.md
T
Joseph Doherty 08a9cfd68a
ci / java (push) Failing after 4m21s
ci / portable (push) Failing after 8m55s
ci / windows-x86 (push) Failing after 21m56s
docs(deploy): PROD vp03/vp04 first install e74918cc (vp04 Galaxy SQL access pending; alarms off; PAR held for WinPlatform)
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-20 15:39:36 -05:00

7.7 KiB
Raw Blame History

2026-11-20 — PROD first install of MxAccessGateway e74918cc on wonder-app-vp03 / wonder-app-vp04

Owner-approved PROD step: "use winrm to bring up mxaccessgw on prod vp03/vp04 … hold off on mxaccessgw on the site servers until the platform is deployed". Plan: scadaproj/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md (§0.4 step 4, §3A.5, §5 rollback). Stage pack: scadaproj/docs/deploy/2026-11-19-prod-stage/mxgw/. Driven from the Mac with the winrm CLI (profiles vp03/vp04, NAM\adm_dohertj2). No firewall changes, and no ScadaBridge services or files were touched (another agent was installing ScadaBridge on the same boxes at the same time).

Build

Item Value
Artifact E:\psrem-stage\mxgw-e74918cc\mxgw-e74918cc.zip, sha256 F0999B75…D847F2 (script-verified on both boxes)
ZB.MOM.WW.MxGateway.Server.exe ProductVersion 0.1.2+e74918cc
MxGateway.Worker.exe (x86) ProductVersion 0.1.2+e74918cc
LmxProxy.dll (MXAccess gate) 6000.0045.2657.1 (sha256 BD75A2FDE222) on both boxes
Service MxAccessGw, NSSM, LocalSystem, Automatic, SCM recovery restart ×3 (5 s/5 s/30 s), failureflag 1
Layout E:\ApiInstall\MxAccessGateway\{app,worker,data,logs}; KEK C:\ProgramData\ZB\kek\mxgateway.kek (32 bytes, per box)
Env 38 entries (ports 5120 h2c gRPC / 5130 dashboard; LDAPS 636 NAM, svc_igntadmin, BindMode=ServiceAccount; UG_WW_APP_*_PROD role map; RequireHttpsCookie=false; Alarms__Enabled=false; OTLP → 127.0.0.1:4317)

Staging-script fix applied before the install (PROD-STAGE PATCH 4)

The first dry run on vp03 (20:32:00Z) aborted:

Install-MxGatewayService  service=MxAccessGw  dry=True
ERROR: Can't open service!
ABORT: Install-MxGatewayService exited 1

Cause: Invoke-Nssm runs & nssm status MxAccessGw 2>&1 under $ErrorActionPreference='Stop'. On Windows PowerShell 5.1 that turns nssm's stderr ("Can't open service!", the expected answer on a fresh box) into a terminating NativeCommandError, so the -AllowFail existence probe can never succeed on a first install. Fix: run the native call under Continue (restored in finally). Only the staged copy was patched (scadaproj/docs/deploy/2026-11-19-prod-stage/mxgw/Install-MxGatewayService.ps1, sha256 FB6F0F474A26…); it was pushed over SMB to both boxes, and the previous copy was kept beside it as Install-MxGatewayService.ps1.patch3. The repo's scripts/deploy/Install-MxGatewayService.ps1 has the same defect (line 65, under EAP=Stop) and needs the same fix before the PAR boxes are installed.

wonder-app-vp03 (central, Galaxy Repository host)

Step (UTC) Result
20:32:50 dry DONE, no ABORT (one benign WARN: dry WorkerDir ≠ the env's canonical worker path); dry\ deleted afterwards
20:33:08 real DONE; service installed STOPPED; KEK generated sha256 567A6AFF
20:33:29 seed nam-ldap-svc-password → E:\ApiInstall\MxAccessGateway\data\secrets.db, set rc=0, readback-match=True, SEED DONE failures=0
20:33:55 start nssm start OK, Running after 3 s; stdout log shows listening on 5130 + 5120 and "Application started", stderr empty
health /health/ready 200 Healthy: auth-store, worker-executable, galaxy-sql all Healthy. /healthz 200, /health/active 200. /health/live is 404 (this build doesn't map that route; not a fault)
GET / loopback 200 (AllowAnonymousLocalhost=true); from the Mac 302 → /login?ReturnUrl=%2F
20:34:53 LDAP login one POST /auth/login as dohertj2 → 302 → / (success branch); dashboard pages 200 as dohertj2, Admin nav (API keys, Secrets) present, /admin/secrets 200. No LDAP bind warnings/errors in the log. Signed out afterwards
Galaxy Galaxy Repository page Healthy: 901 objects, 60 areas, 544,454 attributes, 5,373 historized, 5,326 alarm attributes; last deploy 2026-10-15 15:28:20 UTC; connection Data Source=localhost;Initial Catalog=ZB;Integrated Security=True
Worker none running (no session yet, as expected)

Top-level Galaxy areas (Browse page, for the owner's alarm-area choice): ControlSystem, EUR, KPI, KPI_CNC, NAM. The other top-level entries are platforms/engines, not areas: ArcomHMI_Platform, Monitor_OMI_001, the OI_WAR_* OI servers, the VEHMI0x platforms and view engines, VitaminEPress_HMI_00x, WinPlatformBER01/02 and WinPlatform_PPPHMI_Row2.

wonder-app-vp04 (central)

Step (UTC) Result
20:36:19 dry DONE, no ABORT; dry\ deleted
20:36:31 real DONE; service installed STOPPED; KEK generated sha256 30C69B13; account LocalSystem
20:36:45 seed set rc=0, readback-match=True, SEED DONE failures=0
20:36:55 start Running after 3 s; listening on 5130 + 5120; stderr empty
health /health/ready 503 Unhealthy. auth-store and worker-executable are Healthy; galaxy-sql is Unhealthy ("Galaxy Repository SQL ping timed out after 5s.")
GET / from the Mac 302 → /login
20:37:40 LDAP login one POST as dohertj2 → 302 → / (success); /, /admin/secrets, /galaxy 200. No LDAP errors in the log. Signed out
Galaxy NOT connected. The Galaxy page shows "Awaiting…". Log, repeated every 30 s refresh:
[15:37:08 WRN] Galaxy hierarchy cache refresh failed.
Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open database "ZB" requested by the login. The login failed.
Login failed for user 'ZMR\WONDER-APP-VP04$'.
Error Number:4060,State:1,Class:11

The computer account is in the ZMR domain (ZMR\WONDER-APP-VP04$, not NAM). It reaches vp03's SQL Server but has no access to the ZB database. SQL logins were not changed. Fix (a DBA or owner action): create a login for ZMR\WONDER-APP-VP04$ on WONDER-APP-VP03's SQL Server and give it read access on ZB, as QA did for vq04 → vq03. The service was left running, because the gateway retries every 30 s and picks the database up without a restart once access exists. Until then its readiness answers 503.

Clean-up

  • E:\psrem-stage\_values\mxgw-vp03 and …\mxgw-vp04 deleted (verified absent). The seed file E:\psrem-stage\_values\seed\nam-ldap-mxgw.txt was deleted by seed-secret-prod.ps1 on both boxes (verified). The spare copy taken before seeding was inside the deleted values dirs.
  • vp03 E:\psrem-stage\_values\seed\ still holds site-psk.txt, which belongs to the ScadaBridge install and was left untouched.
  • Login cookies and the temporary password file on the Mac were deleted right after use.

Probe note

curl caps the Cookie header at about 8 KB and silently drops cookies over that limit. The dashboard auth cookie for dohertj2 is about 13 KB in four chunks (many memberOf groups), so a plain curl -b jar follow-up looks like a failed login. Replaying the jar as an explicit Cookie: header (-H @file) shows the session is valid. Kestrel's 32 KB header limit and browsers are fine.

NOT done (open)

  • PAR site boxes (parvp03/parvp04): held until WinPlatform/MXAccess is deployed there from the PROD Galaxy (plan Q3).
  • Alarms disabled (MxGateway__Alarms__Enabled=false) until the owner picks an area. Then set MxGateway__Alarms__SubscriptionExpression=\\WONDER-APP-VP03\Galaxy!<area>, set Enabled=true and restart.
  • No API keys minted yet, so no consumer can open a session and no x86 Worker has started on either box.
  • vp04 Galaxy Repository access (above).
  • Repo copy of Install-MxGatewayService.ps1 still needs PATCH 4 (and PATCHES 2–3) upstreamed.

Rollback (plan §5)

nssm stop MxAccessGw; nssm remove MxAccessGw confirm, then delete E:\ApiInstall\MxAccessGateway and C:\ProgramData\ZB\kek\mxgateway.kek on the box. Nothing else on either box depends on it yet.