7.7 KiB
2026-11-20 — PROD first install of MxAccessGateway e74918cc on wonder-app-vp03 / wonder-app-vp04
Owner-approved PROD step: "use winrm to bring up mxaccessgw on prod vp03/vp04 … hold off on mxaccessgw on the
site servers until the platform is deployed". Plan: scadaproj/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md
(§0.4 step 4, §3A.5, §5 rollback). Stage pack: scadaproj/docs/deploy/2026-11-19-prod-stage/mxgw/.
Driven from the Mac with the winrm CLI (profiles vp03/vp04, NAM\adm_dohertj2). No firewall changes, and no
ScadaBridge services or files were touched (another agent was installing ScadaBridge on the same boxes at the same time).
Build
| Item | Value |
|---|---|
| Artifact | E:\psrem-stage\mxgw-e74918cc\mxgw-e74918cc.zip, sha256 F0999B75…D847F2 (script-verified on both boxes) |
ZB.MOM.WW.MxGateway.Server.exe ProductVersion |
0.1.2+e74918cc |
MxGateway.Worker.exe (x86) ProductVersion |
0.1.2+e74918cc |
| LmxProxy.dll (MXAccess gate) | 6000.0045.2657.1 (sha256 BD75A2FDE222) on both boxes |
| Service | MxAccessGw, NSSM, LocalSystem, Automatic, SCM recovery restart ×3 (5 s/5 s/30 s), failureflag 1 |
| Layout | E:\ApiInstall\MxAccessGateway\{app,worker,data,logs}; KEK C:\ProgramData\ZB\kek\mxgateway.kek (32 bytes, per box) |
| Env | 38 entries (ports 5120 h2c gRPC / 5130 dashboard; LDAPS 636 NAM, svc_igntadmin, BindMode=ServiceAccount; UG_WW_APP_*_PROD role map; RequireHttpsCookie=false; Alarms__Enabled=false; OTLP → 127.0.0.1:4317) |
Staging-script fix applied before the install (PROD-STAGE PATCH 4)
The first dry run on vp03 (20:32:00Z) aborted:
Install-MxGatewayService service=MxAccessGw dry=True
ERROR: Can't open service!
ABORT: Install-MxGatewayService exited 1
Cause: Invoke-Nssm runs & nssm status MxAccessGw 2>&1 under $ErrorActionPreference='Stop'. On Windows PowerShell
5.1 that turns nssm's stderr ("Can't open service!", the expected answer on a fresh box) into a terminating
NativeCommandError, so the -AllowFail existence probe can never succeed on a first install. Fix: run the native call
under Continue (restored in finally). Only the staged copy was patched
(scadaproj/docs/deploy/2026-11-19-prod-stage/mxgw/Install-MxGatewayService.ps1, sha256 FB6F0F474A26…); it was pushed
over SMB to both boxes, and the previous copy was kept beside it as Install-MxGatewayService.ps1.patch3. The repo's
scripts/deploy/Install-MxGatewayService.ps1 has the same defect (line 65, under EAP=Stop) and needs the same fix before the PAR boxes are installed.
wonder-app-vp03 (central, Galaxy Repository host)
| Step (UTC) | Result |
|---|---|
| 20:32:50 dry | DONE, no ABORT (one benign WARN: dry WorkerDir ≠ the env's canonical worker path); dry\ deleted afterwards |
| 20:33:08 real | DONE; service installed STOPPED; KEK generated sha256 567A6AFF |
| 20:33:29 seed | nam-ldap-svc-password → E:\ApiInstall\MxAccessGateway\data\secrets.db, set rc=0, readback-match=True, SEED DONE failures=0 |
| 20:33:55 start | nssm start OK, Running after 3 s; stdout log shows listening on 5130 + 5120 and "Application started", stderr empty |
| health | /health/ready 200 Healthy: auth-store, worker-executable, galaxy-sql all Healthy. /healthz 200, /health/active 200. /health/live is 404 (this build doesn't map that route; not a fault) |
GET / |
loopback 200 (AllowAnonymousLocalhost=true); from the Mac 302 → /login?ReturnUrl=%2F |
| 20:34:53 LDAP login | one POST /auth/login as dohertj2 → 302 → / (success branch); dashboard pages 200 as dohertj2, Admin nav (API keys, Secrets) present, /admin/secrets 200. No LDAP bind warnings/errors in the log. Signed out afterwards |
| Galaxy | Galaxy Repository page Healthy: 901 objects, 60 areas, 544,454 attributes, 5,373 historized, 5,326 alarm attributes; last deploy 2026-10-15 15:28:20 UTC; connection Data Source=localhost;Initial Catalog=ZB;Integrated Security=True |
| Worker | none running (no session yet, as expected) |
Top-level Galaxy areas (Browse page, for the owner's alarm-area choice): ControlSystem, EUR, KPI, KPI_CNC, NAM. The other top-level entries are platforms/engines, not areas: ArcomHMI_Platform, Monitor_OMI_001, the OI_WAR_* OI servers, the VEHMI0x platforms and view engines, VitaminEPress_HMI_00x, WinPlatformBER01/02 and WinPlatform_PPPHMI_Row2.
wonder-app-vp04 (central)
| Step (UTC) | Result |
|---|---|
| 20:36:19 dry | DONE, no ABORT; dry\ deleted |
| 20:36:31 real | DONE; service installed STOPPED; KEK generated sha256 30C69B13; account LocalSystem |
| 20:36:45 seed | set rc=0, readback-match=True, SEED DONE failures=0 |
| 20:36:55 start | Running after 3 s; listening on 5130 + 5120; stderr empty |
| health | /health/ready 503 Unhealthy. auth-store and worker-executable are Healthy; galaxy-sql is Unhealthy ("Galaxy Repository SQL ping timed out after 5s.") |
GET / |
from the Mac 302 → /login |
| 20:37:40 LDAP login | one POST as dohertj2 → 302 → / (success); /, /admin/secrets, /galaxy 200. No LDAP errors in the log. Signed out |
| Galaxy | NOT connected. The Galaxy page shows "Awaiting…". Log, repeated every 30 s refresh: |
[15:37:08 WRN] Galaxy hierarchy cache refresh failed.
Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open database "ZB" requested by the login. The login failed.
Login failed for user 'ZMR\WONDER-APP-VP04$'.
Error Number:4060,State:1,Class:11
The computer account is in the ZMR domain (ZMR\WONDER-APP-VP04$, not NAM). It reaches vp03's SQL Server but has
no access to the ZB database. SQL logins were not changed. Fix (a DBA or owner action): create a login for
ZMR\WONDER-APP-VP04$ on WONDER-APP-VP03's SQL Server and give it read access on ZB, as QA did for vq04 → vq03. The
service was left running, because the gateway retries every 30 s and picks the database up without a restart once access
exists. Until then its readiness answers 503.
Clean-up
E:\psrem-stage\_values\mxgw-vp03and…\mxgw-vp04deleted (verified absent). The seed fileE:\psrem-stage\_values\seed\nam-ldap-mxgw.txtwas deleted byseed-secret-prod.ps1on both boxes (verified). The spare copy taken before seeding was inside the deleted values dirs.- vp03
E:\psrem-stage\_values\seed\still holdssite-psk.txt, which belongs to the ScadaBridge install and was left untouched. - Login cookies and the temporary password file on the Mac were deleted right after use.
Probe note
curl caps the Cookie header at about 8 KB and silently drops cookies over that limit. The dashboard auth cookie for
dohertj2 is about 13 KB in four chunks (many memberOf groups), so a plain curl -b jar follow-up looks like a failed
login. Replaying the jar as an explicit Cookie: header (-H @file) shows the session is valid. Kestrel's 32 KB header
limit and browsers are fine.
NOT done (open)
- PAR site boxes (parvp03/parvp04): held until WinPlatform/MXAccess is deployed there from the PROD Galaxy (plan Q3).
- Alarms disabled (
MxGateway__Alarms__Enabled=false) until the owner picks an area. Then setMxGateway__Alarms__SubscriptionExpression=\\WONDER-APP-VP03\Galaxy!<area>, setEnabled=trueand restart. - No API keys minted yet, so no consumer can open a session and no x86 Worker has started on either box.
- vp04 Galaxy Repository access (above).
- Repo copy of
Install-MxGatewayService.ps1still needs PATCH 4 (and PATCHES 2–3) upstreamed.
Rollback (plan §5)
nssm stop MxAccessGw; nssm remove MxAccessGw confirm, then delete E:\ApiInstall\MxAccessGateway and
C:\ProgramData\ZB\kek\mxgateway.kek on the box. Nothing else on either box depends on it yet.