diff --git a/docs/deploy/2026-11-20-prod-e74918cc.md b/docs/deploy/2026-11-20-prod-e74918cc.md new file mode 100644 index 00000000..331d3bab --- /dev/null +++ b/docs/deploy/2026-11-20-prod-e74918cc.md @@ -0,0 +1,110 @@ +# 2026-11-20 — PROD first install of MxAccessGateway `e74918cc` on wonder-app-vp03 / wonder-app-vp04 + +Owner-approved PROD step: "use winrm to bring up mxaccessgw on prod vp03/vp04 … hold off on mxaccessgw on the +site servers until the platform is deployed". Plan: `scadaproj/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md` +(§0.4 step 4, §3A.5, §5 rollback). Stage pack: `scadaproj/docs/deploy/2026-11-19-prod-stage/mxgw/`. +Driven from the Mac with the `winrm` CLI (profiles `vp03`/`vp04`, NAM\adm_dohertj2). No firewall changes, and no +ScadaBridge services or files were touched (another agent was installing ScadaBridge on the same boxes at the same time). + +## Build + +| Item | Value | +|---|---| +| Artifact | `E:\psrem-stage\mxgw-e74918cc\mxgw-e74918cc.zip`, sha256 `F0999B75…D847F2` (script-verified on both boxes) | +| `ZB.MOM.WW.MxGateway.Server.exe` ProductVersion | `0.1.2+e74918cc` | +| `MxGateway.Worker.exe` (x86) ProductVersion | `0.1.2+e74918cc` | +| LmxProxy.dll (MXAccess gate) | `6000.0045.2657.1` (sha256 `BD75A2FDE222`) on both boxes | +| Service | `MxAccessGw`, NSSM, `LocalSystem`, Automatic, SCM recovery restart ×3 (5 s/5 s/30 s), failureflag 1 | +| Layout | `E:\ApiInstall\MxAccessGateway\{app,worker,data,logs}`; KEK `C:\ProgramData\ZB\kek\mxgateway.kek` (32 bytes, per box) | +| Env | 38 entries (ports 5120 h2c gRPC / 5130 dashboard; LDAPS 636 NAM, `svc_igntadmin`, `BindMode=ServiceAccount`; `UG_WW_APP_*_PROD` role map; `RequireHttpsCookie=false`; **`Alarms__Enabled=false`**; OTLP → 127.0.0.1:4317) | + +## Staging-script fix applied before the install (PROD-STAGE PATCH 4) + +The first dry run on vp03 (20:32:00Z) aborted: + +``` +Install-MxGatewayService service=MxAccessGw dry=True +ERROR: Can't open service! +ABORT: Install-MxGatewayService exited 1 +``` + +Cause: `Invoke-Nssm` runs `& nssm status MxAccessGw 2>&1` under `$ErrorActionPreference='Stop'`. On Windows PowerShell +5.1 that turns nssm's stderr ("Can't open service!", the *expected* answer on a fresh box) into a terminating +NativeCommandError, so the `-AllowFail` existence probe can never succeed on a first install. Fix: run the native call +under `Continue` (restored in `finally`). Only the staged copy was patched +(`scadaproj/docs/deploy/2026-11-19-prod-stage/mxgw/Install-MxGatewayService.ps1`, sha256 `FB6F0F474A26…`); it was pushed +over SMB to both boxes, and the previous copy was kept beside it as `Install-MxGatewayService.ps1.patch3`. **The repo's +`scripts/deploy/Install-MxGatewayService.ps1` has the same defect (line 65, under EAP=Stop) and needs the same fix before the PAR boxes are installed.** + +## wonder-app-vp03 (central, Galaxy Repository host) + +| Step (UTC) | Result | +|---|---| +| 20:32:50 dry | DONE, no ABORT (one benign WARN: dry `WorkerDir` ≠ the env's canonical worker path); `dry\` deleted afterwards | +| 20:33:08 real | DONE; service installed STOPPED; KEK generated sha256 `567A6AFF` | +| 20:33:29 seed | `nam-ldap-svc-password` → `E:\ApiInstall\MxAccessGateway\data\secrets.db`, `set rc=0`, `readback-match=True`, `SEED DONE failures=0` | +| 20:33:55 start | `nssm start` OK, Running after 3 s; stdout log shows listening on 5130 + 5120 and "Application started", stderr empty | +| health | `/health/ready` **200 Healthy**: auth-store, worker-executable, galaxy-sql all Healthy. `/healthz` 200, `/health/active` 200. `/health/live` is **404** (this build doesn't map that route; not a fault) | +| `GET /` | loopback 200 (`AllowAnonymousLocalhost=true`); from the Mac **302 → `/login?ReturnUrl=%2F`** | +| 20:34:53 LDAP login | one POST `/auth/login` as `dohertj2` → **302 → `/` (success branch)**; dashboard pages 200 as `dohertj2`, Admin nav (API keys, Secrets) present, `/admin/secrets` 200. No LDAP bind warnings/errors in the log. Signed out afterwards | +| Galaxy | Galaxy Repository page **Healthy**: 901 objects, 60 areas, 544,454 attributes, 5,373 historized, 5,326 alarm attributes; last deploy 2026-10-15 15:28:20 UTC; connection `Data Source=localhost;Initial Catalog=ZB;Integrated Security=True` | +| Worker | none running (no session yet, as expected) | + +Top-level Galaxy areas (Browse page, for the owner's alarm-area choice): **ControlSystem, EUR, KPI, KPI_CNC, NAM**. +The other top-level entries are platforms/engines, not areas: ArcomHMI_Platform, Monitor_OMI_001, the OI_WAR_* OI servers, +the VEHMI0x platforms and view engines, VitaminEPress_HMI_00x, WinPlatformBER01/02 and WinPlatform_PPPHMI_Row2. + +## wonder-app-vp04 (central) + +| Step (UTC) | Result | +|---|---| +| 20:36:19 dry | DONE, no ABORT; `dry\` deleted | +| 20:36:31 real | DONE; service installed STOPPED; KEK generated sha256 `30C69B13`; account LocalSystem | +| 20:36:45 seed | `set rc=0`, `readback-match=True`, `SEED DONE failures=0` | +| 20:36:55 start | Running after 3 s; listening on 5130 + 5120; stderr empty | +| health | `/health/ready` **503 Unhealthy**. auth-store and worker-executable are Healthy; `galaxy-sql` is Unhealthy ("Galaxy Repository SQL ping timed out after 5s.") | +| `GET /` | from the Mac 302 → `/login` | +| 20:37:40 LDAP login | one POST as `dohertj2` → **302 → `/` (success)**; `/`, `/admin/secrets`, `/galaxy` 200. No LDAP errors in the log. Signed out | +| Galaxy | **NOT connected**. The Galaxy page shows "Awaiting…". Log, repeated every 30 s refresh: | + +``` +[15:37:08 WRN] Galaxy hierarchy cache refresh failed. +Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open database "ZB" requested by the login. The login failed. +Login failed for user 'ZMR\WONDER-APP-VP04$'. +Error Number:4060,State:1,Class:11 +``` + +The computer account is in the **ZMR** domain (`ZMR\WONDER-APP-VP04$`, not NAM). It reaches vp03's SQL Server but has +no access to the `ZB` database. SQL logins were **not** changed. **Fix (a DBA or owner action):** create a login for +`ZMR\WONDER-APP-VP04$` on WONDER-APP-VP03's SQL Server and give it read access on `ZB`, as QA did for vq04 → vq03. The +service was left running, because the gateway retries every 30 s and picks the database up without a restart once access +exists. Until then its readiness answers 503. + +## Clean-up + +- `E:\psrem-stage\_values\mxgw-vp03` and `…\mxgw-vp04` deleted (verified absent). The seed file + `E:\psrem-stage\_values\seed\nam-ldap-mxgw.txt` was deleted by `seed-secret-prod.ps1` on both boxes (verified). The + spare copy taken before seeding was inside the deleted values dirs. +- vp03 `E:\psrem-stage\_values\seed\` still holds `site-psk.txt`, which belongs to the ScadaBridge install and was left untouched. +- Login cookies and the temporary password file on the Mac were deleted right after use. + +## Probe note + +curl caps the Cookie header at about 8 KB and silently drops cookies over that limit. The dashboard auth cookie for +`dohertj2` is about 13 KB in four chunks (many `memberOf` groups), so a plain `curl -b jar` follow-up looks like a failed +login. Replaying the jar as an explicit `Cookie:` header (`-H @file`) shows the session is valid. Kestrel's 32 KB header +limit and browsers are fine. + +## NOT done (open) + +- **PAR site boxes (parvp03/parvp04): held** until WinPlatform/MXAccess is deployed there from the PROD Galaxy (plan Q3). +- **Alarms disabled** (`MxGateway__Alarms__Enabled=false`) until the owner picks an area. Then set + `MxGateway__Alarms__SubscriptionExpression=\\WONDER-APP-VP03\Galaxy!`, set `Enabled=true` and restart. +- **No API keys minted yet**, so no consumer can open a session and no x86 Worker has started on either box. +- vp04 Galaxy Repository access (above). +- Repo copy of `Install-MxGatewayService.ps1` still needs PATCH 4 (and PATCHES 2–3) upstreamed. + +## Rollback (plan §5) + +`nssm stop MxAccessGw; nssm remove MxAccessGw confirm`, then delete `E:\ApiInstall\MxAccessGateway` and +`C:\ProgramData\ZB\kek\mxgateway.kek` on the box. Nothing else on either box depends on it yet.