Files
mxaccessgw/docs/deploy/2026-11-20-prod-e74918cc.md
T
Joseph Doherty 08a9cfd68a
ci / java (push) Failing after 4m21s
ci / portable (push) Failing after 8m55s
ci / windows-x86 (push) Failing after 21m56s
docs(deploy): PROD vp03/vp04 first install e74918cc (vp04 Galaxy SQL access pending; alarms off; PAR held for WinPlatform)
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-20 15:39:36 -05:00

111 lines
7.7 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 2026-11-20 — PROD first install of MxAccessGateway `e74918cc` on wonder-app-vp03 / wonder-app-vp04
Owner-approved PROD step: "use winrm to bring up mxaccessgw on prod vp03/vp04 … hold off on mxaccessgw on the
site servers until the platform is deployed". Plan: `scadaproj/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md`
(§0.4 step 4, §3A.5, §5 rollback). Stage pack: `scadaproj/docs/deploy/2026-11-19-prod-stage/mxgw/`.
Driven from the Mac with the `winrm` CLI (profiles `vp03`/`vp04`, NAM\adm_dohertj2). No firewall changes, and no
ScadaBridge services or files were touched (another agent was installing ScadaBridge on the same boxes at the same time).
## Build
| Item | Value |
|---|---|
| Artifact | `E:\psrem-stage\mxgw-e74918cc\mxgw-e74918cc.zip`, sha256 `F0999B75…D847F2` (script-verified on both boxes) |
| `ZB.MOM.WW.MxGateway.Server.exe` ProductVersion | `0.1.2+e74918cc` |
| `MxGateway.Worker.exe` (x86) ProductVersion | `0.1.2+e74918cc` |
| LmxProxy.dll (MXAccess gate) | `6000.0045.2657.1` (sha256 `BD75A2FDE222`) on both boxes |
| Service | `MxAccessGw`, NSSM, `LocalSystem`, Automatic, SCM recovery restart ×3 (5 s/5 s/30 s), failureflag 1 |
| Layout | `E:\ApiInstall\MxAccessGateway\{app,worker,data,logs}`; KEK `C:\ProgramData\ZB\kek\mxgateway.kek` (32 bytes, per box) |
| Env | 38 entries (ports 5120 h2c gRPC / 5130 dashboard; LDAPS 636 NAM, `svc_igntadmin`, `BindMode=ServiceAccount`; `UG_WW_APP_*_PROD` role map; `RequireHttpsCookie=false`; **`Alarms__Enabled=false`**; OTLP → 127.0.0.1:4317) |
## Staging-script fix applied before the install (PROD-STAGE PATCH 4)
The first dry run on vp03 (20:32:00Z) aborted:
```
Install-MxGatewayService service=MxAccessGw dry=True
ERROR: Can't open service!
ABORT: Install-MxGatewayService exited 1
```
Cause: `Invoke-Nssm` runs `& nssm status MxAccessGw 2>&1` under `$ErrorActionPreference='Stop'`. On Windows PowerShell
5.1 that turns nssm's stderr ("Can't open service!", the *expected* answer on a fresh box) into a terminating
NativeCommandError, so the `-AllowFail` existence probe can never succeed on a first install. Fix: run the native call
under `Continue` (restored in `finally`). Only the staged copy was patched
(`scadaproj/docs/deploy/2026-11-19-prod-stage/mxgw/Install-MxGatewayService.ps1`, sha256 `FB6F0F474A26…`); it was pushed
over SMB to both boxes, and the previous copy was kept beside it as `Install-MxGatewayService.ps1.patch3`. **The repo's
`scripts/deploy/Install-MxGatewayService.ps1` has the same defect (line 65, under EAP=Stop) and needs the same fix before the PAR boxes are installed.**
## wonder-app-vp03 (central, Galaxy Repository host)
| Step (UTC) | Result |
|---|---|
| 20:32:50 dry | DONE, no ABORT (one benign WARN: dry `WorkerDir` ≠ the env's canonical worker path); `dry\` deleted afterwards |
| 20:33:08 real | DONE; service installed STOPPED; KEK generated sha256 `567A6AFF` |
| 20:33:29 seed | `nam-ldap-svc-password` → `E:\ApiInstall\MxAccessGateway\data\secrets.db`, `set rc=0`, `readback-match=True`, `SEED DONE failures=0` |
| 20:33:55 start | `nssm start` OK, Running after 3 s; stdout log shows listening on 5130 + 5120 and "Application started", stderr empty |
| health | `/health/ready` **200 Healthy**: auth-store, worker-executable, galaxy-sql all Healthy. `/healthz` 200, `/health/active` 200. `/health/live` is **404** (this build doesn't map that route; not a fault) |
| `GET /` | loopback 200 (`AllowAnonymousLocalhost=true`); from the Mac **302 → `/login?ReturnUrl=%2F`** |
| 20:34:53 LDAP login | one POST `/auth/login` as `dohertj2` → **302 → `/` (success branch)**; dashboard pages 200 as `dohertj2`, Admin nav (API keys, Secrets) present, `/admin/secrets` 200. No LDAP bind warnings/errors in the log. Signed out afterwards |
| Galaxy | Galaxy Repository page **Healthy**: 901 objects, 60 areas, 544,454 attributes, 5,373 historized, 5,326 alarm attributes; last deploy 2026-10-15 15:28:20 UTC; connection `Data Source=localhost;Initial Catalog=ZB;Integrated Security=True` |
| Worker | none running (no session yet, as expected) |
Top-level Galaxy areas (Browse page, for the owner's alarm-area choice): **ControlSystem, EUR, KPI, KPI_CNC, NAM**.
The other top-level entries are platforms/engines, not areas: ArcomHMI_Platform, Monitor_OMI_001, the OI_WAR_* OI servers,
the VEHMI0x platforms and view engines, VitaminEPress_HMI_00x, WinPlatformBER01/02 and WinPlatform_PPPHMI_Row2.
## wonder-app-vp04 (central)
| Step (UTC) | Result |
|---|---|
| 20:36:19 dry | DONE, no ABORT; `dry\` deleted |
| 20:36:31 real | DONE; service installed STOPPED; KEK generated sha256 `30C69B13`; account LocalSystem |
| 20:36:45 seed | `set rc=0`, `readback-match=True`, `SEED DONE failures=0` |
| 20:36:55 start | Running after 3 s; listening on 5130 + 5120; stderr empty |
| health | `/health/ready` **503 Unhealthy**. auth-store and worker-executable are Healthy; `galaxy-sql` is Unhealthy ("Galaxy Repository SQL ping timed out after 5s.") |
| `GET /` | from the Mac 302 → `/login` |
| 20:37:40 LDAP login | one POST as `dohertj2` → **302 → `/` (success)**; `/`, `/admin/secrets`, `/galaxy` 200. No LDAP errors in the log. Signed out |
| Galaxy | **NOT connected**. The Galaxy page shows "Awaiting…". Log, repeated every 30 s refresh: |
```
[15:37:08 WRN] Galaxy hierarchy cache refresh failed.
Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open database "ZB" requested by the login. The login failed.
Login failed for user 'ZMR\WONDER-APP-VP04$'.
Error Number:4060,State:1,Class:11
```
The computer account is in the **ZMR** domain (`ZMR\WONDER-APP-VP04$`, not NAM). It reaches vp03's SQL Server but has
no access to the `ZB` database. SQL logins were **not** changed. **Fix (a DBA or owner action):** create a login for
`ZMR\WONDER-APP-VP04$` on WONDER-APP-VP03's SQL Server and give it read access on `ZB`, as QA did for vq04 → vq03. The
service was left running, because the gateway retries every 30 s and picks the database up without a restart once access
exists. Until then its readiness answers 503.
## Clean-up
- `E:\psrem-stage\_values\mxgw-vp03` and `…\mxgw-vp04` deleted (verified absent). The seed file
`E:\psrem-stage\_values\seed\nam-ldap-mxgw.txt` was deleted by `seed-secret-prod.ps1` on both boxes (verified). The
spare copy taken before seeding was inside the deleted values dirs.
- vp03 `E:\psrem-stage\_values\seed\` still holds `site-psk.txt`, which belongs to the ScadaBridge install and was left untouched.
- Login cookies and the temporary password file on the Mac were deleted right after use.
## Probe note
curl caps the Cookie header at about 8 KB and silently drops cookies over that limit. The dashboard auth cookie for
`dohertj2` is about 13 KB in four chunks (many `memberOf` groups), so a plain `curl -b jar` follow-up looks like a failed
login. Replaying the jar as an explicit `Cookie:` header (`-H @file`) shows the session is valid. Kestrel's 32 KB header
limit and browsers are fine.
## NOT done (open)
- **PAR site boxes (parvp03/parvp04): held** until WinPlatform/MXAccess is deployed there from the PROD Galaxy (plan Q3).
- **Alarms disabled** (`MxGateway__Alarms__Enabled=false`) until the owner picks an area. Then set
`MxGateway__Alarms__SubscriptionExpression=\\WONDER-APP-VP03\Galaxy!<area>`, set `Enabled=true` and restart.
- **No API keys minted yet**, so no consumer can open a session and no x86 Worker has started on either box.
- vp04 Galaxy Repository access (above).
- Repo copy of `Install-MxGatewayService.ps1` still needs PATCH 4 (and PATCHES 2–3) upstreamed.
## Rollback (plan §5)
`nssm stop MxAccessGw; nssm remove MxAccessGw confirm`, then delete `E:\ApiInstall\MxAccessGateway` and
`C:\ProgramData\ZB\kek\mxgateway.kek` on the box. Nothing else on either box depends on it yet.