111 lines
7.7 KiB
Markdown
111 lines
7.7 KiB
Markdown
# 2026-11-20 — PROD first install of MxAccessGateway `e74918cc` on wonder-app-vp03 / wonder-app-vp04
|
||
|
||
Owner-approved PROD step: "use winrm to bring up mxaccessgw on prod vp03/vp04 … hold off on mxaccessgw on the
|
||
site servers until the platform is deployed". Plan: `scadaproj/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md`
|
||
(§0.4 step 4, §3A.5, §5 rollback). Stage pack: `scadaproj/docs/deploy/2026-11-19-prod-stage/mxgw/`.
|
||
Driven from the Mac with the `winrm` CLI (profiles `vp03`/`vp04`, NAM\adm_dohertj2). No firewall changes, and no
|
||
ScadaBridge services or files were touched (another agent was installing ScadaBridge on the same boxes at the same time).
|
||
|
||
## Build
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Artifact | `E:\psrem-stage\mxgw-e74918cc\mxgw-e74918cc.zip`, sha256 `F0999B75…D847F2` (script-verified on both boxes) |
|
||
| `ZB.MOM.WW.MxGateway.Server.exe` ProductVersion | `0.1.2+e74918cc` |
|
||
| `MxGateway.Worker.exe` (x86) ProductVersion | `0.1.2+e74918cc` |
|
||
| LmxProxy.dll (MXAccess gate) | `6000.0045.2657.1` (sha256 `BD75A2FDE222`) on both boxes |
|
||
| Service | `MxAccessGw`, NSSM, `LocalSystem`, Automatic, SCM recovery restart ×3 (5 s/5 s/30 s), failureflag 1 |
|
||
| Layout | `E:\ApiInstall\MxAccessGateway\{app,worker,data,logs}`; KEK `C:\ProgramData\ZB\kek\mxgateway.kek` (32 bytes, per box) |
|
||
| Env | 38 entries (ports 5120 h2c gRPC / 5130 dashboard; LDAPS 636 NAM, `svc_igntadmin`, `BindMode=ServiceAccount`; `UG_WW_APP_*_PROD` role map; `RequireHttpsCookie=false`; **`Alarms__Enabled=false`**; OTLP → 127.0.0.1:4317) |
|
||
|
||
## Staging-script fix applied before the install (PROD-STAGE PATCH 4)
|
||
|
||
The first dry run on vp03 (20:32:00Z) aborted:
|
||
|
||
```
|
||
Install-MxGatewayService service=MxAccessGw dry=True
|
||
ERROR: Can't open service!
|
||
ABORT: Install-MxGatewayService exited 1
|
||
```
|
||
|
||
Cause: `Invoke-Nssm` runs `& nssm status MxAccessGw 2>&1` under `$ErrorActionPreference='Stop'`. On Windows PowerShell
|
||
5.1 that turns nssm's stderr ("Can't open service!", the *expected* answer on a fresh box) into a terminating
|
||
NativeCommandError, so the `-AllowFail` existence probe can never succeed on a first install. Fix: run the native call
|
||
under `Continue` (restored in `finally`). Only the staged copy was patched
|
||
(`scadaproj/docs/deploy/2026-11-19-prod-stage/mxgw/Install-MxGatewayService.ps1`, sha256 `FB6F0F474A26…`); it was pushed
|
||
over SMB to both boxes, and the previous copy was kept beside it as `Install-MxGatewayService.ps1.patch3`. **The repo's
|
||
`scripts/deploy/Install-MxGatewayService.ps1` has the same defect (line 65, under EAP=Stop) and needs the same fix before the PAR boxes are installed.**
|
||
|
||
## wonder-app-vp03 (central, Galaxy Repository host)
|
||
|
||
| Step (UTC) | Result |
|
||
|---|---|
|
||
| 20:32:50 dry | DONE, no ABORT (one benign WARN: dry `WorkerDir` ≠ the env's canonical worker path); `dry\` deleted afterwards |
|
||
| 20:33:08 real | DONE; service installed STOPPED; KEK generated sha256 `567A6AFF` |
|
||
| 20:33:29 seed | `nam-ldap-svc-password` → `E:\ApiInstall\MxAccessGateway\data\secrets.db`, `set rc=0`, `readback-match=True`, `SEED DONE failures=0` |
|
||
| 20:33:55 start | `nssm start` OK, Running after 3 s; stdout log shows listening on 5130 + 5120 and "Application started", stderr empty |
|
||
| health | `/health/ready` **200 Healthy**: auth-store, worker-executable, galaxy-sql all Healthy. `/healthz` 200, `/health/active` 200. `/health/live` is **404** (this build doesn't map that route; not a fault) |
|
||
| `GET /` | loopback 200 (`AllowAnonymousLocalhost=true`); from the Mac **302 → `/login?ReturnUrl=%2F`** |
|
||
| 20:34:53 LDAP login | one POST `/auth/login` as `dohertj2` → **302 → `/` (success branch)**; dashboard pages 200 as `dohertj2`, Admin nav (API keys, Secrets) present, `/admin/secrets` 200. No LDAP bind warnings/errors in the log. Signed out afterwards |
|
||
| Galaxy | Galaxy Repository page **Healthy**: 901 objects, 60 areas, 544,454 attributes, 5,373 historized, 5,326 alarm attributes; last deploy 2026-10-15 15:28:20 UTC; connection `Data Source=localhost;Initial Catalog=ZB;Integrated Security=True` |
|
||
| Worker | none running (no session yet, as expected) |
|
||
|
||
Top-level Galaxy areas (Browse page, for the owner's alarm-area choice): **ControlSystem, EUR, KPI, KPI_CNC, NAM**.
|
||
The other top-level entries are platforms/engines, not areas: ArcomHMI_Platform, Monitor_OMI_001, the OI_WAR_* OI servers,
|
||
the VEHMI0x platforms and view engines, VitaminEPress_HMI_00x, WinPlatformBER01/02 and WinPlatform_PPPHMI_Row2.
|
||
|
||
## wonder-app-vp04 (central)
|
||
|
||
| Step (UTC) | Result |
|
||
|---|---|
|
||
| 20:36:19 dry | DONE, no ABORT; `dry\` deleted |
|
||
| 20:36:31 real | DONE; service installed STOPPED; KEK generated sha256 `30C69B13`; account LocalSystem |
|
||
| 20:36:45 seed | `set rc=0`, `readback-match=True`, `SEED DONE failures=0` |
|
||
| 20:36:55 start | Running after 3 s; listening on 5130 + 5120; stderr empty |
|
||
| health | `/health/ready` **503 Unhealthy**. auth-store and worker-executable are Healthy; `galaxy-sql` is Unhealthy ("Galaxy Repository SQL ping timed out after 5s.") |
|
||
| `GET /` | from the Mac 302 → `/login` |
|
||
| 20:37:40 LDAP login | one POST as `dohertj2` → **302 → `/` (success)**; `/`, `/admin/secrets`, `/galaxy` 200. No LDAP errors in the log. Signed out |
|
||
| Galaxy | **NOT connected**. The Galaxy page shows "Awaiting…". Log, repeated every 30 s refresh: |
|
||
|
||
```
|
||
[15:37:08 WRN] Galaxy hierarchy cache refresh failed.
|
||
Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open database "ZB" requested by the login. The login failed.
|
||
Login failed for user 'ZMR\WONDER-APP-VP04$'.
|
||
Error Number:4060,State:1,Class:11
|
||
```
|
||
|
||
The computer account is in the **ZMR** domain (`ZMR\WONDER-APP-VP04$`, not NAM). It reaches vp03's SQL Server but has
|
||
no access to the `ZB` database. SQL logins were **not** changed. **Fix (a DBA or owner action):** create a login for
|
||
`ZMR\WONDER-APP-VP04$` on WONDER-APP-VP03's SQL Server and give it read access on `ZB`, as QA did for vq04 → vq03. The
|
||
service was left running, because the gateway retries every 30 s and picks the database up without a restart once access
|
||
exists. Until then its readiness answers 503.
|
||
|
||
## Clean-up
|
||
|
||
- `E:\psrem-stage\_values\mxgw-vp03` and `…\mxgw-vp04` deleted (verified absent). The seed file
|
||
`E:\psrem-stage\_values\seed\nam-ldap-mxgw.txt` was deleted by `seed-secret-prod.ps1` on both boxes (verified). The
|
||
spare copy taken before seeding was inside the deleted values dirs.
|
||
- vp03 `E:\psrem-stage\_values\seed\` still holds `site-psk.txt`, which belongs to the ScadaBridge install and was left untouched.
|
||
- Login cookies and the temporary password file on the Mac were deleted right after use.
|
||
|
||
## Probe note
|
||
|
||
curl caps the Cookie header at about 8 KB and silently drops cookies over that limit. The dashboard auth cookie for
|
||
`dohertj2` is about 13 KB in four chunks (many `memberOf` groups), so a plain `curl -b jar` follow-up looks like a failed
|
||
login. Replaying the jar as an explicit `Cookie:` header (`-H @file`) shows the session is valid. Kestrel's 32 KB header
|
||
limit and browsers are fine.
|
||
|
||
## NOT done (open)
|
||
|
||
- **PAR site boxes (parvp03/parvp04): held** until WinPlatform/MXAccess is deployed there from the PROD Galaxy (plan Q3).
|
||
- **Alarms disabled** (`MxGateway__Alarms__Enabled=false`) until the owner picks an area. Then set
|
||
`MxGateway__Alarms__SubscriptionExpression=\\WONDER-APP-VP03\Galaxy!<area>`, set `Enabled=true` and restart.
|
||
- **No API keys minted yet**, so no consumer can open a session and no x86 Worker has started on either box.
|
||
- vp04 Galaxy Repository access (above).
|
||
- Repo copy of `Install-MxGatewayService.ps1` still needs PATCH 4 (and PATCHES 2–3) upstreamed.
|
||
|
||
## Rollback (plan §5)
|
||
|
||
`nssm stop MxAccessGw; nssm remove MxAccessGw confirm`, then delete `E:\ApiInstall\MxAccessGateway` and
|
||
`C:\ProgramData\ZB\kek\mxgateway.kek` on the box. Nothing else on either box depends on it yet.
|