From 530eaf32c1d7e760025922ffc8e4f3d410f2bc79 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 20 Nov 2026 16:37:41 -0500 Subject: [PATCH] docs(deploy): PROD cluster config - Site site-par (id 1), mxgw key scadabridge-prod shared vp03/vp04, MxGateway connection id 1 Connected Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ --- .../2026-11-20-prod-cluster-config.md | 92 +++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 docs/deployment/deployment-records/2026-11-20-prod-cluster-config.md diff --git a/docs/deployment/deployment-records/2026-11-20-prod-cluster-config.md b/docs/deployment/deployment-records/2026-11-20-prod-cluster-config.md new file mode 100644 index 00000000..c8ef7960 --- /dev/null +++ b/docs/deployment/deployment-records/2026-11-20-prod-cluster-config.md @@ -0,0 +1,92 @@ +# 2026-11-20: PROD cluster configuration (Site `site-par`, MxGateway API key, site MxGateway connection) + +**Owner request:** "configure the clusters", with the option *Site + PSK + MxGateway conn*: create the Site entity, use +`SB-GRPC-PSK`, verify central↔site gRPC, mint an mxgw API key and add a site MxGateway data connection that points at +the vp03/vp04 gateways (as QA). No templates or instances. + +Builds: ScadaBridge `2e0ce81e` (centrals wonder-app-vp03 10.220.153.205 / vp04 10.220.153.210; sites wonder-par-vp03 +10.201.75.86 / par-vp04 10.201.75.87); MxAccessGateway `0.1.2+e74918cc` on vp03/vp04. Previous record: +[`2026-11-20-wonder-app-vp03-vp04-par-vp03-vp04-2e0ce81e.md`](2026-11-20-wonder-app-vp03-vp04-par-vp03-vp04-2e0ce81e.md). + +Everything was driven from the Mac. ScadaBridge configuration went through the repo CLI (`2e0ce81e`-identical CLI +sources) over HTTP Basic (`SCADABRIDGE_AUTH_MODE=basic`, `--password-file`) against `http://wonder-app-vp03.zmr.zimmer.com:8085` +as `NAM\dohertj2`. WinRM was used only for mxgw deploy steps (key mint, store copy, service stop/start) and for reading logs. +No firewall changes. No secret was put on argv or written into this record. + +## 1. §3A.5 checks on the active central (vp03) + +| Check (UTC) | Result | +|---|---| +| 21:26:37 `GET /` | 302 → `/login`. `/health/active`: vp03 200 (active), vp04 503 (standby) | +| 21:26:55 `/auth/token` | **401**. Operator error: the password file had a trailing newline. The log shows only the 401; there is no LDAP or service-bind line | +| 21:27:16 `/auth/token` | **200**. `username=dohertj2`; roles **Administrator, Designer, Deployer**; claims `zb:username`, `zb:displayname`, `…/identity/claims/role` (×3), `iss`/`aud`/`nbf`/`exp`; `expires_in` 1500 s; `management_bearer=true` | +| 21:27:36 UI `/auth/login` | 302 → `/`, log `AuthenticationScheme: Cookies signed in`; `/` answers 200 with `dohertj2` and *Sign Out* in the page (cookie replayed as an explicit header: the 4-chunk ~10.8 KB cookie goes over curl's 8 KB cookie limit, the same probe note as in the mxgw PROD record) | +| 21:28:11 CLI over Basic | `site list` → `[]`, `data-connection list` → `[]`, rc 0 | +| Log | `E:\ApiInstall\ScadaBridge\central\logs\scadabridge-central-20261120.log` (FileStream shared read): no LDAP, `svc_igntadmin`, bind-failure or `system-side` lines, and no WRN/ERR from 16:28 to 16:36 local | + +Not done: bogus-user refusal and zero-role refusal. We kept LDAP attempts to a minimum; the DEV/QA proofs (#144/#145) apply to the same build. + +## 2. Site entity and central↔site gRPC + +| Item | Value | +|---|---| +| Created 21:30:33 | Site **id 1**, Name `PAR`, SiteIdentifier `site-par`, Description "Wonderware PROD Site (PAR)" | +| Node A / B (Akka) | `akka.tcp://scadabridge@10.201.75.86:8082` / `akka.tcp://scadabridge@10.201.75.87:8082` | +| gRPC node A / B | `http://10.201.75.86:8083` / `http://10.201.75.87:8083` (IPv4 literals, matching QA's site-a shape) | +| PSK | central store secret `sb-grpc-psk-site-par` (seeded earlier); the site env carries the same value. No change was needed | + +Evidence: +- Central log 21:30:34: `gRPC channel pair for site site-par refreshed (nodeA=true, nodeB=true)`. Site→central `CentralControlService/Heartbeat` calls on `:8083` return 200. +- `health site --identifier site-par` at 21:30:54: `isOnline=true`, firstSeen 21:30:20, nodeRole Active `10.201.75.86`, clusterNodes .86 Primary / .87 Standby both online, `localDbReplicationConnected=true`, oplog backlog 0, secret-mirror failures none on node-a and node-b. +- Site logs on both PAR boxes: the last "central endpoint unavailable" warning is from before the centrals started (par-vp03 21:25:35, par-vp04 21:25:10). It is followed by `preferred endpoint http://10.220.153.205:8083 is reachable again; site site-par failed back` (par-vp03 21:25:37, par-vp04 21:25:11). After that there are no refused or auth failures. +- Central→site round trip: `deploy artifacts --site-id 1` at 21:35:22. Deployment `1ba4d975a0b246a19c4e0f58f6ca6cc6` returned success. par-vp03 logged `SiteCommandService/ExecuteLifecycle` → `Deploying system artifacts` → `Applying 0 deployed secret(s)`. Site row `artifactsDeployedAt` = 21:35:22, hash `sha256:77d02460…`. + +## 3. MxAccessGateway API key `scadabridge-prod` + +QA reference (`scadabridge-qa`, read from an SMB copy of vq03's store; the copy was deleted afterwards): scopes +`events:read, invoke:read, invoke:write, metadata:read, session:close, session:open`, no constraints, no expiry, +display name = key id. QA shares one key across vq03/vq04 by copying the store, since both boxes have the same pepper. PROD does the same. + +| Step (UTC) | Result | +|---|---| +| Pepper check | vp03 and vp04 `MxGateway__ApiKeyPepper` have the same sha256 prefix `d98f5191bba2`, which also matches Keychain `zb-prod/mxgw-pepper` | +| 21:31:19 mint on vp03 | `ZB.MOM.WW.MxGateway.Server.exe apikey create-key --key-id scadabridge-prod --display-name scadabridge-prod --scopes `. It ran with the service's own env, exit 0, `Status: created`, created_utc 21:31:21. The token was written to an Administrators/SYSTEM-only file, pulled over SMB, stored in Keychain **`zb-prod/mxgw-apikey-scadabridge`** (account `scadabridge-prod`; read back and matched), and the remote file was deleted | +| 21:31:59 verify vp03 | `galaxy-test-connection` `{ok:true}`; `open-session` OK (first x86 Worker on PROD vp03, pid 20172) then `close-session` CLOSED; bogus key → `Missing or invalid API key` | +| 21:32:25–21:32:48 vp03 | `MxAccessGw` stopped (WAL checkpointed on close), `gateway-auth.db` pulled (sha256 `F71DFD15FD79…`), restarted, ready 200. About 23 s down, with no consumers yet | +| 21:33:09–21:34:07 vp04 | Stopped. Backup `E:\ApiInstall\MxAccessGateway\data\gateway-auth.db.bak-20261120-prod-keycopy` (vp04's own store, holding its earlier dashboard audit rows). Stale -wal/-shm removed, vp03's copy placed (hash verified), started. Ready **503**, as before: auth-store Healthy, worker-executable Healthy, galaxy-sql Unhealthy (scadaproj#60). Staging folder deleted | +| 21:34:16 verify vp04 | `open-session` OK (Worker pid 15684) and `close-session` CLOSED with the same token; bogus key → `Missing or invalid API key`. `galaxy-test-connection` returns `{}` here because of #60 | + +The key store now holds only `scadabridge-prod`, and it is byte-identical on both boxes. + +## 4. Site MxGateway data connection + +QA reference (`data-connection list` on QA, read-only): connection 1 `MxGateway`, protocol `MxGateway`, retry 3, primary +`{"endpoint":"http://10.220.158.2:5120","apiKey":,"clientName":"","writeUserId":0,"useTls":false,"caFile":"","serverName":"","readTimeoutMs":5000}`, +no backup. QA's central secret store holds only `nam-ldap-svc-password`, so QA's apiKey is an **inline literal, not a `secret:` reference**. +PROD does the same and adds a backup endpoint at the owner's request. + +| Item | Value | +|---|---| +| Created 21:35:14 | DataConnection **id 1**, site 1, Name `MxGateway`, Protocol `MxGateway`, FailoverRetryCount 3, `hasSecrets=true` | +| Primary | `http://10.220.153.205:5120` (vp03), apiKey = `scadabridge-prod` token (inline, encrypted at rest), useTls false, readTimeoutMs 5000 | +| Backup | `http://10.220.153.210:5120` (vp04), same token and settings | +| Delivery | Config files were 0600 in the session scratchpad (`--primary-config-file`/`--backup-config-file`) and deleted after use. Artifact deploy `1ba4d975…` at 21:35:22 | + +Evidence: par-vp03 at 21:35:22 logged `Created DCL connection MxGateway (protocol=MxGateway)`. The site health report at +21:35:48 shows `dataConnectionStatuses {MxGateway: Connected}` and `dataConnectionEndpoints {MxGateway: "Connected to primary"}`. +This also proves the `10.201.75.x → 10.220.153.205:5120` path. **Tag quality cannot be checked**, because no instance binds the +connection yet (`dataConnectionTagQuality {}`). The backup (vp04) leg was proven only from the Mac (session open/close). +Its Galaxy browse needs #60. + +## Left open + +- Bogus-user and zero-role refusal on PROD (§3A.5 items 2 and 4): not run, to keep LDAP attempts low. +- vp04 Galaxy SQL access (scadaproj#60). Galaxy browse through vp04 fails until it is fixed. Session/read/write paths do not depend on it. +- When the PAR gateways exist (plan Q3), re-point connection 1 to `PAR-VP03:5120` / `PAR-VP04:5120` and redeploy artifacts. Mint the key on that pair then. +- Templates, instances, inbound API keys, bundle import: Phase 7, not in scope. + +## Rollback + +- Connection: `data-connection delete --id 1`, then `deploy artifacts --site-id 1`. +- Site: `site delete --id 1` (the site nodes go back to unregistered heartbeats). +- Key: `apikey revoke-key --key-id scadabridge-prod` on each box. On vp04 you can instead restore `gateway-auth.db.bak-20261120-prod-keycopy` with the service stopped.