Files
ScadaBridge/docs/deployment/deployment-records/2026-11-20-prod-cluster-config.md
T

93 lines
8.7 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 2026-11-20: PROD cluster configuration (Site `site-par`, MxGateway API key, site MxGateway connection)
**Owner request:** "configure the clusters", with the option *Site + PSK + MxGateway conn*: create the Site entity, use
`SB-GRPC-PSK`, verify central↔site gRPC, mint an mxgw API key and add a site MxGateway data connection that points at
the vp03/vp04 gateways (as QA). No templates or instances.
Builds: ScadaBridge `2e0ce81e` (centrals wonder-app-vp03 10.220.153.205 / vp04 10.220.153.210; sites wonder-par-vp03
10.201.75.86 / par-vp04 10.201.75.87); MxAccessGateway `0.1.2+e74918cc` on vp03/vp04. Previous record:
[`2026-11-20-wonder-app-vp03-vp04-par-vp03-vp04-2e0ce81e.md`](2026-11-20-wonder-app-vp03-vp04-par-vp03-vp04-2e0ce81e.md).
Everything was driven from the Mac. ScadaBridge configuration went through the repo CLI (`2e0ce81e`-identical CLI
sources) over HTTP Basic (`SCADABRIDGE_AUTH_MODE=basic`, `--password-file`) against `http://wonder-app-vp03.zmr.zimmer.com:8085`
as `NAM\dohertj2`. WinRM was used only for mxgw deploy steps (key mint, store copy, service stop/start) and for reading logs.
No firewall changes. No secret was put on argv or written into this record.
## 1. §3A.5 checks on the active central (vp03)
| Check (UTC) | Result |
|---|---|
| 21:26:37 `GET /` | 302 → `/login`. `/health/active`: vp03 200 (active), vp04 503 (standby) |
| 21:26:55 `/auth/token` | **401**. Operator error: the password file had a trailing newline. The log shows only the 401; there is no LDAP or service-bind line |
| 21:27:16 `/auth/token` | **200**. `username=dohertj2`; roles **Administrator, Designer, Deployer**; claims `zb:username`, `zb:displayname`, `…/identity/claims/role` (×3), `iss`/`aud`/`nbf`/`exp`; `expires_in` 1500 s; `management_bearer=true` |
| 21:27:36 UI `/auth/login` | 302 → `/`, log `AuthenticationScheme: Cookies signed in`; `/` answers 200 with `dohertj2` and *Sign Out* in the page (cookie replayed as an explicit header: the 4-chunk ~10.8 KB cookie goes over curl's 8 KB cookie limit, the same probe note as in the mxgw PROD record) |
| 21:28:11 CLI over Basic | `site list` → `[]`, `data-connection list` → `[]`, rc 0 |
| Log | `E:\ApiInstall\ScadaBridge\central\logs\scadabridge-central-20261120.log` (FileStream shared read): no LDAP, `svc_igntadmin`, bind-failure or `system-side` lines, and no WRN/ERR from 16:28 to 16:36 local |
Not done: bogus-user refusal and zero-role refusal. We kept LDAP attempts to a minimum; the DEV/QA proofs (#144/#145) apply to the same build.
## 2. Site entity and central↔site gRPC
| Item | Value |
|---|---|
| Created 21:30:33 | Site **id 1**, Name `PAR`, SiteIdentifier `site-par`, Description "Wonderware PROD Site (PAR)" |
| Node A / B (Akka) | `akka.tcp://scadabridge@10.201.75.86:8082` / `akka.tcp://scadabridge@10.201.75.87:8082` |
| gRPC node A / B | `http://10.201.75.86:8083` / `http://10.201.75.87:8083` (IPv4 literals, matching QA's site-a shape) |
| PSK | central store secret `sb-grpc-psk-site-par` (seeded earlier); the site env carries the same value. No change was needed |
Evidence:
- Central log 21:30:34: `gRPC channel pair for site site-par refreshed (nodeA=true, nodeB=true)`. Site→central `CentralControlService/Heartbeat` calls on `:8083` return 200.
- `health site --identifier site-par` at 21:30:54: `isOnline=true`, firstSeen 21:30:20, nodeRole Active `10.201.75.86`, clusterNodes .86 Primary / .87 Standby both online, `localDbReplicationConnected=true`, oplog backlog 0, secret-mirror failures none on node-a and node-b.
- Site logs on both PAR boxes: the last "central endpoint unavailable" warning is from before the centrals started (par-vp03 21:25:35, par-vp04 21:25:10). It is followed by `preferred endpoint http://10.220.153.205:8083 is reachable again; site site-par failed back` (par-vp03 21:25:37, par-vp04 21:25:11). After that there are no refused or auth failures.
- Central→site round trip: `deploy artifacts --site-id 1` at 21:35:22. Deployment `1ba4d975a0b246a19c4e0f58f6ca6cc6` returned success. par-vp03 logged `SiteCommandService/ExecuteLifecycle` → `Deploying system artifacts` → `Applying 0 deployed secret(s)`. Site row `artifactsDeployedAt` = 21:35:22, hash `sha256:77d02460…`.
## 3. MxAccessGateway API key `scadabridge-prod`
QA reference (`scadabridge-qa`, read from an SMB copy of vq03's store; the copy was deleted afterwards): scopes
`events:read, invoke:read, invoke:write, metadata:read, session:close, session:open`, no constraints, no expiry,
display name = key id. QA shares one key across vq03/vq04 by copying the store, since both boxes have the same pepper. PROD does the same.
| Step (UTC) | Result |
|---|---|
| Pepper check | vp03 and vp04 `MxGateway__ApiKeyPepper` have the same sha256 prefix `d98f5191bba2`, which also matches Keychain `zb-prod/mxgw-pepper` |
| 21:31:19 mint on vp03 | `ZB.MOM.WW.MxGateway.Server.exe apikey create-key --key-id scadabridge-prod --display-name scadabridge-prod --scopes <the six above>`. It ran with the service's own env, exit 0, `Status: created`, created_utc 21:31:21. The token was written to an Administrators/SYSTEM-only file, pulled over SMB, stored in Keychain **`zb-prod/mxgw-apikey-scadabridge`** (account `scadabridge-prod`; read back and matched), and the remote file was deleted |
| 21:31:59 verify vp03 | `galaxy-test-connection` `{ok:true}`; `open-session` OK (first x86 Worker on PROD vp03, pid 20172) then `close-session` CLOSED; bogus key → `Missing or invalid API key` |
| 21:32:25–21:32:48 vp03 | `MxAccessGw` stopped (WAL checkpointed on close), `gateway-auth.db` pulled (sha256 `F71DFD15FD79…`), restarted, ready 200. About 23 s down, with no consumers yet |
| 21:33:09–21:34:07 vp04 | Stopped. Backup `E:\ApiInstall\MxAccessGateway\data\gateway-auth.db.bak-20261120-prod-keycopy` (vp04's own store, holding its earlier dashboard audit rows). Stale -wal/-shm removed, vp03's copy placed (hash verified), started. Ready **503**, as before: auth-store Healthy, worker-executable Healthy, galaxy-sql Unhealthy (scadaproj#60). Staging folder deleted |
| 21:34:16 verify vp04 | `open-session` OK (Worker pid 15684) and `close-session` CLOSED with the same token; bogus key → `Missing or invalid API key`. `galaxy-test-connection` returns `{}` here because of #60 |
The key store now holds only `scadabridge-prod`, and it is byte-identical on both boxes.
## 4. Site MxGateway data connection
QA reference (`data-connection list` on QA, read-only): connection 1 `MxGateway`, protocol `MxGateway`, retry 3, primary
`{"endpoint":"http://10.220.158.2:5120","apiKey":<literal, encrypted at rest>,"clientName":"","writeUserId":0,"useTls":false,"caFile":"","serverName":"","readTimeoutMs":5000}`,
no backup. QA's central secret store holds only `nam-ldap-svc-password`, so QA's apiKey is an **inline literal, not a `secret:` reference**.
PROD does the same and adds a backup endpoint at the owner's request.
| Item | Value |
|---|---|
| Created 21:35:14 | DataConnection **id 1**, site 1, Name `MxGateway`, Protocol `MxGateway`, FailoverRetryCount 3, `hasSecrets=true` |
| Primary | `http://10.220.153.205:5120` (vp03), apiKey = `scadabridge-prod` token (inline, encrypted at rest), useTls false, readTimeoutMs 5000 |
| Backup | `http://10.220.153.210:5120` (vp04), same token and settings |
| Delivery | Config files were 0600 in the session scratchpad (`--primary-config-file`/`--backup-config-file`) and deleted after use. Artifact deploy `1ba4d975…` at 21:35:22 |
Evidence: par-vp03 at 21:35:22 logged `Created DCL connection MxGateway (protocol=MxGateway)`. The site health report at
21:35:48 shows `dataConnectionStatuses {MxGateway: Connected}` and `dataConnectionEndpoints {MxGateway: "Connected to primary"}`.
This also proves the `10.201.75.x → 10.220.153.205:5120` path. **Tag quality cannot be checked**, because no instance binds the
connection yet (`dataConnectionTagQuality {}`). The backup (vp04) leg was proven only from the Mac (session open/close).
Its Galaxy browse needs #60.
## Left open
- Bogus-user and zero-role refusal on PROD (§3A.5 items 2 and 4): not run, to keep LDAP attempts low.
- vp04 Galaxy SQL access (scadaproj#60). Galaxy browse through vp04 fails until it is fixed. Session/read/write paths do not depend on it.
- When the PAR gateways exist (plan Q3), re-point connection 1 to `PAR-VP03:5120` / `PAR-VP04:5120` and redeploy artifacts. Mint the key on that pair then.
- Templates, instances, inbound API keys, bundle import: Phase 7, not in scope.
## Rollback
- Connection: `data-connection delete --id 1`, then `deploy artifacts --site-id 1`.
- Site: `site delete --id 1` (the site nodes go back to unregistered heartbeats).
- Key: `apikey revoke-key --key-id scadabridge-prod` on each box. On vp04 you can instead restore `gateway-auth.db.bak-20261120-prod-keycopy` with the service stopped.