docs(deploy): PROD cluster config - Site site-par (id 1), mxgw key scadabridge-prod shared vp03/vp04, MxGateway connection id 1 Connected
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
This commit is contained in:
@@ -0,0 +1,92 @@
|
|||||||
|
# 2026-11-20: PROD cluster configuration (Site `site-par`, MxGateway API key, site MxGateway connection)
|
||||||
|
|
||||||
|
**Owner request:** "configure the clusters", with the option *Site + PSK + MxGateway conn*: create the Site entity, use
|
||||||
|
`SB-GRPC-PSK`, verify central↔site gRPC, mint an mxgw API key and add a site MxGateway data connection that points at
|
||||||
|
the vp03/vp04 gateways (as QA). No templates or instances.
|
||||||
|
|
||||||
|
Builds: ScadaBridge `2e0ce81e` (centrals wonder-app-vp03 10.220.153.205 / vp04 10.220.153.210; sites wonder-par-vp03
|
||||||
|
10.201.75.86 / par-vp04 10.201.75.87); MxAccessGateway `0.1.2+e74918cc` on vp03/vp04. Previous record:
|
||||||
|
[`2026-11-20-wonder-app-vp03-vp04-par-vp03-vp04-2e0ce81e.md`](2026-11-20-wonder-app-vp03-vp04-par-vp03-vp04-2e0ce81e.md).
|
||||||
|
|
||||||
|
Everything was driven from the Mac. ScadaBridge configuration went through the repo CLI (`2e0ce81e`-identical CLI
|
||||||
|
sources) over HTTP Basic (`SCADABRIDGE_AUTH_MODE=basic`, `--password-file`) against `http://wonder-app-vp03.zmr.zimmer.com:8085`
|
||||||
|
as `NAM\dohertj2`. WinRM was used only for mxgw deploy steps (key mint, store copy, service stop/start) and for reading logs.
|
||||||
|
No firewall changes. No secret was put on argv or written into this record.
|
||||||
|
|
||||||
|
## 1. §3A.5 checks on the active central (vp03)
|
||||||
|
|
||||||
|
| Check (UTC) | Result |
|
||||||
|
|---|---|
|
||||||
|
| 21:26:37 `GET /` | 302 → `/login`. `/health/active`: vp03 200 (active), vp04 503 (standby) |
|
||||||
|
| 21:26:55 `/auth/token` | **401**. Operator error: the password file had a trailing newline. The log shows only the 401; there is no LDAP or service-bind line |
|
||||||
|
| 21:27:16 `/auth/token` | **200**. `username=dohertj2`; roles **Administrator, Designer, Deployer**; claims `zb:username`, `zb:displayname`, `…/identity/claims/role` (×3), `iss`/`aud`/`nbf`/`exp`; `expires_in` 1500 s; `management_bearer=true` |
|
||||||
|
| 21:27:36 UI `/auth/login` | 302 → `/`, log `AuthenticationScheme: Cookies signed in`; `/` answers 200 with `dohertj2` and *Sign Out* in the page (cookie replayed as an explicit header: the 4-chunk ~10.8 KB cookie goes over curl's 8 KB cookie limit, the same probe note as in the mxgw PROD record) |
|
||||||
|
| 21:28:11 CLI over Basic | `site list` → `[]`, `data-connection list` → `[]`, rc 0 |
|
||||||
|
| Log | `E:\ApiInstall\ScadaBridge\central\logs\scadabridge-central-20261120.log` (FileStream shared read): no LDAP, `svc_igntadmin`, bind-failure or `system-side` lines, and no WRN/ERR from 16:28 to 16:36 local |
|
||||||
|
|
||||||
|
Not done: bogus-user refusal and zero-role refusal. We kept LDAP attempts to a minimum; the DEV/QA proofs (#144/#145) apply to the same build.
|
||||||
|
|
||||||
|
## 2. Site entity and central↔site gRPC
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|---|---|
|
||||||
|
| Created 21:30:33 | Site **id 1**, Name `PAR`, SiteIdentifier `site-par`, Description "Wonderware PROD Site (PAR)" |
|
||||||
|
| Node A / B (Akka) | `akka.tcp://scadabridge@10.201.75.86:8082` / `akka.tcp://scadabridge@10.201.75.87:8082` |
|
||||||
|
| gRPC node A / B | `http://10.201.75.86:8083` / `http://10.201.75.87:8083` (IPv4 literals, matching QA's site-a shape) |
|
||||||
|
| PSK | central store secret `sb-grpc-psk-site-par` (seeded earlier); the site env carries the same value. No change was needed |
|
||||||
|
|
||||||
|
Evidence:
|
||||||
|
- Central log 21:30:34: `gRPC channel pair for site site-par refreshed (nodeA=true, nodeB=true)`. Site→central `CentralControlService/Heartbeat` calls on `:8083` return 200.
|
||||||
|
- `health site --identifier site-par` at 21:30:54: `isOnline=true`, firstSeen 21:30:20, nodeRole Active `10.201.75.86`, clusterNodes .86 Primary / .87 Standby both online, `localDbReplicationConnected=true`, oplog backlog 0, secret-mirror failures none on node-a and node-b.
|
||||||
|
- Site logs on both PAR boxes: the last "central endpoint unavailable" warning is from before the centrals started (par-vp03 21:25:35, par-vp04 21:25:10). It is followed by `preferred endpoint http://10.220.153.205:8083 is reachable again; site site-par failed back` (par-vp03 21:25:37, par-vp04 21:25:11). After that there are no refused or auth failures.
|
||||||
|
- Central→site round trip: `deploy artifacts --site-id 1` at 21:35:22. Deployment `1ba4d975a0b246a19c4e0f58f6ca6cc6` returned success. par-vp03 logged `SiteCommandService/ExecuteLifecycle` → `Deploying system artifacts` → `Applying 0 deployed secret(s)`. Site row `artifactsDeployedAt` = 21:35:22, hash `sha256:77d02460…`.
|
||||||
|
|
||||||
|
## 3. MxAccessGateway API key `scadabridge-prod`
|
||||||
|
|
||||||
|
QA reference (`scadabridge-qa`, read from an SMB copy of vq03's store; the copy was deleted afterwards): scopes
|
||||||
|
`events:read, invoke:read, invoke:write, metadata:read, session:close, session:open`, no constraints, no expiry,
|
||||||
|
display name = key id. QA shares one key across vq03/vq04 by copying the store, since both boxes have the same pepper. PROD does the same.
|
||||||
|
|
||||||
|
| Step (UTC) | Result |
|
||||||
|
|---|---|
|
||||||
|
| Pepper check | vp03 and vp04 `MxGateway__ApiKeyPepper` have the same sha256 prefix `d98f5191bba2`, which also matches Keychain `zb-prod/mxgw-pepper` |
|
||||||
|
| 21:31:19 mint on vp03 | `ZB.MOM.WW.MxGateway.Server.exe apikey create-key --key-id scadabridge-prod --display-name scadabridge-prod --scopes <the six above>`. It ran with the service's own env, exit 0, `Status: created`, created_utc 21:31:21. The token was written to an Administrators/SYSTEM-only file, pulled over SMB, stored in Keychain **`zb-prod/mxgw-apikey-scadabridge`** (account `scadabridge-prod`; read back and matched), and the remote file was deleted |
|
||||||
|
| 21:31:59 verify vp03 | `galaxy-test-connection` `{ok:true}`; `open-session` OK (first x86 Worker on PROD vp03, pid 20172) then `close-session` CLOSED; bogus key → `Missing or invalid API key` |
|
||||||
|
| 21:32:25–21:32:48 vp03 | `MxAccessGw` stopped (WAL checkpointed on close), `gateway-auth.db` pulled (sha256 `F71DFD15FD79…`), restarted, ready 200. About 23 s down, with no consumers yet |
|
||||||
|
| 21:33:09–21:34:07 vp04 | Stopped. Backup `E:\ApiInstall\MxAccessGateway\data\gateway-auth.db.bak-20261120-prod-keycopy` (vp04's own store, holding its earlier dashboard audit rows). Stale -wal/-shm removed, vp03's copy placed (hash verified), started. Ready **503**, as before: auth-store Healthy, worker-executable Healthy, galaxy-sql Unhealthy (scadaproj#60). Staging folder deleted |
|
||||||
|
| 21:34:16 verify vp04 | `open-session` OK (Worker pid 15684) and `close-session` CLOSED with the same token; bogus key → `Missing or invalid API key`. `galaxy-test-connection` returns `{}` here because of #60 |
|
||||||
|
|
||||||
|
The key store now holds only `scadabridge-prod`, and it is byte-identical on both boxes.
|
||||||
|
|
||||||
|
## 4. Site MxGateway data connection
|
||||||
|
|
||||||
|
QA reference (`data-connection list` on QA, read-only): connection 1 `MxGateway`, protocol `MxGateway`, retry 3, primary
|
||||||
|
`{"endpoint":"http://10.220.158.2:5120","apiKey":<literal, encrypted at rest>,"clientName":"","writeUserId":0,"useTls":false,"caFile":"","serverName":"","readTimeoutMs":5000}`,
|
||||||
|
no backup. QA's central secret store holds only `nam-ldap-svc-password`, so QA's apiKey is an **inline literal, not a `secret:` reference**.
|
||||||
|
PROD does the same and adds a backup endpoint at the owner's request.
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|---|---|
|
||||||
|
| Created 21:35:14 | DataConnection **id 1**, site 1, Name `MxGateway`, Protocol `MxGateway`, FailoverRetryCount 3, `hasSecrets=true` |
|
||||||
|
| Primary | `http://10.220.153.205:5120` (vp03), apiKey = `scadabridge-prod` token (inline, encrypted at rest), useTls false, readTimeoutMs 5000 |
|
||||||
|
| Backup | `http://10.220.153.210:5120` (vp04), same token and settings |
|
||||||
|
| Delivery | Config files were 0600 in the session scratchpad (`--primary-config-file`/`--backup-config-file`) and deleted after use. Artifact deploy `1ba4d975…` at 21:35:22 |
|
||||||
|
|
||||||
|
Evidence: par-vp03 at 21:35:22 logged `Created DCL connection MxGateway (protocol=MxGateway)`. The site health report at
|
||||||
|
21:35:48 shows `dataConnectionStatuses {MxGateway: Connected}` and `dataConnectionEndpoints {MxGateway: "Connected to primary"}`.
|
||||||
|
This also proves the `10.201.75.x → 10.220.153.205:5120` path. **Tag quality cannot be checked**, because no instance binds the
|
||||||
|
connection yet (`dataConnectionTagQuality {}`). The backup (vp04) leg was proven only from the Mac (session open/close).
|
||||||
|
Its Galaxy browse needs #60.
|
||||||
|
|
||||||
|
## Left open
|
||||||
|
|
||||||
|
- Bogus-user and zero-role refusal on PROD (§3A.5 items 2 and 4): not run, to keep LDAP attempts low.
|
||||||
|
- vp04 Galaxy SQL access (scadaproj#60). Galaxy browse through vp04 fails until it is fixed. Session/read/write paths do not depend on it.
|
||||||
|
- When the PAR gateways exist (plan Q3), re-point connection 1 to `PAR-VP03:5120` / `PAR-VP04:5120` and redeploy artifacts. Mint the key on that pair then.
|
||||||
|
- Templates, instances, inbound API keys, bundle import: Phase 7, not in scope.
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
- Connection: `data-connection delete --id 1`, then `deploy artifacts --site-id 1`.
|
||||||
|
- Site: `site delete --id 1` (the site nodes go back to unregistered heartbeats).
|
||||||
|
- Key: `apikey revoke-key --key-id scadabridge-prod` on each box. On vp04 you can instead restore `gateway-auth.db.bak-20261120-prod-keycopy` with the service stopped.
|
||||||
Reference in New Issue
Block a user