Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
10 KiB
2026-11-20: PROD cluster configuration (Site site-par, MxGateway API key, site MxGateway connection)
Owner request: "configure the clusters", with the option Site + PSK + MxGateway conn: create the Site entity, use
SB-GRPC-PSK, verify central↔site gRPC, mint an mxgw API key and add a site MxGateway data connection that points at
the vp03/vp04 gateways (as QA). No templates or instances.
Builds: ScadaBridge 2e0ce81e (centrals wonder-app-vp03 10.220.153.205 / vp04 10.220.153.210; sites wonder-par-vp03
10.201.75.86 / par-vp04 10.201.75.87); MxAccessGateway 0.1.2+e74918cc on vp03/vp04. Previous record:
2026-11-20-wonder-app-vp03-vp04-par-vp03-vp04-2e0ce81e.md.
Everything was driven from the Mac. ScadaBridge configuration went through the repo CLI (2e0ce81e-identical CLI
sources) over HTTP Basic (SCADABRIDGE_AUTH_MODE=basic, --password-file) against http://wonder-app-vp03.zmr.zimmer.com:8085
as NAM\dohertj2. WinRM was used only for mxgw deploy steps (key mint, store copy, service stop/start) and for reading logs.
No firewall changes. No secret was put on argv or written into this record.
1. §3A.5 checks on the active central (vp03)
| Check (UTC) | Result |
|---|---|
21:26:37 GET / |
302 → /login. /health/active: vp03 200 (active), vp04 503 (standby) |
21:26:55 /auth/token |
401. Operator error: the password file had a trailing newline. The log shows only the 401; there is no LDAP or service-bind line |
21:27:16 /auth/token |
200. username=dohertj2; roles Administrator, Designer, Deployer; claims zb:username, zb:displayname, …/identity/claims/role (×3), iss/aud/nbf/exp; expires_in 1500 s; management_bearer=true |
21:27:36 UI /auth/login |
302 → /, log AuthenticationScheme: Cookies signed in; / answers 200 with dohertj2 and Sign Out in the page (cookie replayed as an explicit header: the 4-chunk ~10.8 KB cookie goes over curl's 8 KB cookie limit, the same probe note as in the mxgw PROD record) |
| 21:28:11 CLI over Basic | site list → [], data-connection list → [], rc 0 |
| Log | E:\ApiInstall\ScadaBridge\central\logs\scadabridge-central-20261120.log (FileStream shared read): no LDAP, svc_igntadmin, bind-failure or system-side lines, and no WRN/ERR from 16:28 to 16:36 local |
Not done: bogus-user refusal and zero-role refusal. We kept LDAP attempts to a minimum; the DEV/QA proofs (#144/#145) apply to the same build.
2. Site entity and central↔site gRPC
| Item | Value |
|---|---|
| Created 21:30:33 | Site id 1, Name PAR, SiteIdentifier site-par, Description "Wonderware PROD Site (PAR)" |
| Node A / B (Akka) | akka.tcp://scadabridge@10.201.75.86:8082 / akka.tcp://scadabridge@10.201.75.87:8082 |
| gRPC node A / B | http://10.201.75.86:8083 / http://10.201.75.87:8083 (IPv4 literals, matching QA's site-a shape) |
| PSK | central store secret sb-grpc-psk-site-par (seeded earlier); the site env carries the same value. No change was needed |
Evidence:
- Central log 21:30:34:
gRPC channel pair for site site-par refreshed (nodeA=true, nodeB=true). Site→centralCentralControlService/Heartbeatcalls on:8083return 200. health site --identifier site-parat 21:30:54:isOnline=true, firstSeen 21:30:20, nodeRole Active10.201.75.86, clusterNodes .86 Primary / .87 Standby both online,localDbReplicationConnected=true, oplog backlog 0, secret-mirror failures none on node-a and node-b.- Site logs on both PAR boxes: the last "central endpoint unavailable" warning is from before the centrals started (par-vp03 21:25:35, par-vp04 21:25:10). It is followed by
preferred endpoint http://10.220.153.205:8083 is reachable again; site site-par failed back(par-vp03 21:25:37, par-vp04 21:25:11). After that there are no refused or auth failures. - Central→site round trip:
deploy artifacts --site-id 1at 21:35:22. Deployment1ba4d975a0b246a19c4e0f58f6ca6cc6returned success. par-vp03 loggedSiteCommandService/ExecuteLifecycle→Deploying system artifacts→Applying 0 deployed secret(s). Site rowartifactsDeployedAt= 21:35:22, hashsha256:77d02460….
3. MxAccessGateway API key scadabridge-prod
QA reference (scadabridge-qa, read from an SMB copy of vq03's store; the copy was deleted afterwards): scopes
events:read, invoke:read, invoke:write, metadata:read, session:close, session:open, no constraints, no expiry,
display name = key id. QA shares one key across vq03/vq04 by copying the store, since both boxes have the same pepper. PROD does the same.
| Step (UTC) | Result |
|---|---|
| Pepper check | vp03 and vp04 MxGateway__ApiKeyPepper have the same sha256 prefix d98f5191bba2, which also matches Keychain zb-prod/mxgw-pepper |
| 21:31:19 mint on vp03 | ZB.MOM.WW.MxGateway.Server.exe apikey create-key --key-id scadabridge-prod --display-name scadabridge-prod --scopes <the six above>. It ran with the service's own env, exit 0, Status: created, created_utc 21:31:21. The token was written to an Administrators/SYSTEM-only file, pulled over SMB, stored in Keychain zb-prod/mxgw-apikey-scadabridge (account scadabridge-prod; read back and matched), and the remote file was deleted |
| 21:31:59 verify vp03 | galaxy-test-connection {ok:true}; open-session OK (first x86 Worker on PROD vp03, pid 20172) then close-session CLOSED; bogus key → Missing or invalid API key |
| 21:32:25–21:32:48 vp03 | MxAccessGw stopped (WAL checkpointed on close), gateway-auth.db pulled (sha256 F71DFD15FD79…), restarted, ready 200. About 23 s down, with no consumers yet |
| 21:33:09–21:34:07 vp04 | Stopped. Backup E:\ApiInstall\MxAccessGateway\data\gateway-auth.db.bak-20261120-prod-keycopy (vp04's own store, holding its earlier dashboard audit rows). Stale -wal/-shm removed, vp03's copy placed (hash verified), started. Ready 503, as before: auth-store Healthy, worker-executable Healthy, galaxy-sql Unhealthy (scadaproj#60). Staging folder deleted |
| 21:34:16 verify vp04 | open-session OK (Worker pid 15684) and close-session CLOSED with the same token; bogus key → Missing or invalid API key. galaxy-test-connection returns {} here because of #60 |
The key store now holds only scadabridge-prod, and it is byte-identical on both boxes.
4. Site MxGateway data connection
QA reference (data-connection list on QA, read-only): connection 1 MxGateway, protocol MxGateway, retry 3, primary
{"endpoint":"http://10.220.158.2:5120","apiKey":<literal, encrypted at rest>,"clientName":"","writeUserId":0,"useTls":false,"caFile":"","serverName":"","readTimeoutMs":5000},
no backup. QA's central secret store holds only nam-ldap-svc-password, so QA's apiKey is an inline literal, not a secret: reference.
PROD does the same and adds a backup endpoint at the owner's request.
| Item | Value |
|---|---|
| Created 21:35:14 | DataConnection id 1, site 1, Name MxGateway, Protocol MxGateway, FailoverRetryCount 3, hasSecrets=true |
| Primary | http://10.220.153.205:5120 (vp03), apiKey = scadabridge-prod token (inline, encrypted at rest), useTls false, readTimeoutMs 5000 |
| Backup | http://10.220.153.210:5120 (vp04), same token and settings |
| Delivery | Config files were 0600 in the session scratchpad (--primary-config-file/--backup-config-file) and deleted after use. Artifact deploy 1ba4d975… at 21:35:22 |
Evidence: par-vp03 at 21:35:22 logged Created DCL connection MxGateway (protocol=MxGateway). The site health report at
21:35:48 shows dataConnectionStatuses {MxGateway: Connected} and dataConnectionEndpoints {MxGateway: "Connected to primary"}.
This also proves the 10.201.75.x → 10.220.153.205:5120 path. Tag quality cannot be checked, because no instance binds the
connection yet (dataConnectionTagQuality {}). The backup (vp04) leg was proven only from the Mac (session open/close).
Its Galaxy browse needs #60.
Left open
- Bogus-user and zero-role refusal on PROD (§3A.5 items 2 and 4): not run, to keep LDAP attempts low.
- vp04 Galaxy SQL access (scadaproj#60). Galaxy browse through vp04 fails until it is fixed. Session/read/write paths do not depend on it.
- When the PAR gateways exist (plan Q3), re-point connection 1 to
PAR-VP03:5120/PAR-VP04:5120and redeploy artifacts. Mint the key on that pair then. - Templates, instances, inbound API keys, bundle import: Phase 7, not in scope.
Rollback
- Connection:
data-connection delete --id 1, thendeploy artifacts --site-id 1. - Site:
site delete --id 1(the site nodes go back to unregistered heartbeats). - Key:
apikey revoke-key --key-id scadabridge-prodon each box. On vp04 you can instead restoregateway-auth.db.bak-20261120-prod-keycopywith the service stopped.
2026-11-20 22:22Z: connection re-pointed to the PAR gateways (owner: "the par mxgw on the par scadabridge instances", chose "Re-point to PAR only")
data-connection update --id 1: primaryhttp://10.201.75.86:5120(par-vp03), backuphttp://10.201.75.87:5120(par-vp04), samescadabridge-prodtoken and settings, retry 3. Config files 0600 in the scratchpad, deleted after use. CLI built from a detached2e0ce81eworktree (assemblyscadabridge.dll).deploy artifacts --site-id 1→ deployment2264d02f4b21450fa1ac61ad8161fb35, success. par-vp03 logged "Reconfigured DCL connection MxGateway ... in place" (no restart), thenConnect failed: Error connecting to subchannelretries. Expected: mxgw is not installed on the PAR boxes (held for WinPlatform, plan Q3). Site-par has no working gateway until then; the connection retries (WRN log lines) and fails over between the two PAR endpoints.- When mxgw is installed on par-vp03/04: use the same
MxGateway__ApiKeyPepper(Keychainzb-prod/mxgw-pepper) and copy vp03'sgateway-auth.db(service stopped) so thescadabridge-prodtoken is valid there; then the connection comes up on its own. Install script fix first: mxaccessgw#234. - Rollback to the app-box gateways: the same
updatewith10.220.153.205:5120/10.220.153.210:5120, thendeploy artifacts --site-id 1.