Files
ScadaBridge/docs/deployment/deployment-records/2026-11-20-prod-cluster-config.md
T

8.7 KiB
Raw Blame History

2026-11-20: PROD cluster configuration (Site site-par, MxGateway API key, site MxGateway connection)

Owner request: "configure the clusters", with the option Site + PSK + MxGateway conn: create the Site entity, use SB-GRPC-PSK, verify central↔site gRPC, mint an mxgw API key and add a site MxGateway data connection that points at the vp03/vp04 gateways (as QA). No templates or instances.

Builds: ScadaBridge 2e0ce81e (centrals wonder-app-vp03 10.220.153.205 / vp04 10.220.153.210; sites wonder-par-vp03 10.201.75.86 / par-vp04 10.201.75.87); MxAccessGateway 0.1.2+e74918cc on vp03/vp04. Previous record: 2026-11-20-wonder-app-vp03-vp04-par-vp03-vp04-2e0ce81e.md.

Everything was driven from the Mac. ScadaBridge configuration went through the repo CLI (2e0ce81e-identical CLI sources) over HTTP Basic (SCADABRIDGE_AUTH_MODE=basic, --password-file) against http://wonder-app-vp03.zmr.zimmer.com:8085 as NAM\dohertj2. WinRM was used only for mxgw deploy steps (key mint, store copy, service stop/start) and for reading logs. No firewall changes. No secret was put on argv or written into this record.

1. §3A.5 checks on the active central (vp03)

Check (UTC) Result
21:26:37 GET / 302 → /login. /health/active: vp03 200 (active), vp04 503 (standby)
21:26:55 /auth/token 401. Operator error: the password file had a trailing newline. The log shows only the 401; there is no LDAP or service-bind line
21:27:16 /auth/token 200. username=dohertj2; roles Administrator, Designer, Deployer; claims zb:username, zb:displayname, …/identity/claims/role (×3), iss/aud/nbf/exp; expires_in 1500 s; management_bearer=true
21:27:36 UI /auth/login 302 → /, log AuthenticationScheme: Cookies signed in; / answers 200 with dohertj2 and Sign Out in the page (cookie replayed as an explicit header: the 4-chunk ~10.8 KB cookie goes over curl's 8 KB cookie limit, the same probe note as in the mxgw PROD record)
21:28:11 CLI over Basic site list → [], data-connection list → [], rc 0
Log E:\ApiInstall\ScadaBridge\central\logs\scadabridge-central-20261120.log (FileStream shared read): no LDAP, svc_igntadmin, bind-failure or system-side lines, and no WRN/ERR from 16:28 to 16:36 local

Not done: bogus-user refusal and zero-role refusal. We kept LDAP attempts to a minimum; the DEV/QA proofs (#144/#145) apply to the same build.

2. Site entity and central↔site gRPC

Item Value
Created 21:30:33 Site id 1, Name PAR, SiteIdentifier site-par, Description "Wonderware PROD Site (PAR)"
Node A / B (Akka) akka.tcp://scadabridge@10.201.75.86:8082 / akka.tcp://scadabridge@10.201.75.87:8082
gRPC node A / B http://10.201.75.86:8083 / http://10.201.75.87:8083 (IPv4 literals, matching QA's site-a shape)
PSK central store secret sb-grpc-psk-site-par (seeded earlier); the site env carries the same value. No change was needed

Evidence:

  • Central log 21:30:34: gRPC channel pair for site site-par refreshed (nodeA=true, nodeB=true). Site→central CentralControlService/Heartbeat calls on :8083 return 200.
  • health site --identifier site-par at 21:30:54: isOnline=true, firstSeen 21:30:20, nodeRole Active 10.201.75.86, clusterNodes .86 Primary / .87 Standby both online, localDbReplicationConnected=true, oplog backlog 0, secret-mirror failures none on node-a and node-b.
  • Site logs on both PAR boxes: the last "central endpoint unavailable" warning is from before the centrals started (par-vp03 21:25:35, par-vp04 21:25:10). It is followed by preferred endpoint http://10.220.153.205:8083 is reachable again; site site-par failed back (par-vp03 21:25:37, par-vp04 21:25:11). After that there are no refused or auth failures.
  • Central→site round trip: deploy artifacts --site-id 1 at 21:35:22. Deployment 1ba4d975a0b246a19c4e0f58f6ca6cc6 returned success. par-vp03 logged SiteCommandService/ExecuteLifecycle → Deploying system artifacts → Applying 0 deployed secret(s). Site row artifactsDeployedAt = 21:35:22, hash sha256:77d02460….

3. MxAccessGateway API key scadabridge-prod

QA reference (scadabridge-qa, read from an SMB copy of vq03's store; the copy was deleted afterwards): scopes events:read, invoke:read, invoke:write, metadata:read, session:close, session:open, no constraints, no expiry, display name = key id. QA shares one key across vq03/vq04 by copying the store, since both boxes have the same pepper. PROD does the same.

Step (UTC) Result
Pepper check vp03 and vp04 MxGateway__ApiKeyPepper have the same sha256 prefix d98f5191bba2, which also matches Keychain zb-prod/mxgw-pepper
21:31:19 mint on vp03 ZB.MOM.WW.MxGateway.Server.exe apikey create-key --key-id scadabridge-prod --display-name scadabridge-prod --scopes <the six above>. It ran with the service's own env, exit 0, Status: created, created_utc 21:31:21. The token was written to an Administrators/SYSTEM-only file, pulled over SMB, stored in Keychain zb-prod/mxgw-apikey-scadabridge (account scadabridge-prod; read back and matched), and the remote file was deleted
21:31:59 verify vp03 galaxy-test-connection {ok:true}; open-session OK (first x86 Worker on PROD vp03, pid 20172) then close-session CLOSED; bogus key → Missing or invalid API key
21:32:25–21:32:48 vp03 MxAccessGw stopped (WAL checkpointed on close), gateway-auth.db pulled (sha256 F71DFD15FD79…), restarted, ready 200. About 23 s down, with no consumers yet
21:33:09–21:34:07 vp04 Stopped. Backup E:\ApiInstall\MxAccessGateway\data\gateway-auth.db.bak-20261120-prod-keycopy (vp04's own store, holding its earlier dashboard audit rows). Stale -wal/-shm removed, vp03's copy placed (hash verified), started. Ready 503, as before: auth-store Healthy, worker-executable Healthy, galaxy-sql Unhealthy (scadaproj#60). Staging folder deleted
21:34:16 verify vp04 open-session OK (Worker pid 15684) and close-session CLOSED with the same token; bogus key → Missing or invalid API key. galaxy-test-connection returns {} here because of #60

The key store now holds only scadabridge-prod, and it is byte-identical on both boxes.

4. Site MxGateway data connection

QA reference (data-connection list on QA, read-only): connection 1 MxGateway, protocol MxGateway, retry 3, primary {"endpoint":"http://10.220.158.2:5120","apiKey":<literal, encrypted at rest>,"clientName":"","writeUserId":0,"useTls":false,"caFile":"","serverName":"","readTimeoutMs":5000}, no backup. QA's central secret store holds only nam-ldap-svc-password, so QA's apiKey is an inline literal, not a secret: reference. PROD does the same and adds a backup endpoint at the owner's request.

Item Value
Created 21:35:14 DataConnection id 1, site 1, Name MxGateway, Protocol MxGateway, FailoverRetryCount 3, hasSecrets=true
Primary http://10.220.153.205:5120 (vp03), apiKey = scadabridge-prod token (inline, encrypted at rest), useTls false, readTimeoutMs 5000
Backup http://10.220.153.210:5120 (vp04), same token and settings
Delivery Config files were 0600 in the session scratchpad (--primary-config-file/--backup-config-file) and deleted after use. Artifact deploy 1ba4d975… at 21:35:22

Evidence: par-vp03 at 21:35:22 logged Created DCL connection MxGateway (protocol=MxGateway). The site health report at 21:35:48 shows dataConnectionStatuses {MxGateway: Connected} and dataConnectionEndpoints {MxGateway: "Connected to primary"}. This also proves the 10.201.75.x → 10.220.153.205:5120 path. Tag quality cannot be checked, because no instance binds the connection yet (dataConnectionTagQuality {}). The backup (vp04) leg was proven only from the Mac (session open/close). Its Galaxy browse needs #60.

Left open

  • Bogus-user and zero-role refusal on PROD (§3A.5 items 2 and 4): not run, to keep LDAP attempts low.
  • vp04 Galaxy SQL access (scadaproj#60). Galaxy browse through vp04 fails until it is fixed. Session/read/write paths do not depend on it.
  • When the PAR gateways exist (plan Q3), re-point connection 1 to PAR-VP03:5120 / PAR-VP04:5120 and redeploy artifacts. Mint the key on that pair then.
  • Templates, instances, inbound API keys, bundle import: Phase 7, not in scope.

Rollback

  • Connection: data-connection delete --id 1, then deploy artifacts --site-id 1.
  • Site: site delete --id 1 (the site nodes go back to unregistered heartbeats).
  • Key: apikey revoke-key --key-id scadabridge-prod on each box. On vp04 you can instead restore gateway-auth.db.bak-20261120-prod-keycopy with the service stopped.