Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
8.6 KiB
2026-11-20: PROD site site-app co-located with the centrals on wonder-app-vp03 / vp04
Owner request: "prod should have a site too, configure it for mxaccessgw on central nodes". The owner chose
SiteIdentifier site-app and Name APP. The site follows the QA pattern: a site node runs next to the central on each
box and uses the mxgw gateway already running there.
Build: ScadaBridge 2e0ce81e (1.0.0+2e0ce81ea04ac5e85ddb654cf33068594ba5c3c4). The site reuses the central's
E:\ApiInstall\ScadaBridge\app. Boxes: wonder-app-vp03 10.220.153.205 (active central) and wonder-app-vp04
10.220.153.210 (standby central). Previous record: 2026-11-20-prod-cluster-config.md.
Everything was driven from the Mac: winrm (NAM\adm_dohertj2) for the on-box deploy steps and log reads, and the
2e0ce81e CLI over HTTP Basic as NAM\dohertj2 against http://wonder-app-vp03.zmr.zimmer.com:8085 for configuration.
Nothing was done to ScadaBridge-Central, MxAccessGw, the PAR ScadaBridge-Site services, Site 1 or DataConnection 1.
No firewall changes. No secret went on argv or into this record.
Ports per box
| Service | Akka | gRPC | HTTP |
|---|---|---|---|
| ScadaBridge-Central (unchanged) | 8081 | 8083 | 8085 (UI + health) |
| ScadaBridge-Site (new, site-app) | 8082 | 8086 | 8084 (health) |
| MxAccessGw / Overview / aaGR (unchanged) | 5120 | 5130 / 5320 / 8090 |
Site paths: config E:\ApiInstall\ScadaBridge\app\appsettings.Site.json (the central loads only
appsettings.Central.json), and data and logs under E:\ApiInstall\ScadaBridge\site\{data,logs}. The site is a separate
Akka cluster from the centrals, with central memberCount still 2. LocalDb replication runs vp03 → http://10.220.153.210:8086.
Steps (UTC)
| Time | Step | Result |
|---|---|---|
| 22:33:53 | install-sb-site-colocated.ps1 -Mode dry on vp03 |
Aborted at the env step: sb-site.env was missing from E:\psrem-stage\sb-2e0ce81e on both boxes, because it had only been staged for PAR. The non-secret local file (sha256 186210EE…) was pushed to both boxes and the dry run was repeated |
| 22:34:26 | dry vp03 | DONE mode=dry. Host ProductVersion ok, KEK FA4511BE, ports 8082/8084/8086 free, 8 env entries |
| 22:34:35 | real vp03 | service ScadaBridge-Site created STOPPED, env 8 entries, DONE mode=real. Central Running, /health/ready 200, /health/active 200 |
| 22:34:43–22:34:46 | dry + real on vp04 | DONE mode=dry, then DONE mode=real. Central Running, ready 200, active 503 (standby, as before) |
| 22:35:04 | Seed on vp03: seed-secret-prod.ps1 -Target ScadaBridge-Central -SecretName SB-GRPC-PSK-site-app -AllowRunning |
Shared SQL store zbsecrets, KEK FA4511BE, set rc=0 {"name":"sb-grpc-psk-site-app","action":"set"} readback-match=True, value file removed, failures=0 |
| 22:35:20 | site create |
Site id 2: Name APP, identifier site-app, description "PROD site co-located on the central nodes", Akka akka.tcp://scadabridge@10.220.153.205:8082 / @10.220.153.210:8082, gRPC http://10.220.153.205:8086 / http://10.220.153.210:8086. Central log at 22:35:20: gRPC channel pair for site site-app refreshed (nodeA=true, nodeB=true) |
| 22:35:25 | Started ScadaBridge-Site on vp03 and vp04 in parallel | Both Running by 22:35:28. vp03 founded the cluster (bootstrap guard "preferred founder") and vp04 was Up at 22:35:34 |
| 22:37:14 | PROD-STAGE PATCH (see below): vp04 NodeName changed to node-b, and vp04's site restarted |
Started again by about 22:37:28. Bootstrap guard: "partner … REACHABLE … joining peer-first". SiteReconciliationActor … node node-b |
| 22:38:4x | Verification | See the evidence below |
| 22:38:55 | data-connection create --site-id 2 |
DataConnection id 2, site 2, MxGateway / protocol MxGateway, FailoverRetryCount 3, hasSecrets=true. Primary http://10.220.153.205:5120, backup http://10.220.153.210:5120, config shape as for site-par (clientName "", writeUserId 0, useTls false, readTimeoutMs 5000). apiKey = the scadabridge-prod token (Keychain zb-prod/mxgw-apikey-scadabridge), inline and encrypted at rest. Config files were 0600 in the scratchpad and deleted after use |
| 22:39:00 | deploy artifacts --site-id 2 |
Deployment 63fdb88f49134bab89fa9892bb338fcb, site-app success. vp03 site log: Deploying system artifacts → Applying 0 deployed secret(s) → Created DCL connection MxGateway (protocol=MxGateway) → [MxGateway] Connection established / Entering Connected state (22:39:04) |
| 22:40:26 / 22:40:58 | Overview registry on vp03, then vp04 | Old appsettings.Prod.json (sha 3CED9601…) backed up as .bak-20261120. New file sha 964EE099… placed in app\ and in E:\psrem-stage\overview-818775b3\, then Restart-Service ZbOverview. Running, /health/ready 200 |
| 22:42 | Cleanup | E:\psrem-stage\_values\sb-site-app deleted on both boxes, and _values\seed\site-app-psk.txt is gone on vp03. Temporary log script removed |
PROD-STAGE PATCH: vp04 NodeName
The staged appsettings.Site.json.prod-vp04 had "NodeName": "node-a", a copy of vp03's value. The PAR pair uses
node-a / node-b. NodeName feeds the audit SourceNode, the per-node secret_mirror_status row and the
reconciliation actor's identity, so two node-a nodes would overwrite each other's status. The fix changed the value to node-b and added a
_comment_patch key in three places: the local stage
(scadaproj/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Site.json.prod-vp04), the vp04 staged copy and vp04's
installed app\appsettings.Site.json (sha 6B67CCFE…; the earlier file is kept as appsettings.Site.json.bak-20261120-nodename).
Only vp04's new ScadaBridge-Site was restarted. vp04's staged SHA256SUMS.txt is now stale for that file and does not
list sb-site.env.
Afterwards the site health report lists secretMirrorStatuses for node-a and node-b, with no failures on either.
Cosmetic leftover: _comment_Cluster in both site-app configs still describes the PAR pair. It was not edited.
Evidence
/health/readyon :8084 returns 200 on both nodes: akka-cluster Up, memberCount 2, unreachable 0, leader10.220.153.205:8082, rolesSite, site-site-app. LocalDb showsreplicationConnected=trueand backlog 0 on both./health/activeon :8084 returns 200 on vp03 and 503 on vp04 ("Standby: active node … is …205:8082"), so exactly one node is active.- vp03 logged
Replication session … faultedwarnings only while vp04 was restarting (Unavailable: This node is shutting down). Replication reconnected after the restart. - CLI
health site --identifier site-app:isOnline=true, firstSeen 22:35:29, nodeRole Active10.220.153.205. clusterNodes .205 Primary / .210 Standby, both online.localDbReplicationConnected=true, oplog backlog 0. After the deploy (report 22:39:27):dataConnectionEndpoints {MxGateway: "Connected to primary"}.dataConnectionTagQuality {}because no instance binds the connection yet. - Site log:
Sitecentral transport: gRPC to 2 central endpoint(s) for site site-app. NoUnauthenticatedorPermissionDeniedlines and no PSK errors. - ScadaBridge-Central
/health/readyreturned 200 on both boxes at every check, with vp03 active (200) and vp04 standby (503) throughout. The central log has nosvc_igntadmin, bind-failure or LDAP-exception lines from this work. The only[ERR]is at 21:24:23Z (akka-cluster … Removed), which predates this work. - site-par is unaffected. CLI
health site --identifier site-parat 22:39:48 showsisOnline=true, Active10.201.75.86, and both nodes online. - Overview on both centrals reports 10 instances: Up 7 / Down 1 (mxgw vp04, scadaproj#60) / Unreachable 2 (the PAR mxgw,
not installed). The ScadaBridge card has 3 clusters.
site-appshows leader…205:8082, 2 members, node-a (vp03) Active Up, and node-b (vp04) Standby Up. The group formerly namedsiteis nowsite-parand is still Active/Standby Up. This is a display/grouping label only.
Rollback
- On both boxes:
Stop-Service ScadaBridge-Site; sc.exe delete ScadaBridge-Site. Then removeE:\ApiInstall\ScadaBridge\app\appsettings.Site.json(plus.bak-20261120-nodenameon vp04) and theE:\ApiInstall\ScadaBridge\site\folder. Do not touchapp\otherwise, because the central uses it. - CLI:
data-connection delete --id 2, thensite delete --id 2. Optionally delete the central-store secretsb-grpc-psk-site-app. - Overview: restore
E:\ApiInstall\Overview\app\appsettings.Prod.json.bak-20261120on both boxes (and the staged copy), thenRestart-Service ZbOverview. Revert the scadaproj stage file.