Files
ScadaBridge/docs/deployment/deployment-records/2026-11-20-prod-site-app-colocated.md
T

8.6 KiB
Raw Blame History

2026-11-20: PROD site site-app co-located with the centrals on wonder-app-vp03 / vp04

Owner request: "prod should have a site too, configure it for mxaccessgw on central nodes". The owner chose SiteIdentifier site-app and Name APP. The site follows the QA pattern: a site node runs next to the central on each box and uses the mxgw gateway already running there.

Build: ScadaBridge 2e0ce81e (1.0.0+2e0ce81ea04ac5e85ddb654cf33068594ba5c3c4). The site reuses the central's E:\ApiInstall\ScadaBridge\app. Boxes: wonder-app-vp03 10.220.153.205 (active central) and wonder-app-vp04 10.220.153.210 (standby central). Previous record: 2026-11-20-prod-cluster-config.md.

Everything was driven from the Mac: winrm (NAM\adm_dohertj2) for the on-box deploy steps and log reads, and the 2e0ce81e CLI over HTTP Basic as NAM\dohertj2 against http://wonder-app-vp03.zmr.zimmer.com:8085 for configuration. Nothing was done to ScadaBridge-Central, MxAccessGw, the PAR ScadaBridge-Site services, Site 1 or DataConnection 1. No firewall changes. No secret went on argv or into this record.

Ports per box

Service Akka gRPC HTTP
ScadaBridge-Central (unchanged) 8081 8083 8085 (UI + health)
ScadaBridge-Site (new, site-app) 8082 8086 8084 (health)
MxAccessGw / Overview / aaGR (unchanged) 5120 5130 / 5320 / 8090

Site paths: config E:\ApiInstall\ScadaBridge\app\appsettings.Site.json (the central loads only appsettings.Central.json), and data and logs under E:\ApiInstall\ScadaBridge\site\{data,logs}. The site is a separate Akka cluster from the centrals, with central memberCount still 2. LocalDb replication runs vp03 → http://10.220.153.210:8086.

Steps (UTC)

Time Step Result
22:33:53 install-sb-site-colocated.ps1 -Mode dry on vp03 Aborted at the env step: sb-site.env was missing from E:\psrem-stage\sb-2e0ce81e on both boxes, because it had only been staged for PAR. The non-secret local file (sha256 186210EE…) was pushed to both boxes and the dry run was repeated
22:34:26 dry vp03 DONE mode=dry. Host ProductVersion ok, KEK FA4511BE, ports 8082/8084/8086 free, 8 env entries
22:34:35 real vp03 service ScadaBridge-Site created STOPPED, env 8 entries, DONE mode=real. Central Running, /health/ready 200, /health/active 200
22:34:43–22:34:46 dry + real on vp04 DONE mode=dry, then DONE mode=real. Central Running, ready 200, active 503 (standby, as before)
22:35:04 Seed on vp03: seed-secret-prod.ps1 -Target ScadaBridge-Central -SecretName SB-GRPC-PSK-site-app -AllowRunning Shared SQL store zbsecrets, KEK FA4511BE, set rc=0 {"name":"sb-grpc-psk-site-app","action":"set"} readback-match=True, value file removed, failures=0
22:35:20 site create Site id 2: Name APP, identifier site-app, description "PROD site co-located on the central nodes", Akka akka.tcp://scadabridge@10.220.153.205:8082 / @10.220.153.210:8082, gRPC http://10.220.153.205:8086 / http://10.220.153.210:8086. Central log at 22:35:20: gRPC channel pair for site site-app refreshed (nodeA=true, nodeB=true)
22:35:25 Started ScadaBridge-Site on vp03 and vp04 in parallel Both Running by 22:35:28. vp03 founded the cluster (bootstrap guard "preferred founder") and vp04 was Up at 22:35:34
22:37:14 PROD-STAGE PATCH (see below): vp04 NodeName changed to node-b, and vp04's site restarted Started again by about 22:37:28. Bootstrap guard: "partner … REACHABLE … joining peer-first". SiteReconciliationActor … node node-b
22:38:4x Verification See the evidence below
22:38:55 data-connection create --site-id 2 DataConnection id 2, site 2, MxGateway / protocol MxGateway, FailoverRetryCount 3, hasSecrets=true. Primary http://10.220.153.205:5120, backup http://10.220.153.210:5120, config shape as for site-par (clientName "", writeUserId 0, useTls false, readTimeoutMs 5000). apiKey = the scadabridge-prod token (Keychain zb-prod/mxgw-apikey-scadabridge), inline and encrypted at rest. Config files were 0600 in the scratchpad and deleted after use
22:39:00 deploy artifacts --site-id 2 Deployment 63fdb88f49134bab89fa9892bb338fcb, site-app success. vp03 site log: Deploying system artifacts → Applying 0 deployed secret(s) → Created DCL connection MxGateway (protocol=MxGateway) → [MxGateway] Connection established / Entering Connected state (22:39:04)
22:40:26 / 22:40:58 Overview registry on vp03, then vp04 Old appsettings.Prod.json (sha 3CED9601…) backed up as .bak-20261120. New file sha 964EE099… placed in app\ and in E:\psrem-stage\overview-818775b3\, then Restart-Service ZbOverview. Running, /health/ready 200
22:42 Cleanup E:\psrem-stage\_values\sb-site-app deleted on both boxes, and _values\seed\site-app-psk.txt is gone on vp03. Temporary log script removed

PROD-STAGE PATCH: vp04 NodeName

The staged appsettings.Site.json.prod-vp04 had "NodeName": "node-a", a copy of vp03's value. The PAR pair uses node-a / node-b. NodeName feeds the audit SourceNode, the per-node secret_mirror_status row and the reconciliation actor's identity, so two node-a nodes would overwrite each other's status. The fix changed the value to node-b and added a _comment_patch key in three places: the local stage (scadaproj/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Site.json.prod-vp04), the vp04 staged copy and vp04's installed app\appsettings.Site.json (sha 6B67CCFE…; the earlier file is kept as appsettings.Site.json.bak-20261120-nodename). Only vp04's new ScadaBridge-Site was restarted. vp04's staged SHA256SUMS.txt is now stale for that file and does not list sb-site.env.

Afterwards the site health report lists secretMirrorStatuses for node-a and node-b, with no failures on either.

Cosmetic leftover: _comment_Cluster in both site-app configs still describes the PAR pair. It was not edited.

Evidence

  • /health/ready on :8084 returns 200 on both nodes: akka-cluster Up, memberCount 2, unreachable 0, leader 10.220.153.205:8082, roles Site, site-site-app. LocalDb shows replicationConnected=true and backlog 0 on both.
  • /health/active on :8084 returns 200 on vp03 and 503 on vp04 ("Standby: active node … is …205:8082"), so exactly one node is active.
  • vp03 logged Replication session … faulted warnings only while vp04 was restarting (Unavailable: This node is shutting down). Replication reconnected after the restart.
  • CLI health site --identifier site-app: isOnline=true, firstSeen 22:35:29, nodeRole Active 10.220.153.205. clusterNodes .205 Primary / .210 Standby, both online. localDbReplicationConnected=true, oplog backlog 0. After the deploy (report 22:39:27): dataConnectionEndpoints {MxGateway: "Connected to primary"}. dataConnectionTagQuality {} because no instance binds the connection yet.
  • Site log: Sitecentral transport: gRPC to 2 central endpoint(s) for site site-app. No Unauthenticated or PermissionDenied lines and no PSK errors.
  • ScadaBridge-Central /health/ready returned 200 on both boxes at every check, with vp03 active (200) and vp04 standby (503) throughout. The central log has no svc_igntadmin, bind-failure or LDAP-exception lines from this work. The only [ERR] is at 21:24:23Z (akka-cluster … Removed), which predates this work.
  • site-par is unaffected. CLI health site --identifier site-par at 22:39:48 shows isOnline=true, Active 10.201.75.86, and both nodes online.
  • Overview on both centrals reports 10 instances: Up 7 / Down 1 (mxgw vp04, scadaproj#60) / Unreachable 2 (the PAR mxgw, not installed). The ScadaBridge card has 3 clusters. site-app shows leader …205:8082, 2 members, node-a (vp03) Active Up, and node-b (vp04) Standby Up. The group formerly named site is now site-par and is still Active/Standby Up. This is a display/grouping label only.

Rollback

  1. On both boxes: Stop-Service ScadaBridge-Site; sc.exe delete ScadaBridge-Site. Then remove E:\ApiInstall\ScadaBridge\app\appsettings.Site.json (plus .bak-20261120-nodename on vp04) and the E:\ApiInstall\ScadaBridge\site\ folder. Do not touch app\ otherwise, because the central uses it.
  2. CLI: data-connection delete --id 2, then site delete --id 2. Optionally delete the central-store secret sb-grpc-psk-site-app.
  3. Overview: restore E:\ApiInstall\Overview\app\appsettings.Prod.json.bak-20261120 on both boxes (and the staged copy), then Restart-Service ZbOverview. Revert the scadaproj stage file.