From dc87289fafef8bca174ab1b5d7896f4ee7439f14 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 20 Nov 2026 17:43:47 -0500 Subject: [PATCH] docs(deploy): PROD site-app (id 2) co-located on app-vp03/vp04, MxGateway connection id 2 Connected Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ --- .../2026-11-20-prod-site-app-colocated.md | 91 +++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 docs/deployment/deployment-records/2026-11-20-prod-site-app-colocated.md diff --git a/docs/deployment/deployment-records/2026-11-20-prod-site-app-colocated.md b/docs/deployment/deployment-records/2026-11-20-prod-site-app-colocated.md new file mode 100644 index 00000000..93f59a9c --- /dev/null +++ b/docs/deployment/deployment-records/2026-11-20-prod-site-app-colocated.md @@ -0,0 +1,91 @@ +# 2026-11-20: PROD site `site-app` co-located with the centrals on wonder-app-vp03 / vp04 + +**Owner request:** "prod should have a site too, configure it for mxaccessgw on central nodes". The owner chose +SiteIdentifier `site-app` and Name `APP`. The site follows the QA pattern: a site node runs next to the central on each +box and uses the mxgw gateway already running there. + +Build: ScadaBridge `2e0ce81e` (`1.0.0+2e0ce81ea04ac5e85ddb654cf33068594ba5c3c4`). The site reuses the central's +`E:\ApiInstall\ScadaBridge\app`. Boxes: wonder-app-vp03 `10.220.153.205` (active central) and wonder-app-vp04 +`10.220.153.210` (standby central). Previous record: [`2026-11-20-prod-cluster-config.md`](2026-11-20-prod-cluster-config.md). + +Everything was driven from the Mac: `winrm` (NAM\adm_dohertj2) for the on-box deploy steps and log reads, and the +`2e0ce81e` CLI over HTTP Basic as `NAM\dohertj2` against `http://wonder-app-vp03.zmr.zimmer.com:8085` for configuration. +Nothing was done to `ScadaBridge-Central`, `MxAccessGw`, the PAR `ScadaBridge-Site` services, Site 1 or DataConnection 1. +No firewall changes. No secret went on argv or into this record. + +## Ports per box + +| Service | Akka | gRPC | HTTP | +|---|---|---|---| +| ScadaBridge-Central (unchanged) | 8081 | 8083 | 8085 (UI + health) | +| **ScadaBridge-Site (new, site-app)** | **8082** | **8086** | **8084** (health) | +| MxAccessGw / Overview / aaGR (unchanged) | | 5120 | 5130 / 5320 / 8090 | + +Site paths: config `E:\ApiInstall\ScadaBridge\app\appsettings.Site.json` (the central loads only +`appsettings.Central.json`), and data and logs under `E:\ApiInstall\ScadaBridge\site\{data,logs}`. The site is a separate +Akka cluster from the centrals, with central memberCount still 2. LocalDb replication runs vp03 → `http://10.220.153.210:8086`. + +## Steps (UTC) + +| Time | Step | Result | +|---|---|---| +| 22:33:53 | `install-sb-site-colocated.ps1 -Mode dry` on vp03 | Aborted at the env step: `sb-site.env` was missing from `E:\psrem-stage\sb-2e0ce81e` on both boxes, because it had only been staged for PAR. The non-secret local file (sha256 `186210EE…`) was pushed to both boxes and the dry run was repeated | +| 22:34:26 | dry vp03 | `DONE mode=dry`. Host ProductVersion ok, KEK `FA4511BE`, ports 8082/8084/8086 free, 8 env entries | +| 22:34:35 | real vp03 | `service ScadaBridge-Site created STOPPED, env 8 entries`, `DONE mode=real`. Central Running, `/health/ready` 200, `/health/active` 200 | +| 22:34:43–22:34:46 | dry + real on vp04 | `DONE mode=dry`, then `DONE mode=real`. Central Running, ready 200, active 503 (standby, as before) | +| 22:35:04 | Seed on vp03: `seed-secret-prod.ps1 -Target ScadaBridge-Central -SecretName SB-GRPC-PSK-site-app -AllowRunning` | Shared SQL store `zbsecrets`, KEK `FA4511BE`, `set rc=0 {"name":"sb-grpc-psk-site-app","action":"set"} readback-match=True`, value file removed, `failures=0` | +| 22:35:20 | `site create` | **Site id 2**: Name `APP`, identifier `site-app`, description "PROD site co-located on the central nodes", Akka `akka.tcp://scadabridge@10.220.153.205:8082` / `@10.220.153.210:8082`, gRPC `http://10.220.153.205:8086` / `http://10.220.153.210:8086`. Central log at 22:35:20: `gRPC channel pair for site site-app refreshed (nodeA=true, nodeB=true)` | +| 22:35:25 | Started ScadaBridge-Site on vp03 and vp04 in parallel | Both Running by 22:35:28. vp03 founded the cluster (bootstrap guard "preferred founder") and vp04 was Up at 22:35:34 | +| 22:37:14 | **PROD-STAGE PATCH** (see below): vp04 `NodeName` changed to `node-b`, and vp04's site restarted | Started again by about 22:37:28. Bootstrap guard: "partner … REACHABLE … joining peer-first". `SiteReconciliationActor … node node-b` | +| 22:38:4x | Verification | See the evidence below | +| 22:38:55 | `data-connection create --site-id 2` | **DataConnection id 2**, site 2, `MxGateway` / protocol `MxGateway`, FailoverRetryCount 3, `hasSecrets=true`. Primary `http://10.220.153.205:5120`, backup `http://10.220.153.210:5120`, config shape as for site-par (`clientName ""`, `writeUserId 0`, `useTls false`, `readTimeoutMs 5000`). apiKey = the `scadabridge-prod` token (Keychain `zb-prod/mxgw-apikey-scadabridge`), inline and encrypted at rest. Config files were 0600 in the scratchpad and deleted after use | +| 22:39:00 | `deploy artifacts --site-id 2` | Deployment `63fdb88f49134bab89fa9892bb338fcb`, `site-app` success. vp03 site log: `Deploying system artifacts` → `Applying 0 deployed secret(s)` → `Created DCL connection MxGateway (protocol=MxGateway)` → `[MxGateway] Connection established` / `Entering Connected state` (22:39:04) | +| 22:40:26 / 22:40:58 | Overview registry on vp03, then vp04 | Old `appsettings.Prod.json` (sha `3CED9601…`) backed up as `.bak-20261120`. New file sha `964EE099…` placed in `app\` and in `E:\psrem-stage\overview-818775b3\`, then `Restart-Service ZbOverview`. Running, `/health/ready` 200 | +| 22:42 | Cleanup | `E:\psrem-stage\_values\sb-site-app` deleted on both boxes, and `_values\seed\site-app-psk.txt` is gone on vp03. Temporary log script removed | + +## PROD-STAGE PATCH: vp04 NodeName + +The staged `appsettings.Site.json.prod-vp04` had `"NodeName": "node-a"`, a copy of vp03's value. The PAR pair uses +`node-a` / `node-b`. NodeName feeds the audit `SourceNode`, the per-node `secret_mirror_status` row and the +reconciliation actor's identity, so two `node-a` nodes would overwrite each other's status. The fix changed the value to `node-b` and added a +`_comment_patch` key in three places: the local stage +(`scadaproj/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Site.json.prod-vp04`), the vp04 staged copy and vp04's +installed `app\appsettings.Site.json` (sha `6B67CCFE…`; the earlier file is kept as `appsettings.Site.json.bak-20261120-nodename`). +Only vp04's new ScadaBridge-Site was restarted. vp04's staged `SHA256SUMS.txt` is now stale for that file and does not +list `sb-site.env`. + +Afterwards the site health report lists `secretMirrorStatuses` for **node-a and node-b**, with no failures on either. + +Cosmetic leftover: `_comment_Cluster` in both site-app configs still describes the PAR pair. It was not edited. + +## Evidence + +- `/health/ready` on :8084 returns 200 on both nodes: akka-cluster Up, memberCount 2, unreachable 0, leader `10.220.153.205:8082`, + roles `Site, site-site-app`. LocalDb shows `replicationConnected=true` and backlog 0 on both. +- `/health/active` on :8084 returns 200 on vp03 and 503 on vp04 ("Standby: active node … is …205:8082"), so exactly one node is active. +- vp03 logged `Replication session … faulted` warnings only while vp04 was restarting + (`Unavailable: This node is shutting down`). Replication reconnected after the restart. +- CLI `health site --identifier site-app`: `isOnline=true`, firstSeen 22:35:29, nodeRole Active `10.220.153.205`. + clusterNodes .205 Primary / .210 Standby, both online. `localDbReplicationConnected=true`, oplog backlog 0. + After the deploy (report 22:39:27): `dataConnectionEndpoints {MxGateway: "Connected to primary"}`. + `dataConnectionTagQuality {}` because no instance binds the connection yet. +- Site log: `Sitecentral transport: gRPC to 2 central endpoint(s) for site site-app`. No `Unauthenticated` or + `PermissionDenied` lines and no PSK errors. +- ScadaBridge-Central `/health/ready` returned 200 on both boxes at every check, with vp03 active (200) and vp04 standby (503) throughout. + The central log has no `svc_igntadmin`, bind-failure or LDAP-exception lines from this work. The only `[ERR]` is at + 21:24:23Z (`akka-cluster … Removed`), which predates this work. +- site-par is unaffected. CLI `health site --identifier site-par` at 22:39:48 shows `isOnline=true`, Active `10.201.75.86`, and both nodes online. +- Overview on both centrals reports 10 instances: Up 7 / Down 1 (mxgw vp04, scadaproj#60) / Unreachable 2 (the PAR mxgw, + not installed). The ScadaBridge card has 3 clusters. `site-app` shows leader `…205:8082`, 2 members, node-a (vp03) Active Up, + and node-b (vp04) Standby Up. The group formerly named `site` is now `site-par` and is still Active/Standby Up. This is a + display/grouping label only. + +## Rollback + +1. On both boxes: `Stop-Service ScadaBridge-Site; sc.exe delete ScadaBridge-Site`. Then remove + `E:\ApiInstall\ScadaBridge\app\appsettings.Site.json` (plus `.bak-20261120-nodename` on vp04) and the + `E:\ApiInstall\ScadaBridge\site\` folder. Do not touch `app\` otherwise, because the central uses it. +2. CLI: `data-connection delete --id 2`, then `site delete --id 2`. Optionally delete the central-store secret + `sb-grpc-psk-site-app`. +3. Overview: restore `E:\ApiInstall\Overview\app\appsettings.Prod.json.bak-20261120` on both boxes (and the staged copy), then + `Restart-Service ZbOverview`. Revert the scadaproj stage file.