Files
ScadaBridge/docs/deployment/deployment-records/2026-11-20-prod-site-app-colocated.md
T

92 lines
8.6 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 2026-11-20: PROD site `site-app` co-located with the centrals on wonder-app-vp03 / vp04
**Owner request:** "prod should have a site too, configure it for mxaccessgw on central nodes". The owner chose
SiteIdentifier `site-app` and Name `APP`. The site follows the QA pattern: a site node runs next to the central on each
box and uses the mxgw gateway already running there.
Build: ScadaBridge `2e0ce81e` (`1.0.0+2e0ce81ea04ac5e85ddb654cf33068594ba5c3c4`). The site reuses the central's
`E:\ApiInstall\ScadaBridge\app`. Boxes: wonder-app-vp03 `10.220.153.205` (active central) and wonder-app-vp04
`10.220.153.210` (standby central). Previous record: [`2026-11-20-prod-cluster-config.md`](2026-11-20-prod-cluster-config.md).
Everything was driven from the Mac: `winrm` (NAM\adm_dohertj2) for the on-box deploy steps and log reads, and the
`2e0ce81e` CLI over HTTP Basic as `NAM\dohertj2` against `http://wonder-app-vp03.zmr.zimmer.com:8085` for configuration.
Nothing was done to `ScadaBridge-Central`, `MxAccessGw`, the PAR `ScadaBridge-Site` services, Site 1 or DataConnection 1.
No firewall changes. No secret went on argv or into this record.
## Ports per box
| Service | Akka | gRPC | HTTP |
|---|---|---|---|
| ScadaBridge-Central (unchanged) | 8081 | 8083 | 8085 (UI + health) |
| **ScadaBridge-Site (new, site-app)** | **8082** | **8086** | **8084** (health) |
| MxAccessGw / Overview / aaGR (unchanged) | | 5120 | 5130 / 5320 / 8090 |
Site paths: config `E:\ApiInstall\ScadaBridge\app\appsettings.Site.json` (the central loads only
`appsettings.Central.json`), and data and logs under `E:\ApiInstall\ScadaBridge\site\{data,logs}`. The site is a separate
Akka cluster from the centrals, with central memberCount still 2. LocalDb replication runs vp03 → `http://10.220.153.210:8086`.
## Steps (UTC)
| Time | Step | Result |
|---|---|---|
| 22:33:53 | `install-sb-site-colocated.ps1 -Mode dry` on vp03 | Aborted at the env step: `sb-site.env` was missing from `E:\psrem-stage\sb-2e0ce81e` on both boxes, because it had only been staged for PAR. The non-secret local file (sha256 `186210EE…`) was pushed to both boxes and the dry run was repeated |
| 22:34:26 | dry vp03 | `DONE mode=dry`. Host ProductVersion ok, KEK `FA4511BE`, ports 8082/8084/8086 free, 8 env entries |
| 22:34:35 | real vp03 | `service ScadaBridge-Site created STOPPED, env 8 entries`, `DONE mode=real`. Central Running, `/health/ready` 200, `/health/active` 200 |
| 22:34:43–22:34:46 | dry + real on vp04 | `DONE mode=dry`, then `DONE mode=real`. Central Running, ready 200, active 503 (standby, as before) |
| 22:35:04 | Seed on vp03: `seed-secret-prod.ps1 -Target ScadaBridge-Central -SecretName SB-GRPC-PSK-site-app -AllowRunning` | Shared SQL store `zbsecrets`, KEK `FA4511BE`, `set rc=0 {"name":"sb-grpc-psk-site-app","action":"set"} readback-match=True`, value file removed, `failures=0` |
| 22:35:20 | `site create` | **Site id 2**: Name `APP`, identifier `site-app`, description "PROD site co-located on the central nodes", Akka `akka.tcp://scadabridge@10.220.153.205:8082` / `@10.220.153.210:8082`, gRPC `http://10.220.153.205:8086` / `http://10.220.153.210:8086`. Central log at 22:35:20: `gRPC channel pair for site site-app refreshed (nodeA=true, nodeB=true)` |
| 22:35:25 | Started ScadaBridge-Site on vp03 and vp04 in parallel | Both Running by 22:35:28. vp03 founded the cluster (bootstrap guard "preferred founder") and vp04 was Up at 22:35:34 |
| 22:37:14 | **PROD-STAGE PATCH** (see below): vp04 `NodeName` changed to `node-b`, and vp04's site restarted | Started again by about 22:37:28. Bootstrap guard: "partner … REACHABLE … joining peer-first". `SiteReconciliationActor … node node-b` |
| 22:38:4x | Verification | See the evidence below |
| 22:38:55 | `data-connection create --site-id 2` | **DataConnection id 2**, site 2, `MxGateway` / protocol `MxGateway`, FailoverRetryCount 3, `hasSecrets=true`. Primary `http://10.220.153.205:5120`, backup `http://10.220.153.210:5120`, config shape as for site-par (`clientName ""`, `writeUserId 0`, `useTls false`, `readTimeoutMs 5000`). apiKey = the `scadabridge-prod` token (Keychain `zb-prod/mxgw-apikey-scadabridge`), inline and encrypted at rest. Config files were 0600 in the scratchpad and deleted after use |
| 22:39:00 | `deploy artifacts --site-id 2` | Deployment `63fdb88f49134bab89fa9892bb338fcb`, `site-app` success. vp03 site log: `Deploying system artifacts` → `Applying 0 deployed secret(s)` → `Created DCL connection MxGateway (protocol=MxGateway)` → `[MxGateway] Connection established` / `Entering Connected state` (22:39:04) |
| 22:40:26 / 22:40:58 | Overview registry on vp03, then vp04 | Old `appsettings.Prod.json` (sha `3CED9601…`) backed up as `.bak-20261120`. New file sha `964EE099…` placed in `app\` and in `E:\psrem-stage\overview-818775b3\`, then `Restart-Service ZbOverview`. Running, `/health/ready` 200 |
| 22:42 | Cleanup | `E:\psrem-stage\_values\sb-site-app` deleted on both boxes, and `_values\seed\site-app-psk.txt` is gone on vp03. Temporary log script removed |
## PROD-STAGE PATCH: vp04 NodeName
The staged `appsettings.Site.json.prod-vp04` had `"NodeName": "node-a"`, a copy of vp03's value. The PAR pair uses
`node-a` / `node-b`. NodeName feeds the audit `SourceNode`, the per-node `secret_mirror_status` row and the
reconciliation actor's identity, so two `node-a` nodes would overwrite each other's status. The fix changed the value to `node-b` and added a
`_comment_patch` key in three places: the local stage
(`scadaproj/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Site.json.prod-vp04`), the vp04 staged copy and vp04's
installed `app\appsettings.Site.json` (sha `6B67CCFE…`; the earlier file is kept as `appsettings.Site.json.bak-20261120-nodename`).
Only vp04's new ScadaBridge-Site was restarted. vp04's staged `SHA256SUMS.txt` is now stale for that file and does not
list `sb-site.env`.
Afterwards the site health report lists `secretMirrorStatuses` for **node-a and node-b**, with no failures on either.
Cosmetic leftover: `_comment_Cluster` in both site-app configs still describes the PAR pair. It was not edited.
## Evidence
- `/health/ready` on :8084 returns 200 on both nodes: akka-cluster Up, memberCount 2, unreachable 0, leader `10.220.153.205:8082`,
roles `Site, site-site-app`. LocalDb shows `replicationConnected=true` and backlog 0 on both.
- `/health/active` on :8084 returns 200 on vp03 and 503 on vp04 ("Standby: active node … is …205:8082"), so exactly one node is active.
- vp03 logged `Replication session … faulted` warnings only while vp04 was restarting
(`Unavailable: This node is shutting down`). Replication reconnected after the restart.
- CLI `health site --identifier site-app`: `isOnline=true`, firstSeen 22:35:29, nodeRole Active `10.220.153.205`.
clusterNodes .205 Primary / .210 Standby, both online. `localDbReplicationConnected=true`, oplog backlog 0.
After the deploy (report 22:39:27): `dataConnectionEndpoints {MxGateway: "Connected to primary"}`.
`dataConnectionTagQuality {}` because no instance binds the connection yet.
- Site log: `Sitecentral transport: gRPC to 2 central endpoint(s) for site site-app`. No `Unauthenticated` or
`PermissionDenied` lines and no PSK errors.
- ScadaBridge-Central `/health/ready` returned 200 on both boxes at every check, with vp03 active (200) and vp04 standby (503) throughout.
The central log has no `svc_igntadmin`, bind-failure or LDAP-exception lines from this work. The only `[ERR]` is at
21:24:23Z (`akka-cluster … Removed`), which predates this work.
- site-par is unaffected. CLI `health site --identifier site-par` at 22:39:48 shows `isOnline=true`, Active `10.201.75.86`, and both nodes online.
- Overview on both centrals reports 10 instances: Up 7 / Down 1 (mxgw vp04, scadaproj#60) / Unreachable 2 (the PAR mxgw,
not installed). The ScadaBridge card has 3 clusters. `site-app` shows leader `…205:8082`, 2 members, node-a (vp03) Active Up,
and node-b (vp04) Standby Up. The group formerly named `site` is now `site-par` and is still Active/Standby Up. This is a
display/grouping label only.
## Rollback
1. On both boxes: `Stop-Service ScadaBridge-Site; sc.exe delete ScadaBridge-Site`. Then remove
`E:\ApiInstall\ScadaBridge\app\appsettings.Site.json` (plus `.bak-20261120-nodename` on vp04) and the
`E:\ApiInstall\ScadaBridge\site\` folder. Do not touch `app\` otherwise, because the central uses it.
2. CLI: `data-connection delete --id 2`, then `site delete --id 2`. Optionally delete the central-store secret
`sb-grpc-psk-site-app`.
3. Overview: restore `E:\ApiInstall\Overview\app\appsettings.Prod.json.bak-20261120` on both boxes (and the staged copy), then
`Restart-Service ZbOverview`. Revert the scadaproj stage file.