dc87289faf
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
92 lines
8.6 KiB
Markdown
92 lines
8.6 KiB
Markdown
# 2026-11-20: PROD site `site-app` co-located with the centrals on wonder-app-vp03 / vp04
|
||
|
||
**Owner request:** "prod should have a site too, configure it for mxaccessgw on central nodes". The owner chose
|
||
SiteIdentifier `site-app` and Name `APP`. The site follows the QA pattern: a site node runs next to the central on each
|
||
box and uses the mxgw gateway already running there.
|
||
|
||
Build: ScadaBridge `2e0ce81e` (`1.0.0+2e0ce81ea04ac5e85ddb654cf33068594ba5c3c4`). The site reuses the central's
|
||
`E:\ApiInstall\ScadaBridge\app`. Boxes: wonder-app-vp03 `10.220.153.205` (active central) and wonder-app-vp04
|
||
`10.220.153.210` (standby central). Previous record: [`2026-11-20-prod-cluster-config.md`](2026-11-20-prod-cluster-config.md).
|
||
|
||
Everything was driven from the Mac: `winrm` (NAM\adm_dohertj2) for the on-box deploy steps and log reads, and the
|
||
`2e0ce81e` CLI over HTTP Basic as `NAM\dohertj2` against `http://wonder-app-vp03.zmr.zimmer.com:8085` for configuration.
|
||
Nothing was done to `ScadaBridge-Central`, `MxAccessGw`, the PAR `ScadaBridge-Site` services, Site 1 or DataConnection 1.
|
||
No firewall changes. No secret went on argv or into this record.
|
||
|
||
## Ports per box
|
||
|
||
| Service | Akka | gRPC | HTTP |
|
||
|---|---|---|---|
|
||
| ScadaBridge-Central (unchanged) | 8081 | 8083 | 8085 (UI + health) |
|
||
| **ScadaBridge-Site (new, site-app)** | **8082** | **8086** | **8084** (health) |
|
||
| MxAccessGw / Overview / aaGR (unchanged) | | 5120 | 5130 / 5320 / 8090 |
|
||
|
||
Site paths: config `E:\ApiInstall\ScadaBridge\app\appsettings.Site.json` (the central loads only
|
||
`appsettings.Central.json`), and data and logs under `E:\ApiInstall\ScadaBridge\site\{data,logs}`. The site is a separate
|
||
Akka cluster from the centrals, with central memberCount still 2. LocalDb replication runs vp03 → `http://10.220.153.210:8086`.
|
||
|
||
## Steps (UTC)
|
||
|
||
| Time | Step | Result |
|
||
|---|---|---|
|
||
| 22:33:53 | `install-sb-site-colocated.ps1 -Mode dry` on vp03 | Aborted at the env step: `sb-site.env` was missing from `E:\psrem-stage\sb-2e0ce81e` on both boxes, because it had only been staged for PAR. The non-secret local file (sha256 `186210EE…`) was pushed to both boxes and the dry run was repeated |
|
||
| 22:34:26 | dry vp03 | `DONE mode=dry`. Host ProductVersion ok, KEK `FA4511BE`, ports 8082/8084/8086 free, 8 env entries |
|
||
| 22:34:35 | real vp03 | `service ScadaBridge-Site created STOPPED, env 8 entries`, `DONE mode=real`. Central Running, `/health/ready` 200, `/health/active` 200 |
|
||
| 22:34:43–22:34:46 | dry + real on vp04 | `DONE mode=dry`, then `DONE mode=real`. Central Running, ready 200, active 503 (standby, as before) |
|
||
| 22:35:04 | Seed on vp03: `seed-secret-prod.ps1 -Target ScadaBridge-Central -SecretName SB-GRPC-PSK-site-app -AllowRunning` | Shared SQL store `zbsecrets`, KEK `FA4511BE`, `set rc=0 {"name":"sb-grpc-psk-site-app","action":"set"} readback-match=True`, value file removed, `failures=0` |
|
||
| 22:35:20 | `site create` | **Site id 2**: Name `APP`, identifier `site-app`, description "PROD site co-located on the central nodes", Akka `akka.tcp://scadabridge@10.220.153.205:8082` / `@10.220.153.210:8082`, gRPC `http://10.220.153.205:8086` / `http://10.220.153.210:8086`. Central log at 22:35:20: `gRPC channel pair for site site-app refreshed (nodeA=true, nodeB=true)` |
|
||
| 22:35:25 | Started ScadaBridge-Site on vp03 and vp04 in parallel | Both Running by 22:35:28. vp03 founded the cluster (bootstrap guard "preferred founder") and vp04 was Up at 22:35:34 |
|
||
| 22:37:14 | **PROD-STAGE PATCH** (see below): vp04 `NodeName` changed to `node-b`, and vp04's site restarted | Started again by about 22:37:28. Bootstrap guard: "partner … REACHABLE … joining peer-first". `SiteReconciliationActor … node node-b` |
|
||
| 22:38:4x | Verification | See the evidence below |
|
||
| 22:38:55 | `data-connection create --site-id 2` | **DataConnection id 2**, site 2, `MxGateway` / protocol `MxGateway`, FailoverRetryCount 3, `hasSecrets=true`. Primary `http://10.220.153.205:5120`, backup `http://10.220.153.210:5120`, config shape as for site-par (`clientName ""`, `writeUserId 0`, `useTls false`, `readTimeoutMs 5000`). apiKey = the `scadabridge-prod` token (Keychain `zb-prod/mxgw-apikey-scadabridge`), inline and encrypted at rest. Config files were 0600 in the scratchpad and deleted after use |
|
||
| 22:39:00 | `deploy artifacts --site-id 2` | Deployment `63fdb88f49134bab89fa9892bb338fcb`, `site-app` success. vp03 site log: `Deploying system artifacts` → `Applying 0 deployed secret(s)` → `Created DCL connection MxGateway (protocol=MxGateway)` → `[MxGateway] Connection established` / `Entering Connected state` (22:39:04) |
|
||
| 22:40:26 / 22:40:58 | Overview registry on vp03, then vp04 | Old `appsettings.Prod.json` (sha `3CED9601…`) backed up as `.bak-20261120`. New file sha `964EE099…` placed in `app\` and in `E:\psrem-stage\overview-818775b3\`, then `Restart-Service ZbOverview`. Running, `/health/ready` 200 |
|
||
| 22:42 | Cleanup | `E:\psrem-stage\_values\sb-site-app` deleted on both boxes, and `_values\seed\site-app-psk.txt` is gone on vp03. Temporary log script removed |
|
||
|
||
## PROD-STAGE PATCH: vp04 NodeName
|
||
|
||
The staged `appsettings.Site.json.prod-vp04` had `"NodeName": "node-a"`, a copy of vp03's value. The PAR pair uses
|
||
`node-a` / `node-b`. NodeName feeds the audit `SourceNode`, the per-node `secret_mirror_status` row and the
|
||
reconciliation actor's identity, so two `node-a` nodes would overwrite each other's status. The fix changed the value to `node-b` and added a
|
||
`_comment_patch` key in three places: the local stage
|
||
(`scadaproj/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Site.json.prod-vp04`), the vp04 staged copy and vp04's
|
||
installed `app\appsettings.Site.json` (sha `6B67CCFE…`; the earlier file is kept as `appsettings.Site.json.bak-20261120-nodename`).
|
||
Only vp04's new ScadaBridge-Site was restarted. vp04's staged `SHA256SUMS.txt` is now stale for that file and does not
|
||
list `sb-site.env`.
|
||
|
||
Afterwards the site health report lists `secretMirrorStatuses` for **node-a and node-b**, with no failures on either.
|
||
|
||
Cosmetic leftover: `_comment_Cluster` in both site-app configs still describes the PAR pair. It was not edited.
|
||
|
||
## Evidence
|
||
|
||
- `/health/ready` on :8084 returns 200 on both nodes: akka-cluster Up, memberCount 2, unreachable 0, leader `10.220.153.205:8082`,
|
||
roles `Site, site-site-app`. LocalDb shows `replicationConnected=true` and backlog 0 on both.
|
||
- `/health/active` on :8084 returns 200 on vp03 and 503 on vp04 ("Standby: active node … is …205:8082"), so exactly one node is active.
|
||
- vp03 logged `Replication session … faulted` warnings only while vp04 was restarting
|
||
(`Unavailable: This node is shutting down`). Replication reconnected after the restart.
|
||
- CLI `health site --identifier site-app`: `isOnline=true`, firstSeen 22:35:29, nodeRole Active `10.220.153.205`.
|
||
clusterNodes .205 Primary / .210 Standby, both online. `localDbReplicationConnected=true`, oplog backlog 0.
|
||
After the deploy (report 22:39:27): `dataConnectionEndpoints {MxGateway: "Connected to primary"}`.
|
||
`dataConnectionTagQuality {}` because no instance binds the connection yet.
|
||
- Site log: `Sitecentral transport: gRPC to 2 central endpoint(s) for site site-app`. No `Unauthenticated` or
|
||
`PermissionDenied` lines and no PSK errors.
|
||
- ScadaBridge-Central `/health/ready` returned 200 on both boxes at every check, with vp03 active (200) and vp04 standby (503) throughout.
|
||
The central log has no `svc_igntadmin`, bind-failure or LDAP-exception lines from this work. The only `[ERR]` is at
|
||
21:24:23Z (`akka-cluster … Removed`), which predates this work.
|
||
- site-par is unaffected. CLI `health site --identifier site-par` at 22:39:48 shows `isOnline=true`, Active `10.201.75.86`, and both nodes online.
|
||
- Overview on both centrals reports 10 instances: Up 7 / Down 1 (mxgw vp04, scadaproj#60) / Unreachable 2 (the PAR mxgw,
|
||
not installed). The ScadaBridge card has 3 clusters. `site-app` shows leader `…205:8082`, 2 members, node-a (vp03) Active Up,
|
||
and node-b (vp04) Standby Up. The group formerly named `site` is now `site-par` and is still Active/Standby Up. This is a
|
||
display/grouping label only.
|
||
|
||
## Rollback
|
||
|
||
1. On both boxes: `Stop-Service ScadaBridge-Site; sc.exe delete ScadaBridge-Site`. Then remove
|
||
`E:\ApiInstall\ScadaBridge\app\appsettings.Site.json` (plus `.bak-20261120-nodename` on vp04) and the
|
||
`E:\ApiInstall\ScadaBridge\site\` folder. Do not touch `app\` otherwise, because the central uses it.
|
||
2. CLI: `data-connection delete --id 2`, then `site delete --id 2`. Optionally delete the central-store secret
|
||
`sb-grpc-psk-site-app`.
|
||
3. Overview: restore `E:\ApiInstall\Overview\app\appsettings.Prod.json.bak-20261120` on both boxes (and the staged copy), then
|
||
`Restart-Service ZbOverview`. Revert the scadaproj stage file.
|