docs(deploy): PROD 2e0ce81e installed on vp03/vp04 (stopped, gated on DB indexes) + site-par pair running on par-vp03/04
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
This commit is contained in:
+141
@@ -0,0 +1,141 @@
|
|||||||
|
# 2026-11-20 — PROD first install: centrals wonder-app-vp03 + vp04 (installed, NOT started), site pair wonder-par-vp03 + vp04 `site-par` (installed + started), `2e0ce81e`
|
||||||
|
|
||||||
|
**Owner go (2026-11-20):** "use winrm to bring up ... scadabridge central. then scadabridge on the site servers." Owner decision
|
||||||
|
the same day: **the centrals are installed but not started** until the DBA has created the 104 missing nonclustered indexes in the
|
||||||
|
PROD config DB (scadaproj#59). The zbauth/zbsecrets startup migrators issue `CREATE INDEX`, and the app login `wwpdsbrw` has no
|
||||||
|
DDL rights. The sites are installed and started.
|
||||||
|
|
||||||
|
**Build:** `2e0ce81e`, ProductVersion `1.0.0+2e0ce81ea04ac5e85ddb654cf33068594ba5c3c4` (Host exe, read on all four boxes). Zip
|
||||||
|
`scadabridge-2e0ce81e.zip` SHA-256 `6C706B0A…6572E23`, checked by `install-sb.ps1` on every box. App 1056 files, cli 224.
|
||||||
|
|
||||||
|
**Transport:** `winrm` (deploy scope) running the staged `E:\psrem-stage\sb-2e0ce81e\install-sb.ps1` (dry run first, read, then
|
||||||
|
real) and `E:\psrem-stage\prod-common\seed-secret-prod.ps1` (SHA-256 prefix `3bd6237d0bd3ac00`, same as the scadaproj copy). The
|
||||||
|
KEK moved over SMB. No firewall changes. There were no security-software alerts and no unexpected WinRM errors.
|
||||||
|
|
||||||
|
**Not touched:** MxAccessGateway (`MxAccessGw`, `E:\ApiInstall\MxAccessGateway`, `mxgateway.kek`, `_values\mxgw-*`,
|
||||||
|
`nam-ldap-mxgw.txt`, vp04 `_values\seed`), which another session was installing at the same time. Nothing else on the boxes was
|
||||||
|
touched either.
|
||||||
|
|
||||||
|
## Timeline (UTC)
|
||||||
|
|
||||||
|
| Time | Step |
|
||||||
|
|---|---|
|
||||||
|
| 20:32:14 | vp03 Central dry run: clean (zip ok, config `appsettings.Central.json.prod-vp03` ok, 12 env entries); dry folder removed |
|
||||||
|
| 20:32:34–20:32:49 | vp03 Central real: expanded, **KEK generated**, service `ScadaBridge-Central` created STOPPED |
|
||||||
|
| ~20:33 | KEK pulled from vp03 over SMB (`C$\ProgramData\ZB\kek\scadabridge.kek`, 32 B) to a 0600 temp file, pushed to `E$\psrem-stage\_values\kek\` on vp04, par-vp03 and par-vp04, then the Mac copy was deleted. All three copies were 32 B with prefix `FA4511BE`. The folder ACL was inherited Administrators + SYSTEM FullControl only |
|
||||||
|
| 20:33:19 | vp04 Central dry run (`-KekFrom`): clean; dry folder removed |
|
||||||
|
| 20:33:37–20:33:51 | vp04 Central real: KEK copied from `-KekFrom`, service created STOPPED |
|
||||||
|
| 20:34:07 | vp03 seed `nam-ldap-svc-password` into the shared SQL store: **FAILED** (below) |
|
||||||
|
| 20:34:26 | par-vp03 Site dry run: clean (8 env entries; `SiteId site-par`, `NodeHostname 10.201.75.86`, seeds self-first, `PeerAddress http://10.201.75.87:8083`); dry folder removed |
|
||||||
|
| 20:34:44–20:34:55 | par-vp03 Site real: service `ScadaBridge-Site` created STOPPED |
|
||||||
|
| 20:34:57 | par-vp04 Site dry run: clean (self-first seeds `.87`, then `.86`; no `PeerAddress`, which is correct because only one node sets it); dry folder removed |
|
||||||
|
| 20:35:22–20:35:33 | par-vp04 Site real: service created STOPPED |
|
||||||
|
| 20:35:46 | `Start-Service ScadaBridge-Site` on both at once: par-vp04 Running at 20:35:48.71, par-vp03 at 20:35:48.96 |
|
||||||
|
| 20:36:02 / 20:36:27 | Site health checks (below) |
|
||||||
|
| 20:36:51 | Value cleanup (below) |
|
||||||
|
|
||||||
|
## Service configuration (env names only)
|
||||||
|
|
||||||
|
| | Central (vp03, vp04) | Site (par-vp03, par-vp04) |
|
||||||
|
|---|---|---|
|
||||||
|
| Service | `ScadaBridge-Central`, "ScadaBridge Central Node" | `ScadaBridge-Site`, "ScadaBridge Site Node" |
|
||||||
|
| Binary | `E:\ApiInstall\ScadaBridge\app\ZB.MOM.WW.ScadaBridge.Host.exe` | same |
|
||||||
|
| Account / start | LocalSystem / **Automatic** | LocalSystem / Automatic |
|
||||||
|
| Recovery | restart 5 s / 5 s / 10 s, reset 86 400, `failureflag 1` | same |
|
||||||
|
| Env (REG_MULTI_SZ) | 12: `SCADABRIDGE_CONFIG`, `ASPNETCORE_URLS`, `ScadaBridge__Database__ConfigurationDb`, `ScadaBridge__Security__JwtSigningKey`, `ScadaBridge__InboundApi__ApiKeyPepper`, `ScadaBridge__InboundApi__ApiKeyStore__SqlitePath`, `Secrets__SqlitePath`, `Secrets__MasterKey__Source`, `Secrets__MasterKey__FilePath`, `Secrets__SqlServer__ConnectionString`, `ScadaBridge__Telemetry__Exporter`, `ScadaBridge__Telemetry__OtlpEndpoint` | 8: `SCADABRIDGE_CONFIG`, `ScadaBridge__Communication__GrpcPsk`, `LocalDb__Replication__ApiKey`, `Secrets__SqlitePath`, `Secrets__MasterKey__Source`, `Secrets__MasterKey__FilePath`, `ScadaBridge__Telemetry__Exporter`, `ScadaBridge__Telemetry__OtlpEndpoint` |
|
||||||
|
| Role dir | `E:\ApiInstall\ScadaBridge\central\{data,logs}` | `E:\ApiInstall\ScadaBridge\site\{data,logs}` |
|
||||||
|
|
||||||
|
**KEK:** one `C:\ProgramData\ZB\kek\scadabridge.kek` for all four nodes, generated on vp03. SHA-256 prefix **`FA4511BE`** on
|
||||||
|
vp03, vp04, par-vp03 and par-vp04 (read back from the installed file). The `C:\ProgramData\ZB\kek` ACL has inheritance removed and
|
||||||
|
Administrators + SYSTEM FullControl only.
|
||||||
|
|
||||||
|
**Start type note:** the centrals are **Automatic**. A reboot of vp03/vp04 before scadaproj#59 closes would start them. They
|
||||||
|
would then fail on the zbauth/zbsecrets `CREATE INDEX`, and the failure actions would restart them 3 times. Set them to Manual or
|
||||||
|
Disabled if a reboot is possible before the DBA work.
|
||||||
|
|
||||||
|
## Seeds into the shared central store (vp03): FAILED, stopped as instructed
|
||||||
|
|
||||||
|
`seed-secret-prod.ps1 -Target ScadaBridge-Central -ValueFile E:\psrem-stage\_values\seed\nam-ldap-central.txt` (service
|
||||||
|
Stopped, SQL-aware CLI `E:\ApiInstall\_tools\secret-0.6.6-sql`):
|
||||||
|
|
||||||
|
```
|
||||||
|
value file read, length 15
|
||||||
|
value file removed: True
|
||||||
|
cmd.exe : secret: startup failed: Cannot find the object "zbsecrets.secret" because it does not exist or you do not
|
||||||
|
have permissions.
|
||||||
|
At E:\psrem-stage\prod-common\seed-secret-prod.ps1:46 char:10
|
||||||
|
+ CategoryInfo : NotSpecified: (secret: startup...ve permissions.:String) [], RemoteException
|
||||||
|
+ FullyQualifiedErrorId : NativeCommandError
|
||||||
|
```
|
||||||
|
|
||||||
|
- The CLI fails in its own startup. SQL Server returns this message (error 1088) for `CREATE INDEX`/`ALTER` on a table when
|
||||||
|
the login lacks the permission, so this is almost certainly the zbsecrets startup migrator's index DDL, which is the
|
||||||
|
scadaproj#59 gate. It is not a missing table: the 2026-11-20 schema check found `zbsecrets` present. No SQL workaround was
|
||||||
|
attempted.
|
||||||
|
- No secret was written. PowerShell 5.1 turned the CLI's stderr into a terminating `NativeCommandError` under
|
||||||
|
`$ErrorActionPreference='Stop'`, so the script never reached its readback/`SEED DONE` lines.
|
||||||
|
- **The script's `finally` block deleted `nam-ldap-central.txt` even though the seed failed.** It must be re-staged
|
||||||
|
(Keychain → 0600 → SMB) before the retry.
|
||||||
|
- **`SB-GRPC-PSK-site-par` was NOT attempted.** It would fail the same way and delete its value file. `site-psk.txt` (48 B, ACL
|
||||||
|
Administrators + SYSTEM) was **kept** in `E:\psrem-stage\_values\seed\` on vp03 for the retry.
|
||||||
|
- Script follow-up: the `finally` should delete the value file only on success. The CLI's stderr should be captured with
|
||||||
|
`$ErrorActionPreference='Continue'` around the call so the `rc`/readback line is printed.
|
||||||
|
|
||||||
|
## Site pair `site-par`: started, healthy
|
||||||
|
|
||||||
|
| | par-vp03 (10.201.75.86, node-a) | par-vp04 (10.201.75.87, node-b) |
|
||||||
|
|---|---|---|
|
||||||
|
| Service | Running | Running |
|
||||||
|
| `/health/ready` | **200** Healthy: akka-cluster Up, memberCount 2, unreachable 0, leader `.86`; localdb `replicationConnected=true`, backlog 0 | **200** Healthy: same; leader `.86`; `replicationConnected=true`, backlog 0 |
|
||||||
|
| `/health/active` | **200** active (oldest Up member for `site-site-par`) | **503** Standby, active = `.86` |
|
||||||
|
| `/health/live` | **404** (route not mapped on this build) | **404** |
|
||||||
|
| Roles | `Site`, `site-site-par` | same |
|
||||||
|
|
||||||
|
The two checks at 20:36:02 and 20:36:27 gave the same results.
|
||||||
|
|
||||||
|
**Logs** (`E:\ApiInstall\ScadaBridge\site\logs\scadabridge-site-20261120.log`):
|
||||||
|
- Akka: both started at 20:35:48–49; par-vp04 got `Welcome from [akka.tcp://scadabridge@10.201.75.86:8082]`; the leader `.86`
|
||||||
|
moved both nodes to Up (20:35:49.561 and 20:35:50.216).
|
||||||
|
- LocalDb: par-vp04 received `POST /localdb_sync.v1.LocalDbSync/Sync` from its peer at 20:35:49.430; health shows the stream
|
||||||
|
connected on both sides.
|
||||||
|
- Central gRPC: expected failures, e.g. `Central control-plane endpoint http://10.220.153.205:8083 is unavailable; site site-par
|
||||||
|
failed over to http://10.220.153.210:8083`, then `Status(StatusCode="Unavailable" ... SocketException (10061): No connection
|
||||||
|
could be made because the target machine actively refused it.)`, and `Reconcile pass 1/3 ... did not complete`. On par-vp03
|
||||||
|
(the active node) the only `[ERR]` was `Failed to send health report for site site-par` with the same 10061. **10061 means
|
||||||
|
"refused", not a timeout.** The 10.201.75.x → 10.220.153.x path to `:8083` is open and the centrals are simply not
|
||||||
|
listening.
|
||||||
|
- Other warnings are all routine startup ones: `ClusterSingletonProxy failed to find ... [deployment-manager]/[event-log-handler]`
|
||||||
|
before Up, and `HTTP/2 is not enabled for [::]:8084` (no TLS). There were no other WRN lines and no other ERR/FTL lines
|
||||||
|
(par-vp03: 1 ERR as above; par-vp04: 0).
|
||||||
|
|
||||||
|
The sites were not restarted.
|
||||||
|
|
||||||
|
## Cleanup (20:36:51)
|
||||||
|
|
||||||
|
| Box | Removed | Left |
|
||||||
|
|---|---|---|
|
||||||
|
| vp03 | `_values\sb-central` | `_values\seed\site-psk.txt` (kept for the PSK seed); `nam-ldap-central.txt` was already deleted by the seed script; `_values\mxgw-vp03` not touched |
|
||||||
|
| vp04 | `_values\sb-central`, `_values\kek` | `_values\mxgw-vp04` and `_values\seed` not touched (not this install's) |
|
||||||
|
| par-vp03 | `_values\sb-site`, `_values\kek` | — |
|
||||||
|
| par-vp04 | `_values\sb-site`, `_values\kek` | — |
|
||||||
|
|
||||||
|
The service env in the registry now holds the only on-box copy of the central/site values.
|
||||||
|
|
||||||
|
## State and next steps
|
||||||
|
|
||||||
|
**Centrals installed, NOT started — gated on scadaproj#59 (DBA indexes).** Then:
|
||||||
|
1. Re-stage `nam-ldap-central.txt` on vp03 and run both seeds on vp03 (`nam-ldap-svc-password`, then
|
||||||
|
`-SecretName SB-GRPC-PSK-site-par -Description 'ScadaBridge site-par gRPC PSK (PROD)'` from the kept `site-psk.txt`).
|
||||||
|
Each must show readback-match=True and failures=0.
|
||||||
|
2. Start vp03, wait for `/health/ready` 200, then start vp04.
|
||||||
|
3. Run the §3A.5 checks with `dohertj2` on the active central.
|
||||||
|
4. Create the Site entity `site-par` (`GrpcNodeA/BAddress` `http://<PAR-VP0x>:8083`).
|
||||||
|
5. Mint the mxgw API key and create the MxGateway data connection.
|
||||||
|
|
||||||
|
## Parent follow-up (2026-11-20)
|
||||||
|
|
||||||
|
- `ScadaBridge-Central` start type set to **Manual** on vp03 and vp04 (`sc.exe config … start= demand`), so a reboot
|
||||||
|
before scadaproj#59 closes cannot crash-loop the service. Set it back to Automatic (`start= auto`) at first start.
|
||||||
|
- `seed-secret-prod.ps1` fixed on all four boxes: it now keeps the value file when a seed fails, so a retry needs no
|
||||||
|
re-staging. `nam-ldap-central.txt` must still be re-staged from the Keychain (`btdb-manager/ldap`) before the retry,
|
||||||
|
because the old copy deleted it; `site-psk.txt` is still on vp03.
|
||||||
Reference in New Issue
Block a user