diff --git a/docs/deployment/deployment-records/2026-11-20-wonder-app-vp03-vp04-par-vp03-vp04-2e0ce81e.md b/docs/deployment/deployment-records/2026-11-20-wonder-app-vp03-vp04-par-vp03-vp04-2e0ce81e.md new file mode 100644 index 00000000..42faf0d9 --- /dev/null +++ b/docs/deployment/deployment-records/2026-11-20-wonder-app-vp03-vp04-par-vp03-vp04-2e0ce81e.md @@ -0,0 +1,141 @@ +# 2026-11-20 — PROD first install: centrals wonder-app-vp03 + vp04 (installed, NOT started), site pair wonder-par-vp03 + vp04 `site-par` (installed + started), `2e0ce81e` + +**Owner go (2026-11-20):** "use winrm to bring up ... scadabridge central. then scadabridge on the site servers." Owner decision +the same day: **the centrals are installed but not started** until the DBA has created the 104 missing nonclustered indexes in the +PROD config DB (scadaproj#59). The zbauth/zbsecrets startup migrators issue `CREATE INDEX`, and the app login `wwpdsbrw` has no +DDL rights. The sites are installed and started. + +**Build:** `2e0ce81e`, ProductVersion `1.0.0+2e0ce81ea04ac5e85ddb654cf33068594ba5c3c4` (Host exe, read on all four boxes). Zip +`scadabridge-2e0ce81e.zip` SHA-256 `6C706B0A…6572E23`, checked by `install-sb.ps1` on every box. App 1056 files, cli 224. + +**Transport:** `winrm` (deploy scope) running the staged `E:\psrem-stage\sb-2e0ce81e\install-sb.ps1` (dry run first, read, then +real) and `E:\psrem-stage\prod-common\seed-secret-prod.ps1` (SHA-256 prefix `3bd6237d0bd3ac00`, same as the scadaproj copy). The +KEK moved over SMB. No firewall changes. There were no security-software alerts and no unexpected WinRM errors. + +**Not touched:** MxAccessGateway (`MxAccessGw`, `E:\ApiInstall\MxAccessGateway`, `mxgateway.kek`, `_values\mxgw-*`, +`nam-ldap-mxgw.txt`, vp04 `_values\seed`), which another session was installing at the same time. Nothing else on the boxes was +touched either. + +## Timeline (UTC) + +| Time | Step | +|---|---| +| 20:32:14 | vp03 Central dry run: clean (zip ok, config `appsettings.Central.json.prod-vp03` ok, 12 env entries); dry folder removed | +| 20:32:34–20:32:49 | vp03 Central real: expanded, **KEK generated**, service `ScadaBridge-Central` created STOPPED | +| ~20:33 | KEK pulled from vp03 over SMB (`C$\ProgramData\ZB\kek\scadabridge.kek`, 32 B) to a 0600 temp file, pushed to `E$\psrem-stage\_values\kek\` on vp04, par-vp03 and par-vp04, then the Mac copy was deleted. All three copies were 32 B with prefix `FA4511BE`. The folder ACL was inherited Administrators + SYSTEM FullControl only | +| 20:33:19 | vp04 Central dry run (`-KekFrom`): clean; dry folder removed | +| 20:33:37–20:33:51 | vp04 Central real: KEK copied from `-KekFrom`, service created STOPPED | +| 20:34:07 | vp03 seed `nam-ldap-svc-password` into the shared SQL store: **FAILED** (below) | +| 20:34:26 | par-vp03 Site dry run: clean (8 env entries; `SiteId site-par`, `NodeHostname 10.201.75.86`, seeds self-first, `PeerAddress http://10.201.75.87:8083`); dry folder removed | +| 20:34:44–20:34:55 | par-vp03 Site real: service `ScadaBridge-Site` created STOPPED | +| 20:34:57 | par-vp04 Site dry run: clean (self-first seeds `.87`, then `.86`; no `PeerAddress`, which is correct because only one node sets it); dry folder removed | +| 20:35:22–20:35:33 | par-vp04 Site real: service created STOPPED | +| 20:35:46 | `Start-Service ScadaBridge-Site` on both at once: par-vp04 Running at 20:35:48.71, par-vp03 at 20:35:48.96 | +| 20:36:02 / 20:36:27 | Site health checks (below) | +| 20:36:51 | Value cleanup (below) | + +## Service configuration (env names only) + +| | Central (vp03, vp04) | Site (par-vp03, par-vp04) | +|---|---|---| +| Service | `ScadaBridge-Central`, "ScadaBridge Central Node" | `ScadaBridge-Site`, "ScadaBridge Site Node" | +| Binary | `E:\ApiInstall\ScadaBridge\app\ZB.MOM.WW.ScadaBridge.Host.exe` | same | +| Account / start | LocalSystem / **Automatic** | LocalSystem / Automatic | +| Recovery | restart 5 s / 5 s / 10 s, reset 86 400, `failureflag 1` | same | +| Env (REG_MULTI_SZ) | 12: `SCADABRIDGE_CONFIG`, `ASPNETCORE_URLS`, `ScadaBridge__Database__ConfigurationDb`, `ScadaBridge__Security__JwtSigningKey`, `ScadaBridge__InboundApi__ApiKeyPepper`, `ScadaBridge__InboundApi__ApiKeyStore__SqlitePath`, `Secrets__SqlitePath`, `Secrets__MasterKey__Source`, `Secrets__MasterKey__FilePath`, `Secrets__SqlServer__ConnectionString`, `ScadaBridge__Telemetry__Exporter`, `ScadaBridge__Telemetry__OtlpEndpoint` | 8: `SCADABRIDGE_CONFIG`, `ScadaBridge__Communication__GrpcPsk`, `LocalDb__Replication__ApiKey`, `Secrets__SqlitePath`, `Secrets__MasterKey__Source`, `Secrets__MasterKey__FilePath`, `ScadaBridge__Telemetry__Exporter`, `ScadaBridge__Telemetry__OtlpEndpoint` | +| Role dir | `E:\ApiInstall\ScadaBridge\central\{data,logs}` | `E:\ApiInstall\ScadaBridge\site\{data,logs}` | + +**KEK:** one `C:\ProgramData\ZB\kek\scadabridge.kek` for all four nodes, generated on vp03. SHA-256 prefix **`FA4511BE`** on +vp03, vp04, par-vp03 and par-vp04 (read back from the installed file). The `C:\ProgramData\ZB\kek` ACL has inheritance removed and +Administrators + SYSTEM FullControl only. + +**Start type note:** the centrals are **Automatic**. A reboot of vp03/vp04 before scadaproj#59 closes would start them. They +would then fail on the zbauth/zbsecrets `CREATE INDEX`, and the failure actions would restart them 3 times. Set them to Manual or +Disabled if a reboot is possible before the DBA work. + +## Seeds into the shared central store (vp03): FAILED, stopped as instructed + +`seed-secret-prod.ps1 -Target ScadaBridge-Central -ValueFile E:\psrem-stage\_values\seed\nam-ldap-central.txt` (service +Stopped, SQL-aware CLI `E:\ApiInstall\_tools\secret-0.6.6-sql`): + +``` +value file read, length 15 +value file removed: True +cmd.exe : secret: startup failed: Cannot find the object "zbsecrets.secret" because it does not exist or you do not +have permissions. +At E:\psrem-stage\prod-common\seed-secret-prod.ps1:46 char:10 + + CategoryInfo : NotSpecified: (secret: startup...ve permissions.:String) [], RemoteException + + FullyQualifiedErrorId : NativeCommandError +``` + +- The CLI fails in its own startup. SQL Server returns this message (error 1088) for `CREATE INDEX`/`ALTER` on a table when + the login lacks the permission, so this is almost certainly the zbsecrets startup migrator's index DDL, which is the + scadaproj#59 gate. It is not a missing table: the 2026-11-20 schema check found `zbsecrets` present. No SQL workaround was + attempted. +- No secret was written. PowerShell 5.1 turned the CLI's stderr into a terminating `NativeCommandError` under + `$ErrorActionPreference='Stop'`, so the script never reached its readback/`SEED DONE` lines. +- **The script's `finally` block deleted `nam-ldap-central.txt` even though the seed failed.** It must be re-staged + (Keychain → 0600 → SMB) before the retry. +- **`SB-GRPC-PSK-site-par` was NOT attempted.** It would fail the same way and delete its value file. `site-psk.txt` (48 B, ACL + Administrators + SYSTEM) was **kept** in `E:\psrem-stage\_values\seed\` on vp03 for the retry. +- Script follow-up: the `finally` should delete the value file only on success. The CLI's stderr should be captured with + `$ErrorActionPreference='Continue'` around the call so the `rc`/readback line is printed. + +## Site pair `site-par`: started, healthy + +| | par-vp03 (10.201.75.86, node-a) | par-vp04 (10.201.75.87, node-b) | +|---|---|---| +| Service | Running | Running | +| `/health/ready` | **200** Healthy: akka-cluster Up, memberCount 2, unreachable 0, leader `.86`; localdb `replicationConnected=true`, backlog 0 | **200** Healthy: same; leader `.86`; `replicationConnected=true`, backlog 0 | +| `/health/active` | **200** active (oldest Up member for `site-site-par`) | **503** Standby, active = `.86` | +| `/health/live` | **404** (route not mapped on this build) | **404** | +| Roles | `Site`, `site-site-par` | same | + +The two checks at 20:36:02 and 20:36:27 gave the same results. + +**Logs** (`E:\ApiInstall\ScadaBridge\site\logs\scadabridge-site-20261120.log`): +- Akka: both started at 20:35:48–49; par-vp04 got `Welcome from [akka.tcp://scadabridge@10.201.75.86:8082]`; the leader `.86` + moved both nodes to Up (20:35:49.561 and 20:35:50.216). +- LocalDb: par-vp04 received `POST /localdb_sync.v1.LocalDbSync/Sync` from its peer at 20:35:49.430; health shows the stream + connected on both sides. +- Central gRPC: expected failures, e.g. `Central control-plane endpoint http://10.220.153.205:8083 is unavailable; site site-par + failed over to http://10.220.153.210:8083`, then `Status(StatusCode="Unavailable" ... SocketException (10061): No connection + could be made because the target machine actively refused it.)`, and `Reconcile pass 1/3 ... did not complete`. On par-vp03 + (the active node) the only `[ERR]` was `Failed to send health report for site site-par` with the same 10061. **10061 means + "refused", not a timeout.** The 10.201.75.x → 10.220.153.x path to `:8083` is open and the centrals are simply not + listening. +- Other warnings are all routine startup ones: `ClusterSingletonProxy failed to find ... [deployment-manager]/[event-log-handler]` + before Up, and `HTTP/2 is not enabled for [::]:8084` (no TLS). There were no other WRN lines and no other ERR/FTL lines + (par-vp03: 1 ERR as above; par-vp04: 0). + +The sites were not restarted. + +## Cleanup (20:36:51) + +| Box | Removed | Left | +|---|---|---| +| vp03 | `_values\sb-central` | `_values\seed\site-psk.txt` (kept for the PSK seed); `nam-ldap-central.txt` was already deleted by the seed script; `_values\mxgw-vp03` not touched | +| vp04 | `_values\sb-central`, `_values\kek` | `_values\mxgw-vp04` and `_values\seed` not touched (not this install's) | +| par-vp03 | `_values\sb-site`, `_values\kek` | — | +| par-vp04 | `_values\sb-site`, `_values\kek` | — | + +The service env in the registry now holds the only on-box copy of the central/site values. + +## State and next steps + +**Centrals installed, NOT started — gated on scadaproj#59 (DBA indexes).** Then: +1. Re-stage `nam-ldap-central.txt` on vp03 and run both seeds on vp03 (`nam-ldap-svc-password`, then + `-SecretName SB-GRPC-PSK-site-par -Description 'ScadaBridge site-par gRPC PSK (PROD)'` from the kept `site-psk.txt`). + Each must show readback-match=True and failures=0. +2. Start vp03, wait for `/health/ready` 200, then start vp04. +3. Run the §3A.5 checks with `dohertj2` on the active central. +4. Create the Site entity `site-par` (`GrpcNodeA/BAddress` `http://:8083`). +5. Mint the mxgw API key and create the MxGateway data connection. + +## Parent follow-up (2026-11-20) + +- `ScadaBridge-Central` start type set to **Manual** on vp03 and vp04 (`sc.exe config … start= demand`), so a reboot + before scadaproj#59 closes cannot crash-loop the service. Set it back to Automatic (`start= auto`) at first start. +- `seed-secret-prod.ps1` fixed on all four boxes: it now keeps the value file when a seed fails, so a retry needs no + re-staging. `nam-ldap-central.txt` must still be re-staged from the Keychain (`btdb-manager/ldap`) before the retry, + because the old copy deleted it; `site-psk.txt` is still on vp03.