Joseph Doherty
17864d1ae4
Both driven from the Mac over the winrm CLI (WinRM 5985 + SMB 445, no helper on the boxes). Migration 53->55; the first Central boot encrypts DataConnections.PrimaryConfiguration (dp:protocol:v1:), so rollback is the COPY_ONLY DB restore, not a binary swap. QA: all four nodes stopped together, one shared-DB migration, 2 min 35 s window, preflight against a restored clone. Claude-Session: https://claude.ai/code/session_01RgthVLVU5CJ8SUuEwruTkt
141 lines
12 KiB
Markdown
141 lines
12 KiB
Markdown
# Deployment record — wonder-app-vq03 + wonder-app-vq04 (QA) — 2026-09-10 — `afb27e77` → `2b9a7c3c`
|
||
|
||
Point-in-time record of a ScadaBridge binary + schema upgrade of the **QA two-host topology**:
|
||
a real central pair (`central-a` = wonder-app-vq03 `10.220.158.2`, `central-b` = wonder-app-vq04
|
||
`10.220.158.3`, :8085, Akka 8081, CentralGrpcPort 8086) and a site-a pair (`node-a`/`node-b` on
|
||
the same two boxes, gRPC :8083, health :8084, Akka 8082, LocalDb replication ON), each box running
|
||
`ScadaBridge-Central`, `ScadaBridge-Site` and `ScadaBridge-LDAP` (native services, env in the
|
||
service key) out of `E:\ApiInstall\ScadaBridge`. Shared ConfigDb `ScadaBridge` on
|
||
`wonder-sql-vq03` (both centrals; DataProtection key ring lives in the same DB). Performed from the
|
||
Mac entirely via the **`winrm` CLI** (WinRM 5985/NTLM for commands; the zip had been pushed over SMB
|
||
beforehand) — no psbridge, no servecli, nothing installed on the boxes. Same build as the DEV record
|
||
`2026-09-10-wonder-app-vd03-2b9a7c3c.md` (read that for the build/delta description).
|
||
|
||
## 1. Binary + schema upgrade
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Build deployed | `main` @ **`2b9a7c3c`** (from `afb27e77`, 2026-08-21; 253 commits) — all four nodes. ProductVersion `1.0.0+2b9a7c3ca05fde8fe58f647d8f91e5ab9452807b` on Host, CLI and all 25 `ZB.MOM.WW.ScadaBridge.*.dll` on both boxes. |
|
||
| Headline change | Arch-review remediation (13 High + 33 Medium + 46 Low), management API / CLI parity (CLI gains `shared-schema` + `flow-debug`), Flow Debugger v1.4, **`DataConnections` protocol config encrypted at rest** (`dp:protocol:v1:`). |
|
||
| Artifact | `E:\psrem-stage\2b9a7c3c\scadabridge.zip` on both boxes, 125 543 056 B, SHA-256 `371B3271A57EC1BA…` re-verified on-box; unpacked to `E:\psrem-stage\2b9a7c3c\unpacked` (app 1029 files, cli 223); staged exe ProductVersion checked before the swap. |
|
||
| EF migrations applied | 2 (53 → 55) from the from→to script `sb-mig-53to55.sql` (`20260819201210` → `20260909203046`, NOT idempotent, identical to vd03's) via `sqlcmd -C -b -I` on vq03: `20260906034113_AddApiMethodTraceAllCalls`, `20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration` (THROW 51002 guard passed — 0 duplicate `UniqueName`; `IX_Instances_SiteId_UniqueName` dropped, `IX_Instances_SiteId` + unique `IX_Instances_UniqueName` created; `Primary/BackupConfiguration` widened to `nvarchar(max)`). |
|
||
| Coordinated data upgrade | The first Central boot on the new binary (vq03, 12:19:43) ran `DataConnectionSecretUpgrade`: `DataConnections` id 1 (`MxGateway`) `PrimaryConfiguration` 210 chars plaintext → **426 chars `dp:protocol:v1:`**, `BackupConfiguration` NULL, `DataProtectionKeys` unchanged at 2. **`afb27e77` cannot read the protected row and the migration `Down` throws over it — rollback is the DB restore, not a binary swap.** |
|
||
| New required config | **None.** The box's three `appsettings*.json` were kept, restored after the swap and SHA-256-matched on both boxes (vq03: `F94000B1…`/`92524782…`/`4CBAF66C…`; vq04: `F94000B1…`/`2FD3EC4C…`/`D0C1AF05…`). Service registry environment untouched (`reg export` of both service keys to `E:\psrem-stage\2b9a7c3c\backup\`). `ApiKeyStore:Provider` deliberately still UNSET (per-node SQLite) — separate planned cutover; the boot `[WRN] Inbound API keys use the per-node SQLite store on a 2-node central cluster` is real on QA (unlike vd03) and is the reason for that cutover. |
|
||
| DB backup | `F:\MSSQL\BACKUP\ScadaBridge_pre-2b9a7c3c_20260910T121009.bak` on wonder-sql-vq03 (COPY_ONLY, CHECKSUM, COMPRESSION; `RESTORE VERIFYONLY … WITH CHECKSUM` valid; 688 MB logical / 113 MB compressed), taken 12:11:41 with everything still running. |
|
||
| Binary backup | `E:\ApiInstall\ScadaBridge\app.bak.20260910T121009` + `cli.bak.20260910T121009` on each box (old `afb27e77` binaries + appsettings); `_appsettings-keep-20260910T121009` copies alongside. |
|
||
| Notes on box | `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-20260910T121009.txt` on each box (baseline, preflight, window, verify, rollback). |
|
||
|
||
### Preflight (vq04, before the window, 12:14–12:17)
|
||
|
||
The staged Host was booted as an **isolated single-node Central** on vq04 (`:18085`, Akka `18081`,
|
||
seeds = self + a dead `18087`, `BootstrapGuard` off, `CentralGrpcPort` 18086, own copies of the
|
||
inbound-api-key store and `secrets.db`, real service env otherwise) against a **clone**
|
||
`ScadaBridge_pf2b9a7c3c` restored from the pre-upgrade backup (`WITH MOVE`) and migrated 53 → 55
|
||
with the same script. Result: `/health/ready` 200 in 15 s (memberCount 1, 5/5 singletons),
|
||
`POST /management` no-auth 401, clone row encrypted 210 → 426 `dp:protocol:v1:`, `DataProtectionKeys`
|
||
2, live central cluster stayed at memberCount 2 / unreachable 0 throughout. Clone dropped and
|
||
`pf-app` removed afterwards (`pf-data\` logs left on vq04).
|
||
|
||
**Near miss, recorded for next time:** attempt 1 set the clone catalog with
|
||
`$csb.InitialCatalog = …` on a `SqlConnectionStringBuilder`, which PowerShell 5.1 rejects
|
||
(`Keyword not supported: 'InitialCatalog'`) — with `$ErrorActionPreference='Continue'` the script
|
||
went on and started the Host against the **live** `ScadaBridge` DB. The production-mode
|
||
pending-migration check (`MigrationHelper.ApplyOrValidateMigrationsAsync`) refused the boot with
|
||
"2 pending migration(s)" before `DataConnectionSecretUpgrade` or any write; the live DB was verified
|
||
untouched (53 migrations, row still plaintext). Use `$csb['Database'] = …`, assert
|
||
`$csb.InitialCatalog` equals the clone name, and run the setup under `Stop`.
|
||
|
||
### Sequence (maintenance window)
|
||
|
||
Stop Site vq04 → Site vq03 → Central vq04 → Central vq03 (all four stopped, no
|
||
`ZB.MOM.WW.ScadaBridge.Host` process on either box, 0 sessions on the DB) → migration from vq03 →
|
||
swap `app\`/`cli\` on vq03 then vq04 → start Central vq03 (was active) → Central vq04 → Site vq03 →
|
||
Site vq04. Pairs stopped and started **together** (no rolling upgrade of a site pair; no mixed
|
||
versions against the encrypted row).
|
||
|
||
| Step | Local time | Duration |
|
||
|---|---|---|
|
||
| Stop `ScadaBridge-Site` vq04 (**window opens**) | 12:17:57 | 30 s |
|
||
| Stop `ScadaBridge-Site` vq03 | 12:18:28 | 10 s |
|
||
| Stop `ScadaBridge-Central` vq04 | 12:18:46 | 10 s |
|
||
| Stop `ScadaBridge-Central` vq03 | 12:18:58 | 10 s |
|
||
| Migration 53 → 55 (vq03 → wonder-sql-vq03) | 12:19:15 | 1 s |
|
||
| Swap vq03 / vq04 (rename, copy, restore appsettings, SHA match) | 12:19:21 / 12:19:30 | 7 s / 8 s |
|
||
| Start Central vq03 → `/health/ready` 200 (protocol-row encryption ran here) | 12:19:43 → 12:19:57 | 14 s |
|
||
| Start Central vq04 → ready (Joining, memberCount 2) | 12:20:02 → 12:20:10 | 8 s |
|
||
| Start Site vq03 → ready | 12:20:15 → 12:20:23 | 8 s |
|
||
| Start Site vq04 → ready (**window closes**) | 12:20:28 → 12:20:32 | 4 s |
|
||
|
||
**Downtime (first stop → last site ready): 2 min 35 s.**
|
||
|
||
### Verification (12:21–12:22, both boxes)
|
||
|
||
- `:8085 /health/ready` **200 Healthy** on both centrals — database, akka-cluster Up,
|
||
**memberCount 2, unreachable 0**, all 5 required singletons; `/health/active` 200 on vq03
|
||
(oldest Up member), 503 Standby on vq04 — same active node as before the window.
|
||
- `:8084 /health/ready` **200 Healthy** on both sites — akka-cluster Up memberCount 2, `localdb`
|
||
Healthy with **`replicationConnected:true`, backlog 0** on both (vq03 briefly showed
|
||
`replicationConnected:false` / backlog 2 while vq04's site was still down — expected, cleared on
|
||
vq04 start); `/health/active` 200 vq03 / 503 vq04.
|
||
- DB: 55 migrations (max `20260909203046_…`), indexes `IX_Instances_AreaId, IX_Instances_SiteId,
|
||
IX_Instances_TemplateId, IX_Instances_UniqueName`; `DataConnections` id 1 LEN 426 `dp:protocol:v1:`;
|
||
`DataProtectionKeys` 2; 4 instances (Z28061, Z28062 Enabled; Z28063, ZIPSEN Disabled).
|
||
- Auth posture unchanged: `POST /management` no-auth → **401** before and after; `GET :8085/` 200
|
||
(title ScadaBridge); on-box CLI with the QA test user works — `--version` = `2b9a7c3c`,
|
||
`instance list` → 4 rows, groups vs old: `+shared-schema +flow-debug`.
|
||
- **Snapshots**: baseline (12:10, via the CLI against the active central) Z28061(1) attrs=244
|
||
Good=244, Z28062(2) attrs=244 Good=244, Z28063(9)/ZIPSEN(7) attrs=0 (not deployed); **after:
|
||
identical on the first poll** at 12:21:46 (vq03) and 12:22:27 (vq04) — the DCL re-subscribed
|
||
(`Instance Z28061/Z28062 subscribing to 244 tags on connection MxGateway` at 12:20:26).
|
||
- Site runtime: `Loaded 2 deployed configs (2 enabled) from SQLite`; reconcile passes on node-a and
|
||
node-b `0 fetched, 0 failed, 0 orphan(s)`.
|
||
- Logs since the new boot: vq03 central 3 ERR (all the known first-probe `member status: Removed`),
|
||
vq04 central 0; vq03 site 5 ERR (all `member status: Removed`), vq04 site 1 (same); the only FTL is
|
||
the outgoing build's `host terminated unexpectedly` at 12:18:37 on vq04's site during its stop
|
||
(known, cosmetic). WRNs are the usual set (HTTP/2-without-TLS notice, `RequireHttpsCookie` disabled,
|
||
the SQLite-key-store warning, `Debug snapshot for unknown instance Z28063/ZIPSEN` from the
|
||
snapshot probe of the two undeployed instances).
|
||
- QA Overview on vq03 `:5320/health/ready` 200, still `Watching 10 instance(s) across 4 application(s)`.
|
||
|
||
### Pitfalls hit (for next time)
|
||
|
||
- `SqlConnectionStringBuilder` property *sets* do not work in PS 5.1 (`$b.InitialCatalog = …` throws
|
||
`Keyword not supported`); use the indexer with the real keyword (`$b['Database']`) and assert.
|
||
- When a preflight targets a clone, put the catalog assertion **before** `Start-Process` and run the
|
||
setup section with `$ErrorActionPreference = 'Stop'` — see the near miss above.
|
||
- Masking by *key name* is not enough: a key like `ScadaBridge__Database__ConfigurationDb` carries a
|
||
`Password=` inside its value. Mask values by pattern (`Password=[^;]*`) too.
|
||
- The migration script from `dotnet ef migrations script <from> <to>` does not contain the `<from>`
|
||
id — checking that it "starts from" `20260819201210` means checking the DB's `MAX(MigrationId)`
|
||
equals it, which it did.
|
||
- Windows reports today's open Serilog file as 0 bytes (`Get-ChildItem`); `Get-Content` reads it
|
||
fine — do not treat the 0-byte size as "nothing logged".
|
||
|
||
## 2. Rollback
|
||
|
||
Both boxes, in this order:
|
||
|
||
1. `Stop-Service ScadaBridge-Site` on vq04 then vq03; `Stop-Service ScadaBridge-Central` on vq04
|
||
then vq03; confirm no `ZB.MOM.WW.ScadaBridge.Host` process remains on either box.
|
||
2. From vq03 (`sqlcmd -C -b`): `RESTORE DATABASE [ScadaBridge] FROM DISK =
|
||
'F:\MSSQL\BACKUP\ScadaBridge_pre-2b9a7c3c_20260910T121009.bak' WITH REPLACE` — **required**: the
|
||
protected protocol row is unreadable by `afb27e77` and the migration `Down` throws over it, so a
|
||
binary-only rollback is not possible for this build.
|
||
3. On each box: `app` → `app.failed`, `app.bak.20260910T121009` → `app`; `cli` → `cli.failed`,
|
||
`cli.bak.20260910T121009` → `cli`.
|
||
4. `Start-Service ScadaBridge-Central` on vq03 then vq04 (wait for `:8085/health/ready` 200), then
|
||
`ScadaBridge-Site` on vq03 then vq04 (`:8084`).
|
||
|
||
Service env untouched; `.reg` exports of both service keys are in `E:\psrem-stage\2b9a7c3c\backup\`
|
||
on each box if a service ever has to be re-created.
|
||
|
||
## 3. Outstanding
|
||
|
||
- `ApiKeyStore:Provider` still unset on the QA pair (per-node SQLite inbound-API keys on a real
|
||
2-node central — a failover rejects inbound clients until keys are re-issued): the SqlServer
|
||
cutover is the separate planned record.
|
||
- ZIPSEN (id 7) and Z28063 (id 9) remain non-deployed (Disabled) instances.
|
||
- Housekeeping once the build has soaked: remove `app.bak.*`/`cli.bak.*`/`_appsettings-keep-*` from
|
||
2026-08-2x and this deploy under `E:\ApiInstall\ScadaBridge`, and `E:\psrem-stage\2b9a7c3c\`
|
||
(zip + unpacked ≈ 500 MB per box).
|