From 17864d1ae41f759b78932f797ae3a308365642bf Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 10 Sep 2026 12:25:53 -0400 Subject: [PATCH] docs(deploy): records for 2b9a7c3c on DEV vd03 and the QA pair vq03/vq04 Both driven from the Mac over the winrm CLI (WinRM 5985 + SMB 445, no helper on the boxes). Migration 53->55; the first Central boot encrypts DataConnections.PrimaryConfiguration (dp:protocol:v1:), so rollback is the COPY_ONLY DB restore, not a binary swap. QA: all four nodes stopped together, one shared-DB migration, 2 min 35 s window, preflight against a restored clone. Claude-Session: https://claude.ai/code/session_01RgthVLVU5CJ8SUuEwruTkt --- .../2026-09-10-wonder-app-vd03-2b9a7c3c.md | 102 +++++++++++++ ...026-09-10-wonder-app-vq03-vq04-2b9a7c3c.md | 140 ++++++++++++++++++ 2 files changed, 242 insertions(+) create mode 100644 docs/deployment/deployment-records/2026-09-10-wonder-app-vd03-2b9a7c3c.md create mode 100644 docs/deployment/deployment-records/2026-09-10-wonder-app-vq03-vq04-2b9a7c3c.md diff --git a/docs/deployment/deployment-records/2026-09-10-wonder-app-vd03-2b9a7c3c.md b/docs/deployment/deployment-records/2026-09-10-wonder-app-vd03-2b9a7c3c.md new file mode 100644 index 00000000..0a0de74f --- /dev/null +++ b/docs/deployment/deployment-records/2026-09-10-wonder-app-vd03-2b9a7c3c.md @@ -0,0 +1,102 @@ +# 2026-09-10 — DEV (wonder-app-vd03) → `2b9a7c3c` (arch-review remediation + CLI parity + Flow Debugger v1.4) + +**Build:** `main` @ `2b9a7c3c` (= `origin/main`), ProductVersion `1.0.0+2b9a7c3ca05fde8fe58f647d8f91e5ab9452807b`, +published from a detached worktree at that SHA (`dotnet publish` Host + CLI `-c Release -r win-x64 +--self-contained true -p:PublishSingleFile=false`; 1029 app files, 223 cli files, zip 125 543 056 B, +SHA-256 `371b3271…`). +**Delta from the box's previous build `afb27e77`:** 253 commits — Flow Debugger instance picker + v1.4 +step logs, the 2026-09-09 whole-system arch review remediation (13 High + 33 Medium + 46 Low), and +management API / CLI parity (`docs/plans/2026-09-10-management-api-cli-parity-review.md`). The CLI +gains the `shared-schema` and `flow-debug` groups. +**QA (vq03/vq04) NOT touched** — this record covers vd03 only. + +**Transport:** first ScadaBridge deploy driven entirely by the Mac-side `winrm` CLI (WinRM 5985/NTLM +for commands, `winrm smb push` over 445 for the zip) — no psbridge, no servecli. + +## Schema: 53 → 55 migrations, one of them a coordinated data upgrade + +| Migration | Effect | Pre-check | +|---|---|---| +| `20260906034113_AddApiMethodTraceAllCalls` | `ApiMethods.TraceAllCalls bit NOT NULL DEFAULT 0` | — | +| `20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration` | `THROW 51002` if `Instances.UniqueName` is not globally unique; drops `IX_Instances_SiteId_UniqueName`, adds `IX_Instances_SiteId` + unique `IX_Instances_UniqueName`; widens `DataConnections.Primary/BackupConfiguration` to `nvarchar(max)` | 0 duplicate names (4 instances, all site 2); 1 DataConnection (`MxGateway`, 210 chars, unprotected) | + +The migration is schema-only. **The first Central boot on the new binary ran +`DataConnectionSecretUpgrade`** and rewrote `DataConnections.PrimaryConfiguration` as +`dp:protocol:v1:…` (LEN 210 → 426) using the Data Protection key ring, which on this install lives +**in the same database** (`DataProtectionKeys`, 2 keys, unchanged). So the single +`BACKUP DATABASE … COPY_ONLY, CHECKSUM` taken beforehand *is* the coordinated database/key-ring +backup the component doc asks for. **Old binaries cannot read the protected row, and the migration's +`Down` refuses while protected rows exist** — rollback is the DB restore, not `dotnet ef`. + +From→to script generated with `dotnet ef migrations script 20260819201210_AddNotificationsListSortIndex +20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration` (NOT idempotent — the +full idempotent script does not compile against this DB, see the 2026-06-16 record), applied with +`sqlcmd -C -b -I` while both services were stopped. + +## Config + +**No appsettings change.** Every new option in the delta has a passing default +(`InboundApi:StandbyForwardPeerEndpoints`/`StandbyForwardTimeout`/`MaxLogLinesPerRun`, +`SiteRuntime:TagSubscribeRetryMaxIntervalMs`/`TagSubscribeInFlightCeilingMs`/`TraceArmWindowSeconds`/ +`MaxLogLinesPerRun`, `Communication:ConnectedHeaderTimeout`, `MxGateway:ClientNamePrefix`, +`AuditLogPurge:MaxPurgedRowsPerChannelPerTick`), verified by reading the validators and then by the +clean boot. `ScadaBridge:InboundApi:ApiKeyStore:Provider` **left unset (= Sqlite)** — the SqlServer +cutover is the separate planned record `2026-09-XX-zb-servers-inbound-key-store.md`. The box's three +`appsettings*.json` were copied out, restored after the swap and SHA-256-matched. Service registry +environment untouched. + +## Procedure (`E:\psrem-stage\2b9a7c3c\sb-swap.ps1`, log `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-20260910T104529.txt`) + +1. 10:43 — `reg export` of both service keys (`_envbak-ScadaBridge-{Central,Site}-20260910T104349.reg`); + `BACKUP DATABASE [ScadaBridge] … COPY_ONLY, CHECKSUM, COMPRESSION` → + `E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-2b9a7c3c_20260910T104349.bak` (120 MB), + `RESTORE VERIFYONLY … WITH CHECKSUM` ok. Zip pushed over SMB, SHA-256 verified on the box, unpacked, + staged `Host.exe`/`scadabridge.exe` ProductVersion checked = `2b9a7c3c`. +2. Baseline: `/health/ready` 200/200; `POST /management ListInstances` no-auth → 200; bogus + `X-API-Key` → 401; `debug snapshot` Z28061Sim(8) attrs=270 Bad=28 Good=242, Z28061(1) attrs=270 + Bad=126 Good=144 (via the on-box CLI with `--username multi-role --password password`). +3. **10:45:29** Site stopped (10 s), Central stopped (10 s), no `Host.exe` left. +4. 10:45:50 migration script: 53 → 55 rows in `__EFMigrationsHistory`. +5. `app`/`cli` → `app.bak.20260910T104529` / `cli.bak.20260910T104529`; staged copy in; the three + appsettings restored (SHA match). +6. 10:45:57 Central started → ready 10:46:11 (14 s, includes the protocol-row encryption upgrade); + Site started → ready 10:46:19. **~50 s total downtime.** + +## Verification + +| Check | Result | +|---|---| +| `:8085 /health/ready` | 200 Healthy — database, akka-cluster Up (members=1), required-singletons 5/5 | +| `:8084 /health/ready` | 200 Healthy — akka-cluster, localdb | +| ProductVersion | Host + CLI + 25 `ZB.MOM.WW.ScadaBridge.*` DLLs at `2b9a7c3c` | +| DB | 55 migrations; new indexes present; `DataConnections` id 1 `PrimaryConfiguration` protected (`dp:protocol:v1:`), backup null; `DataProtectionKeys` = 2 | +| Auth posture | `POST /management` no-auth 200 (DisableLogin preserved); bogus `X-API-Key` 401 (key store loaded); `GET /` 200 | +| Site runtime | Loaded 3 deployed configs; reconcile 0 fetched / 0 failed / 0 orphans; Z28061, Z28061Sim, Z28062 each `subscribing to 270 tags on connection MxGateway` | +| Snapshots after | Z28061Sim attrs=270 alarms=1 Bad=28 Good=242; Z28061 attrs=270 Bad=126 Good=144 — **identical to baseline** | +| Logs after boot | central 0 ERR/FTL, site 0 ERR/FTL (only the usual first-probe `member status: Removed` during startup) | +| New CLI | `--version` = `2b9a7c3c`; `instance list` → 4 rows; groups vs old: `+shared-schema +flow-debug` | + +### Observed, not a regression + +- Boot `[WRN] Startup configuration warning: Inbound API keys use the per-node SQLite store on a + 2-node central cluster` — **spurious on this box.** `StartupValidator.CollectWarnings` counts + `Cluster:SeedNodes`, and vd03 carries the phantom second seed (8091/8092) that the same validator + requires (`>= 2` seed nodes, unconditionally). The node is single-member; nothing to act on until + the SqlServer key-store cutover. +- The CLI refuses to run without `--username/--password` (or `SCADABRIDGE_USERNAME/_PASSWORD`) even + though the server ignores them under DisableLogin — pass any value. +- The site's known `SiteLocalDbHealthCheck replicationConnected:false` (unreplicated single node) is + unchanged and cosmetic. + +## Rollback + +Stop Site, stop Central. `RESTORE DATABASE [ScadaBridge] FROM DISK = +'E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-2b9a7c3c_20260910T104349.bak' WITH REPLACE` +(**required** — the protected protocol row is unreadable by `afb27e77` and the migration `Down` +throws over it). Rename `app` → `app.failed`, `app.bak.20260910T104529` → `app`, same for `cli`. +Start Central, then Site. Service env untouched; `.reg` exports are there if a service is ever +re-created. + +Staging left on the box under `E:\psrem-stage\2b9a7c3c\` (zip, unpacked tree, migration SQL, swap +script, before/after snapshot JSON). The `_appsettings-keep-20260910T104529` copy and the +`*.bak.20260910T104529` directories can go once the build has soaked. diff --git a/docs/deployment/deployment-records/2026-09-10-wonder-app-vq03-vq04-2b9a7c3c.md b/docs/deployment/deployment-records/2026-09-10-wonder-app-vq03-vq04-2b9a7c3c.md new file mode 100644 index 00000000..c8688053 --- /dev/null +++ b/docs/deployment/deployment-records/2026-09-10-wonder-app-vq03-vq04-2b9a7c3c.md @@ -0,0 +1,140 @@ +# Deployment record — wonder-app-vq03 + wonder-app-vq04 (QA) — 2026-09-10 — `afb27e77` → `2b9a7c3c` + +Point-in-time record of a ScadaBridge binary + schema upgrade of the **QA two-host topology**: +a real central pair (`central-a` = wonder-app-vq03 `10.220.158.2`, `central-b` = wonder-app-vq04 +`10.220.158.3`, :8085, Akka 8081, CentralGrpcPort 8086) and a site-a pair (`node-a`/`node-b` on +the same two boxes, gRPC :8083, health :8084, Akka 8082, LocalDb replication ON), each box running +`ScadaBridge-Central`, `ScadaBridge-Site` and `ScadaBridge-LDAP` (native services, env in the +service key) out of `E:\ApiInstall\ScadaBridge`. Shared ConfigDb `ScadaBridge` on +`wonder-sql-vq03` (both centrals; DataProtection key ring lives in the same DB). Performed from the +Mac entirely via the **`winrm` CLI** (WinRM 5985/NTLM for commands; the zip had been pushed over SMB +beforehand) — no psbridge, no servecli, nothing installed on the boxes. Same build as the DEV record +`2026-09-10-wonder-app-vd03-2b9a7c3c.md` (read that for the build/delta description). + +## 1. Binary + schema upgrade + +| Item | Value | +|---|---| +| Build deployed | `main` @ **`2b9a7c3c`** (from `afb27e77`, 2026-08-21; 253 commits) — all four nodes. ProductVersion `1.0.0+2b9a7c3ca05fde8fe58f647d8f91e5ab9452807b` on Host, CLI and all 25 `ZB.MOM.WW.ScadaBridge.*.dll` on both boxes. | +| Headline change | Arch-review remediation (13 High + 33 Medium + 46 Low), management API / CLI parity (CLI gains `shared-schema` + `flow-debug`), Flow Debugger v1.4, **`DataConnections` protocol config encrypted at rest** (`dp:protocol:v1:`). | +| Artifact | `E:\psrem-stage\2b9a7c3c\scadabridge.zip` on both boxes, 125 543 056 B, SHA-256 `371B3271A57EC1BA…` re-verified on-box; unpacked to `E:\psrem-stage\2b9a7c3c\unpacked` (app 1029 files, cli 223); staged exe ProductVersion checked before the swap. | +| EF migrations applied | 2 (53 → 55) from the from→to script `sb-mig-53to55.sql` (`20260819201210` → `20260909203046`, NOT idempotent, identical to vd03's) via `sqlcmd -C -b -I` on vq03: `20260906034113_AddApiMethodTraceAllCalls`, `20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration` (THROW 51002 guard passed — 0 duplicate `UniqueName`; `IX_Instances_SiteId_UniqueName` dropped, `IX_Instances_SiteId` + unique `IX_Instances_UniqueName` created; `Primary/BackupConfiguration` widened to `nvarchar(max)`). | +| Coordinated data upgrade | The first Central boot on the new binary (vq03, 12:19:43) ran `DataConnectionSecretUpgrade`: `DataConnections` id 1 (`MxGateway`) `PrimaryConfiguration` 210 chars plaintext → **426 chars `dp:protocol:v1:`**, `BackupConfiguration` NULL, `DataProtectionKeys` unchanged at 2. **`afb27e77` cannot read the protected row and the migration `Down` throws over it — rollback is the DB restore, not a binary swap.** | +| New required config | **None.** The box's three `appsettings*.json` were kept, restored after the swap and SHA-256-matched on both boxes (vq03: `F94000B1…`/`92524782…`/`4CBAF66C…`; vq04: `F94000B1…`/`2FD3EC4C…`/`D0C1AF05…`). Service registry environment untouched (`reg export` of both service keys to `E:\psrem-stage\2b9a7c3c\backup\`). `ApiKeyStore:Provider` deliberately still UNSET (per-node SQLite) — separate planned cutover; the boot `[WRN] Inbound API keys use the per-node SQLite store on a 2-node central cluster` is real on QA (unlike vd03) and is the reason for that cutover. | +| DB backup | `F:\MSSQL\BACKUP\ScadaBridge_pre-2b9a7c3c_20260910T121009.bak` on wonder-sql-vq03 (COPY_ONLY, CHECKSUM, COMPRESSION; `RESTORE VERIFYONLY … WITH CHECKSUM` valid; 688 MB logical / 113 MB compressed), taken 12:11:41 with everything still running. | +| Binary backup | `E:\ApiInstall\ScadaBridge\app.bak.20260910T121009` + `cli.bak.20260910T121009` on each box (old `afb27e77` binaries + appsettings); `_appsettings-keep-20260910T121009` copies alongside. | +| Notes on box | `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-20260910T121009.txt` on each box (baseline, preflight, window, verify, rollback). | + +### Preflight (vq04, before the window, 12:14–12:17) + +The staged Host was booted as an **isolated single-node Central** on vq04 (`:18085`, Akka `18081`, +seeds = self + a dead `18087`, `BootstrapGuard` off, `CentralGrpcPort` 18086, own copies of the +inbound-api-key store and `secrets.db`, real service env otherwise) against a **clone** +`ScadaBridge_pf2b9a7c3c` restored from the pre-upgrade backup (`WITH MOVE`) and migrated 53 → 55 +with the same script. Result: `/health/ready` 200 in 15 s (memberCount 1, 5/5 singletons), +`POST /management` no-auth 401, clone row encrypted 210 → 426 `dp:protocol:v1:`, `DataProtectionKeys` +2, live central cluster stayed at memberCount 2 / unreachable 0 throughout. Clone dropped and +`pf-app` removed afterwards (`pf-data\` logs left on vq04). + +**Near miss, recorded for next time:** attempt 1 set the clone catalog with +`$csb.InitialCatalog = …` on a `SqlConnectionStringBuilder`, which PowerShell 5.1 rejects +(`Keyword not supported: 'InitialCatalog'`) — with `$ErrorActionPreference='Continue'` the script +went on and started the Host against the **live** `ScadaBridge` DB. The production-mode +pending-migration check (`MigrationHelper.ApplyOrValidateMigrationsAsync`) refused the boot with +"2 pending migration(s)" before `DataConnectionSecretUpgrade` or any write; the live DB was verified +untouched (53 migrations, row still plaintext). Use `$csb['Database'] = …`, assert +`$csb.InitialCatalog` equals the clone name, and run the setup under `Stop`. + +### Sequence (maintenance window) + +Stop Site vq04 → Site vq03 → Central vq04 → Central vq03 (all four stopped, no +`ZB.MOM.WW.ScadaBridge.Host` process on either box, 0 sessions on the DB) → migration from vq03 → +swap `app\`/`cli\` on vq03 then vq04 → start Central vq03 (was active) → Central vq04 → Site vq03 → +Site vq04. Pairs stopped and started **together** (no rolling upgrade of a site pair; no mixed +versions against the encrypted row). + +| Step | Local time | Duration | +|---|---|---| +| Stop `ScadaBridge-Site` vq04 (**window opens**) | 12:17:57 | 30 s | +| Stop `ScadaBridge-Site` vq03 | 12:18:28 | 10 s | +| Stop `ScadaBridge-Central` vq04 | 12:18:46 | 10 s | +| Stop `ScadaBridge-Central` vq03 | 12:18:58 | 10 s | +| Migration 53 → 55 (vq03 → wonder-sql-vq03) | 12:19:15 | 1 s | +| Swap vq03 / vq04 (rename, copy, restore appsettings, SHA match) | 12:19:21 / 12:19:30 | 7 s / 8 s | +| Start Central vq03 → `/health/ready` 200 (protocol-row encryption ran here) | 12:19:43 → 12:19:57 | 14 s | +| Start Central vq04 → ready (Joining, memberCount 2) | 12:20:02 → 12:20:10 | 8 s | +| Start Site vq03 → ready | 12:20:15 → 12:20:23 | 8 s | +| Start Site vq04 → ready (**window closes**) | 12:20:28 → 12:20:32 | 4 s | + +**Downtime (first stop → last site ready): 2 min 35 s.** + +### Verification (12:21–12:22, both boxes) + +- `:8085 /health/ready` **200 Healthy** on both centrals — database, akka-cluster Up, + **memberCount 2, unreachable 0**, all 5 required singletons; `/health/active` 200 on vq03 + (oldest Up member), 503 Standby on vq04 — same active node as before the window. +- `:8084 /health/ready` **200 Healthy** on both sites — akka-cluster Up memberCount 2, `localdb` + Healthy with **`replicationConnected:true`, backlog 0** on both (vq03 briefly showed + `replicationConnected:false` / backlog 2 while vq04's site was still down — expected, cleared on + vq04 start); `/health/active` 200 vq03 / 503 vq04. +- DB: 55 migrations (max `20260909203046_…`), indexes `IX_Instances_AreaId, IX_Instances_SiteId, + IX_Instances_TemplateId, IX_Instances_UniqueName`; `DataConnections` id 1 LEN 426 `dp:protocol:v1:`; + `DataProtectionKeys` 2; 4 instances (Z28061, Z28062 Enabled; Z28063, ZIPSEN Disabled). +- Auth posture unchanged: `POST /management` no-auth → **401** before and after; `GET :8085/` 200 + (title ScadaBridge); on-box CLI with the QA test user works — `--version` = `2b9a7c3c`, + `instance list` → 4 rows, groups vs old: `+shared-schema +flow-debug`. +- **Snapshots**: baseline (12:10, via the CLI against the active central) Z28061(1) attrs=244 + Good=244, Z28062(2) attrs=244 Good=244, Z28063(9)/ZIPSEN(7) attrs=0 (not deployed); **after: + identical on the first poll** at 12:21:46 (vq03) and 12:22:27 (vq04) — the DCL re-subscribed + (`Instance Z28061/Z28062 subscribing to 244 tags on connection MxGateway` at 12:20:26). +- Site runtime: `Loaded 2 deployed configs (2 enabled) from SQLite`; reconcile passes on node-a and + node-b `0 fetched, 0 failed, 0 orphan(s)`. +- Logs since the new boot: vq03 central 3 ERR (all the known first-probe `member status: Removed`), + vq04 central 0; vq03 site 5 ERR (all `member status: Removed`), vq04 site 1 (same); the only FTL is + the outgoing build's `host terminated unexpectedly` at 12:18:37 on vq04's site during its stop + (known, cosmetic). WRNs are the usual set (HTTP/2-without-TLS notice, `RequireHttpsCookie` disabled, + the SQLite-key-store warning, `Debug snapshot for unknown instance Z28063/ZIPSEN` from the + snapshot probe of the two undeployed instances). +- QA Overview on vq03 `:5320/health/ready` 200, still `Watching 10 instance(s) across 4 application(s)`. + +### Pitfalls hit (for next time) + +- `SqlConnectionStringBuilder` property *sets* do not work in PS 5.1 (`$b.InitialCatalog = …` throws + `Keyword not supported`); use the indexer with the real keyword (`$b['Database']`) and assert. +- When a preflight targets a clone, put the catalog assertion **before** `Start-Process` and run the + setup section with `$ErrorActionPreference = 'Stop'` — see the near miss above. +- Masking by *key name* is not enough: a key like `ScadaBridge__Database__ConfigurationDb` carries a + `Password=` inside its value. Mask values by pattern (`Password=[^;]*`) too. +- The migration script from `dotnet ef migrations script ` does not contain the `` + id — checking that it "starts from" `20260819201210` means checking the DB's `MAX(MigrationId)` + equals it, which it did. +- Windows reports today's open Serilog file as 0 bytes (`Get-ChildItem`); `Get-Content` reads it + fine — do not treat the 0-byte size as "nothing logged". + +## 2. Rollback + +Both boxes, in this order: + +1. `Stop-Service ScadaBridge-Site` on vq04 then vq03; `Stop-Service ScadaBridge-Central` on vq04 + then vq03; confirm no `ZB.MOM.WW.ScadaBridge.Host` process remains on either box. +2. From vq03 (`sqlcmd -C -b`): `RESTORE DATABASE [ScadaBridge] FROM DISK = + 'F:\MSSQL\BACKUP\ScadaBridge_pre-2b9a7c3c_20260910T121009.bak' WITH REPLACE` — **required**: the + protected protocol row is unreadable by `afb27e77` and the migration `Down` throws over it, so a + binary-only rollback is not possible for this build. +3. On each box: `app` → `app.failed`, `app.bak.20260910T121009` → `app`; `cli` → `cli.failed`, + `cli.bak.20260910T121009` → `cli`. +4. `Start-Service ScadaBridge-Central` on vq03 then vq04 (wait for `:8085/health/ready` 200), then + `ScadaBridge-Site` on vq03 then vq04 (`:8084`). + +Service env untouched; `.reg` exports of both service keys are in `E:\psrem-stage\2b9a7c3c\backup\` +on each box if a service ever has to be re-created. + +## 3. Outstanding + +- `ApiKeyStore:Provider` still unset on the QA pair (per-node SQLite inbound-API keys on a real + 2-node central — a failover rejects inbound clients until keys are re-issued): the SqlServer + cutover is the separate planned record. +- ZIPSEN (id 7) and Z28063 (id 9) remain non-deployed (Disabled) instances. +- Housekeeping once the build has soaked: remove `app.bak.*`/`cli.bak.*`/`_appsettings-keep-*` from + 2026-08-2x and this deploy under `E:\ApiInstall\ScadaBridge`, and `E:\psrem-stage\2b9a7c3c\` + (zip + unpacked ≈ 500 MB per box).