ScadaBridge/docs/deployment/deployment-records/2026-09-10-wonder-app-vq03-vq04-2b9a7c3c.md

Deployment record — wonder-app-vq03 + wonder-app-vq04 (QA) — 2026-09-10 — afb27e77 → 2b9a7c3c

Point-in-time record of a ScadaBridge binary + schema upgrade of the QA two-host topology: a real central pair (central-a = wonder-app-vq03 10.220.158.2, central-b = wonder-app-vq04 10.220.158.3, :8085, Akka 8081, CentralGrpcPort 8086) and a site-a pair (node-a/node-b on the same two boxes, gRPC :8083, health :8084, Akka 8082, LocalDb replication ON), each box running ScadaBridge-Central, ScadaBridge-Site and ScadaBridge-LDAP (native services, env in the service key) out of E:\ApiInstall\ScadaBridge. Shared ConfigDb ScadaBridge on wonder-sql-vq03 (both centrals; DataProtection key ring lives in the same DB). Performed from the Mac entirely via the winrm CLI (WinRM 5985/NTLM for commands; the zip had been pushed over SMB beforehand) — no psbridge, no servecli, nothing installed on the boxes. Same build as the DEV record 2026-09-10-wonder-app-vd03-2b9a7c3c.md (read that for the build/delta description).

1. Binary + schema upgrade

Item	Value
Build deployed	main @ 2b9a7c3c (from afb27e77, 2026-08-21; 253 commits) — all four nodes. ProductVersion 1.0.0+2b9a7c3ca05fde8fe58f647d8f91e5ab9452807b on Host, CLI and all 25 ZB.MOM.WW.ScadaBridge.*.dll on both boxes.
Headline change	Arch-review remediation (13 High + 33 Medium + 46 Low), management API / CLI parity (CLI gains shared-schema + flow-debug), Flow Debugger v1.4, DataConnections protocol config encrypted at rest (dp:protocol:v1:).
Artifact	E:\psrem-stage\2b9a7c3c\scadabridge.zip on both boxes, 125 543 056 B, SHA-256 371B3271A57EC1BA… re-verified on-box; unpacked to E:\psrem-stage\2b9a7c3c\unpacked (app 1029 files, cli 223); staged exe ProductVersion checked before the swap.
EF migrations applied	2 (53 → 55) from the from→to script sb-mig-53to55.sql (20260819201210 → 20260909203046, NOT idempotent, identical to vd03's) via sqlcmd -C -b -I on vq03: 20260906034113_AddApiMethodTraceAllCalls, 20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration (THROW 51002 guard passed — 0 duplicate UniqueName; IX_Instances_SiteId_UniqueName dropped, IX_Instances_SiteId + unique IX_Instances_UniqueName created; Primary/BackupConfiguration widened to nvarchar(max)).
Coordinated data upgrade	The first Central boot on the new binary (vq03, 12:19:43) ran DataConnectionSecretUpgrade: DataConnections id 1 (MxGateway) PrimaryConfiguration 210 chars plaintext → 426 chars dp:protocol:v1:, BackupConfiguration NULL, DataProtectionKeys unchanged at 2. afb27e77 cannot read the protected row and the migration Down throws over it — rollback is the DB restore, not a binary swap.
New required config	None. The box's three appsettings*.json were kept, restored after the swap and SHA-256-matched on both boxes (vq03: F94000B1…/92524782…/4CBAF66C…; vq04: F94000B1…/2FD3EC4C…/D0C1AF05…). Service registry environment untouched (reg export of both service keys to E:\psrem-stage\2b9a7c3c\backup\). ApiKeyStore:Provider deliberately still UNSET (per-node SQLite) — separate planned cutover; the boot [WRN] Inbound API keys use the per-node SQLite store on a 2-node central cluster is real on QA (unlike vd03) and is the reason for that cutover.
DB backup	F:\MSSQL\BACKUP\ScadaBridge_pre-2b9a7c3c_20260910T121009.bak on wonder-sql-vq03 (COPY_ONLY, CHECKSUM, COMPRESSION; RESTORE VERIFYONLY … WITH CHECKSUM valid; 688 MB logical / 113 MB compressed), taken 12:11:41 with everything still running.
Binary backup	E:\ApiInstall\ScadaBridge\app.bak.20260910T121009 + cli.bak.20260910T121009 on each box (old afb27e77 binaries + appsettings); _appsettings-keep-20260910T121009 copies alongside.
Notes on box	E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-20260910T121009.txt on each box (baseline, preflight, window, verify, rollback).

Preflight (vq04, before the window, 12:14–12:17)

The staged Host was booted as an isolated single-node Central on vq04 (:18085, Akka 18081, seeds = self + a dead 18087, BootstrapGuard off, CentralGrpcPort 18086, own copies of the inbound-api-key store and secrets.db, real service env otherwise) against a clone ScadaBridge_pf2b9a7c3c restored from the pre-upgrade backup (WITH MOVE) and migrated 53 → 55 with the same script. Result: /health/ready 200 in 15 s (memberCount 1, 5/5 singletons), POST /management no-auth 401, clone row encrypted 210 → 426 dp:protocol:v1:, DataProtectionKeys 2, live central cluster stayed at memberCount 2 / unreachable 0 throughout. Clone dropped and pf-app removed afterwards (pf-data\ logs left on vq04).

Near miss, recorded for next time: attempt 1 set the clone catalog with $csb.InitialCatalog = … on a SqlConnectionStringBuilder, which PowerShell 5.1 rejects (Keyword not supported: 'InitialCatalog') — with $ErrorActionPreference='Continue' the script went on and started the Host against the live ScadaBridge DB. The production-mode pending-migration check (MigrationHelper.ApplyOrValidateMigrationsAsync) refused the boot with "2 pending migration(s)" before DataConnectionSecretUpgrade or any write; the live DB was verified untouched (53 migrations, row still plaintext). Use $csb['Database'] = …, assert $csb.InitialCatalog equals the clone name, and run the setup under Stop.

Sequence (maintenance window)

Stop Site vq04 → Site vq03 → Central vq04 → Central vq03 (all four stopped, no ZB.MOM.WW.ScadaBridge.Host process on either box, 0 sessions on the DB) → migration from vq03 → swap app\/cli\ on vq03 then vq04 → start Central vq03 (was active) → Central vq04 → Site vq03 → Site vq04. Pairs stopped and started together (no rolling upgrade of a site pair; no mixed versions against the encrypted row).

Step	Local time	Duration
Stop ScadaBridge-Site vq04 (window opens)	12:17:57	30 s
Stop ScadaBridge-Site vq03	12:18:28	10 s
Stop ScadaBridge-Central vq04	12:18:46	10 s
Stop ScadaBridge-Central vq03	12:18:58	10 s
Migration 53 → 55 (vq03 → wonder-sql-vq03)	12:19:15	1 s
Swap vq03 / vq04 (rename, copy, restore appsettings, SHA match)	12:19:21 / 12:19:30	7 s / 8 s
Start Central vq03 → /health/ready 200 (protocol-row encryption ran here)	12:19:43 → 12:19:57	14 s
Start Central vq04 → ready (Joining, memberCount 2)	12:20:02 → 12:20:10	8 s
Start Site vq03 → ready	12:20:15 → 12:20:23	8 s
Start Site vq04 → ready (window closes)	12:20:28 → 12:20:32	4 s

Downtime (first stop → last site ready): 2 min 35 s.

Verification (12:21–12:22, both boxes)

• :8085 /health/ready 200 Healthy on both centrals — database, akka-cluster Up, memberCount 2, unreachable 0, all 5 required singletons; /health/active 200 on vq03 (oldest Up member), 503 Standby on vq04 — same active node as before the window.
• :8084 /health/ready 200 Healthy on both sites — akka-cluster Up memberCount 2, localdb Healthy with replicationConnected:true, backlog 0 on both (vq03 briefly showed replicationConnected:false / backlog 2 while vq04's site was still down — expected, cleared on vq04 start); /health/active 200 vq03 / 503 vq04.
• DB: 55 migrations (max 20260909203046_…), indexes IX_Instances_AreaId, IX_Instances_SiteId, IX_Instances_TemplateId, IX_Instances_UniqueName; DataConnections id 1 LEN 426 dp:protocol:v1:; DataProtectionKeys 2; 4 instances (Z28061, Z28062 Enabled; Z28063, ZIPSEN Disabled).
• Auth posture unchanged: POST /management no-auth → 401 before and after; GET :8085/ 200 (title ScadaBridge); on-box CLI with the QA test user works — --version = 2b9a7c3c, instance list → 4 rows, groups vs old: +shared-schema +flow-debug.
• Snapshots: baseline (12:10, via the CLI against the active central) Z28061(1) attrs=244 Good=244, Z28062(2) attrs=244 Good=244, Z28063(9)/ZIPSEN(7) attrs=0 (not deployed); after: identical on the first poll at 12:21:46 (vq03) and 12:22:27 (vq04) — the DCL re-subscribed (Instance Z28061/Z28062 subscribing to 244 tags on connection MxGateway at 12:20:26).
• Site runtime: Loaded 2 deployed configs (2 enabled) from SQLite; reconcile passes on node-a and node-b 0 fetched, 0 failed, 0 orphan(s).
• Logs since the new boot: vq03 central 3 ERR (all the known first-probe member status: Removed), vq04 central 0; vq03 site 5 ERR (all member status: Removed), vq04 site 1 (same); the only FTL is the outgoing build's host terminated unexpectedly at 12:18:37 on vq04's site during its stop (known, cosmetic). WRNs are the usual set (HTTP/2-without-TLS notice, RequireHttpsCookie disabled, the SQLite-key-store warning, Debug snapshot for unknown instance Z28063/ZIPSEN from the snapshot probe of the two undeployed instances).
• QA Overview on vq03 :5320/health/ready 200, still Watching 10 instance(s) across 4 application(s).

Pitfalls hit (for next time)

• SqlConnectionStringBuilder property sets do not work in PS 5.1 ($b.InitialCatalog = … throws Keyword not supported); use the indexer with the real keyword ($b['Database']) and assert.
• When a preflight targets a clone, put the catalog assertion before Start-Process and run the setup section with $ErrorActionPreference = 'Stop' — see the near miss above.
• Masking by key name is not enough: a key like ScadaBridge__Database__ConfigurationDb carries a Password= inside its value. Mask values by pattern (Password=[^;]*) too.
• The migration script from dotnet ef migrations script <from> <to> does not contain the <from> id — checking that it "starts from" 20260819201210 means checking the DB's MAX(MigrationId) equals it, which it did.
• Windows reports today's open Serilog file as 0 bytes (Get-ChildItem); Get-Content reads it fine — do not treat the 0-byte size as "nothing logged".

2. Rollback

Both boxes, in this order:

1. Stop-Service ScadaBridge-Site on vq04 then vq03; Stop-Service ScadaBridge-Central on vq04 then vq03; confirm no ZB.MOM.WW.ScadaBridge.Host process remains on either box.
2. From vq03 (sqlcmd -C -b): RESTORE DATABASE [ScadaBridge] FROM DISK = 'F:\MSSQL\BACKUP\ScadaBridge_pre-2b9a7c3c_20260910T121009.bak' WITH REPLACE — required: the protected protocol row is unreadable by afb27e77 and the migration Down throws over it, so a binary-only rollback is not possible for this build.
3. On each box: app → app.failed, app.bak.20260910T121009 → app; cli → cli.failed, cli.bak.20260910T121009 → cli.
4. Start-Service ScadaBridge-Central on vq03 then vq04 (wait for :8085/health/ready 200), then ScadaBridge-Site on vq03 then vq04 (:8084).

Service env untouched; .reg exports of both service keys are in E:\psrem-stage\2b9a7c3c\backup\ on each box if a service ever has to be re-created.

3. Outstanding

• ApiKeyStore:Provider still unset on the QA pair (per-node SQLite inbound-API keys on a real 2-node central — a failover rejects inbound clients until keys are re-issued): the SqlServer cutover is the separate planned record.
• ZIPSEN (id 7) and Z28063 (id 9) remain non-deployed (Disabled) instances.
• Housekeeping once the build has soaked: remove app.bak.*/cli.bak.*/_appsettings-keep-* from 2026-08-2x and this deploy under E:\ApiInstall\ScadaBridge, and E:\psrem-stage\2b9a7c3c\ (zip + unpacked ≈ 500 MB per box).