Files
ScadaBridge/docs/deployment/deployment-records/2026-09-10-wonder-app-vd03-2b9a7c3c.md
T
Joseph Doherty 17864d1ae4 docs(deploy): records for 2b9a7c3c on DEV vd03 and the QA pair vq03/vq04
Both driven from the Mac over the winrm CLI (WinRM 5985 + SMB 445, no helper
on the boxes). Migration 53->55; the first Central boot encrypts
DataConnections.PrimaryConfiguration (dp:protocol:v1:), so rollback is the
COPY_ONLY DB restore, not a binary swap. QA: all four nodes stopped together,
one shared-DB migration, 2 min 35 s window, preflight against a restored clone.

Claude-Session: https://claude.ai/code/session_01RgthVLVU5CJ8SUuEwruTkt
2026-09-10 12:25:53 -04:00

7.3 KiB

2026-09-10 — DEV (wonder-app-vd03) → 2b9a7c3c (arch-review remediation + CLI parity + Flow Debugger v1.4)

Build: main @ 2b9a7c3c (= origin/main), ProductVersion 1.0.0+2b9a7c3ca05fde8fe58f647d8f91e5ab9452807b, published from a detached worktree at that SHA (dotnet publish Host + CLI -c Release -r win-x64 --self-contained true -p:PublishSingleFile=false; 1029 app files, 223 cli files, zip 125 543 056 B, SHA-256 371b3271…). Delta from the box's previous build afb27e77: 253 commits — Flow Debugger instance picker + v1.4 step logs, the 2026-09-09 whole-system arch review remediation (13 High + 33 Medium + 46 Low), and management API / CLI parity (docs/plans/2026-09-10-management-api-cli-parity-review.md). The CLI gains the shared-schema and flow-debug groups. QA (vq03/vq04) NOT touched — this record covers vd03 only.

Transport: first ScadaBridge deploy driven entirely by the Mac-side winrm CLI (WinRM 5985/NTLM for commands, winrm smb push over 445 for the zip) — no psbridge, no servecli.

Schema: 53 → 55 migrations, one of them a coordinated data upgrade

Migration Effect Pre-check
20260906034113_AddApiMethodTraceAllCalls ApiMethods.TraceAllCalls bit NOT NULL DEFAULT 0 —
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration THROW 51002 if Instances.UniqueName is not globally unique; drops IX_Instances_SiteId_UniqueName, adds IX_Instances_SiteId + unique IX_Instances_UniqueName; widens DataConnections.Primary/BackupConfiguration to nvarchar(max) 0 duplicate names (4 instances, all site 2); 1 DataConnection (MxGateway, 210 chars, unprotected)

The migration is schema-only. The first Central boot on the new binary ran DataConnectionSecretUpgrade and rewrote DataConnections.PrimaryConfiguration as dp:protocol:v1:… (LEN 210 → 426) using the Data Protection key ring, which on this install lives in the same database (DataProtectionKeys, 2 keys, unchanged). So the single BACKUP DATABASE … COPY_ONLY, CHECKSUM taken beforehand is the coordinated database/key-ring backup the component doc asks for. Old binaries cannot read the protected row, and the migration's Down refuses while protected rows exist — rollback is the DB restore, not dotnet ef.

From→to script generated with dotnet ef migrations script 20260819201210_AddNotificationsListSortIndex 20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration (NOT idempotent — the full idempotent script does not compile against this DB, see the 2026-06-16 record), applied with sqlcmd -C -b -I while both services were stopped.

Config

No appsettings change. Every new option in the delta has a passing default (InboundApi:StandbyForwardPeerEndpoints/StandbyForwardTimeout/MaxLogLinesPerRun, SiteRuntime:TagSubscribeRetryMaxIntervalMs/TagSubscribeInFlightCeilingMs/TraceArmWindowSeconds/ MaxLogLinesPerRun, Communication:ConnectedHeaderTimeout, MxGateway:ClientNamePrefix, AuditLogPurge:MaxPurgedRowsPerChannelPerTick), verified by reading the validators and then by the clean boot. ScadaBridge:InboundApi:ApiKeyStore:Provider left unset (= Sqlite) — the SqlServer cutover is the separate planned record 2026-09-XX-zb-servers-inbound-key-store.md. The box's three appsettings*.json were copied out, restored after the swap and SHA-256-matched. Service registry environment untouched.

Procedure (E:\psrem-stage\2b9a7c3c\sb-swap.ps1, log E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-20260910T104529.txt)

  1. 10:43 — reg export of both service keys (_envbak-ScadaBridge-{Central,Site}-20260910T104349.reg); BACKUP DATABASE [ScadaBridge] … COPY_ONLY, CHECKSUM, COMPRESSION → E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-2b9a7c3c_20260910T104349.bak (120 MB), RESTORE VERIFYONLY … WITH CHECKSUM ok. Zip pushed over SMB, SHA-256 verified on the box, unpacked, staged Host.exe/scadabridge.exe ProductVersion checked = 2b9a7c3c.
  2. Baseline: /health/ready 200/200; POST /management ListInstances no-auth → 200; bogus X-API-Key → 401; debug snapshot Z28061Sim(8) attrs=270 Bad=28 Good=242, Z28061(1) attrs=270 Bad=126 Good=144 (via the on-box CLI with --username multi-role --password password).
  3. 10:45:29 Site stopped (10 s), Central stopped (10 s), no Host.exe left.
  4. 10:45:50 migration script: 53 → 55 rows in __EFMigrationsHistory.
  5. app/cli → app.bak.20260910T104529 / cli.bak.20260910T104529; staged copy in; the three appsettings restored (SHA match).
  6. 10:45:57 Central started → ready 10:46:11 (14 s, includes the protocol-row encryption upgrade); Site started → ready 10:46:19. ~50 s total downtime.

Verification

Check Result
:8085 /health/ready 200 Healthy — database, akka-cluster Up (members=1), required-singletons 5/5
:8084 /health/ready 200 Healthy — akka-cluster, localdb
ProductVersion Host + CLI + 25 ZB.MOM.WW.ScadaBridge.* DLLs at 2b9a7c3c
DB 55 migrations; new indexes present; DataConnections id 1 PrimaryConfiguration protected (dp:protocol:v1:), backup null; DataProtectionKeys = 2
Auth posture POST /management no-auth 200 (DisableLogin preserved); bogus X-API-Key 401 (key store loaded); GET / 200
Site runtime Loaded 3 deployed configs; reconcile 0 fetched / 0 failed / 0 orphans; Z28061, Z28061Sim, Z28062 each subscribing to 270 tags on connection MxGateway
Snapshots after Z28061Sim attrs=270 alarms=1 Bad=28 Good=242; Z28061 attrs=270 Bad=126 Good=144 — identical to baseline
Logs after boot central 0 ERR/FTL, site 0 ERR/FTL (only the usual first-probe member status: Removed during startup)
New CLI --version = 2b9a7c3c; instance list → 4 rows; groups vs old: +shared-schema +flow-debug

Observed, not a regression

  • Boot [WRN] Startup configuration warning: Inbound API keys use the per-node SQLite store on a 2-node central cluster — spurious on this box. StartupValidator.CollectWarnings counts Cluster:SeedNodes, and vd03 carries the phantom second seed (8091/8092) that the same validator requires (>= 2 seed nodes, unconditionally). The node is single-member; nothing to act on until the SqlServer key-store cutover.
  • The CLI refuses to run without --username/--password (or SCADABRIDGE_USERNAME/_PASSWORD) even though the server ignores them under DisableLogin — pass any value.
  • The site's known SiteLocalDbHealthCheck replicationConnected:false (unreplicated single node) is unchanged and cosmetic.

Rollback

Stop Site, stop Central. RESTORE DATABASE [ScadaBridge] FROM DISK = 'E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-2b9a7c3c_20260910T104349.bak' WITH REPLACE (required — the protected protocol row is unreadable by afb27e77 and the migration Down throws over it). Rename app → app.failed, app.bak.20260910T104529 → app, same for cli. Start Central, then Site. Service env untouched; .reg exports are there if a service is ever re-created.

Staging left on the box under E:\psrem-stage\2b9a7c3c\ (zip, unpacked tree, migration SQL, swap script, before/after snapshot JSON). The _appsettings-keep-20260910T104529 copy and the *.bak.20260910T104529 directories can go once the build has soaked.