ScadaBridge/docs/deployment/deployment-records/2026-08-20-wonder-app-vq03-vq04.md

Deployment record — wonder-app-vq03 + wonder-app-vq04 (QA) — 2026-08-20

Point-in-time record of a ScadaBridge binary + schema upgrade of the QA two-host topology: a real central pair (central-a = wonder-app-vq03 10.220.158.2, central-b = wonder-app-vq04 10.220.158.3, :8085) and a site-a pair (node-a/node-b on the same two boxes, gRPC :8083, metrics :8084), each box running ScadaBridge-Central, ScadaBridge-Site and ScadaBridge-LDAP out of E:\ApiInstall\ScadaBridge. Shared ConfigDb on wonder-sql-vq03. Performed from the Mac via psbridge on vq04 (:6150, root E:\psrem); vq03 reached from vq04 over WinRM (Invoke-Command) and the E$ admin share.

1. Binary + schema upgrade

Item	Value
Build deployed	main @ a24fa38e (from d4bb4e17, 2026-08-12; 246 commits) — all four nodes
Headline change	Grouped connection bindings (docs/plans/2026-08-20-grouped-connection-bindings-design.md): the Configure page sources its binding rows from the flattened configuration, grouped per module with a per-group Object apply; CLI instance bind-module. Plus everything since 08-12 (arch-review Medium/Low waves, LocalDb 0.3.2→0.3.4, gRPC stream batching, bulk-deploy up-to-date skip, …).
Publish	dotnet publish -c Release -r win-x64 --self-contained true for Host (→ app\, 520 files) and CLI (→ cli\), zipped and pushed via psbridge push (SHA-256 verified), unpacked under E:\psrem\releases\a24fa38e\, mirrored to \\wonder-app-vq03\E$\psrem-stage\a24fa38e with robocopy.
EF migrations applied	6, via sqlcmd -C -b -I on vq03 from an idempotent delta script (dotnet ef migrations script 20260810100836_AddSmtpConfigurationTransport --idempotent): 20260814235335_AddAuditLogAndNotificationCoveringIndexes, 20260815004957_AlignAuditLogEventIdUniqueness, 20260815154803_AddBundleUnlockAttemptTable, 20260815165452_AddDeploymentRecordListSortIndex, 20260819183326_AddSiteCallsListSortIndex, 20260819201210_AddNotificationsListSortIndex. All additive (indexes + one table + one index drop).
New required config	AuditLog:SiteWriter:DatabasePath (top-level section, Site nodes; StartupValidator refuses to boot without it since 2e4e41a8) — added to appsettings.Site.json on both boxes as E:\ApiInstall\ScadaBridge\site\data\auditlog.db. The only appsettings delta across the 246 commits.
Data relocated	The site audit SQLite file had been written CWD-relative, i.e. to C:\Windows\System32\auditlog.db (the exact hazard 2e4e41a8 closed). Moved, with its -wal/-shm, into site\data\ on both boxes while the Site service was stopped.
Preserved across swap	box appsettings.json / appsettings.Central.json / appsettings.Site.json (copied back from the backup; service env vars untouched), central\data, site\data (secrets.db, site-localdb.db).
DB backup	ScadaBridge_pre-a24fa38e_20260820T162535.bak (COPY_ONLY, SQL Server default backup dir on wonder-sql-vq03)
Binary backup	E:\ApiInstall\ScadaBridge\app.bak.20260820T162659 + cli.bak.20260820T162659 on each box (old d4bb4e17 binaries + appsettings)

Sequence

Stop Site pair (both boxes) → stop Central pair → swap app\/cli\ on vq04 then vq03 → start Central pair → wait 20 s → start Site pair. Pairs were stopped and started together (required since LocalDb Phase 2 — no rolling upgrade of a site pair).

Pitfalls hit (for next time)

• A full-history --idempotent script fails on SQL Server batch-compile (Invalid column name 'Configuration' from an early migration) — script from the DB's current migration.
• sqlcmd needs -I (QUOTED_IDENTIFIER ON) or the filtered/covering index migrations fail with Msg 1934.
• Inside a psbridge exec single-quoted script, doubled '' quotes are eaten — use [char]39.
• Invoke-Command output must be piped through Out-String to come back through psbridge.

Verification

• /health/ready = 200 Healthy on both centrals (database + akka-cluster Up, memberCount 2, unreachable 0, all 5 required singletons) and both sites (akka-cluster Up, memberCount 2, localdb Healthy with replicationConnected=true, backlog 0).
• Zero [ERR]/[WRN]/[FTL] on any of the four node logs after the new boot (the [FTL] host terminated unexpectedly at 16:27:39 is the old build's stop).
• Feature proven on the QA data that motivated it: /deployment/instances/1/configure (Z28061, CvdReactor) now lists 264 data-sourced attributes in 8 groups — (instance) 16, DelmiaReceiver 10, LeftDelmiaReceiver 10, LeftMESReceiver 26, LeftReactorSide 83 (incl. LeakTest), RightDelmiaReceiver 10, RightMESReceiver 26, RightReactorSide 83 — all bound, every group's Object inferred and pre-filled (Z28061, DelmiaReceiver_043, Left_002.DelmiaReceiver, MESReceiver_023, Left_002, Right_002.DelmiaReceiver, MESReceiver_022, Right_002), Test Bindings (264). Before this build the page showed only the 16 direct attributes.

2. Rollback

• Binaries: on each box stop both services, rename app → app.failed, app.bak.20260820T162659 → app (same for cli), start Central then Site. The old build tolerates the new (additive) schema and ignores the added AuditLog key — but note it will again write auditlog.db CWD-relative into System32.
• Schema: restore ScadaBridge_pre-a24fa38e_20260820T162535.bak (not needed for a binary rollback).

3. Outstanding

• None from this deploy. ZIPSEN (id 7) remains the known non-deployable draft instance.

4. Hotfix da0bfe2b — same day, 16:53 (QA only so far)

Incident (reported on vq03 Z28061 MoveIn): SetAttribute('RightMESReceiver.MoveInCompleteFlag') failed: Session session-736b2d47… was not found while the attribute showed GOOD quality. Root cause (from both logs): at 15:29:03 the gateway dashboard user admin (client 10.181.200.169) closed ScadaBridge's MxGateway session. The gateway completes a closed session's event stream gracefully (OK status), and MxGatewayDataConnection.RunEventLoopAsync only raised Disconnected on a fault — so the adapter stayed Connected on a dead session for an hour (values frozen at last GOOD, every write NotFound; the 10 s NotFound heartbeat was the unresolved-tag resolution probe re-subscribing on the dead session). The 16:28 upgrade restart cleared it incidentally. A graceful gateway shutdown takes the same path.

Fix: da0bfe2b — a non-cancelled end of the event stream now signals Disconnected (pinned by MxGatewayDataConnectionReconnectTests.EventStreamGracefulCompletion_*). Deployed to all four QA nodes as a binary-only swap (no migrations, no config): backups app.bak.20260820T165357 / cli.bak.20260820T165357 on both boxes; /health/ready 200 Healthy, 2/2 members, site replication connected. Not yet on prod vd03. Live gate still to run: close the site's session from the gateway dashboard and confirm the site reconnects (MxGateway event stream ended without a fault … signalling disconnect → reconnect → values resume).