docs(deployment): record QA upgrade of wonder-app-vq03 + vq04 (all 4 nodes) to a24fa38e — 6 EF migrations, AuditLog:SiteWriter path added, System32 auditlog.db relocated; grouped bindings verified on Z28061 (264 rows / 8 groups)

2026-08-20 16:31:57 -04:00
parent a24fa38eae
commit 5b9cf772db
1 changed files with 61 additions and 0 deletions
@@ -0,0 +1,61 @@
+# Deployment record — wonder-app-vq03 + wonder-app-vq04 (QA) — 2026-08-20
+
+Point-in-time record of a ScadaBridge binary + schema upgrade of the **QA two-host
+topology**: a real central pair (`central-a` = wonder-app-vq03 `10.220.158.2`, `central-b` =
+wonder-app-vq04 `10.220.158.3`, :8085) and a site-a pair (`node-a`/`node-b` on the same two
+boxes, gRPC :8083, metrics :8084), each box running `ScadaBridge-Central`, `ScadaBridge-Site`
+and `ScadaBridge-LDAP` out of `E:\ApiInstall\ScadaBridge`. Shared ConfigDb on
+`wonder-sql-vq03`. Performed from the Mac via **psbridge** on vq04 (`:6150`, root `E:\psrem`);
+vq03 reached from vq04 over WinRM (`Invoke-Command`) and the `E$` admin share.
+
+## 1. Binary + schema upgrade
+
+| Item | Value |
+|---|---|
+| Build deployed | `main` @ **`a24fa38e`** (from `d4bb4e17`, 2026-08-12; 246 commits) — all four nodes |
+| Headline change | **Grouped connection bindings** (`docs/plans/2026-08-20-grouped-connection-bindings-design.md`): the Configure page sources its binding rows from the flattened configuration, grouped per module with a per-group Object apply; CLI `instance bind-module`. Plus everything since 08-12 (arch-review Medium/Low waves, LocalDb 0.3.2→0.3.4, gRPC stream batching, bulk-deploy up-to-date skip, …). |
+| Publish | `dotnet publish -c Release -r win-x64 --self-contained true` for Host (→ `app\`, 520 files) and CLI (→ `cli\`), zipped and pushed via `psbridge push` (SHA-256 verified), unpacked under `E:\psrem\releases\a24fa38e\`, mirrored to `\\wonder-app-vq03\E$\psrem-stage\a24fa38e` with robocopy. |
+| EF migrations applied | 6, via `sqlcmd -C -b -I` on vq03 from an **idempotent delta script** (`dotnet ef migrations script 20260810100836_AddSmtpConfigurationTransport --idempotent`): `20260814235335_AddAuditLogAndNotificationCoveringIndexes`, `20260815004957_AlignAuditLogEventIdUniqueness`, `20260815154803_AddBundleUnlockAttemptTable`, `20260815165452_AddDeploymentRecordListSortIndex`, `20260819183326_AddSiteCallsListSortIndex`, `20260819201210_AddNotificationsListSortIndex`. All additive (indexes + one table + one index drop). |
+| New required config | **`AuditLog:SiteWriter:DatabasePath`** (top-level section, Site nodes; `StartupValidator` refuses to boot without it since `2e4e41a8`) — added to `appsettings.Site.json` on both boxes as `E:\ApiInstall\ScadaBridge\site\data\auditlog.db`. The only appsettings delta across the 246 commits. |
+| Data relocated | The site audit SQLite file had been written CWD-relative, i.e. to **`C:\Windows\System32\auditlog.db`** (the exact hazard `2e4e41a8` closed). Moved, with its `-wal`/`-shm`, into `site\data\` on both boxes while the Site service was stopped. |
+| Preserved across swap | box `appsettings.json` / `appsettings.Central.json` / `appsettings.Site.json` (copied back from the backup; service env vars untouched), `central\data`, `site\data` (secrets.db, site-localdb.db). |
+| DB backup | `ScadaBridge_pre-a24fa38e_20260820T162535.bak` (COPY_ONLY, SQL Server default backup dir on wonder-sql-vq03) |
+| Binary backup | `E:\ApiInstall\ScadaBridge\app.bak.20260820T162659` + `cli.bak.20260820T162659` on each box (old `d4bb4e17` binaries + appsettings) |
+
+### Sequence
+Stop Site pair (both boxes) → stop Central pair → swap `app\`/`cli\` on vq04 then vq03 →
+start Central pair → wait 20 s → start Site pair. Pairs were stopped and started **together**
+(required since LocalDb Phase 2 — no rolling upgrade of a site pair).
+
+### Pitfalls hit (for next time)
+- A **full-history** `--idempotent` script fails on SQL Server batch-compile (`Invalid column
+  name 'Configuration'` from an early migration) — script **from the DB's current migration**.
+- `sqlcmd` needs **`-I`** (QUOTED_IDENTIFIER ON) or the filtered/covering index migrations fail
+  with Msg 1934.
+- Inside a `psbridge exec` single-quoted script, doubled `''` quotes are eaten — use `[char]39`.
+- `Invoke-Command` output must be piped through `Out-String` to come back through psbridge.
+
+### Verification
+- `/health/ready` = **200 Healthy** on both centrals (database + akka-cluster Up, memberCount 2,
+  unreachable 0, all 5 required singletons) and both sites (akka-cluster Up, memberCount 2,
+  `localdb` Healthy with `replicationConnected=true`, backlog 0).
+- Zero `[ERR]`/`[WRN]`/`[FTL]` on any of the four node logs after the new boot (the `[FTL]
+  host terminated unexpectedly` at 16:27:39 is the *old* build's stop).
+- **Feature proven on the QA data that motivated it:** `/deployment/instances/1/configure`
+  (Z28061, CvdReactor) now lists **264 data-sourced attributes in 8 groups** — `(instance)` 16,
+  `DelmiaReceiver` 10, `LeftDelmiaReceiver` 10, `LeftMESReceiver` 26, `LeftReactorSide` 83
+  (incl. `LeakTest`), `RightDelmiaReceiver` 10, `RightMESReceiver` 26, `RightReactorSide` 83 —
+  all bound, every group's Object inferred and pre-filled (`Z28061`, `DelmiaReceiver_043`,
+  `Left_002.DelmiaReceiver`, `MESReceiver_023`, `Left_002`, `Right_002.DelmiaReceiver`,
+  `MESReceiver_022`, `Right_002`), `Test Bindings (264)`. Before this build the page showed
+  only the 16 direct attributes.
+
+## 2. Rollback
+- Binaries: on each box stop both services, rename `app` → `app.failed`, `app.bak.20260820T162659`
+  → `app` (same for `cli`), start Central then Site. The old build tolerates the new (additive)
+  schema and ignores the added `AuditLog` key — but note it will again write `auditlog.db`
+  CWD-relative into System32.
+- Schema: restore `ScadaBridge_pre-a24fa38e_20260820T162535.bak` (not needed for a binary rollback).
+
+## 3. Outstanding
+- None from this deploy. ZIPSEN (id 7) remains the known non-deployable draft instance.