docs(deploy): record vd03 + QA rolling upgrade 3808b49f -> 0e540291 (no migration; QA one box at a time, no full outage)

2026-10-22 18:06:26 -04:00
parent 0e5402915b
commit 10c04a23d7
1 changed files with 68 additions and 0 deletions
@@ -0,0 +1,68 @@
+# 2026-10-22 (evening) — DEV vd03 + QA vq03/vq04 — `3808b49f` → `0e540291`
+
+Same-day follow-up to `2026-10-22-wonder-app-vd03-3808b49f.md` and `2026-10-22-wonder-app-vq03-vq04-3808b49f.md`.
+
+**Build:** `main` @ `0e540291` (= `origin/main`), ProductVersion `1.0.0+0e5402915ba7d1f17a39dc896b8ee19a5e9cb72f`.
+- Published from a detached worktree with the same flags as before; 1028 app files and 223 cli files.
+- Zip is 125 560 852 B, SHA-256 `B65FE228…7EF7D8D9B`; the hash was re-verified on every box.
+
+**Delta** (6 commits, 2 code):
+- `6ed6800b` — the inbound-key import clears pooled SQLite handles before the `*.imported` rename, so Windows no longer refuses it.
+- `0e540291` — the pending-updates rail clears as soon as a deployment finishes.
+
+## Pre-flight (breaking-change check)
+
+| Area | Finding | Verdict |
+|---|---|---|
+| Schema | No migrations in the delta; 58 before and after on both DBs | OK (asserted on the box, nothing applied) |
+| Config | No appsettings or options changes; box appsettings kept and restored with hash matches | OK |
+| Wire / LocalDb / Akka messages | Delta touches only `CentralUI` (pending-updates cache + indicator) and `Host/InboundApiKeyStoreImportService.cs` | Mixed `3808b49f`/`0e540291` pair is wire-identical, so a rolling upgrade is safe |
+| Inbound-key import | QA's files were already renamed `*.imported` by hand, so the importer finds nothing; vd03 runs SQLite keys, no import | No-op on all three boxes |
+
+## vd03 (single node)
+
+- `sb-prep.ps1` (17:54:59): unpacked; reg export of both services; DB backup `E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-0e540291_20261022T175459.bak` (VERIFYONLY valid).
+- `sb-swap.ps1`: Site stop 17:55:25 → Central ready 17:56:04 → Site ready **17:56:13**. **About 48 s of downtime.** Binary backups `app.bak.20261022T175525` / `cli.bak.20261022T175525`.
+
+**Verification:**
+- `/health/ready` 200 on :8085 and :8084; live Host and CLI are `0e540291`.
+- Snapshots: SBTEST01/02 Good 86, 5 alarms each, as before. Z28061/Z28062/Z28061Sim went Uncertain during startup, then back to Bad. That's the pre-existing MxGateway `AddItemBulk` timeouts.
+- `docker/verify-mes-test-machine.sh` with the vd03 `RECIPE_*` env: **PASS 31/31**.
+- Logs: the usual first-probe `akka-cluster … Removed` ERRs and the two known placeholder-seed startup Warnings. The EF `FirstOrDefault without OrderBy` Warning predates this build.
+
+## QA (vq03 + vq04) — rolling, one box at a time
+
+The delta is wire-identical, so this ran **one box at a time** instead of the pair-wise window that `3808b49f` needed. Each box: Site stop → Central stop (wait for the Host processes to exit) → swap → Central start → ready → Site start → ready. The script rolls back to the old binaries if the swap fails.
+
+Before the run, vq04 held the active central and vq03 the active site. vq04 went first: that meant two central handovers and only one site handover.
+
+| Step | Local time |
+|---|---|
+| Prep vq03 (incl. DB backup `F:\MSSQL\BACKUP\ScadaBridge_pre-0e540291_20261022T175848.bak`, verified; migrations 58) | 17:59:39 → 17:59:54 |
+| Prep vq04 | 18:00:04 → 18:00:17 |
+| vq04 box: Site stop 18:00:53, Central stop 18:01:24 (central handed over to vq03), swap, Central ready 18:01:51, Site ready | → 18:01:54 |
+| Replication reconnected; 2 members / 0 unreachable on both clusters | 18:02:03 |
+| vq03 box: Site stop 18:02:36 (site handed over to vq04), Central stop 18:02:47 (central handed over to vq04), swap, Central ready 18:03:10, Site ready | → 18:03:12 |
+
+**No full outage:** at every point one central and one site node were serving. vq04 is now active for both central and site; nothing depends on which node is active.
+
+The first vq04 attempt (18:00:31) aborted **before stopping anything**: vq04 has no `sqlcmd` for the migration-count assert. The assert now skips on a box without `sqlcmd`; the count was proven 58 from vq03.
+
+Binary backups `app.bak.20261022T175848` / `cli.bak.20261022T175848` and `_appsettings-keep-20261022T175848` on both boxes.
+
+**Verification:**
+- `/health/ready` 200 on all four services; `/health/active` 200 on vq04 (central + site), 503 on vq03.
+- Both clusters show 2 members / 0 unreachable; `localdb` `replicationConnected:true` on both site nodes.
+- `security api-key list` returns the same 3 keys (`MESTestKey`, `MES_Key`, `delmia-notifier-qa`) on **both** centrals (shared SQL store).
+- A bogus-key inbound call gets 401 on both nodes (active answers, standby relays).
+- Snapshots Z28061–Z28069 + ZIPSEN are **identical** before and after (242 Good / 2 Bad; alarms 16, or 10 on Z28062).
+- Logs: each service stop logs the pre-existing `[FTL] host terminated unexpectedly` (an `OperationCanceledException` in `WindowsServiceLifetime.StopAsync`, after `actor system shutdown complete`). The same line appears at every stop today (06:43, 15:30), so it isn't new.
+
+## Rollback
+
+On any box: stop Site then Central, rename `app` → `app.failed` and `app.bak.<ts>` → `app` (same for `cli`), start Central then Site. vd03 `<ts>` = `20261022T175525`; QA `<ts>` = `20261022T175848`. No DB change, so there's nothing to restore. Since the two versions are wire-identical, QA can roll back one box at a time too.
+
+## Outstanding
+
+- Housekeeping after soak: delete older `app.bak.*` / `cli.bak.*` / `_appsettings-keep-*` and `E:\psrem-stage\*` folders on all three boxes, plus the QA `*.imported` key files.
+- Follow-up candidate: the `[FTL] host terminated unexpectedly` line on a normal service stop. `WindowsServiceLifetime.StopAsync` gets a cancelled token after shutdown has already finished, so a routine stop logs as Fatal.