From 10c04a23d7a2d0443131c8dc10c3c330242a5016 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 22 Oct 2026 18:06:26 -0400 Subject: [PATCH] docs(deploy): record vd03 + QA rolling upgrade 3808b49f -> 0e540291 (no migration; QA one box at a time, no full outage) --- ...0-22-wonder-app-vd03-vq03-vq04-0e540291.md | 68 +++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 docs/deployment/deployment-records/2026-10-22-wonder-app-vd03-vq03-vq04-0e540291.md diff --git a/docs/deployment/deployment-records/2026-10-22-wonder-app-vd03-vq03-vq04-0e540291.md b/docs/deployment/deployment-records/2026-10-22-wonder-app-vd03-vq03-vq04-0e540291.md new file mode 100644 index 00000000..bce4ff6d --- /dev/null +++ b/docs/deployment/deployment-records/2026-10-22-wonder-app-vd03-vq03-vq04-0e540291.md @@ -0,0 +1,68 @@ +# 2026-10-22 (evening) — DEV vd03 + QA vq03/vq04 — `3808b49f` → `0e540291` + +Same-day follow-up to `2026-10-22-wonder-app-vd03-3808b49f.md` and `2026-10-22-wonder-app-vq03-vq04-3808b49f.md`. + +**Build:** `main` @ `0e540291` (= `origin/main`), ProductVersion `1.0.0+0e5402915ba7d1f17a39dc896b8ee19a5e9cb72f`. +- Published from a detached worktree with the same flags as before; 1028 app files and 223 cli files. +- Zip is 125 560 852 B, SHA-256 `B65FE228…7EF7D8D9B`; the hash was re-verified on every box. + +**Delta** (6 commits, 2 code): +- `6ed6800b` — the inbound-key import clears pooled SQLite handles before the `*.imported` rename, so Windows no longer refuses it. +- `0e540291` — the pending-updates rail clears as soon as a deployment finishes. + +## Pre-flight (breaking-change check) + +| Area | Finding | Verdict | +|---|---|---| +| Schema | No migrations in the delta; 58 before and after on both DBs | OK (asserted on the box, nothing applied) | +| Config | No appsettings or options changes; box appsettings kept and restored with hash matches | OK | +| Wire / LocalDb / Akka messages | Delta touches only `CentralUI` (pending-updates cache + indicator) and `Host/InboundApiKeyStoreImportService.cs` | Mixed `3808b49f`/`0e540291` pair is wire-identical, so a rolling upgrade is safe | +| Inbound-key import | QA's files were already renamed `*.imported` by hand, so the importer finds nothing; vd03 runs SQLite keys, no import | No-op on all three boxes | + +## vd03 (single node) + +- `sb-prep.ps1` (17:54:59): unpacked; reg export of both services; DB backup `E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-0e540291_20261022T175459.bak` (VERIFYONLY valid). +- `sb-swap.ps1`: Site stop 17:55:25 → Central ready 17:56:04 → Site ready **17:56:13**. **About 48 s of downtime.** Binary backups `app.bak.20261022T175525` / `cli.bak.20261022T175525`. + +**Verification:** +- `/health/ready` 200 on :8085 and :8084; live Host and CLI are `0e540291`. +- Snapshots: SBTEST01/02 Good 86, 5 alarms each, as before. Z28061/Z28062/Z28061Sim went Uncertain during startup, then back to Bad. That's the pre-existing MxGateway `AddItemBulk` timeouts. +- `docker/verify-mes-test-machine.sh` with the vd03 `RECIPE_*` env: **PASS 31/31**. +- Logs: the usual first-probe `akka-cluster … Removed` ERRs and the two known placeholder-seed startup Warnings. The EF `FirstOrDefault without OrderBy` Warning predates this build. + +## QA (vq03 + vq04) — rolling, one box at a time + +The delta is wire-identical, so this ran **one box at a time** instead of the pair-wise window that `3808b49f` needed. Each box: Site stop → Central stop (wait for the Host processes to exit) → swap → Central start → ready → Site start → ready. The script rolls back to the old binaries if the swap fails. + +Before the run, vq04 held the active central and vq03 the active site. vq04 went first: that meant two central handovers and only one site handover. + +| Step | Local time | +|---|---| +| Prep vq03 (incl. DB backup `F:\MSSQL\BACKUP\ScadaBridge_pre-0e540291_20261022T175848.bak`, verified; migrations 58) | 17:59:39 → 17:59:54 | +| Prep vq04 | 18:00:04 → 18:00:17 | +| vq04 box: Site stop 18:00:53, Central stop 18:01:24 (central handed over to vq03), swap, Central ready 18:01:51, Site ready | → 18:01:54 | +| Replication reconnected; 2 members / 0 unreachable on both clusters | 18:02:03 | +| vq03 box: Site stop 18:02:36 (site handed over to vq04), Central stop 18:02:47 (central handed over to vq04), swap, Central ready 18:03:10, Site ready | → 18:03:12 | + +**No full outage:** at every point one central and one site node were serving. vq04 is now active for both central and site; nothing depends on which node is active. + +The first vq04 attempt (18:00:31) aborted **before stopping anything**: vq04 has no `sqlcmd` for the migration-count assert. The assert now skips on a box without `sqlcmd`; the count was proven 58 from vq03. + +Binary backups `app.bak.20261022T175848` / `cli.bak.20261022T175848` and `_appsettings-keep-20261022T175848` on both boxes. + +**Verification:** +- `/health/ready` 200 on all four services; `/health/active` 200 on vq04 (central + site), 503 on vq03. +- Both clusters show 2 members / 0 unreachable; `localdb` `replicationConnected:true` on both site nodes. +- `security api-key list` returns the same 3 keys (`MESTestKey`, `MES_Key`, `delmia-notifier-qa`) on **both** centrals (shared SQL store). +- A bogus-key inbound call gets 401 on both nodes (active answers, standby relays). +- Snapshots Z28061–Z28069 + ZIPSEN are **identical** before and after (242 Good / 2 Bad; alarms 16, or 10 on Z28062). +- Logs: each service stop logs the pre-existing `[FTL] host terminated unexpectedly` (an `OperationCanceledException` in `WindowsServiceLifetime.StopAsync`, after `actor system shutdown complete`). The same line appears at every stop today (06:43, 15:30), so it isn't new. + +## Rollback + +On any box: stop Site then Central, rename `app` → `app.failed` and `app.bak.` → `app` (same for `cli`), start Central then Site. vd03 `` = `20261022T175525`; QA `` = `20261022T175848`. No DB change, so there's nothing to restore. Since the two versions are wire-identical, QA can roll back one box at a time too. + +## Outstanding + +- Housekeeping after soak: delete older `app.bak.*` / `cli.bak.*` / `_appsettings-keep-*` and `E:\psrem-stage\*` folders on all three boxes, plus the QA `*.imported` key files. +- Follow-up candidate: the `[FTL] host terminated unexpectedly` line on a normal service stop. `WindowsServiceLifetime.StopAsync` gets a cancelled token after shutdown has already finished, so a routine stop logs as Fatal.