docs(deploy): record QA vq03/vq04 0e540291 -> 8e32f353, migrations 58->64, #116 leader-auto-down (#107, #116)

Pair window per pack 8ffa6394: COPY_ONLY backup + VERIFYONLY, stop both pairs,
idempotent EF script 58->64 from the Mac (flip in ~2 s), swap, centrals then
sites started together. Site data path ~2 min 6 s, central API ~58 s.
leader-auto-down started on all 4 nodes; no #116 drills; hash chain off;
recovery flag TRUE on all 4 (actions kept). Env digest-identical; only the
strategy key changed in appsettings; per-node JWT/idle values preserved.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

docs/deployment/deployment-records/2026-11-13-wonder-app-vq03-vq04-8e32f353.md

@@ -0,0 +1,117 @@
+# 2026-11-13 — QA (wonder-app-vq03 + vq04) `0e540291` → `8e32f353`, migrations 58 → 64, #116 leader-auto-down
+
+**Owner go (2026-11-13, "just deploy"):** the pack `8ffa6394` (`docs/deployment/deploy-prep/2026-11-13-qa-vq03-vq04/`), deploy only.
+- **In:** the upgrade, the six migrations, the #116 `leader-auto-down` flip and the exit-75 service recovery.
+- **Out:** no #116 drills (G14 not run), hash chain and head export stay **off** (`AuditLog__HashChain__*` not set), #52 not bundled, Z28061 not deployed to, written or called.
+
+**Build:** `main` @ `8e32f353`, ProductVersion `1.0.0+8e32f3538ce940e349be3701efd0784a7ebeb551`. This is the same zip DEV received (`2026-11-13-wonder-app-vd03-8e32f353.md`), pulled from vd03's stage over SMB: 130 520 104 B, SHA-256 `0cf06cf5cfe5c1a3da2f1c6ce8db77b43f22cbcb79e5237abc4fe51cbc7fdebc`, re-verified on each box. App 1050 files, cli 224.
+
+**Transport:** `winrm` (deploy scope) for one rendered script per step (`prep`, `stop-site`, `stop-central`, `swap`, `start-central`, `start-site`, `verify`), SMB for staging, logs and appsettings, direct SQL from the Mac for the backup, migration and schema checks, CLI and HTTP from the Mac for the gates. Stage `E:\psrem-stage\sb-8e32f353\` on both boxes; on-box notes `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-8e32f353-20261113T080106.txt`.
+
+**Not touched:** OtOpcUa, mxaccessgw, HistorianGateway, Overview, the OTel collector, the `ScadaBridge-LDAP` (GLAuth) service, DEV vd03 and the local docker rigs.
+
+## Preflight (read-only, 07:55–08:01)
+
+| Check | Result |
+|---|---|
+| Live build | Host + CLI `0e540291` on both boxes (= pack §0) |
+| `preflight.sql` (wonder-sql-vq03) | 58 migrations, last `20261022133907_ReviewRemediationIndexes`; FULL recovery, `log_reuse_wait` NOTHING; AuditLog 3 441 rows / 18.3 MB; clustered key `EventId`-first; log 89 MB used of 199 (44.9 %); no chain objects; roles `_writer` + `_purger` only. Matches §3 apart from log usage |
+| Migration script | SHA-256 `488da93d…c34ecc0` (= pack) |
+| `ASPNETCORE_URLS` (#108) | no `https://` on either Central |
+| Service env | Central 15 entries, Site 8 entries, **identical digests on both boxes** (name + SHA-256 prefix only). `LocalDb__Replication__ApiKey` present on both Site services, same prefix `19fd2d0910cd` |
+| Service recovery | All four services already had restart 5 s / 5 s / 10 s, reset 86 400 s, and `FAILURE_ACTIONS_ON_NONCRASH_FAILURES: TRUE` |
+| appsettings (live) | `appsettings.json` `C0C653D6…` (both); `appsettings.Central.json` vq03 `4BB3385F…`, vq04 `C1D77243…`; `appsettings.Site.json` vq03 `4CBAF66C…`, vq04 `D0C1AF05…` |
+| Central drift (kept, not normalised) | vq03 `JwtExpiryMinutes` 25 / `IdleTimeoutMinutes` 60; vq04 15 / 30 |
+| Health | all four `/health/ready` 200; vq03 active on both pairs, vq04 503; central memberCount 2, leader vq03 |
+| Baseline | Z28062–Z28069: 244 Good / 0 Bad each (1 952 Good), 10 alarms each. Site `MxGateway` Connected, 2196/2196 resolved, 2196 good. `security api-key list` on both centrals: `MESTestKey`, `MES_Key`, `delmia-notifier-qa` |
+| Disk | vq03 E: 212 GB, vq04 E: 228 GB; SQL F: 249 GB, G: 246 GB free |
+
+**#116 edit, prepared on the Mac:** one byte-exact replacement per file, `"auto-down"` → `"leader-auto-down"` on `ScadaBridge:Cluster:SplitBrainResolverStrategy`, encodings kept (Central no BOM, Site BOM + CRLF). A key-level digest diff (every flattened JSON key, SHA-256 prefix of the value) shows that key as the only change in each file. Staged as `E:\psrem-stage\sb-8e32f353\leader-auto-down\`.
+
+| Box | `appsettings.Central.json` after | `appsettings.Site.json` after |
+|---|---|---|
+| vq03 | `EF01B129…` | `D0E202F9…` |
+| vq04 | `F333D48F…` | `E9E6336F…` |
+
+**Prep step (08:01, no downtime):** staged zip hash + ProductVersions verified, live appsettings checked against the inspected hashes, `reg export` of both service keys (`_envbak-ScadaBridge-{Central,Site}-20261113T080106.reg` in the stage), appsettings kept in `E:\ApiInstall\ScadaBridge\_appsettings-keep-20261113T080106\` (hash-verified).
+
+## Window
+
+| Step | Local time (EST) |
+|---|---|
+| W0 backup (from the Mac): `F:\MSSQL\BACKUP\ScadaBridge_pre-8e32f353_20261113T080106.bak` on wonder-sql-vq03, `COPY_ONLY, CHECKSUM, COMPRESSION`, 153 995 429 B compressed, `RESTORE VERIFYONLY … WITH CHECKSUM` valid | 08:02:08 → 08:02:12 |
+| Stop Site vq04 (standby; **window opens**) | 08:02:46 → 08:03:16 |
+| Stop Site vq03 (active; **site data path down**) | 08:03:28 → 08:03:38 |
+| Stop Central vq04 | 08:03:54 → 08:04:04 |
+| Stop Central vq03 (**central API down**); no `Host.exe` left on either box | 08:04:15 → 08:04:26 |
+| W3 migrate from the Mac: `sqlcmd -I -b` with `SET XACT_ABORT ON` + `migrate-58-to-64.idempotent.sql`. **64**, last `20261110120000_AuditMaintenanceGrants`, clustered key `OccurredAtUtc`-first | 08:04:27 → 08:04:29 |
+| W4–W6 swap on both boxes (parallel): `failureflag 1` re-asserted on all four services (existing actions kept), `app`/`cli` → `*.bak.20261113T080106`, new build in, three kept appsettings restored (hash match), then the #116 files copied in (hash match) | 08:04:39 → 08:04:51 |
+| W7 start both Centrals (vq03, vq04 six seconds later) → both ready | 08:05:03 → 08:05:15 |
+| W8 start both Sites → both ready (**window closes**) | 08:05:27 → 08:05:36 |
+
+**Downtime:** site data path **~2 min 6 s** (08:03:28 → 08:05:34); central API **~58 s** (08:04:15 → 08:05:13); whole window 2 min 50 s. Most of the gaps between steps are the ~10–15 s WinRM session set-up per call.
+
+The window's stops ran the **old** binaries, as the pack predicted (§1.2):
+- vq04's Site stop took 30 s, waiting out the drain on the peer's LocalDb sync stream (#140 fixes this from the new build on).
+- That stop logged the old `[FTL] ScadaBridge host terminated unexpectedly` at 08:03:26 after `Akka.NET actor system shutdown complete` (#76, fixed in the new build).
+
+The migration printed the known 1 700-byte key-length warnings for `IX_AuditLog_Site_Occurred`, `IX_AuditLog_Channel_Status_Occurred` and `IX_AuditLog_Kind_Occurred`. They are benign: these columns hold short values.
+
+## Gates
+
+| Gate | Result |
+|---|---|
+| G1 Build | Host + CLI `1.0.0+8e32f353…` on both boxes (swap log and post-deploy inspect) |
+| G2 Schema | 64 migrations; AuditLog clustered key `(OccurredAtUtc, EventId)`; `AuditLogChainHead` + `AuditLogChainTombstone` exist (0 rows each); roles `scadabridge_audit_writer`, `_purger`, `_chainer` |
+| G3 Health | `/health/ready` 200 ×4. Central: `database`, `akka-cluster`, `required-singletons`, `bundle-signing-policy`, `audit-chain-head-export` ("Chain-head export disabled") Healthy. Site: `akka-cluster`, `localdb` Healthy |
+| G4 Cluster | Central memberCount 2, unreachable 0, leader vq03 (`10.220.158.2:8081`) on both nodes. Site memberCount 2, leader vq03 (`:8082`). **LocalDb `replicationConnected: true`, backlog 0 on both site nodes**; the sync ApiKey prefix `19fd2d0910cd` is the same on both and unchanged |
+| G5 Active node | `/health/active` sampled 10×, 1 s apart: vq03 200 / vq04 503 for both pairs at every sample (exactly one active per pair) |
+| G6 Boot logs (#116) | `[leader-auto-down] started on akka.tcp://scadabridge@10.220.158.{2,3}:{8081,8082}` on **all four** nodes (13:05:07Z, 13:05:14Z, 13:05:28Z, 13:05:35Z), each with `shut-out restart is ON`; vq03 logged `gained leadership` for both pairs. No `keep-majority` / `auto-down-unreachable-after` line in today's logs. ERR/FTL after start: only the usual first-probe `akka-cluster … Removed` lines on vq03 (5 central, 7 site, 08:05:08–08:05:33); none on vq04 after start |
+| G7 CLI | `POST /auth/token` 200 on both centrals; `/management` without credentials 401 on both; `security api-key list` on **both** centrals: `MESTestKey`, `MES_Key`, `delmia-notifier-qa` |
+| G8 Inbound API | A bogus `X-API-Key` on each node gets 401 `UNAUTHORIZED`, vq04 through the standby relay. `flow-debug run-method --method-id 9` (`SimpleAlarmStatusRequest`, `SAPID=000000`): InboundRequest (113 ms) → DbCall BTDB 1 row → Response Ok |
+| G9 MxGateway | Site health: `MxGateway` Connected, 2196/2196 resolved, good 2196 / bad 0 (= baseline); 9 deployed instances. Z28062–Z28069: 244 Good each, 10 alarms each (= baseline) |
+| G10 AuditLog | Rows after the window from `central-a`: InboundRequest 1, DbWrite 1, ScriptLog 8, InboundAuthFailure 2 (the two bogus calls), `SourceNode` set, **`ChainSeq` NULL on all 3 441+ rows** (chain off). Site audit backlog 0 |
+| G11 Overview | `:5320` on vq03 and vq04: 12/12 Up; ScadaBridge 4 instances / 2 clusters, central-a (vq03) Active + leader chip, central-b (vq04) Standby, 5/5 checks; no "Split brain" |
+| G12 Artifacts / staleness | `health pending-updates`: `hasPendingUpdates false`, 0 stale instances, site-a artifacts Current, hash unchanged `sha256:d79ba1a3…` (deployed 2026-10-27). No redeploy |
+| G13 OTLP | Box-local collector `logs.jsonl` now holds **logs** with `service.name scadabridge`, `service.version 1.0.0+8e32f353…`: vq03 Central 1 675 / Site 808 records, vq04 Central 762 / Site 537 |
+| ~~G14 #116 drill~~ | **Not run** (owner's go). #116: QA is on leader-auto-down, **not drilled** |
+| G15 Recovery | `sc qfailure` / `qfailureflag` on all four: restart 5 000 / 5 000 / 10 000 ms, reset 86 400 s, `FAILURE_ACTIONS_ON_NONCRASH_FAILURES: TRUE` |
+
+**Env and appsettings, before vs after (key name + SHA-256 prefix of the value only):**
+- **Service env:** identical on both boxes (23 entries per box: Central 15, Site 8). `AuditLog__HashChain__Enabled` is not set.
+- **appsettings:** the only changed key is `ScadaBridge:Cluster:SplitBrainResolverStrategy` (`77d79bbc1fb2` = `"auto-down"` → `ea78c76eff79` = `"leader-auto-down"`), in `appsettings.Central.json` and `appsettings.Site.json` on both boxes. `appsettings.json` is byte-identical.
+- **Per-node values preserved:** `JwtExpiryMinutes` / `IdleTimeoutMinutes` are still 25 / 60 on vq03 and 15 / 30 on vq04.
+
+## Deviations from the pack
+
+1. **No new build:** the DEV zip (same SHA, same hash as the DEV record) was reused instead of a fresh publish (§5.1 step 1).
+2. **Service recovery:** §5.3 / G15 call for `sc.exe failure … restart/10000/restart/30000/restart/60000`. As on DEV, the existing actions (5 s / 5 s / 10 s, reset 1 day) were kept, and only `failureflag 1` was re-asserted. It was already TRUE on all four services, so the service definitions are unchanged. Any non-zero exit (75 included) restarts the node.
+3. **#116 files:** the edited files were staged over SMB before the window and copied in by the swap script, with hash checks, after the kept originals were restored. The pack has them pushed over SMB while the services are stopped. The outcome is the same, and both the originals and the edited copies are hash-pinned.
+4. **Start order:** vq03 Central was started about 6 s before vq04 (and the same for the Sites), not simultaneously. vq03 stayed the leader and active node on both pairs.
+
+## Rollback
+
+Schema 64 cannot run under `0e540291`, so a full rollback is **binaries + DB restore**, never binaries alone. Never roll back past `0e540291`.
+
+1. On **each** box: `winrm --timeout 900 ps <box> -f qa-rb-swap.ps1` (in `E:\psrem-stage\sb-8e32f353\`). It stops Site then Central, waits for no `Host.exe`, renames `app`/`cli` → `*.failed.<now>` and `*.bak.20261113T080106` → `app`/`cli`, and restores the pre-window appsettings from `_appsettings-keep-20261113T080106` (hash-checked; strategy back to `auto-down`). Services are left **stopped**.
+2. From the Mac, with all four stopped:
+   ```sql
+   ALTER DATABASE [ScadaBridge] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
+   RESTORE DATABASE [ScadaBridge] FROM DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-8e32f353_20261113T080106.bak' WITH REPLACE, CHECKSUM;
+   ALTER DATABASE [ScadaBridge] SET MULTI_USER;
+   ```
+   This loses every central DB write since 08:02.
+3. Start both Centrals together (`qa-start-central.ps1` on each box), then both Sites (`qa-start-site.ps1`). Confirm `0e540291`, 58 migrations and health.
+
+- **#116 only** (keep the new build): copy `appsettings.Central.json` and `appsettings.Site.json` back from `_appsettings-keep-20261113T080106` on both boxes, then stop and start **each pair together**.
+- **Recovery flag:** it stays TRUE on either build.
+- **Env:** the `.reg` exports are only needed if the env is found changed. The deploy did not change it.
+
+`app.bak.20261113T080106`, `cli.bak.20261113T080106` and `_appsettings-keep-20261113T080106` can go once the build has soaked. The older `*.bak.20261022T175848` folders are still on both boxes.
+
+## Follow-ups
+
+- **#116:** central and site-pair kill drills on QA, which need an owner slot (the site drill re-hosts Z28061).
+- **#107 §7:** hash chain on QA, later and separately. This build has the #139 cutoff. Run the SourceNode backfill first (843 NULL rows).
+- **#52:** still open on QA and needs a Z28061 slot.
+- **Drift:** the JWT / idle-timeout difference between the centrals is unchanged, as instructed.