From 5ad6f8317674e997e06a9613b8eda8a25ce0313e Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 22 Oct 2026 15:38:27 -0400 Subject: [PATCH] docs(deploy): record QA wonder-app-vq03/vq04 upgrade b3021e7c -> 3808b49f (migration 58, pair-wise window 2m35s) --- ...026-10-22-wonder-app-vq03-vq04-3808b49f.md | 98 +++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 docs/deployment/deployment-records/2026-10-22-wonder-app-vq03-vq04-3808b49f.md diff --git a/docs/deployment/deployment-records/2026-10-22-wonder-app-vq03-vq04-3808b49f.md b/docs/deployment/deployment-records/2026-10-22-wonder-app-vq03-vq04-3808b49f.md new file mode 100644 index 00000000..48dc3dd9 --- /dev/null +++ b/docs/deployment/deployment-records/2026-10-22-wonder-app-vq03-vq04-3808b49f.md @@ -0,0 +1,98 @@ +# Deployment record — wonder-app-vq03 + wonder-app-vq04 (QA) — 2026-10-22 — `b3021e7c` → `3808b49f` + +This record covers the **QA two-host topology**: + +| Box | Central node | Site-a node | +|---|---|---| +| vq03 `10.220.158.2` | central-a | node-a | +| vq04 `10.220.158.3` | central-b | node-b | + +- Centrals on :8085; site gRPC on :8083, health on :8084. +- LocalDb replication is on. +- The shared config DB `ScadaBridge` is on `wonder-sql-vq03`. + +The package is the one DEV received the same day. See `2026-10-22-wonder-app-vd03-3808b49f.md` for the build delta: 191 commits, including site secrets via Deploy Artifacts, the 2026-10-21 review remediation, and Topology #38/#39/#40. + +The deploy ran from the Mac with the `winrm` CLI. The zip and SQL went over SMB, followed by one prep script per box and seven window calls, all from one mode-switched script. + +## Pre-flight (breaking-change check) + +| Area | Change in delta | QA state | Verdict | +|---|---|---|---| +| Schema | 57 → 58 `ReviewRemediationIndexes` (5 additive indexes) | 57; AuditLog 2 132 rows, AuditLogEntries 515; `ps_AuditLog_Month` present; Enterprise | OK | +| `Secrets:Replication:Enabled` ≠ false → startup failure | — | `false` in appsettings.json on both boxes; not in the service env | OK (stale `Mode`/`GrpcHub` keys log one ignore-Warning) | +| `JwtSigningKey` ≥ 32 bytes; must match on both centrals now that the CLI uses bearer tokens | — | 64 chars on both; identical hash | OK | +| `MachineDataDb`, `Transport:SchemaVersionMajor`, `JwtRefreshThresholdMinutes` removed | — | `MachineDataDb` still in the Central env and appsettings; the others are unset | Ignored, harmless | +| Actor-name deploy gate | — | 33 instance, script, alarm, native-source and connection names | All valid | +| LocalDb replicated set 10 → 12 tables (`deployed_secrets`, `secret_mirror_status`) | Registered-table digests are compared fail-closed | Both site nodes upgraded in one window | OK; the pair reconnected, backlog 0 | +| Data-connection `secret:` references | Artifact deploy resolves them centrally | The artifact hash is unchanged after the deploy, so there are no references | OK | +| Service env | — | Re-exported during prep; identical to the 2026-09-28 export on all four services | OK | + +**No appsettings change.** The three files were kept and restored on each box, with matching hashes: + +| Box | appsettings.json | appsettings.Central.json | appsettings.Site.json | +|---|---|---|---| +| vq03 | `F94000B1` | `D1320FA8` | `4CBAF66C` | +| vq04 | `F94000B1` | `2FD3EC4C` | `D0C1AF05` | + +## Upgrade + +| Item | Value | +|---|---| +| Build | `main` @ **`3808b49f`**, ProductVersion `1.0.0+3808b49fc7558a177219d00eac97747b28850a80`, all four nodes | +| Artifact | `E:\psrem-stage\3808b49f\scadabridge.zip` on both boxes; 125 543 156 B; SHA-256 `770BB992…5FB1098E47`, re-verified on each box; app 1028 files, cli 223 | +| Migration | `ReviewRemediationIndexes.sql` (idempotent) applied with `sqlcmd -C -b -I` from vq03 while both centrals were stopped; 57 → 58. SQL Server warned that `IX_AuditLog_Kind_Occurred` has a maximum key of 8 008 bytes against the 1 700-byte limit. `Kind` holds short enum names, so this is benign | +| DB backup | `F:\MSSQL\BACKUP\ScadaBridge_pre-3808b49f_20261022T152457.bak` on wonder-sql-vq03 (COPY_ONLY, CHECKSUM, COMPRESSION; VERIFYONLY valid) | +| Binary backups | `app.bak.20261022T152457` / `cli.bak.20261022T152457` plus `_appsettings-keep-20261022T152457` on both boxes | +| Notes on box | `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-3808b49f.txt` | + +| Step | Local time | +|---|---| +| Stop Site vq04 (**window opens**) | 15:29:58 → 15:30:28 | +| Stop Site vq03 | 15:30:37 → 15:30:47 | +| Stop Central vq04, swap vq04 | 15:30:55 → 15:31:12 | +| Stop Central vq03, migrate, swap vq03, start Central vq03 → ready | 15:31:19 → 15:31:49 | +| Start Central vq04 → ready | 15:32:00 → 15:32:05 | +| Start Site vq03 → ready | 15:32:12 → 15:32:21 | +| Start Site vq04 → ready (**window closes**) | 15:32:30 → 15:32:33 | + +**Downtime: 2 min 35 s.** + +## Verification (from the Mac) + +- **Centrals:** `/health/ready` 200 on both. `database`, `akka-cluster` (memberCount 2, unreachable 0) and `required-singletons` are all Healthy. `/health/active` is 200 on vq03 and 503 on vq04, the same active node as before. +- **Sites:** `/health/ready` 200 on both. `localdb` shows `replicationConnected:true`, backlog 0. `/health/active` is 200 on vq03 and 503 on vq04. +- **Auth:** + - `POST /auth/token` returns 200; the CLI now runs on bearer tokens against QA. + - `/management` without credentials returns 401, unchanged. +- **Snapshots:** + - Z28061–Z28069: 242 Good / 2 Bad on both sides of the window. Z28063 showed 114 Uncertain on the first read while its subscriptions settled, then matched. + - Alarm counts are unchanged (14, or 8 on Z28062). + - ZIPSEN is not deployed, as before. +- **Deploy Artifacts:** deployment `d7505cd8…`, site-a success. The hash is unchanged (`sha256:d79ba1a3…`); `ArtifactsDeployedAt` is 19:32:56Z. +- **Logs** (pulled over SMB): + - The usual first-probe `akka-cluster … Removed` ERR lines. + - vq03 site logged four `Replication session … faulted` Warnings between 15:32:16 and 15:32:29, while vq04's site was still down. The session has been connected since. + - New startup Warnings: Secrets per-node SQLite on a clustered central, and the stale `Secrets:Replication`/`GrpcHub` keys. + - No other ERR or FTL lines. +- **Smoke:** `flow-debug run-method` of `SimpleAlarmStatusRequest` (id 9) with `SAPID=000000` returned InboundRequest → BTDB DbCall (1 row) → Response Ok. + +## Rollback + +1. Stop Site vq04 → vq03, then Central vq04 → vq03. +2. On each box rename `app` → `app.failed` and `app.bak.20261022T152457` → `app`; do the same for `cli`. +3. Start Central vq03 → vq04, then Site vq03 → vq04. + +The indexes are additive, so `b3021e7c` runs against 58 migrations. For an exact DB, `RESTORE DATABASE [ScadaBridge] FROM DISK = 'F:\MSSQL\BACKUP\ScadaBridge_pre-3808b49f_20261022T152457.bak' WITH REPLACE` with everything stopped. + +## Outstanding + +These pre-existing items now carry startup Warnings: +- **Shared stores.** Set both, then recreate: + - `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer` for inbound keys. + - `Secrets:Store:Provider=SqlServer` plus `Secrets:SqlServer:ConnectionString` for secrets. + + Neither is needed for current QA operation: the site holds no deployed secrets. +- **Stale keys.** Remove the `Secrets:Replication` `Mode`/`GrpcHub` keys from both boxes' appsettings.json. + +Housekeeping after soak: older `app.bak.*`, `cli.bak.*`, `_appsettings-keep-*` and `E:\psrem-stage\*` folders on both boxes.