dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(deploy): record QA wonder-app-vq03/vq04 upgrade b3021e7c -> 3808b49f (migration 58, pair-wise window 2m35s)

Browse Source
This commit is contained in:
Joseph Doherty
2026-10-22 15:38:27 -04:00
parent 51b111a35f
commit 5ad6f83176
1 changed files with 98 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deployment/deployment-records/2026-10-22-wonder-app-vq03-vq04-3808b49f.md
+98
View File
@@ -0,0 +1,98 @@
# Deployment record — wonder-app-vq03 + wonder-app-vq04 (QA) — 2026-10-22 — `b3021e7c` → `3808b49f`
This record covers the **QA two-host topology**:
| Box | Central node | Site-a node |
|---|---|---|
| vq03 `10.220.158.2` | central-a | node-a |
| vq04 `10.220.158.3` | central-b | node-b |
- Centrals on :8085; site gRPC on :8083, health on :8084.
- LocalDb replication is on.
- The shared config DB `ScadaBridge` is on `wonder-sql-vq03`.
The package is the one DEV received the same day. See `2026-10-22-wonder-app-vd03-3808b49f.md` for the build delta: 191 commits, including site secrets via Deploy Artifacts, the 2026-10-21 review remediation, and Topology #38/#39/#40.
The deploy ran from the Mac with the `winrm` CLI. The zip and SQL went over SMB, followed by one prep script per box and seven window calls, all from one mode-switched script.
## Pre-flight (breaking-change check)
| Area | Change in delta | QA state | Verdict |
|---|---|---|---|
| Schema | 57 → 58 `ReviewRemediationIndexes` (5 additive indexes) | 57; AuditLog 2 132 rows, AuditLogEntries 515; `ps_AuditLog_Month` present; Enterprise | OK |
| `Secrets:Replication:Enabled` ≠ false → startup failure | — | `false` in appsettings.json on both boxes; not in the service env | OK (stale `Mode`/`GrpcHub` keys log one ignore-Warning) |
| `JwtSigningKey` ≥ 32 bytes; must match on both centrals now that the CLI uses bearer tokens | — | 64 chars on both; identical hash | OK |
| `MachineDataDb`, `Transport:SchemaVersionMajor`, `JwtRefreshThresholdMinutes` removed | — | `MachineDataDb` still in the Central env and appsettings; the others are unset | Ignored, harmless |
| Actor-name deploy gate | — | 33 instance, script, alarm, native-source and connection names | All valid |
| LocalDb replicated set 10 → 12 tables (`deployed_secrets`, `secret_mirror_status`) | Registered-table digests are compared fail-closed | Both site nodes upgraded in one window | OK; the pair reconnected, backlog 0 |
| Data-connection `secret:` references | Artifact deploy resolves them centrally | The artifact hash is unchanged after the deploy, so there are no references | OK |
| Service env | — | Re-exported during prep; identical to the 2026-09-28 export on all four services | OK |
**No appsettings change.** The three files were kept and restored on each box, with matching hashes:
| Box | appsettings.json | appsettings.Central.json | appsettings.Site.json |
|---|---|---|---|
| vq03 | `F94000B1` | `D1320FA8` | `4CBAF66C` |
| vq04 | `F94000B1` | `2FD3EC4C` | `D0C1AF05` |
## Upgrade
| Item | Value |
|---|---|
| Build | `main` @ **`3808b49f`**, ProductVersion `1.0.0+3808b49fc7558a177219d00eac97747b28850a80`, all four nodes |
| Artifact | `E:\psrem-stage\3808b49f\scadabridge.zip` on both boxes; 125 543 156 B; SHA-256 `770BB992…5FB1098E47`, re-verified on each box; app 1028 files, cli 223 |
| Migration | `ReviewRemediationIndexes.sql` (idempotent) applied with `sqlcmd -C -b -I` from vq03 while both centrals were stopped; 57 → 58. SQL Server warned that `IX_AuditLog_Kind_Occurred` has a maximum key of 8 008 bytes against the 1 700-byte limit. `Kind` holds short enum names, so this is benign |
| DB backup | `F:\MSSQL\BACKUP\ScadaBridge_pre-3808b49f_20261022T152457.bak` on wonder-sql-vq03 (COPY_ONLY, CHECKSUM, COMPRESSION; VERIFYONLY valid) |
| Binary backups | `app.bak.20261022T152457` / `cli.bak.20261022T152457` plus `_appsettings-keep-20261022T152457` on both boxes |
| Notes on box | `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-3808b49f.txt` |
| Step | Local time |
|---|---|
| Stop Site vq04 (**window opens**) | 15:29:58 → 15:30:28 |
| Stop Site vq03 | 15:30:37 → 15:30:47 |
| Stop Central vq04, swap vq04 | 15:30:55 → 15:31:12 |
| Stop Central vq03, migrate, swap vq03, start Central vq03 → ready | 15:31:19 → 15:31:49 |
| Start Central vq04 → ready | 15:32:00 → 15:32:05 |
| Start Site vq03 → ready | 15:32:12 → 15:32:21 |
| Start Site vq04 → ready (**window closes**) | 15:32:30 → 15:32:33 |
**Downtime: 2 min 35 s.**
## Verification (from the Mac)
- **Centrals:** `/health/ready` 200 on both. `database`, `akka-cluster` (memberCount 2, unreachable 0) and `required-singletons` are all Healthy. `/health/active` is 200 on vq03 and 503 on vq04, the same active node as before.
- **Sites:** `/health/ready` 200 on both. `localdb` shows `replicationConnected:true`, backlog 0. `/health/active` is 200 on vq03 and 503 on vq04.
- **Auth:**
- `POST /auth/token` returns 200; the CLI now runs on bearer tokens against QA.
- `/management` without credentials returns 401, unchanged.
- **Snapshots:**
- Z28061–Z28069: 242 Good / 2 Bad on both sides of the window. Z28063 showed 114 Uncertain on the first read while its subscriptions settled, then matched.
- Alarm counts are unchanged (14, or 8 on Z28062).
- ZIPSEN is not deployed, as before.
- **Deploy Artifacts:** deployment `d7505cd8…`, site-a success. The hash is unchanged (`sha256:d79ba1a3…`); `ArtifactsDeployedAt` is 19:32:56Z.
- **Logs** (pulled over SMB):
- The usual first-probe `akka-cluster … Removed` ERR lines.
- vq03 site logged four `Replication session … faulted` Warnings between 15:32:16 and 15:32:29, while vq04's site was still down. The session has been connected since.
- New startup Warnings: Secrets per-node SQLite on a clustered central, and the stale `Secrets:Replication`/`GrpcHub` keys.
- No other ERR or FTL lines.
- **Smoke:** `flow-debug run-method` of `SimpleAlarmStatusRequest` (id 9) with `SAPID=000000` returned InboundRequest → BTDB DbCall (1 row) → Response Ok.
## Rollback
1. Stop Site vq04 → vq03, then Central vq04 → vq03.
2. On each box rename `app` → `app.failed` and `app.bak.20261022T152457` → `app`; do the same for `cli`.
3. Start Central vq03 → vq04, then Site vq03 → vq04.
The indexes are additive, so `b3021e7c` runs against 58 migrations. For an exact DB, `RESTORE DATABASE [ScadaBridge] FROM DISK = 'F:\MSSQL\BACKUP\ScadaBridge_pre-3808b49f_20261022T152457.bak' WITH REPLACE` with everything stopped.
## Outstanding
These pre-existing items now carry startup Warnings:
- **Shared stores.** Set both, then recreate:
- `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer` for inbound keys.
- `Secrets:Store:Provider=SqlServer` plus `Secrets:SqlServer:ConnectionString` for secrets.
Neither is needed for current QA operation: the site holds no deployed secrets.
- **Stale keys.** Remove the `Secrets:Replication` `Mode`/`GrpcHub` keys from both boxes' appsettings.json.
Housekeeping after soak: older `app.bak.*`, `cli.bak.*`, `_appsettings-keep-*` and `E:\psrem-stage\*` folders on both boxes.