deploy(#143): DEV vd03 + QA vq03/vq04 to 061432c8 — hash chain removed (migration 65), #116 QA site-pair drills

DEV: ~48 s site / ~29 s central outage; QA: pair stop/start, ~2m40s site / ~1m37s central.
Chain schema + chainer role gone, env key removed, all gates pass on both tiers.
#116 QA site drills (all reactors idle): kill → takeover +7.7 s; stop-and-hold →
[leader-auto-down] DOWNING +27.2 s, takeover +27.4 s, recovery restored byte-identical.

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
This commit is contained in:
Joseph Doherty
2026-11-15 19:36:04 -05:00
parent 061432c8c4
commit 50a5291b64
7 changed files with 710 additions and 0 deletions
@@ -0,0 +1,68 @@
# 2026-11-15: DEV (wonder-app-vd03) `8e32f353` → `061432c8`, migration 64 → 65, audit hash chain removed (ScadaBridge#143)
**Owner go (2026-11-15):** "yes 143, with the 116 dependent on idle". IT security asked for the AuditLog hash chain to be removed. The pack is `docs/deployment/deploy-prep/2026-11-15-143/`.
**Build:** `main` @ `061432c8`, ProductVersion `1.0.0+061432c8c4299505a45d4cf396c3d62139865080`.
- Host and CLI were published self-contained win-x64 from a clean detached worktree: 1056 app files and 224 cli files.
- The zip is 130 414 127 B, SHA-256 `b6e85ecc24186af8b658fa6f1e38f080588ea5f78dd856a4356e696897ff4095`.
- Pins: LocalDb 0.3.8, Theme 1.0.1.
**Delta:** 25 commits. See the pack: the hash-chain, head-export and two-login removal (`3dea146c`), #141, #105, and the UI-consistency / Theme 1.0.x work. There is one migration, `20261115221558_RemoveAuditLogHashChain`. The script `migrate-64-to-65.idempotent.sql` has SHA-256 `a16a150d…98f7a7`, is identical to `docs/plans/sql/RemoveAuditLogHashChain.sql` once comments are stripped, and was run with `SET XACT_ABORT ON` and `sqlcmd -I -b`.
**Transport:** one deploy script over `winrm` (deploy scope, owner-authorized), `dev-deploy.ps1` in `E:\ApiInstall\_stage-sb-061432c8\`. Staging went over SMB, and the gates were checked over HTTP and with the CLI from the Mac. The on-box notes are `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-061432c8-20261115T191258.txt`.
**Not touched:** QA (see its own record), OtOpcUa, mxaccessgw, HistorianGateway, Overview, the OTel collector. Nothing was deployed to any instance.
## Preflight (in the script, before anything changed)
| Check | Result |
|---|---|
| Live build | `8e32f353` |
| Staged build | `061432c8` Host + CLI |
| Schema | 64 migrations; `ChainSeq` (8) / `RowHash` (32) present, both chain tables, 2 chain indexes, roles `_chainer`, `_purger`, `_writer`; AuditLog 2 853 rows |
| Central env | 13 entries; retired key present: `AuditLog__HashChain__Enabled` (= `true`) |
| appsettings | `A6B6CF05…`, `3382DD83…`, `3E7AD9A4…` (unchanged since 11-13) |
| Health / baseline | central and site ready/active 200; Z28062 270 attrs, Good 144, 8 alarms |
## Window
| Step | Local (EST) |
|---|---|
| `reg export` of both service keys; backup `E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-061432c8_20261115T191258.bak` (131 757 056 B, `COPY_ONLY, CHECKSUM, COMPRESSION`, VERIFYONLY ok); appsettings kept in `_appsettings-keep-20261115T191258` | 19:13:11 → 19:13:14 |
| Stop Site, then Central; no `Host.exe` left | 19:13:14 → 19:13:35 |
| Migrate 64 → 65 | 19:13:35 (< 1 s) |
| `failureflag 1` re-asserted; `app`/`cli` → `*.bak.20261115T191258`; new build in; appsettings restored (hash match) | 19:13:35 → 19:13:41 |
| Central env: `AuditLog__HashChain__Enabled` removed, 13 → 12 entries; the other 12 byte-identical (digest-checked) | 19:13:41 |
| Central ready, then Site ready | 19:13:53 / 19:14:02 |
**Downtime:** the site data path was down for about 48 s (19:13:14 → 19:14:02) and the central API for about 29 s (19:13:25 → 19:13:54).
## Gates (all pass)
| Gate | Result |
|---|---|
| Build | Host + CLI `1.0.0+061432c8…` |
| Schema | **65** migrations; `ChainSeq`, `RowHash`, `AuditLogChainHead`, `AuditLogChainTombstone`, `IX_AuditLog_ChainSeq`, `IX_AuditLog_Unchained` and `scadabridge_audit_chainer` all **absent**; `scadabridge_audit_purger` + `_writer` present; 2 853 rows kept |
| Health | ready/active 200/200 (central `:8085`, site `:8084`). Central checks: `database`, `akka-cluster`, `required-singletons`, `bundle-signing-policy` Healthy (`audit-chain-head-export` gone). Site checks: `akka-cluster`, `localdb` Healthy |
| Retired-key warning | 0 in `scadabridge-central-20261115.log`, because the env is clean |
| AuditLog | new rows after the start: 2 `InboundAuthFailure` from `central-a` (the bogus-key probes) |
| Inbound API | bogus `X-API-Key` → **401** ×2 |
| MxGateway / instances | `MxGateway` Connected, 810/810 resolved, good 398 / bad 412 (= baseline); Z28062 Good 144 / 270, 8 alarms (= baseline); 5 deployed instances |
| CLI | `audit` group: query, export, tree, backfill-source-node, archive; **no** `verify-chain` / `verify-heads` |
| OTLP | box collector `logs.jsonl` carries `service.version 1.0.0+061432c8…` (79 records at check time) |
| Overview `:5320` | 6/6 Up; ScadaBridge central Active/Up, leader chip, 4/4 checks |
| Recovery | both services `FAILURE_ACTIONS_ON_NONCRASH_FAILURES: TRUE` |
| Pending updates | 1 stale instance: Z28061, expected. Its machine moved to QA, and it was left on the old #52 body on purpose (#142 closed) |
The env block still has `ScadaBridge__Database__MachineDataDb`, which has been unused since its removal on 2026-10-22. It is harmless and was not touched.
## Rollback
Schema 65 cannot run under `8e32f353`, so a rollback is binaries **plus** a DB restore:
1. Stop Site, then Central.
2. `RESTORE DATABASE [ScadaBridge] FROM DISK = N'E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-061432c8_20261115T191258.bak' WITH REPLACE, CHECKSUM` (single-user first).
3. `app`/`cli` ← `*.bak.20261115T191258`.
4. `reg import E:\ApiInstall\_stage-sb-061432c8\_envbak-ScadaBridge-Central-20261115T191258.reg`, which puts `AuditLog__HashChain__Enabled=true` back.
5. Start Central, then Site.
The restore loses every DB write since 19:13. The `*.bak.20261115T191258` and `_appsettings-keep-20261115T191258` folders can go once the build has soaked.
@@ -0,0 +1,116 @@
# 2026-11-15: QA (wonder-app-vq03 + vq04) `8e32f353` → `061432c8`, migration 64 → 65, audit hash chain removed (ScadaBridge#143), #116 site-pair drills
**Owner go (2026-11-15):** "yes 143, with the 116 dependent on idle". The pack is `docs/deployment/deploy-prep/2026-11-15-143/`. DEV went first (`2026-11-15-wonder-app-vd03-061432c8.md`, all gates passed).
**Build:** the same zip as DEV, `061432c8`, SHA-256 `b6e85ecc…ff4095`. It was pushed to `E:\psrem-stage\sb-061432c8\` on both boxes and its hash re-verified on each box: 1056 app files, 224 cli files.
**Transport:** one rendered script per step over `winrm` (deploy scope, owner-authorized): `prep`, `backup`, `stop-site`, `stop-central`, `migrate`, `swap`, `start-central`, `start-site`, `verify`. Staging and logs went over SMB, and the gates were checked over HTTP and with the CLI from the Mac. The backup and migration ran on vq03 against wonder-sql-vq03 (vq04 has no `sqlcmd`), using the Central service's own connection string with the password in `SQLCMDPASSWORD` only. The on-box notes are `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-061432c8-20261115T191851.txt`.
**Not touched:** DEV (separate record), OtOpcUa, mxaccessgw, HistorianGateway, Overview, the OTel collector, `ScadaBridge-LDAP`. No instance was deployed to.
## Preflight
| Check | Result |
|---|---|
| Live / staged | `8e32f353` / `061432c8` on both boxes |
| Schema (wonder-sql-vq03) | 64 migrations; chain columns, tables, indexes and `_chainer` present; AuditLog 3 606 rows |
| Service env | Central 16 entries, Site 8 entries, **digest-identical on both boxes**; retired key `AuditLog__HashChain__Enabled` (`true`) on both Centrals |
| appsettings | `C0C653D6…` (both); Central vq03 `EF01B129…`, vq04 `F333D48F…`; Site vq03 `D0E202F9…`, vq04 `E9E6336F…`. These equal the 11-13 post-deploy hashes, and the per-node JWT/idle drift is kept |
| Recovery | all four: restart 5 s / 5 s / 10 s, reset 86 400, non-crash flag TRUE |
| Active | central vq03, site vq04 (since the #116 central drills earlier today) |
| Baseline | Z28061–Z28069 244/244 Good, 10 alarms each; WarmCanary 2/2 Good; MxGateway 2196/2196 good; LocalDb sync connected |
## Window
| Step | Local (EST) |
|---|---|
| Backup `F:\MSSQL\BACKUP\ScadaBridge_pre-061432c8_20261115T191851.bak` (`COPY_ONLY, CHECKSUM, COMPRESSION`, VERIFYONLY ok); `reg export` + appsettings kept on both boxes in the prep step | 19:19:00 → 19:19:48 |
| Stop Site vq03 (standby) | 19:20:06 → 19:20:17 |
| Stop Site vq04 (active, **site data path down**) | 19:20:34 → 19:20:45 |
| Stop Central vq04 (standby) | 19:21:02 → 19:21:12 |
| Stop Central vq03 (active, **central API down**); no `Host.exe` on either box | 19:21:23 → 19:21:33 |
| Migrate 64 → 65 (it refuses to run while any ScadaBridge port still answers) | 19:21:54 (< 1 s) |
| Swap on both boxes in parallel: `failureflag 1`, `app`/`cli` → `*.bak.20261115T191851`, new build, appsettings restored (hash match), Central env 16 → 15 | 19:22:16 → 19:22:31 |
| Start Centrals (vq03, vq04 3 s later) → ready | 19:23:00 / 19:23:05 |
| Start Sites (vq03, vq04 3 s later) → ready | 19:23:25 / 19:23:30 |
**Downtime:** the site data path was down for about 2 min 40 s (19:20:35 → 19:23:25) and the central API for about 1 min 37 s (19:21:23 → 19:23:00). Most of the gaps between steps are the ~15 s WinRM session set-up per call.
## Gates (all pass)
| Gate | Result |
|---|---|
| Build | Host + CLI `1.0.0+061432c8…` on both boxes |
| Schema | **65** migrations; `ChainSeq`, `RowHash`, both chain tables, both chain indexes and `scadabridge_audit_chainer` **absent**; `_purger` + `_writer` present; 3 606 rows kept |
| Health | ready 200 ×4; active: central vq03 200 / vq04 503, site vq03 200 / vq04 503. Central checks: `database`, `akka-cluster`, `required-singletons`, `bundle-signing-policy` Healthy (`audit-chain-head-export` gone) |
| Env | Central env = the pre-window entries minus `AuditLog__HashChain__Enabled`, byte-identical otherwise, identical on both boxes; Site env unchanged |
| Retired-key warning | 0 on both Centrals |
| Cluster / LocalDb | central Primary vq03 / Standby vq04, both online; site LocalDb `replicationConnected: true`, backlog 0 on both nodes |
| Inbound API | bogus key → **401** on vq03 and vq04 (through the standby relay); 2 new `InboundAuthFailure` rows from `central-a` |
| API keys | `MESTestKey`, `MES_Key`, `delmia-notifier-qa` |
| Data | MxGateway 2196/2196 good; WarmCanary connected, 2/2 good; Z28061–Z28069 244/244 Good, 10 alarms each (= baseline); 10 deployed instances |
| CLI | `audit` group has no `verify-chain` / `verify-heads` |
| OTLP | box collectors carry `service.version 1.0.0+061432c8…` (vq03 95, vq04 63 log records at check time) |
| Overview `:5320` | 12/12 Up on vq03 and vq04; ScadaBridge 4 instances / 2 clusters, central leader vq03 |
| Recovery | unchanged on all four services |
| Pending updates | 1 stale: **WarmCanary** (16). Its OPC UA connection config was edited after its 2026-11-13 deploy (#138 work). This deploy did not cause it, and it was left alone |
**Purge/partition sanity:** skipped. There is no read-only way to exercise a purge or switch-out, and nothing is due: archive and retention are off.
## #116 site-pair drills (owner condition: only if idle)
**Idle check:** two snapshots of all nine CvdReactors (Z28061–Z28069), at 00:27:18Z and 00:28:38Z.
- Every side showed `CycleRunning` False (Good) and `MachineCycleID` 0.
- FurnaceTemp was 23–25 °C, and no `RecipeDownloadFlag` was True.
- Between the snapshots the only changes were ±1 argon purge flow (Z28061, Z28067) and ±1 °C (Z28065).
So the drills ran. ZIPSEN is disabled and hosts no attributes.
**Kill drill.** The active Site, vq03 (PID 8896, picked through the service), was killed at 00:29:06.0Z:
| Event | UTC | After kill |
|---|---|---|
| vq03 Site `/health` stops answering | 00:29:06.4 | +0.4 s |
| vq04 `New incarnation of existing member … is trying to join`; `[leader-auto-down] … already "Down"; not scheduling a Down`; vq04 gains leadership | 00:29:13.05 | +7.0 s |
| vq04 Site active 200 | 00:29:13.7 | **+7.7 s** |
| vq03 back as standby (ready 200 / active 503) | 00:29:24.3 | +18.3 s |
As in the central kill drill, the SCM restarts the killed process after 5 s, before the 15 s downing timer runs out, so Akka downs the old incarnation itself. After the drill all 10 instances were 2198/2198 Good on vq04, and LocalDb was connected.
**Stop-and-hold drill** on the new active Site, vq04 (PID 18956), after about 1 min 20 s stable:
1. Recovery config recorded: RESTART 5000/5000/10000, reset 86400, non-crash flag TRUE.
2. Set to no actions. The kill script refuses to run while any RESTART action remains.
3. Killed at 00:30:32.7Z.
4. Held down for 113 s.
5. Recovery restored **byte-identical** to the recorded config (diff-checked), then started.
| Event | UTC | After kill |
|---|---|---|
| Kill vq04 Site | 00:30:32.72 | 0 |
| vq03 `[leader-auto-down] … 10.220.158.3:8082 is unreachable; it will be downed if still unreachable after "00:00:15"` | 00:30:44.86 | +12.1 s |
| vq03 `[WRN] [leader-auto-down] DOWNING akka.tcp://scadabridge@10.220.158.3:8082: unreachable for "00:00:15" and this node (akka.tcp://scadabridge@10.220.158.2:8082) is leader` | 00:30:59.87 | **+27.2 s** |
| vq03 Site active 200 | 00:31:00.17 | **+27.4 s** |
| Recovery restored, `Start-Service` | 00:31:56 / 00:32:25 | — |
| vq04 `Welcome from …10.220.158.2:8082`; standby (ready 200 / active 503) | 00:32:27.8 / 00:32:28.2 | 2.7 s after the start |
vq04 rejoined without the #132 shut-out path (no exit 75), because it started after it had been removed.
**After the drills (all pass):**
- ready 200 ×4;
- one active per pair: central vq03 and site vq03;
- 10 instances at 2198/2198 Good;
- MxGateway 2196 good, WarmCanary 2 good;
- LocalDb connected, backlog 0, on both nodes;
- both Site services back on RESTART 5 s / 5 s / 10 s, reset 86 400, flag TRUE;
- Overview 12/12 Up.
**What this proves on QA:** a killed site node fails over in about 8 s through the SCM-restart path. A site node that stays down is downed by `leader-auto-down` after 15 s unreachable, and the pair takes over in about 27 s. Both match the central drills (`2026-11-15-qa-116-central-drill.md`).
## Rollback (deploy)
Schema 65 cannot run under `8e32f353`, so a rollback is binaries **plus** a DB restore, with the pair stopped together:
1. On each box, run `qa.tmpl.ps1` (in the pack and in `E:\psrem-stage\sb-061432c8\`) rendered with `__ACTION__`=`rollback-swap` and `__TS__`=`20261115T191851`. It stops Site then Central, puts back `app`/`cli` ← `*.bak.20261115T191851` and re-imports the pre-window Central key (`_envbak-ScadaBridge-Central-20261115T191851.reg`). Services are left stopped.
2. With all four stopped, from vq03: `ALTER DATABASE [ScadaBridge] SET SINGLE_USER WITH ROLLBACK IMMEDIATE; RESTORE DATABASE [ScadaBridge] FROM DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-061432c8_20261115T191851.bak' WITH REPLACE, CHECKSUM; ALTER DATABASE [ScadaBridge] SET MULTI_USER;`. This loses every DB write since 19:19.
3. Start both Centrals together, then both Sites.
The `*.bak.20261115T191851` and `_appsettings-keep-20261115T191851` folders can go once the build has soaked.