Files
ScadaBridge/docs/deployment/deploy-prep/2026-11-15-143/README.md
T
Joseph Doherty 50a5291b64 deploy(#143): DEV vd03 + QA vq03/vq04 to 061432c8 — hash chain removed (migration 65), #116 QA site-pair drills
DEV: ~48 s site / ~29 s central outage; QA: pair stop/start, ~2m40s site / ~1m37s central.
Chain schema + chainer role gone, env key removed, all gates pass on both tiers.
#116 QA site drills (all reactors idle): kill → takeover +7.7 s; stop-and-hold →
[leader-auto-down] DOWNING +27.2 s, takeover +27.4 s, recovery restored byte-identical.

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 19:36:04 -05:00

77 lines
6.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Deploy pack: ScadaBridge#143, hash-chain removal (`8e32f353` → `061432c8`), DEV vd03 then QA vq03/vq04
**Owner go (2026-11-15):** "yes 143, with the 116 dependent on idle". Deploy current `main` to DEV, then QA. The #116 QA site-pair drill follows QA **only if** every QA CvdReactor instance is idle.
## Build
`main` @ `061432c8`. Host and CLI are published with `-c Release -r win-x64 --self-contained true -p:PublishSingleFile=false` (the boxes have no .NET 10). The app has 1056 files and the cli 224. The pins are `ZB.MOM.WW.LocalDb 0.3.8` and `ZB.MOM.WW.Theme 1.0.1`.
## Delta `8e32f353..061432c8` (25 commits)
- **#143 / #107:** the AuditLog hash chain, head export and two-login maintenance hardening are removed (`ed2ed0a3`, `b418408c`, merge `3dea146c`). The archive is kept at format v2. A retired `AuditLog:HashChain*` / `MaintenanceConnectionString` key is ignored and logs one warning.
- **#141:** the DCL fails over within seconds when the connected OPC UA server is going away (`c4ec51e8`).
- **#105:** the legacy site-psk-invalidation topic is removed (`89d2becb`).
- **UI consistency:** Theme 1.0.0 → 1.0.1 (dark palette, reflow), dialog naming and focus containment, controls and editor errors, semantic tables, the audit-window validation and the dashboard count reads (`cdd3adf6`…`6489c9fc`).
- **Docs and deploy records only:** #52/#142, #116 drills, #107/#138 records.
- **No appsettings change.** The only option change is a removal (`AuditLog:HashChain`).
## Migration 64 → 65
- `migrate-64-to-65.idempotent.sql` was generated at `061432c8` with `dotnet ef migrations script 20261110120000_AuditMaintenanceGrants --idempotent`. Its SHA-256 is `a16a150de1e1f6c4cb9153e1c53bed95a029d656f18c3dc3b61ec6575198f7a7`, and with comments stripped it is identical to `docs/plans/sql/RemoveAuditLogHashChain.sql`.
- It applies one migration, `20261115221558_RemoveAuditLogHashChain`, in one transaction, and every statement is existence-guarded.
- It is run with `SET XACT_ABORT ON` and `sqlcmd -I -b`. On DEV it runs on vd03; on QA it runs on vq03 against wonder-sql-vq03 (vq04 has no `sqlcmd`). The login is the Central service's own connection string, with the password only in `SQLCMDPASSWORD`.
- **Gate:** `8e32f353` maps `ChainSeq` / `RowHash`, so it cannot run on schema 65. The migration therefore runs with every ScadaBridge service on the tier stopped, and only the new binaries start afterwards.
## Env change (Central service only)
Remove `AuditLog__HashChain__Enabled` and any other `AuditLog__HashChain__*` / `AuditLog__MaintenanceConnectionString` entries. Every other entry must stay byte-identical, and this is proven by a digest (name + SHA-256 prefix, values never printed). On QA the per-node `JwtExpiryMinutes` / `IdleTimeoutMinutes` drift sits in appsettings and is kept.
## Order per tier
1. **Preflight**, read-only:
- live Host is `8e32f353`;
- 64 migrations;
- the chain objects are present: 2 columns, 2 tables, 2 indexes, the `_chainer` role;
- the env holds `AuditLog__HashChain__Enabled=true`;
- health and the active node.
2. **Backup:** `COPY_ONLY, CHECKSUM, COMPRESSION` + `RESTORE VERIFYONLY`; `reg export` of the service keys; appsettings kept with hashes.
3. **Stop:** DEV stops Site, then Central. QA stops both Sites, then both Centrals, so the pair goes down together.
4. **Migrate** 64 → 65.
5. **Swap:** `app` / `cli` → `*.bak.<ts>`, copy in the staged build, restore the kept appsettings (hash match). Re-assert `failureflag 1`.
6. **Env:** remove the retired keys from the Central env.
7. **Start:** Central(s) first, then Site(s). On QA vq03 goes first, then vq04.
## Gates
- `/health/ready` 200 on every service, with one active node per pair.
- ProductVersion `+061432c8` on Host and CLI.
- 65 migrations.
- `ChainSeq` / `RowHash`, `AuditLogChainHead` / `AuditLogChainTombstone`, `IX_AuditLog_ChainSeq` / `IX_AuditLog_Unchained` and `scadabridge_audit_chainer` are **absent**; `scadabridge_audit_purger` and `_writer` are present.
- New AuditLog rows are written after the start.
- No `no longer supported` (retired-key) warning in the Central log.
- A bogus inbound key gets 401.
- Data connections are at baseline. On QA: LocalDb sync connected, Z28062–Z28069 at 244 Good, WarmCanary connected.
- OTLP `service.version` is `+061432c8`.
- Overview is Up.
- The CLI `audit` group has no `verify-chain` / `verify-heads`.
## Rollback
Schema 65 cannot run under `8e32f353`, so a rollback is binaries **plus** a DB restore:
1. Stop all services.
2. `RESTORE … WITH REPLACE, CHECKSUM` from the pre-window backup.
3. `app` / `cli` ← `*.bak.<ts>`.
4. `reg import` the pre-window Central key, which puts `AuditLog__HashChain__Enabled=true` back.
5. Start the services: on QA the pair starts together.
Never roll back past `0e540291`. The swap scripts auto-roll-back if the start fails. Once the migration has run, that auto-rollback restores the DB.
## #116 site-pair drill (QA, conditional)
Two snapshots about 60 s apart are taken on every QA CvdReactor instance. If any instance has a cycle running, a cycle ID, FurnaceTemp ≥ 60 or a `RecipeDownloadFlag` True, the drill is **skipped**.
If every instance is idle:
1. A kill drill of the active `ScadaBridge-Site` process, picked by service PID.
2. After about 2 min stable, a stop-and-hold drill: disable the SCM restart, kill, hold ≥ 60 s, capture `[leader-auto-down] DOWNING`, restore the recovery config byte-identical, then start.
The gates afterwards are: instances re-hosted at 244 Good, one active site, LocalDb sync connected, and auto-restart restored.
## Scripts (as run)
- `dev-deploy.ps1`: DEV, in one call.
- `qa.tmpl.ps1`: QA, rendered per step with `__ACTION__` / `__TS__`. The actions are `prep`, `backup`, `stop-site`, `stop-central`, `migrate`, `swap`, `start-central`, `start-site`, `verify` and `rollback-swap`.
- `drill.tmpl.ps1`: the #116 site drill helper, with actions `status`, `kill`, `disable`, `kill-held`, `restore` and `start`.
Results are in `deployment-records/2026-11-15-wonder-app-vd03-061432c8.md` and `…-vq03-vq04-061432c8.md`.