DEV: ~48 s site / ~29 s central outage; QA: pair stop/start, ~2m40s site / ~1m37s central. Chain schema + chainer role gone, env key removed, all gates pass on both tiers. #116 QA site drills (all reactors idle): kill → takeover +7.7 s; stop-and-hold → [leader-auto-down] DOWNING +27.2 s, takeover +27.4 s, recovery restored byte-identical. Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
6.0 KiB
Deploy pack: ScadaBridge#143, hash-chain removal (8e32f353 → 061432c8), DEV vd03 then QA vq03/vq04
Owner go (2026-11-15): "yes 143, with the 116 dependent on idle". Deploy current main to DEV, then QA. The #116 QA site-pair drill follows QA only if every QA CvdReactor instance is idle.
Build
main @ 061432c8. Host and CLI are published with -c Release -r win-x64 --self-contained true -p:PublishSingleFile=false (the boxes have no .NET 10). The app has 1056 files and the cli 224. The pins are ZB.MOM.WW.LocalDb 0.3.8 and ZB.MOM.WW.Theme 1.0.1.
Delta 8e32f353..061432c8 (25 commits)
- #143 / #107: the AuditLog hash chain, head export and two-login maintenance hardening are removed (
ed2ed0a3,b418408c, merge3dea146c). The archive is kept at format v2. A retiredAuditLog:HashChain*/MaintenanceConnectionStringkey is ignored and logs one warning. - #141: the DCL fails over within seconds when the connected OPC UA server is going away (
c4ec51e8). - #105: the legacy site-psk-invalidation topic is removed (
89d2becb). - UI consistency: Theme 1.0.0 → 1.0.1 (dark palette, reflow), dialog naming and focus containment, controls and editor errors, semantic tables, the audit-window validation and the dashboard count reads (
cdd3adf6…6489c9fc). - Docs and deploy records only: #52/#142, #116 drills, #107/#138 records.
- No appsettings change. The only option change is a removal (
AuditLog:HashChain).
Migration 64 → 65
migrate-64-to-65.idempotent.sqlwas generated at061432c8withdotnet ef migrations script 20261110120000_AuditMaintenanceGrants --idempotent. Its SHA-256 isa16a150de1e1f6c4cb9153e1c53bed95a029d656f18c3dc3b61ec6575198f7a7, and with comments stripped it is identical todocs/plans/sql/RemoveAuditLogHashChain.sql.- It applies one migration,
20261115221558_RemoveAuditLogHashChain, in one transaction, and every statement is existence-guarded. - It is run with
SET XACT_ABORT ONandsqlcmd -I -b. On DEV it runs on vd03; on QA it runs on vq03 against wonder-sql-vq03 (vq04 has nosqlcmd). The login is the Central service's own connection string, with the password only inSQLCMDPASSWORD. - Gate:
8e32f353mapsChainSeq/RowHash, so it cannot run on schema 65. The migration therefore runs with every ScadaBridge service on the tier stopped, and only the new binaries start afterwards.
Env change (Central service only)
Remove AuditLog__HashChain__Enabled and any other AuditLog__HashChain__* / AuditLog__MaintenanceConnectionString entries. Every other entry must stay byte-identical, and this is proven by a digest (name + SHA-256 prefix, values never printed). On QA the per-node JwtExpiryMinutes / IdleTimeoutMinutes drift sits in appsettings and is kept.
Order per tier
- Preflight, read-only:
- live Host is
8e32f353; - 64 migrations;
- the chain objects are present: 2 columns, 2 tables, 2 indexes, the
_chainerrole; - the env holds
AuditLog__HashChain__Enabled=true; - health and the active node.
- live Host is
- Backup:
COPY_ONLY, CHECKSUM, COMPRESSION+RESTORE VERIFYONLY;reg exportof the service keys; appsettings kept with hashes. - Stop: DEV stops Site, then Central. QA stops both Sites, then both Centrals, so the pair goes down together.
- Migrate 64 → 65.
- Swap:
app/cli→*.bak.<ts>, copy in the staged build, restore the kept appsettings (hash match). Re-assertfailureflag 1. - Env: remove the retired keys from the Central env.
- Start: Central(s) first, then Site(s). On QA vq03 goes first, then vq04.
Gates
/health/ready200 on every service, with one active node per pair.- ProductVersion
+061432c8on Host and CLI. - 65 migrations.
ChainSeq/RowHash,AuditLogChainHead/AuditLogChainTombstone,IX_AuditLog_ChainSeq/IX_AuditLog_Unchainedandscadabridge_audit_chainerare absent;scadabridge_audit_purgerand_writerare present.- New AuditLog rows are written after the start.
- No
no longer supported(retired-key) warning in the Central log. - A bogus inbound key gets 401.
- Data connections are at baseline. On QA: LocalDb sync connected, Z28062–Z28069 at 244 Good, WarmCanary connected.
- OTLP
service.versionis+061432c8. - Overview is Up.
- The CLI
auditgroup has noverify-chain/verify-heads.
Rollback
Schema 65 cannot run under 8e32f353, so a rollback is binaries plus a DB restore:
- Stop all services.
RESTORE … WITH REPLACE, CHECKSUMfrom the pre-window backup.app/cli←*.bak.<ts>.reg importthe pre-window Central key, which putsAuditLog__HashChain__Enabled=trueback.- Start the services: on QA the pair starts together.
Never roll back past 0e540291. The swap scripts auto-roll-back if the start fails. Once the migration has run, that auto-rollback restores the DB.
#116 site-pair drill (QA, conditional)
Two snapshots about 60 s apart are taken on every QA CvdReactor instance. If any instance has a cycle running, a cycle ID, FurnaceTemp ≥ 60 or a RecipeDownloadFlag True, the drill is skipped.
If every instance is idle:
- A kill drill of the active
ScadaBridge-Siteprocess, picked by service PID. - After about 2 min stable, a stop-and-hold drill: disable the SCM restart, kill, hold ≥ 60 s, capture
[leader-auto-down] DOWNING, restore the recovery config byte-identical, then start.
The gates afterwards are: instances re-hosted at 244 Good, one active site, LocalDb sync connected, and auto-restart restored.
Scripts (as run)
dev-deploy.ps1: DEV, in one call.qa.tmpl.ps1: QA, rendered per step with__ACTION__/__TS__. The actions areprep,backup,stop-site,stop-central,migrate,swap,start-central,start-site,verifyandrollback-swap.drill.tmpl.ps1: the #116 site drill helper, with actionsstatus,kill,disable,kill-held,restoreandstart.
Results are in deployment-records/2026-11-15-wonder-app-vd03-061432c8.md and …-vq03-vq04-061432c8.md.