Files
ScadaBridge/docs/deployment/deploy-prep/2026-11-15-143

Deploy pack: ScadaBridge#143, hash-chain removal (8e32f353 → 061432c8), DEV vd03 then QA vq03/vq04

Owner go (2026-11-15): "yes 143, with the 116 dependent on idle". Deploy current main to DEV, then QA. The #116 QA site-pair drill follows QA only if every QA CvdReactor instance is idle.

Build

main @ 061432c8. Host and CLI are published with -c Release -r win-x64 --self-contained true -p:PublishSingleFile=false (the boxes have no .NET 10). The app has 1056 files and the cli 224. The pins are ZB.MOM.WW.LocalDb 0.3.8 and ZB.MOM.WW.Theme 1.0.1.

Delta 8e32f353..061432c8 (25 commits)

  • #143 / #107: the AuditLog hash chain, head export and two-login maintenance hardening are removed (ed2ed0a3, b418408c, merge 3dea146c). The archive is kept at format v2. A retired AuditLog:HashChain* / MaintenanceConnectionString key is ignored and logs one warning.
  • #141: the DCL fails over within seconds when the connected OPC UA server is going away (c4ec51e8).
  • #105: the legacy site-psk-invalidation topic is removed (89d2becb).
  • UI consistency: Theme 1.0.0 → 1.0.1 (dark palette, reflow), dialog naming and focus containment, controls and editor errors, semantic tables, the audit-window validation and the dashboard count reads (cdd3adf6…6489c9fc).
  • Docs and deploy records only: #52/#142, #116 drills, #107/#138 records.
  • No appsettings change. The only option change is a removal (AuditLog:HashChain).

Migration 64 → 65

  • migrate-64-to-65.idempotent.sql was generated at 061432c8 with dotnet ef migrations script 20261110120000_AuditMaintenanceGrants --idempotent. Its SHA-256 is a16a150de1e1f6c4cb9153e1c53bed95a029d656f18c3dc3b61ec6575198f7a7, and with comments stripped it is identical to docs/plans/sql/RemoveAuditLogHashChain.sql.
  • It applies one migration, 20261115221558_RemoveAuditLogHashChain, in one transaction, and every statement is existence-guarded.
  • It is run with SET XACT_ABORT ON and sqlcmd -I -b. On DEV it runs on vd03; on QA it runs on vq03 against wonder-sql-vq03 (vq04 has no sqlcmd). The login is the Central service's own connection string, with the password only in SQLCMDPASSWORD.
  • Gate: 8e32f353 maps ChainSeq / RowHash, so it cannot run on schema 65. The migration therefore runs with every ScadaBridge service on the tier stopped, and only the new binaries start afterwards.

Env change (Central service only)

Remove AuditLog__HashChain__Enabled and any other AuditLog__HashChain__* / AuditLog__MaintenanceConnectionString entries. Every other entry must stay byte-identical, and this is proven by a digest (name + SHA-256 prefix, values never printed). On QA the per-node JwtExpiryMinutes / IdleTimeoutMinutes drift sits in appsettings and is kept.

Order per tier

  1. Preflight, read-only:
    • live Host is 8e32f353;
    • 64 migrations;
    • the chain objects are present: 2 columns, 2 tables, 2 indexes, the _chainer role;
    • the env holds AuditLog__HashChain__Enabled=true;
    • health and the active node.
  2. Backup: COPY_ONLY, CHECKSUM, COMPRESSION + RESTORE VERIFYONLY; reg export of the service keys; appsettings kept with hashes.
  3. Stop: DEV stops Site, then Central. QA stops both Sites, then both Centrals, so the pair goes down together.
  4. Migrate 64 → 65.
  5. Swap: app / cli → *.bak.<ts>, copy in the staged build, restore the kept appsettings (hash match). Re-assert failureflag 1.
  6. Env: remove the retired keys from the Central env.
  7. Start: Central(s) first, then Site(s). On QA vq03 goes first, then vq04.

Gates

  • /health/ready 200 on every service, with one active node per pair.
  • ProductVersion +061432c8 on Host and CLI.
  • 65 migrations.
  • ChainSeq / RowHash, AuditLogChainHead / AuditLogChainTombstone, IX_AuditLog_ChainSeq / IX_AuditLog_Unchained and scadabridge_audit_chainer are absent; scadabridge_audit_purger and _writer are present.
  • New AuditLog rows are written after the start.
  • No no longer supported (retired-key) warning in the Central log.
  • A bogus inbound key gets 401.
  • Data connections are at baseline. On QA: LocalDb sync connected, Z28062–Z28069 at 244 Good, WarmCanary connected.
  • OTLP service.version is +061432c8.
  • Overview is Up.
  • The CLI audit group has no verify-chain / verify-heads.

Rollback

Schema 65 cannot run under 8e32f353, so a rollback is binaries plus a DB restore:

  1. Stop all services.
  2. RESTORE … WITH REPLACE, CHECKSUM from the pre-window backup.
  3. app / cli ← *.bak.<ts>.
  4. reg import the pre-window Central key, which puts AuditLog__HashChain__Enabled=true back.
  5. Start the services: on QA the pair starts together.

Never roll back past 0e540291. The swap scripts auto-roll-back if the start fails. Once the migration has run, that auto-rollback restores the DB.

#116 site-pair drill (QA, conditional)

Two snapshots about 60 s apart are taken on every QA CvdReactor instance. If any instance has a cycle running, a cycle ID, FurnaceTemp ≥ 60 or a RecipeDownloadFlag True, the drill is skipped.

If every instance is idle:

  1. A kill drill of the active ScadaBridge-Site process, picked by service PID.
  2. After about 2 min stable, a stop-and-hold drill: disable the SCM restart, kill, hold ≥ 60 s, capture [leader-auto-down] DOWNING, restore the recovery config byte-identical, then start.

The gates afterwards are: instances re-hosted at 244 Good, one active site, LocalDb sync connected, and auto-restart restored.

Scripts (as run)

  • dev-deploy.ps1: DEV, in one call.
  • qa.tmpl.ps1: QA, rendered per step with __ACTION__ / __TS__. The actions are prep, backup, stop-site, stop-central, migrate, swap, start-central, start-site, verify and rollback-swap.
  • drill.tmpl.ps1: the #116 site drill helper, with actions status, kill, disable, kill-held, restore and start.

Results are in deployment-records/2026-11-15-wonder-app-vd03-061432c8.md and …-vq03-vq04-061432c8.md.