Files
ScadaBridge/docs/deployment/deployment-records/2026-11-13-wonder-app-vd03-8e32f353.md
T
Joseph Doherty 581e25caaf docs(deploy): record DEV vd03 424db155 -> 8e32f353, #131 applied, audit hash chain enabled (#107, #139)
No migration (64). ~45 s site outage. SCM non-crash recovery flag set for exit 75 (#132).
#131: SBTEST01/02 AlarmStatus on ScriptAlarm.Priority, verify 13/13 before and after.
Hash chain: SourceNode backfill 843 rows, Enabled=true (ChainBacklog unset, head export off),
one anchor at 2026-11-13T12:16:53Z, 0 pre-cutoff rows chained, verify-chain 2026-11 Intact.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 07:21:31 -05:00

9.5 KiB
Raw Blame History

2026-11-13 — DEV (wonder-app-vd03) 424db155 → 8e32f353, #131 applied, audit hash chain enabled

Owner go: #107, 2026-11-13: "Deploy, then enable chain". Head export stays off, single SQL login kept.

Build: main @ 8e32f353 (= origin/main, the LocalDb 0.3.8 pin, scadaproj#57), ProductVersion 1.0.0+8e32f3538ce940e349be3701efd0784a7ebeb551.

  • Published from a clean detached worktree at that SHA: dotnet publish Host + CLI -c Release -r win-x64 --self-contained true -p:PublishSingleFile=false.
  • 1050 app files, 224 cli files. Zip 130 520 104 B, SHA-256 0cf06cf5cfe5c1a3da2f1c6ce8db77b43f22cbcb79e5237abc4fe51cbc7fdebc.
  • Host.deps.json carries ZB.MOM.WW.LocalDb{,.Replication,.Contracts}/0.3.8.

Delta 424db155..8e32f353: 37 commits, no migration (64 → 64; the range only edits already-applied migration files, #126/#127). Highlights:

  • #129 alarm Severity banding (raw value on Priority) — hence the #131 gate below;
  • #139 hash-chain cutoff (52da02a7), #140 graceful stop, #138 ServiceLevel-aware OPC UA failover;
  • #132 leader-auto-down shut-out restart (Host exits 75 on an unplanned ActorSystem termination), #135/#136 site runtime;
  • #46 LDAP group refresh (inert here: DisableLogin), #125 OPC UA SDK 2.0.1 (DEV has no OPC UA connection), #128 MxGateway resume cursor;
  • Auth 0.5.0, LocalDb 0.3.4 → 0.3.8 (DEV site is single-node, replication off), Telemetry 0.3.1, Theme 0.5.3.

Not touched: QA vq03/vq04, OtOpcUa, mxaccessgw, HistorianGateway, Overview, the OTel collector. Z28061 / Z28061Sim: nothing deployed, written or called. #52 not applied.

Transport: winrm (deploy scope) for three on-box scripts; SMB for staging and logs; direct SQL and the CLI from the Mac for checks. Stage E:\ApiInstall\_stage-sb-8e32f353\.

Preflight (read-only, before the window)

Check Result
Live build Host + CLI 424db155
__EFMigrationsHistory 64, last 20261110120000_AuditMaintenanceGrants; AuditLog clustered key leads with OccurredAtUtc; 3 scadabridge_audit_* roles
Migrations in range none (no new file under Migrations/)
Option changes in range all optional with safe defaults (AuditLog:HashChain:ChainBacklog, Security:LdapGroupRefresh, DataConnection:ServiceLevel*, SiteRuntime:InstanceSweepSettle*, Cluster:LeaderAutoDown)
ASPNETCORE_URLS (#108) no https://
appsettings (kept + restored) A6B6CF05…, 3382DD83…, 3E7AD9A4… — same as 2026-10-22 / 11-11
Service recovery Both services: restart 5 s / 5 s / 10 s, reset 1 day, FAILURE_ACTIONS_ON_NONCRASH_FAILURES: FALSE → a #132 exit 75 would have left the node stopped
AuditLog 2 394 rows (newest 2026-11-06 — DEV is quiet), 0 chained, 0 heads/tombstones, 843 SourceNode IS NULL (newest 2026-07-07)
Baseline snapshots Z28062 (2) 270 attrs Good 144 / Bad 126, 8 alarms; SBTEST01/02 (9/10) Good 86, 5 alarms each. Site MxGateway Connected, 810/810 resolved, good 398 / bad 412
#131 baseline (old binary, old body) verify-mes-test-machine.sh groups C+D (C13 removed — SAPID 103559 routes to Z28061): PASS 13/13

Window (swap.ps1, notes E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-20261113T071302.txt)

  1. 07:13:14 staged zip hash + ProductVersion re-verified on the box; reg export of both service keys (_envbak-ScadaBridge-{Central,Site}-20261113T071302.reg in the stage).
  2. 07:13:17 BACKUP DATABASE … COPY_ONLY, CHECKSUM, COMPRESSION → E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-8e32f353_20261113T071302.bak (131 219 456 B), VERIFYONLY ok; appsettings kept in _appsettings-keep-20261113T071302.
  3. 07:13:17 Site stopped, 07:13:27 Central stopped; no Host.exe left.
  4. 07:13:37 sc.exe failureflag <svc> 1 on both services (owner-accepted as part of the deploy) — existing restart actions kept.
  5. app/cli → *.bak.20261113T071302, staged copy in, three appsettings restored (SHA match).
  6. 07:13:52 Central ready, 07:14:02 Site ready.

Downtime: site data path ~45 s (07:13:17 → 07:14:02); central API ~25 s. The routine stops logged Application is shutting down… at Information, no [FTL] (#76).

#131 (deploy gate) — applied and verified, DEV SBTEST01/02 only

docs/deployment/deploy-prep/2026-11-12-gates-52-131/apply.sh apply 131 straight after the swap:

  • TestMesMachine.AlarmStatus (template 30, script 42): ORIGINAL → FIXED (a.Severity → a.Priority ×3); metadata read back unchanged.
  • SBTEST01 deployment 95, SBTEST02 deployment 96, both status 2; check 131 afterwards: FIXED, not stale.
  • Verify (groups C+D, same harness as the baseline): PASS 13/13 — C3 Severity == 800, C7 MinSeverity 500 → HighVacuumFault + SensorFault, C9 MaxSeverity 750 → SensorFault, i.e. identical to the pre-#129 binary. Alarm sources reset (D1).

Gates

Gate Result
Health :8085 and :8084 /health/ready 200 / 200, /health/active 200 / 200; central database, akka-cluster, required-singletons, bundle-signing-policy, audit-chain-head-export Healthy; site akka-cluster, localdb Healthy
ProductVersion Host + CLI 1.0.0+8e32f353… (swap log and a post-deploy inspection)
Inbound API bogus X-API-Key and missing key → 401 UNAUTHORIZED; the inbound executor itself answered the 13 #131 calls (flow-debug run-method, InboundRequest → BTDB → Route → Response)
MxGateway / Z28062 MxGateway Connected, 810/810 resolved, good 398 / bad 412 (= baseline); Z28062 Good 144 / Bad 126, 8 alarms (= baseline); SBTEST01/02 Good 86 ×2
AuditLog new rows after the swap from central-a (InboundRequest, RouteCall, DbWrite, AttributeWrite, ScriptLog, InboundAuthFailure) and node-a (ScriptRun), SourceNode set; site audit backlog drained
Logs only the usual first-probe akka-cluster … Removed ERRs at boot and the placeholder-seed warnings; no FTL
Overview (:5320) 6/6 Up; ScadaBridge central and site both Active/Up, leader chips present, 5/5 and 2/2 checks
Env block SHA-256-prefix digest of every service-env entry identical before and after (Central 12, Site 6). The only service-definition change is the recovery flag FALSE → TRUE

Audit hash chain (#107 item 2, #139)

  1. SourceNode backfill: scadabridge audit backfill-source-node --before 2026-07-08T00:00:00Z --sentinel unknown → rows updated : 843; SourceNode IS NULL now 0.

  2. Enable (enable-chain.enable.ps1, 07:16:39): one Central env entry added, AuditLog__HashChain__Enabled=true (digest b5bea41b6c62 = true). ChainBacklog unset, HeadExport unset (off), no MaintenanceConnectionString. All 12 existing entries unchanged; .reg backup _envbak-ScadaBridge-Central-hashchain-enable-20261113T071639.reg. Central-only restart, ready 07:17:01 (site data path not interrupted).

  3. Central log: AuditLog hash chain started at 2026-11-13T12:16:53.1143422Z: only rows dated from then on are chained….

  4. SQL (direct, read-only):

    Check Result
    Anchor rows (AuditLogChainTombstone, ChainSeq 0) exactly 1: month 2026-11-01, cutoff 2026-11-13T12:16:53.1143422, EventId 5CADAB1D-C4A1-4C0F-A000-000000000139
    Chained rows dated before the cutoff 0
    Rows 2 845 total, 1 chained (a post-cutoff InboundAuthFailure, ChainSeq 1), backlog unchained; AuditLogChainHead 2026-11 LastSeq 1; 0 non-anchor tombstones
  5. scadabridge audit verify-chain (DEV, DisableLogin, Basic fallback):

    Partition         : 2026-11            Partition         : 2026-10
    Status            : Intact             Status            : NotChained
    Rows verified     : 1                  Rows verified     : 0
    Head sequence     : 1                  Head sequence     : -
    Unchained rows    : 0                  Unchained rows    : 0
    Chain from        : 2026-11-13T12:16:53.1143422Z (both)
    Pre-cutoff rows   : 507                Pre-cutoff rows   : 667
    

    Exit 0 for both. 507 and 667 match direct SQL counts of the November-before-cutoff and October rows.

  6. Head export stays off (audit-chain-head-export: "Chain-head export disabled").

Rollback

All scripts are in E:\ApiInstall\_stage-sb-8e32f353\ (run with winrm --timeout 900 ps vd03 -f <script>):

  • Chain only: enable-chain.disable.ps1 removes the env entry and restarts Central. The anchor and chained rows stay (harmless; do not delete the anchor — verify-chain reports that as tampering).
  • Deploy (documented path): rollback-Restore.ps1 — stop Site + Central, remove AuditLog__HashChain__* from the Central env (424db155 has no #139 cutoff and would chain the whole backlog), app/cli ← *.bak.20261113T071302, RESTORE … ScadaBridge_pre-8e32f353_20261113T071302.bak WITH REPLACE, CHECKSUM (loses every DB write since 07:13, incl. the backfill, the anchor and the #131 script edit), start both.
  • rollback-None.ps1: the same without the restore (no schema change in the range, so 424db155 runs on schema 64).
  • #131 after a binary rollback: 424db155 has no ScriptAlarm.Priority, and the site still holds the fixed body from deployments 95/96.
    • After rollback-None.ps1: apply.sh rollback 131 (writes the original body back and redeploys SBTEST01/02).
    • After rollback-Restore.ps1: the central template is already the original body again, but the site is not; redeploy SBTEST01 (9) and SBTEST02 (10) with scadabridge instance deploy --id <n>.
  • The recovery flag can stay TRUE on either build. Never roll back past 0e540291.

The _appsettings-keep-20261113T071302 and *.bak.20261113T071302 folders can go once the build has soaked.