No migration (64). ~45 s site outage. SCM non-crash recovery flag set for exit 75 (#132). #131: SBTEST01/02 AlarmStatus on ScriptAlarm.Priority, verify 13/13 before and after. Hash chain: SourceNode backfill 843 rows, Enabled=true (ChainBacklog unset, head export off), one anchor at 2026-11-13T12:16:53Z, 0 pre-cutoff rows chained, verify-chain 2026-11 Intact. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
9.5 KiB
2026-11-13 — DEV (wonder-app-vd03) 424db155 → 8e32f353, #131 applied, audit hash chain enabled
Owner go: #107, 2026-11-13: "Deploy, then enable chain". Head export stays off, single SQL login kept.
Build: main @ 8e32f353 (= origin/main, the LocalDb 0.3.8 pin, scadaproj#57), ProductVersion 1.0.0+8e32f3538ce940e349be3701efd0784a7ebeb551.
- Published from a clean detached worktree at that SHA:
dotnet publishHost + CLI-c Release -r win-x64 --self-contained true -p:PublishSingleFile=false. - 1050 app files, 224 cli files. Zip 130 520 104 B, SHA-256
0cf06cf5cfe5c1a3da2f1c6ce8db77b43f22cbcb79e5237abc4fe51cbc7fdebc. Host.deps.jsoncarriesZB.MOM.WW.LocalDb{,.Replication,.Contracts}/0.3.8.
Delta 424db155..8e32f353: 37 commits, no migration (64 → 64; the range only edits already-applied migration files, #126/#127). Highlights:
- #129 alarm Severity banding (raw value on
Priority) — hence the #131 gate below; - #139 hash-chain cutoff (
52da02a7), #140 graceful stop, #138 ServiceLevel-aware OPC UA failover; - #132 leader-auto-down shut-out restart (Host exits 75 on an unplanned ActorSystem termination), #135/#136 site runtime;
- #46 LDAP group refresh (inert here: DisableLogin), #125 OPC UA SDK 2.0.1 (DEV has no OPC UA connection), #128 MxGateway resume cursor;
- Auth 0.5.0, LocalDb 0.3.4 → 0.3.8 (DEV site is single-node, replication off), Telemetry 0.3.1, Theme 0.5.3.
Not touched: QA vq03/vq04, OtOpcUa, mxaccessgw, HistorianGateway, Overview, the OTel collector. Z28061 / Z28061Sim: nothing deployed, written or called. #52 not applied.
Transport: winrm (deploy scope) for three on-box scripts; SMB for staging and logs; direct SQL and the CLI from the Mac for checks. Stage E:\ApiInstall\_stage-sb-8e32f353\.
Preflight (read-only, before the window)
| Check | Result |
|---|---|
| Live build | Host + CLI 424db155 |
__EFMigrationsHistory |
64, last 20261110120000_AuditMaintenanceGrants; AuditLog clustered key leads with OccurredAtUtc; 3 scadabridge_audit_* roles |
| Migrations in range | none (no new file under Migrations/) |
| Option changes in range | all optional with safe defaults (AuditLog:HashChain:ChainBacklog, Security:LdapGroupRefresh, DataConnection:ServiceLevel*, SiteRuntime:InstanceSweepSettle*, Cluster:LeaderAutoDown) |
ASPNETCORE_URLS (#108) |
no https:// |
| appsettings (kept + restored) | A6B6CF05…, 3382DD83…, 3E7AD9A4… — same as 2026-10-22 / 11-11 |
| Service recovery | Both services: restart 5 s / 5 s / 10 s, reset 1 day, FAILURE_ACTIONS_ON_NONCRASH_FAILURES: FALSE → a #132 exit 75 would have left the node stopped |
| AuditLog | 2 394 rows (newest 2026-11-06 — DEV is quiet), 0 chained, 0 heads/tombstones, 843 SourceNode IS NULL (newest 2026-07-07) |
| Baseline snapshots | Z28062 (2) 270 attrs Good 144 / Bad 126, 8 alarms; SBTEST01/02 (9/10) Good 86, 5 alarms each. Site MxGateway Connected, 810/810 resolved, good 398 / bad 412 |
| #131 baseline (old binary, old body) | verify-mes-test-machine.sh groups C+D (C13 removed — SAPID 103559 routes to Z28061): PASS 13/13 |
Window (swap.ps1, notes E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-20261113T071302.txt)
- 07:13:14 staged zip hash + ProductVersion re-verified on the box;
reg exportof both service keys (_envbak-ScadaBridge-{Central,Site}-20261113T071302.regin the stage). - 07:13:17
BACKUP DATABASE … COPY_ONLY, CHECKSUM, COMPRESSION→E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-8e32f353_20261113T071302.bak(131 219 456 B),VERIFYONLYok; appsettings kept in_appsettings-keep-20261113T071302. - 07:13:17 Site stopped, 07:13:27 Central stopped; no Host.exe left.
- 07:13:37
sc.exe failureflag <svc> 1on both services (owner-accepted as part of the deploy) — existing restart actions kept. app/cli→*.bak.20261113T071302, staged copy in, three appsettings restored (SHA match).- 07:13:52 Central ready, 07:14:02 Site ready.
Downtime: site data path ~45 s (07:13:17 → 07:14:02); central API ~25 s. The routine stops logged Application is shutting down… at Information, no [FTL] (#76).
#131 (deploy gate) — applied and verified, DEV SBTEST01/02 only
docs/deployment/deploy-prep/2026-11-12-gates-52-131/apply.sh apply 131 straight after the swap:
TestMesMachine.AlarmStatus(template 30, script 42): ORIGINAL → FIXED (a.Severity→a.Priority×3); metadata read back unchanged.- SBTEST01 deployment 95, SBTEST02 deployment 96, both status 2;
check 131afterwards: FIXED, not stale. - Verify (groups C+D, same harness as the baseline): PASS 13/13 — C3
Severity == 800, C7 MinSeverity 500 → HighVacuumFault + SensorFault, C9 MaxSeverity 750 → SensorFault, i.e. identical to the pre-#129 binary. Alarm sources reset (D1).
Gates
| Gate | Result |
|---|---|
| Health | :8085 and :8084 /health/ready 200 / 200, /health/active 200 / 200; central database, akka-cluster, required-singletons, bundle-signing-policy, audit-chain-head-export Healthy; site akka-cluster, localdb Healthy |
| ProductVersion | Host + CLI 1.0.0+8e32f353… (swap log and a post-deploy inspection) |
| Inbound API | bogus X-API-Key and missing key → 401 UNAUTHORIZED; the inbound executor itself answered the 13 #131 calls (flow-debug run-method, InboundRequest → BTDB → Route → Response) |
| MxGateway / Z28062 | MxGateway Connected, 810/810 resolved, good 398 / bad 412 (= baseline); Z28062 Good 144 / Bad 126, 8 alarms (= baseline); SBTEST01/02 Good 86 ×2 |
| AuditLog | new rows after the swap from central-a (InboundRequest, RouteCall, DbWrite, AttributeWrite, ScriptLog, InboundAuthFailure) and node-a (ScriptRun), SourceNode set; site audit backlog drained |
| Logs | only the usual first-probe akka-cluster … Removed ERRs at boot and the placeholder-seed warnings; no FTL |
Overview (:5320) |
6/6 Up; ScadaBridge central and site both Active/Up, leader chips present, 5/5 and 2/2 checks |
| Env block | SHA-256-prefix digest of every service-env entry identical before and after (Central 12, Site 6). The only service-definition change is the recovery flag FALSE → TRUE |
Audit hash chain (#107 item 2, #139)
-
SourceNode backfill:
scadabridge audit backfill-source-node --before 2026-07-08T00:00:00Z --sentinel unknown→rows updated : 843;SourceNode IS NULLnow 0. -
Enable (
enable-chain.enable.ps1, 07:16:39): one Central env entry added,AuditLog__HashChain__Enabled=true(digestb5bea41b6c62=true).ChainBacklogunset,HeadExportunset (off), noMaintenanceConnectionString. All 12 existing entries unchanged;.regbackup_envbak-ScadaBridge-Central-hashchain-enable-20261113T071639.reg. Central-only restart, ready 07:17:01 (site data path not interrupted). -
Central log:
AuditLog hash chain started at 2026-11-13T12:16:53.1143422Z: only rows dated from then on are chained…. -
SQL (direct, read-only):
Check Result Anchor rows ( AuditLogChainTombstone,ChainSeq 0)exactly 1: month 2026-11-01, cutoff2026-11-13T12:16:53.1143422, EventId5CADAB1D-C4A1-4C0F-A000-000000000139Chained rows dated before the cutoff 0 Rows 2 845 total, 1 chained (a post-cutoff InboundAuthFailure, ChainSeq 1), backlog unchained; AuditLogChainHead2026-11 LastSeq 1; 0 non-anchor tombstones -
scadabridge audit verify-chain(DEV, DisableLogin, Basic fallback):Partition : 2026-11 Partition : 2026-10 Status : Intact Status : NotChained Rows verified : 1 Rows verified : 0 Head sequence : 1 Head sequence : - Unchained rows : 0 Unchained rows : 0 Chain from : 2026-11-13T12:16:53.1143422Z (both) Pre-cutoff rows : 507 Pre-cutoff rows : 667Exit 0 for both. 507 and 667 match direct SQL counts of the November-before-cutoff and October rows.
-
Head export stays off (
audit-chain-head-export: "Chain-head export disabled").
Rollback
All scripts are in E:\ApiInstall\_stage-sb-8e32f353\ (run with winrm --timeout 900 ps vd03 -f <script>):
- Chain only:
enable-chain.disable.ps1removes the env entry and restarts Central. The anchor and chained rows stay (harmless; do not delete the anchor —verify-chainreports that as tampering). - Deploy (documented path):
rollback-Restore.ps1— stop Site + Central, removeAuditLog__HashChain__*from the Central env (424db155has no #139 cutoff and would chain the whole backlog),app/cli←*.bak.20261113T071302,RESTORE … ScadaBridge_pre-8e32f353_20261113T071302.bak WITH REPLACE, CHECKSUM(loses every DB write since 07:13, incl. the backfill, the anchor and the #131 script edit), start both. rollback-None.ps1: the same without the restore (no schema change in the range, so424db155runs on schema 64).- #131 after a binary rollback:
424db155has noScriptAlarm.Priority, and the site still holds the fixed body from deployments 95/96.- After
rollback-None.ps1:apply.sh rollback 131(writes the original body back and redeploys SBTEST01/02). - After
rollback-Restore.ps1: the central template is already the original body again, but the site is not; redeploy SBTEST01 (9) and SBTEST02 (10) withscadabridge instance deploy --id <n>.
- After
- The recovery flag can stay TRUE on either build. Never roll back past
0e540291.
The _appsettings-keep-20261113T071302 and *.bak.20261113T071302 folders can go once the build has soaked.