diff --git a/docs/deployment/deployment-records/2026-11-13-wonder-app-vd03-8e32f353.md b/docs/deployment/deployment-records/2026-11-13-wonder-app-vd03-8e32f353.md new file mode 100644 index 00000000..97624545 --- /dev/null +++ b/docs/deployment/deployment-records/2026-11-13-wonder-app-vd03-8e32f353.md @@ -0,0 +1,104 @@ +# 2026-11-13 — DEV (wonder-app-vd03) `424db155` → `8e32f353`, #131 applied, audit hash chain enabled + +**Owner go:** #107, 2026-11-13: "Deploy, then enable chain". Head export stays off, single SQL login kept. + +**Build:** `main` @ `8e32f353` (= `origin/main`, the LocalDb 0.3.8 pin, scadaproj#57), ProductVersion `1.0.0+8e32f3538ce940e349be3701efd0784a7ebeb551`. +- Published from a clean detached worktree at that SHA: `dotnet publish` Host + CLI `-c Release -r win-x64 --self-contained true -p:PublishSingleFile=false`. +- 1050 app files, 224 cli files. Zip 130 520 104 B, SHA-256 `0cf06cf5cfe5c1a3da2f1c6ce8db77b43f22cbcb79e5237abc4fe51cbc7fdebc`. +- `Host.deps.json` carries `ZB.MOM.WW.LocalDb{,.Replication,.Contracts}/0.3.8`. + +**Delta `424db155..8e32f353`:** 37 commits, **no migration** (64 → 64; the range only edits already-applied migration files, #126/#127). Highlights: +- #129 alarm Severity banding (raw value on `Priority`) — hence the #131 gate below; +- #139 hash-chain cutoff (`52da02a7`), #140 graceful stop, #138 ServiceLevel-aware OPC UA failover; +- #132 leader-auto-down shut-out restart (Host exits **75** on an unplanned ActorSystem termination), #135/#136 site runtime; +- #46 LDAP group refresh (inert here: DisableLogin), #125 OPC UA SDK 2.0.1 (DEV has no OPC UA connection), #128 MxGateway resume cursor; +- Auth 0.5.0, LocalDb 0.3.4 → 0.3.8 (DEV site is single-node, replication off), Telemetry 0.3.1, Theme 0.5.3. + +**Not touched:** QA vq03/vq04, OtOpcUa, mxaccessgw, HistorianGateway, Overview, the OTel collector. **Z28061 / Z28061Sim: nothing deployed, written or called.** #52 not applied. + +**Transport:** `winrm` (deploy scope) for three on-box scripts; SMB for staging and logs; direct SQL and the CLI from the Mac for checks. Stage `E:\ApiInstall\_stage-sb-8e32f353\`. + +## Preflight (read-only, before the window) + +| Check | Result | +|---|---| +| Live build | Host + CLI `424db155` | +| `__EFMigrationsHistory` | 64, last `20261110120000_AuditMaintenanceGrants`; AuditLog clustered key leads with `OccurredAtUtc`; 3 `scadabridge_audit_*` roles | +| Migrations in range | none (no new file under `Migrations/`) | +| Option changes in range | all optional with safe defaults (`AuditLog:HashChain:ChainBacklog`, `Security:LdapGroupRefresh`, `DataConnection:ServiceLevel*`, `SiteRuntime:InstanceSweepSettle*`, `Cluster:LeaderAutoDown`) | +| `ASPNETCORE_URLS` (#108) | no `https://` | +| appsettings (kept + restored) | `A6B6CF05…`, `3382DD83…`, `3E7AD9A4…` — same as 2026-10-22 / 11-11 | +| Service recovery | Both services: restart 5 s / 5 s / 10 s, reset 1 day, **`FAILURE_ACTIONS_ON_NONCRASH_FAILURES: FALSE`** → a #132 exit 75 would have left the node stopped | +| AuditLog | 2 394 rows (newest 2026-11-06 — DEV is quiet), 0 chained, 0 heads/tombstones, **843 `SourceNode IS NULL`** (newest 2026-07-07) | +| Baseline snapshots | Z28062 (2) 270 attrs Good 144 / Bad 126, 8 alarms; SBTEST01/02 (9/10) Good 86, 5 alarms each. Site `MxGateway` Connected, 810/810 resolved, good 398 / bad 412 | +| #131 baseline (old binary, old body) | `verify-mes-test-machine.sh` groups C+D (C13 removed — SAPID 103559 routes to Z28061): **PASS 13/13** | + +## Window (`swap.ps1`, notes `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-20261113T071302.txt`) + +1. 07:13:14 staged zip hash + ProductVersion re-verified on the box; `reg export` of both service keys (`_envbak-ScadaBridge-{Central,Site}-20261113T071302.reg` in the stage). +2. 07:13:17 `BACKUP DATABASE … COPY_ONLY, CHECKSUM, COMPRESSION` → `E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-8e32f353_20261113T071302.bak` (131 219 456 B), `VERIFYONLY` ok; appsettings kept in `_appsettings-keep-20261113T071302`. +3. 07:13:17 Site stopped, 07:13:27 Central stopped; no Host.exe left. +4. 07:13:37 **`sc.exe failureflag 1`** on both services (owner-accepted as part of the deploy) — existing restart actions kept. +5. `app`/`cli` → `*.bak.20261113T071302`, staged copy in, three appsettings restored (SHA match). +6. 07:13:52 Central ready, **07:14:02 Site ready**. + +**Downtime:** site data path ~45 s (07:13:17 → 07:14:02); central API ~25 s. The routine stops logged `Application is shutting down…` at Information, no `[FTL]` (#76). + +## #131 (deploy gate) — applied and verified, DEV SBTEST01/02 only + +`docs/deployment/deploy-prep/2026-11-12-gates-52-131/apply.sh apply 131` straight after the swap: +- `TestMesMachine.AlarmStatus` (template 30, script 42): ORIGINAL → **FIXED** (`a.Severity` → `a.Priority` ×3); metadata read back unchanged. +- SBTEST01 deployment 95, SBTEST02 deployment 96, both status 2; `check 131` afterwards: FIXED, not stale. +- Verify (groups C+D, same harness as the baseline): **PASS 13/13** — C3 `Severity == 800`, C7 MinSeverity 500 → HighVacuumFault + SensorFault, C9 MaxSeverity 750 → SensorFault, i.e. identical to the pre-#129 binary. Alarm sources reset (D1). + +## Gates + +| Gate | Result | +|---|---| +| Health | `:8085` and `:8084` `/health/ready` 200 / 200, `/health/active` 200 / 200; central `database`, `akka-cluster`, `required-singletons`, `bundle-signing-policy`, `audit-chain-head-export` Healthy; site `akka-cluster`, `localdb` Healthy | +| ProductVersion | Host + CLI `1.0.0+8e32f353…` (swap log and a post-deploy inspection) | +| Inbound API | bogus `X-API-Key` and missing key → **401** `UNAUTHORIZED`; the inbound executor itself answered the 13 #131 calls (`flow-debug run-method`, InboundRequest → BTDB → Route → Response) | +| MxGateway / Z28062 | `MxGateway` Connected, 810/810 resolved, good 398 / bad 412 (= baseline); Z28062 Good 144 / Bad 126, 8 alarms (= baseline); SBTEST01/02 Good 86 ×2 | +| AuditLog | new rows after the swap from central-a (InboundRequest, RouteCall, DbWrite, AttributeWrite, ScriptLog, InboundAuthFailure) and node-a (ScriptRun), `SourceNode` set; site audit backlog drained | +| Logs | only the usual first-probe `akka-cluster … Removed` ERRs at boot and the placeholder-seed warnings; no FTL | +| Overview (`:5320`) | 6/6 Up; ScadaBridge central and site both Active/Up, leader chips present, 5/5 and 2/2 checks | +| Env block | SHA-256-prefix digest of every service-env entry identical before and after (Central 12, Site 6). The only service-definition change is the recovery flag FALSE → TRUE | + +## Audit hash chain (#107 item 2, #139) + +1. **SourceNode backfill:** `scadabridge audit backfill-source-node --before 2026-07-08T00:00:00Z --sentinel unknown` → `rows updated : 843`; `SourceNode IS NULL` now 0. +2. **Enable** (`enable-chain.enable.ps1`, 07:16:39): one Central env entry added, `AuditLog__HashChain__Enabled=true` (digest `b5bea41b6c62` = `true`). `ChainBacklog` unset, `HeadExport` unset (off), no `MaintenanceConnectionString`. All 12 existing entries unchanged; `.reg` backup `_envbak-ScadaBridge-Central-hashchain-enable-20261113T071639.reg`. Central-only restart, ready 07:17:01 (site data path not interrupted). +3. Central log: `AuditLog hash chain started at 2026-11-13T12:16:53.1143422Z: only rows dated from then on are chained…`. +4. **SQL (direct, read-only):** + + | Check | Result | + |---|---| + | Anchor rows (`AuditLogChainTombstone`, `ChainSeq 0`) | **exactly 1**: month `2026-11-01`, cutoff `2026-11-13T12:16:53.1143422`, EventId `5CADAB1D-C4A1-4C0F-A000-000000000139` | + | Chained rows dated before the cutoff | **0** | + | Rows | 2 845 total, 1 chained (a post-cutoff InboundAuthFailure, ChainSeq 1), backlog unchained; `AuditLogChainHead` 2026-11 LastSeq 1; 0 non-anchor tombstones | + +5. **`scadabridge audit verify-chain`** (DEV, DisableLogin, Basic fallback): + ``` + Partition : 2026-11 Partition : 2026-10 + Status : Intact Status : NotChained + Rows verified : 1 Rows verified : 0 + Head sequence : 1 Head sequence : - + Unchained rows : 0 Unchained rows : 0 + Chain from : 2026-11-13T12:16:53.1143422Z (both) + Pre-cutoff rows : 507 Pre-cutoff rows : 667 + ``` + Exit 0 for both. 507 and 667 match direct SQL counts of the November-before-cutoff and October rows. +6. Head export stays **off** (`audit-chain-head-export`: "Chain-head export disabled"). + +## Rollback + +All scripts are in `E:\ApiInstall\_stage-sb-8e32f353\` (run with `winrm --timeout 900 ps vd03 -f