Files
ScadaBridge/docs/deployment/deployment-records/2026-11-13-wonder-app-vd03-8e32f353.md
T
Joseph Doherty 581e25caaf docs(deploy): record DEV vd03 424db155 -> 8e32f353, #131 applied, audit hash chain enabled (#107, #139)
No migration (64). ~45 s site outage. SCM non-crash recovery flag set for exit 75 (#132).
#131: SBTEST01/02 AlarmStatus on ScriptAlarm.Priority, verify 13/13 before and after.
Hash chain: SourceNode backfill 843 rows, Enabled=true (ChainBacklog unset, head export off),
one anchor at 2026-11-13T12:16:53Z, 0 pre-cutoff rows chained, verify-chain 2026-11 Intact.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-13 07:21:31 -05:00

105 lines
9.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 2026-11-13 — DEV (wonder-app-vd03) `424db155` → `8e32f353`, #131 applied, audit hash chain enabled
**Owner go:** #107, 2026-11-13: "Deploy, then enable chain". Head export stays off, single SQL login kept.
**Build:** `main` @ `8e32f353` (= `origin/main`, the LocalDb 0.3.8 pin, scadaproj#57), ProductVersion `1.0.0+8e32f3538ce940e349be3701efd0784a7ebeb551`.
- Published from a clean detached worktree at that SHA: `dotnet publish` Host + CLI `-c Release -r win-x64 --self-contained true -p:PublishSingleFile=false`.
- 1050 app files, 224 cli files. Zip 130 520 104 B, SHA-256 `0cf06cf5cfe5c1a3da2f1c6ce8db77b43f22cbcb79e5237abc4fe51cbc7fdebc`.
- `Host.deps.json` carries `ZB.MOM.WW.LocalDb{,.Replication,.Contracts}/0.3.8`.
**Delta `424db155..8e32f353`:** 37 commits, **no migration** (64 → 64; the range only edits already-applied migration files, #126/#127). Highlights:
- #129 alarm Severity banding (raw value on `Priority`) — hence the #131 gate below;
- #139 hash-chain cutoff (`52da02a7`), #140 graceful stop, #138 ServiceLevel-aware OPC UA failover;
- #132 leader-auto-down shut-out restart (Host exits **75** on an unplanned ActorSystem termination), #135/#136 site runtime;
- #46 LDAP group refresh (inert here: DisableLogin), #125 OPC UA SDK 2.0.1 (DEV has no OPC UA connection), #128 MxGateway resume cursor;
- Auth 0.5.0, LocalDb 0.3.4 → 0.3.8 (DEV site is single-node, replication off), Telemetry 0.3.1, Theme 0.5.3.
**Not touched:** QA vq03/vq04, OtOpcUa, mxaccessgw, HistorianGateway, Overview, the OTel collector. **Z28061 / Z28061Sim: nothing deployed, written or called.** #52 not applied.
**Transport:** `winrm` (deploy scope) for three on-box scripts; SMB for staging and logs; direct SQL and the CLI from the Mac for checks. Stage `E:\ApiInstall\_stage-sb-8e32f353\`.
## Preflight (read-only, before the window)
| Check | Result |
|---|---|
| Live build | Host + CLI `424db155` |
| `__EFMigrationsHistory` | 64, last `20261110120000_AuditMaintenanceGrants`; AuditLog clustered key leads with `OccurredAtUtc`; 3 `scadabridge_audit_*` roles |
| Migrations in range | none (no new file under `Migrations/`) |
| Option changes in range | all optional with safe defaults (`AuditLog:HashChain:ChainBacklog`, `Security:LdapGroupRefresh`, `DataConnection:ServiceLevel*`, `SiteRuntime:InstanceSweepSettle*`, `Cluster:LeaderAutoDown`) |
| `ASPNETCORE_URLS` (#108) | no `https://` |
| appsettings (kept + restored) | `A6B6CF05…`, `3382DD83…`, `3E7AD9A4…` — same as 2026-10-22 / 11-11 |
| Service recovery | Both services: restart 5 s / 5 s / 10 s, reset 1 day, **`FAILURE_ACTIONS_ON_NONCRASH_FAILURES: FALSE`** → a #132 exit 75 would have left the node stopped |
| AuditLog | 2 394 rows (newest 2026-11-06 — DEV is quiet), 0 chained, 0 heads/tombstones, **843 `SourceNode IS NULL`** (newest 2026-07-07) |
| Baseline snapshots | Z28062 (2) 270 attrs Good 144 / Bad 126, 8 alarms; SBTEST01/02 (9/10) Good 86, 5 alarms each. Site `MxGateway` Connected, 810/810 resolved, good 398 / bad 412 |
| #131 baseline (old binary, old body) | `verify-mes-test-machine.sh` groups C+D (C13 removed — SAPID 103559 routes to Z28061): **PASS 13/13** |
## Window (`swap.ps1`, notes `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-20261113T071302.txt`)
1. 07:13:14 staged zip hash + ProductVersion re-verified on the box; `reg export` of both service keys (`_envbak-ScadaBridge-{Central,Site}-20261113T071302.reg` in the stage).
2. 07:13:17 `BACKUP DATABASE … COPY_ONLY, CHECKSUM, COMPRESSION` → `E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-8e32f353_20261113T071302.bak` (131 219 456 B), `VERIFYONLY` ok; appsettings kept in `_appsettings-keep-20261113T071302`.
3. 07:13:17 Site stopped, 07:13:27 Central stopped; no Host.exe left.
4. 07:13:37 **`sc.exe failureflag <svc> 1`** on both services (owner-accepted as part of the deploy) — existing restart actions kept.
5. `app`/`cli` → `*.bak.20261113T071302`, staged copy in, three appsettings restored (SHA match).
6. 07:13:52 Central ready, **07:14:02 Site ready**.
**Downtime:** site data path ~45 s (07:13:17 → 07:14:02); central API ~25 s. The routine stops logged `Application is shutting down…` at Information, no `[FTL]` (#76).
## #131 (deploy gate) — applied and verified, DEV SBTEST01/02 only
`docs/deployment/deploy-prep/2026-11-12-gates-52-131/apply.sh apply 131` straight after the swap:
- `TestMesMachine.AlarmStatus` (template 30, script 42): ORIGINAL → **FIXED** (`a.Severity` → `a.Priority` ×3); metadata read back unchanged.
- SBTEST01 deployment 95, SBTEST02 deployment 96, both status 2; `check 131` afterwards: FIXED, not stale.
- Verify (groups C+D, same harness as the baseline): **PASS 13/13** — C3 `Severity == 800`, C7 MinSeverity 500 → HighVacuumFault + SensorFault, C9 MaxSeverity 750 → SensorFault, i.e. identical to the pre-#129 binary. Alarm sources reset (D1).
## Gates
| Gate | Result |
|---|---|
| Health | `:8085` and `:8084` `/health/ready` 200 / 200, `/health/active` 200 / 200; central `database`, `akka-cluster`, `required-singletons`, `bundle-signing-policy`, `audit-chain-head-export` Healthy; site `akka-cluster`, `localdb` Healthy |
| ProductVersion | Host + CLI `1.0.0+8e32f353…` (swap log and a post-deploy inspection) |
| Inbound API | bogus `X-API-Key` and missing key → **401** `UNAUTHORIZED`; the inbound executor itself answered the 13 #131 calls (`flow-debug run-method`, InboundRequest → BTDB → Route → Response) |
| MxGateway / Z28062 | `MxGateway` Connected, 810/810 resolved, good 398 / bad 412 (= baseline); Z28062 Good 144 / Bad 126, 8 alarms (= baseline); SBTEST01/02 Good 86 ×2 |
| AuditLog | new rows after the swap from central-a (InboundRequest, RouteCall, DbWrite, AttributeWrite, ScriptLog, InboundAuthFailure) and node-a (ScriptRun), `SourceNode` set; site audit backlog drained |
| Logs | only the usual first-probe `akka-cluster … Removed` ERRs at boot and the placeholder-seed warnings; no FTL |
| Overview (`:5320`) | 6/6 Up; ScadaBridge central and site both Active/Up, leader chips present, 5/5 and 2/2 checks |
| Env block | SHA-256-prefix digest of every service-env entry identical before and after (Central 12, Site 6). The only service-definition change is the recovery flag FALSE → TRUE |
## Audit hash chain (#107 item 2, #139)
1. **SourceNode backfill:** `scadabridge audit backfill-source-node --before 2026-07-08T00:00:00Z --sentinel unknown` → `rows updated : 843`; `SourceNode IS NULL` now 0.
2. **Enable** (`enable-chain.enable.ps1`, 07:16:39): one Central env entry added, `AuditLog__HashChain__Enabled=true` (digest `b5bea41b6c62` = `true`). `ChainBacklog` unset, `HeadExport` unset (off), no `MaintenanceConnectionString`. All 12 existing entries unchanged; `.reg` backup `_envbak-ScadaBridge-Central-hashchain-enable-20261113T071639.reg`. Central-only restart, ready 07:17:01 (site data path not interrupted).
3. Central log: `AuditLog hash chain started at 2026-11-13T12:16:53.1143422Z: only rows dated from then on are chained…`.
4. **SQL (direct, read-only):**
| Check | Result |
|---|---|
| Anchor rows (`AuditLogChainTombstone`, `ChainSeq 0`) | **exactly 1**: month `2026-11-01`, cutoff `2026-11-13T12:16:53.1143422`, EventId `5CADAB1D-C4A1-4C0F-A000-000000000139` |
| Chained rows dated before the cutoff | **0** |
| Rows | 2 845 total, 1 chained (a post-cutoff InboundAuthFailure, ChainSeq 1), backlog unchained; `AuditLogChainHead` 2026-11 LastSeq 1; 0 non-anchor tombstones |
5. **`scadabridge audit verify-chain`** (DEV, DisableLogin, Basic fallback):
```
Partition : 2026-11 Partition : 2026-10
Status : Intact Status : NotChained
Rows verified : 1 Rows verified : 0
Head sequence : 1 Head sequence : -
Unchained rows : 0 Unchained rows : 0
Chain from : 2026-11-13T12:16:53.1143422Z (both)
Pre-cutoff rows : 507 Pre-cutoff rows : 667
```
Exit 0 for both. 507 and 667 match direct SQL counts of the November-before-cutoff and October rows.
6. Head export stays **off** (`audit-chain-head-export`: "Chain-head export disabled").
## Rollback
All scripts are in `E:\ApiInstall\_stage-sb-8e32f353\` (run with `winrm --timeout 900 ps vd03 -f <script>`):
- **Chain only:** `enable-chain.disable.ps1` removes the env entry and restarts Central. The anchor and chained rows stay (harmless; do not delete the anchor — `verify-chain` reports that as tampering).
- **Deploy (documented path):** `rollback-Restore.ps1` — stop Site + Central, remove `AuditLog__HashChain__*` from the Central env (424db155 has no #139 cutoff and would chain the whole backlog), `app`/`cli` ← `*.bak.20261113T071302`, `RESTORE … ScadaBridge_pre-8e32f353_20261113T071302.bak WITH REPLACE, CHECKSUM` (loses every DB write since 07:13, incl. the backfill, the anchor and the #131 script edit), start both.
- `rollback-None.ps1`: the same without the restore (no schema change in the range, so 424db155 runs on schema 64).
- #131 after a binary rollback: 424db155 has no `ScriptAlarm.Priority`, and the site still holds the fixed body from deployments 95/96.
- After `rollback-None.ps1`: `apply.sh rollback 131` (writes the original body back and redeploys SBTEST01/02).
- After `rollback-Restore.ps1`: the central template is already the original body again, but the site is not; redeploy SBTEST01 (9) and SBTEST02 (10) with `scadabridge instance deploy --id <n>`.
- The recovery flag can stay TRUE on either build. Never roll back past `0e540291`.
The `_appsettings-keep-20261113T071302` and `*.bak.20261113T071302` folders can go once the build has soaked.