docs(deploy): record DEV vd03 424db155 -> 8e32f353, #131 applied, audit hash chain enabled (#107, #139)
No migration (64). ~45 s site outage. SCM non-crash recovery flag set for exit 75 (#132). #131: SBTEST01/02 AlarmStatus on ScriptAlarm.Priority, verify 13/13 before and after. Hash chain: SourceNode backfill 843 rows, Enabled=true (ChainBacklog unset, head export off), one anchor at 2026-11-13T12:16:53Z, 0 pre-cutoff rows chained, verify-chain 2026-11 Intact. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,104 @@
|
|||||||
|
# 2026-11-13 — DEV (wonder-app-vd03) `424db155` → `8e32f353`, #131 applied, audit hash chain enabled
|
||||||
|
|
||||||
|
**Owner go:** #107, 2026-11-13: "Deploy, then enable chain". Head export stays off, single SQL login kept.
|
||||||
|
|
||||||
|
**Build:** `main` @ `8e32f353` (= `origin/main`, the LocalDb 0.3.8 pin, scadaproj#57), ProductVersion `1.0.0+8e32f3538ce940e349be3701efd0784a7ebeb551`.
|
||||||
|
- Published from a clean detached worktree at that SHA: `dotnet publish` Host + CLI `-c Release -r win-x64 --self-contained true -p:PublishSingleFile=false`.
|
||||||
|
- 1050 app files, 224 cli files. Zip 130 520 104 B, SHA-256 `0cf06cf5cfe5c1a3da2f1c6ce8db77b43f22cbcb79e5237abc4fe51cbc7fdebc`.
|
||||||
|
- `Host.deps.json` carries `ZB.MOM.WW.LocalDb{,.Replication,.Contracts}/0.3.8`.
|
||||||
|
|
||||||
|
**Delta `424db155..8e32f353`:** 37 commits, **no migration** (64 → 64; the range only edits already-applied migration files, #126/#127). Highlights:
|
||||||
|
- #129 alarm Severity banding (raw value on `Priority`) — hence the #131 gate below;
|
||||||
|
- #139 hash-chain cutoff (`52da02a7`), #140 graceful stop, #138 ServiceLevel-aware OPC UA failover;
|
||||||
|
- #132 leader-auto-down shut-out restart (Host exits **75** on an unplanned ActorSystem termination), #135/#136 site runtime;
|
||||||
|
- #46 LDAP group refresh (inert here: DisableLogin), #125 OPC UA SDK 2.0.1 (DEV has no OPC UA connection), #128 MxGateway resume cursor;
|
||||||
|
- Auth 0.5.0, LocalDb 0.3.4 → 0.3.8 (DEV site is single-node, replication off), Telemetry 0.3.1, Theme 0.5.3.
|
||||||
|
|
||||||
|
**Not touched:** QA vq03/vq04, OtOpcUa, mxaccessgw, HistorianGateway, Overview, the OTel collector. **Z28061 / Z28061Sim: nothing deployed, written or called.** #52 not applied.
|
||||||
|
|
||||||
|
**Transport:** `winrm` (deploy scope) for three on-box scripts; SMB for staging and logs; direct SQL and the CLI from the Mac for checks. Stage `E:\ApiInstall\_stage-sb-8e32f353\`.
|
||||||
|
|
||||||
|
## Preflight (read-only, before the window)
|
||||||
|
|
||||||
|
| Check | Result |
|
||||||
|
|---|---|
|
||||||
|
| Live build | Host + CLI `424db155` |
|
||||||
|
| `__EFMigrationsHistory` | 64, last `20261110120000_AuditMaintenanceGrants`; AuditLog clustered key leads with `OccurredAtUtc`; 3 `scadabridge_audit_*` roles |
|
||||||
|
| Migrations in range | none (no new file under `Migrations/`) |
|
||||||
|
| Option changes in range | all optional with safe defaults (`AuditLog:HashChain:ChainBacklog`, `Security:LdapGroupRefresh`, `DataConnection:ServiceLevel*`, `SiteRuntime:InstanceSweepSettle*`, `Cluster:LeaderAutoDown`) |
|
||||||
|
| `ASPNETCORE_URLS` (#108) | no `https://` |
|
||||||
|
| appsettings (kept + restored) | `A6B6CF05…`, `3382DD83…`, `3E7AD9A4…` — same as 2026-10-22 / 11-11 |
|
||||||
|
| Service recovery | Both services: restart 5 s / 5 s / 10 s, reset 1 day, **`FAILURE_ACTIONS_ON_NONCRASH_FAILURES: FALSE`** → a #132 exit 75 would have left the node stopped |
|
||||||
|
| AuditLog | 2 394 rows (newest 2026-11-06 — DEV is quiet), 0 chained, 0 heads/tombstones, **843 `SourceNode IS NULL`** (newest 2026-07-07) |
|
||||||
|
| Baseline snapshots | Z28062 (2) 270 attrs Good 144 / Bad 126, 8 alarms; SBTEST01/02 (9/10) Good 86, 5 alarms each. Site `MxGateway` Connected, 810/810 resolved, good 398 / bad 412 |
|
||||||
|
| #131 baseline (old binary, old body) | `verify-mes-test-machine.sh` groups C+D (C13 removed — SAPID 103559 routes to Z28061): **PASS 13/13** |
|
||||||
|
|
||||||
|
## Window (`swap.ps1`, notes `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-20261113T071302.txt`)
|
||||||
|
|
||||||
|
1. 07:13:14 staged zip hash + ProductVersion re-verified on the box; `reg export` of both service keys (`_envbak-ScadaBridge-{Central,Site}-20261113T071302.reg` in the stage).
|
||||||
|
2. 07:13:17 `BACKUP DATABASE … COPY_ONLY, CHECKSUM, COMPRESSION` → `E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-8e32f353_20261113T071302.bak` (131 219 456 B), `VERIFYONLY` ok; appsettings kept in `_appsettings-keep-20261113T071302`.
|
||||||
|
3. 07:13:17 Site stopped, 07:13:27 Central stopped; no Host.exe left.
|
||||||
|
4. 07:13:37 **`sc.exe failureflag <svc> 1`** on both services (owner-accepted as part of the deploy) — existing restart actions kept.
|
||||||
|
5. `app`/`cli` → `*.bak.20261113T071302`, staged copy in, three appsettings restored (SHA match).
|
||||||
|
6. 07:13:52 Central ready, **07:14:02 Site ready**.
|
||||||
|
|
||||||
|
**Downtime:** site data path ~45 s (07:13:17 → 07:14:02); central API ~25 s. The routine stops logged `Application is shutting down…` at Information, no `[FTL]` (#76).
|
||||||
|
|
||||||
|
## #131 (deploy gate) — applied and verified, DEV SBTEST01/02 only
|
||||||
|
|
||||||
|
`docs/deployment/deploy-prep/2026-11-12-gates-52-131/apply.sh apply 131` straight after the swap:
|
||||||
|
- `TestMesMachine.AlarmStatus` (template 30, script 42): ORIGINAL → **FIXED** (`a.Severity` → `a.Priority` ×3); metadata read back unchanged.
|
||||||
|
- SBTEST01 deployment 95, SBTEST02 deployment 96, both status 2; `check 131` afterwards: FIXED, not stale.
|
||||||
|
- Verify (groups C+D, same harness as the baseline): **PASS 13/13** — C3 `Severity == 800`, C7 MinSeverity 500 → HighVacuumFault + SensorFault, C9 MaxSeverity 750 → SensorFault, i.e. identical to the pre-#129 binary. Alarm sources reset (D1).
|
||||||
|
|
||||||
|
## Gates
|
||||||
|
|
||||||
|
| Gate | Result |
|
||||||
|
|---|---|
|
||||||
|
| Health | `:8085` and `:8084` `/health/ready` 200 / 200, `/health/active` 200 / 200; central `database`, `akka-cluster`, `required-singletons`, `bundle-signing-policy`, `audit-chain-head-export` Healthy; site `akka-cluster`, `localdb` Healthy |
|
||||||
|
| ProductVersion | Host + CLI `1.0.0+8e32f353…` (swap log and a post-deploy inspection) |
|
||||||
|
| Inbound API | bogus `X-API-Key` and missing key → **401** `UNAUTHORIZED`; the inbound executor itself answered the 13 #131 calls (`flow-debug run-method`, InboundRequest → BTDB → Route → Response) |
|
||||||
|
| MxGateway / Z28062 | `MxGateway` Connected, 810/810 resolved, good 398 / bad 412 (= baseline); Z28062 Good 144 / Bad 126, 8 alarms (= baseline); SBTEST01/02 Good 86 ×2 |
|
||||||
|
| AuditLog | new rows after the swap from central-a (InboundRequest, RouteCall, DbWrite, AttributeWrite, ScriptLog, InboundAuthFailure) and node-a (ScriptRun), `SourceNode` set; site audit backlog drained |
|
||||||
|
| Logs | only the usual first-probe `akka-cluster … Removed` ERRs at boot and the placeholder-seed warnings; no FTL |
|
||||||
|
| Overview (`:5320`) | 6/6 Up; ScadaBridge central and site both Active/Up, leader chips present, 5/5 and 2/2 checks |
|
||||||
|
| Env block | SHA-256-prefix digest of every service-env entry identical before and after (Central 12, Site 6). The only service-definition change is the recovery flag FALSE → TRUE |
|
||||||
|
|
||||||
|
## Audit hash chain (#107 item 2, #139)
|
||||||
|
|
||||||
|
1. **SourceNode backfill:** `scadabridge audit backfill-source-node --before 2026-07-08T00:00:00Z --sentinel unknown` → `rows updated : 843`; `SourceNode IS NULL` now 0.
|
||||||
|
2. **Enable** (`enable-chain.enable.ps1`, 07:16:39): one Central env entry added, `AuditLog__HashChain__Enabled=true` (digest `b5bea41b6c62` = `true`). `ChainBacklog` unset, `HeadExport` unset (off), no `MaintenanceConnectionString`. All 12 existing entries unchanged; `.reg` backup `_envbak-ScadaBridge-Central-hashchain-enable-20261113T071639.reg`. Central-only restart, ready 07:17:01 (site data path not interrupted).
|
||||||
|
3. Central log: `AuditLog hash chain started at 2026-11-13T12:16:53.1143422Z: only rows dated from then on are chained…`.
|
||||||
|
4. **SQL (direct, read-only):**
|
||||||
|
|
||||||
|
| Check | Result |
|
||||||
|
|---|---|
|
||||||
|
| Anchor rows (`AuditLogChainTombstone`, `ChainSeq 0`) | **exactly 1**: month `2026-11-01`, cutoff `2026-11-13T12:16:53.1143422`, EventId `5CADAB1D-C4A1-4C0F-A000-000000000139` |
|
||||||
|
| Chained rows dated before the cutoff | **0** |
|
||||||
|
| Rows | 2 845 total, 1 chained (a post-cutoff InboundAuthFailure, ChainSeq 1), backlog unchained; `AuditLogChainHead` 2026-11 LastSeq 1; 0 non-anchor tombstones |
|
||||||
|
|
||||||
|
5. **`scadabridge audit verify-chain`** (DEV, DisableLogin, Basic fallback):
|
||||||
|
```
|
||||||
|
Partition : 2026-11 Partition : 2026-10
|
||||||
|
Status : Intact Status : NotChained
|
||||||
|
Rows verified : 1 Rows verified : 0
|
||||||
|
Head sequence : 1 Head sequence : -
|
||||||
|
Unchained rows : 0 Unchained rows : 0
|
||||||
|
Chain from : 2026-11-13T12:16:53.1143422Z (both)
|
||||||
|
Pre-cutoff rows : 507 Pre-cutoff rows : 667
|
||||||
|
```
|
||||||
|
Exit 0 for both. 507 and 667 match direct SQL counts of the November-before-cutoff and October rows.
|
||||||
|
6. Head export stays **off** (`audit-chain-head-export`: "Chain-head export disabled").
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
All scripts are in `E:\ApiInstall\_stage-sb-8e32f353\` (run with `winrm --timeout 900 ps vd03 -f <script>`):
|
||||||
|
- **Chain only:** `enable-chain.disable.ps1` removes the env entry and restarts Central. The anchor and chained rows stay (harmless; do not delete the anchor — `verify-chain` reports that as tampering).
|
||||||
|
- **Deploy (documented path):** `rollback-Restore.ps1` — stop Site + Central, remove `AuditLog__HashChain__*` from the Central env (424db155 has no #139 cutoff and would chain the whole backlog), `app`/`cli` ← `*.bak.20261113T071302`, `RESTORE … ScadaBridge_pre-8e32f353_20261113T071302.bak WITH REPLACE, CHECKSUM` (loses every DB write since 07:13, incl. the backfill, the anchor and the #131 script edit), start both.
|
||||||
|
- `rollback-None.ps1`: the same without the restore (no schema change in the range, so 424db155 runs on schema 64).
|
||||||
|
- #131 after a binary rollback: 424db155 has no `ScriptAlarm.Priority`, and the site still holds the fixed body from deployments 95/96.
|
||||||
|
- After `rollback-None.ps1`: `apply.sh rollback 131` (writes the original body back and redeploys SBTEST01/02).
|
||||||
|
- After `rollback-Restore.ps1`: the central template is already the original body again, but the site is not; redeploy SBTEST01 (9) and SBTEST02 (10) with `scadabridge instance deploy --id <n>`.
|
||||||
|
- The recovery flag can stay TRUE on either build. Never roll back past `0e540291`.
|
||||||
|
|
||||||
|
The `_appsettings-keep-20261113T071302` and `*.bak.20261113T071302` folders can go once the build has soaked.
|
||||||
Reference in New Issue
Block a user