From b19836788bae32c3f091005296141a78c029e45e Mon Sep 17 00:00:00 2001
From: Joseph Doherty <dohejw01@gmail.com>
Date: Fri, 13 Nov 2026 09:13:00 -0500
Subject: [PATCH] docs(deploy): record QA vq03/vq04 audit hash chain enablement
 (#107, #139)

SourceNode backfill 843 rows (0 NULL), AuditLog__HashChain__Enabled=true on both
centrals (ChainBacklog unset, head export off, single login), pair stopped together and
started vq03 first: central API ~29 s, sites not restarted. One anchor at
2026-11-13T14:07:43.3208169Z, 0 pre-cutoff rows chained, 20 new rows ChainSeq 1-20,
verify-chain 2026-11 Intact on both centrals, 2026-10 NotChained. MxGateway 2196 Good,
WarmCanary 2 Good, LocalDb sync connected. Backup ScadaBridge_pre-hashchain_20261113T090524.bak.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
---
 .../2026-11-13-qa-audit-hash-chain.md         | 106 ++++++++++++++++++
 1 file changed, 106 insertions(+)
 create mode 100644 docs/deployment/deployment-records/2026-11-13-qa-audit-hash-chain.md

diff --git a/docs/deployment/deployment-records/2026-11-13-qa-audit-hash-chain.md b/docs/deployment/deployment-records/2026-11-13-qa-audit-hash-chain.md
new file mode 100644
index 00000000..8616372d
--- /dev/null
+++ b/docs/deployment/deployment-records/2026-11-13-qa-audit-hash-chain.md
@@ -0,0 +1,106 @@
+# 2026-11-13: QA (wonder-app-vq03 + vq04) audit hash chain enabled (#107 item 2, #139 cutoff)
+
+**Owner go:** #107, 2026-11-13: "`AuditLog:HashChain:Enabled` on DEV now, **QA after its window**. No backlog chaining. Head export off. Maintenance login later." The QA window ran earlier today (`2026-11-13-wonder-app-vq03-vq04-8e32f353.md`). This record follows the DEV enablement (`2026-11-13-wonder-app-vd03-8e32f353.md`, "Audit hash chain") and the QA pack §7.
+
+**Build:** unchanged. `8e32f353` (`1.0.0+8e32f3538ce9…`, which contains the #139 cutoff `52da02a7`) is on both centrals, re-checked on the box before the change.
+
+**Scope:**
+- One service-env entry was added to **both** `ScadaBridge-Central` services: `AuditLog__HashChain__Enabled=true`. Both centrals were then stopped together and started together.
+- `ChainBacklog` is unset, so the backlog is not chained. `HeadExport` is unset, so the export is off. `MaintenanceConnectionString` is unset, so the single SQL login stays.
+- Not touched:
+  - the Site services (not restarted), appsettings, and every other env entry;
+  - OtOpcUa (Warm: vq04 Primary, vq03 Standby), mxaccessgw, HistorianGateway, Overview, the OTel collector and the `ScadaBridge-LDAP` service;
+  - DEV vd03 and the docker rigs;
+  - Z28061: nothing was deployed, written or called.
+
+**Transport:**
+- `winrm` (deploy scope) ran one rendered script per step on the box: `qa-chain-{inspect,set-env,stop-central,start-central}.ps1`.
+- SMB was used for logs and appsettings, and to stage the scripts in `E:\psrem-stage\sb-8e32f353\`.
+- The backup, backfill checks and chain checks ran from the Mac over direct SQL to `wonder-sql-vq03`. The CLI (`8e32f353` build) and HTTP also ran from the Mac.
+- On-box notes are in `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-hashchain-20261113T090524.txt`.
+
+## Preflight (read-only, 09:04–09:06 EST)
+
+| Check | Result |
+|---|---|
+| Live build | Host `1.0.0+8e32f353…` on both boxes |
+| Schema | 64 migrations, last `20261110120000_AuditMaintenanceGrants` |
+| AuditLog | 3 453 rows, **0 chained**. `AuditLogChainHead` 0 and `AuditLogChainTombstone` 0. **843 `SourceNode IS NULL`** (2026-05-29 → 2026-07-07) |
+| Service env | Central 15 entries, Site 8. Digests are **identical on both boxes** (key name + SHA-256 prefix only). No `AuditLog__*` key |
+| appsettings | Unchanged since the window (`C0C653D6…` on both; Central vq03 `EF01B129…`, vq04 `F333D48F…`; Site vq03 `D0E202F9…`, vq04 `E9E6336F…`). Neither file contains `HashChain` |
+| Per-node values | `JwtExpiryMinutes` / `IdleTimeoutMinutes` are 25 / 60 on vq03 and 15 / 30 on vq04 |
+| Health | `/health/ready` 200 ×4. `/health/active`: vq03 200 / vq04 503 on both pairs |
+| Functional baseline | Site `MxGateway` Connected, 2196/2196 Good. `WarmCanary` "Connected to backup", 2 Good. LocalDb `replicationConnected true`, oplog backlog 0, site audit backlog 0. Z28062–Z28069: 244 Good / 0 Bad each, 10 alarms each. 10 deployed instances |
+
+## Steps
+
+| Step | Local time (EST) | Result |
+|---|---|---|
+| 1. Backup from the Mac | 09:06:22 → 09:06:26 | `BACKUP DATABASE [ScadaBridge] TO DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-hashchain_20261113T090524.bak' WITH COPY_ONLY, CHECKSUM, COMPRESSION`: 154 268 127 B compressed. `RESTORE VERIFYONLY … WITH CHECKSUM` reports the set valid; msdb records `is_copy_only 1` and `has_backup_checksums 1` |
+| 2. SourceNode backfill (CLI → active central vq03) | 09:06:33 | `audit backfill-source-node --before 2026-07-08T00:00:00Z --sentinel unknown` → `rows updated : 843`. SQL afterwards: **`SourceNode IS NULL` = 0**, 843 rows `unknown`, still 0 chained |
+| 3. `set-env` on vq03, then vq04 (no restart) | 09:06:40 → 09:06:57 | `reg export` → `E:\psrem-stage\sb-8e32f353\_envbak-ScadaBridge-Central-hashchain-set-env-20261113T090524.reg` on each box. Central env went from 15 to 16 entries; every existing entry was asserted byte-identical |
+| 4. Stop Central vq04 (standby) | 09:07:05 → 09:07:15 | Graceful (`CoordinatedShutdown`). Site service still Running |
+| 5. Stop Central vq03 (active; **central API down**) | 09:07:22 → 09:07:33 | Graceful. Site service still Running |
+| 6. Start Central vq03, then vq04 2 s later | 09:07:39 → 09:07:52 | vq03 ready 09:07:51.7, vq04 ready 09:07:52.7 |
+
+**Downtime:**
+- Central API (active node): **about 29 s** (09:07:22.9 → 09:07:51.7).
+- Whole central pair: about 48 s, from the vq04 stop at 09:07:05 to vq04 ready.
+- The site data path was **not interrupted**, because the Site services were not stopped. Site audit rows buffer in the site backlog while the central is down, and the backlog was 0 afterwards.
+
+## Gates
+
+| Gate | Result |
+|---|---|
+| Health | `/health/ready` 200 ×4. `/health/active`: central vq03 200 / vq04 503, site vq03 200 / vq04 503, so **exactly one active central**. Central checks on both nodes are Healthy: `database`, `akka-cluster`, `required-singletons`, `bundle-signing-policy`, `audit-chain-head-export` |
+| Cluster | `health summary`: central nodes 10.220.158.2 Primary and .3 Standby, both online. Akka log: `[leader-auto-down] started` with `shut-out restart is ON` on both centrals (14:07:43Z and 14:07:49Z), and vq03 `gained leadership` at 14:07:48Z |
+| **Anchor cutoff** | Exactly **1** `AuditLogChainTombstone` row: `ChainMonth 2026-11-01`, `ChainSeq 0`, EventId `5CADAB1D-C4A1-4C0F-A000-000000000139`, **`PurgedAtUtc` (cutoff) `2026-11-13T14:07:43.3208169Z`**. vq03 Central log, 09:07:43.325 EST: `AuditLog hash chain started at 2026-11-13T14:07:43.3208169Z: only rows dated from then on are chained; earlier rows stay unchained and out of scope (ScadaBridge#139).` vq04 wrote no second anchor |
+| New rows chained, old rows not | Post-cutoff traffic: 4 bogus/missing-key inbound calls (2 per node; vq04's reach the active node through the standby relay) and one `flow-debug run-method --method-id 9` (`SimpleAlarmStatusRequest`, `SAPID=000000`; InboundRequest → DbWrite → ScriptLog ×14 → completed). SQL: **20 post-cutoff rows, `ChainSeq` 1–20**, all `central-a`: InboundAuthFailure ×4, InboundRequest, DbWrite, ScriptLog ×14. **Pre-cutoff rows chained: 0**; pre-cutoff rows unchained: 3 453 (all of them). Unchained post-cutoff rows: 0. `AuditLogChainHead` 2026-11 `LastSeq 20` |
+| `verify-chain` | Run against **both** centrals. **2026-11:** `Intact`, rows verified 20, head sequence 20, unchained 0, chain from `2026-11-13T14:07:43.3208169Z`, pre-cutoff rows **662**. **2026-10:** `NotChained`, pre-cutoff rows **1 466**. Exit 0 on all four runs. Direct SQL gives the same counts: 662 November rows before the cutoff and 1 466 October rows |
+| Head export | `audit-chain-head-export`: "Chain-head export disabled." on both centrals |
+| Inbound API | A bogus `X-API-Key` and a missing key both get **401** `UNAUTHORIZED` on vq03 and on vq04 (relay). Method 9 completed: InboundRequest → BTDB → Ok |
+| MxGateway | Site health: `MxGateway` Connected, **2196/2196 resolved, 2196 Good / 0 Bad** (= baseline). Z28062–Z28069: 244 Good each, 10 alarms each (= baseline) |
+| WarmCanary | "Connected to backup" (vq04 OtOpcUa, the Warm Primary), **2 Good / 0 Bad**, instance 16 snapshot 2 Good |
+| Site sync | `localDbReplicationConnected true`, oplog backlog 0, site audit backlog 0. Site role Active on vq03, both site nodes online. Site services were not restarted |
+| Logs | Since 09:06 the only ERR/FTL lines are the usual first-probe `akka-cluster … Removed` health ERRs at boot: 7 on vq03 (09:07:44–48) and 4 on vq04 (09:07:49–51). No FTL |
+
+**Env and appsettings, before vs after (key name + SHA-256 prefix only):**
+- The only change on each box is **one added Central entry**, `AuditLog__HashChain__Enabled` `sha256:b5bea41b6c62` len 4. This is the same digest as DEV and equals `true`.
+- The 14 other Central entries and all 8 Site entries are unchanged.
+- The env digests are identical on vq03 and vq04.
+- All three appsettings files have the same hashes as before on both boxes, so the per-node `JwtExpiryMinutes` / `IdleTimeoutMinutes` (25 / 60 on vq03, 15 / 30 on vq04) are preserved.
+
+## Rollback
+
+Every on-box script is in `E:\psrem-stage\sb-8e32f353\` on both boxes. Run each with `winrm --timeout 900 ps <box> -f <script>`.
+
+**Chain only (keep the build and the DB):**
+1. Run `qa-chain-unset-env.ps1` on vq03 and on vq04. It exports the key to `.reg`, removes `AuditLog__HashChain__Enabled` from the Central env and asserts the other entries unchanged. It does not restart anything.
+2. Run `qa-chain-stop-central.ps1` on vq04, then on vq03.
+3. Run `qa-chain-start-central.ps1` on vq03, then on vq04.
+
+The anchor and the chained rows stay. They are harmless, and chaining resumes from them if the chain is re-enabled. **Do not delete the anchor row:** `verify-chain` reports that as tampering. The `.reg` exports from step 3 hold the pre-change key if the env ever needs restoring by hand.
+
+**DB restore to before the enablement (loses every central DB write since 09:06:25):**
+1. Do the chain-only steps 1–2 above. Both centrals are now stopped. The sites can stay up, because they don't connect to this DB.
+2. From the Mac:
+   ```sql
+   ALTER DATABASE [ScadaBridge] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
+   RESTORE DATABASE [ScadaBridge] FROM DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-hashchain_20261113T090524.bak' WITH REPLACE, CHECKSUM;
+   ALTER DATABASE [ScadaBridge] SET MULTI_USER;
+   ```
+   This removes the backfill, the anchor and the chained rows.
+3. Start vq03 Central, then vq04. Confirm 64 migrations, 0 chained rows, 0 tombstones and 843 NULL `SourceNode` rows.
+
+**Full rollback of the 8e32f353 window:** follow the QA window record (`qa-rb-swap.ps1` on both boxes, then restore `ScadaBridge_pre-8e32f353_20261113T080106.bak`, then start the centrals together and the sites together). First run `qa-chain-unset-env.ps1` on both boxes. `0e540291` has no chain code, but no binary may come up with the key set. Never roll back past `0e540291`.
+
+## Deviations from the pack (§7) and DEV
+
+1. **Both centrals instead of DEV's single Central.** As the pack says, both centrals got the setting and the pair was stopped together and started together. The stop order was standby vq04 first, then vq03. The start order was vq03 first, then vq04 2 s later, which matches the window's pattern.
+2. **The env edit was split from the restart.** The setting was written on both boxes before either restart, so both nodes came up with the same configuration. DEV's `enable-chain.enable.ps1` edits and restarts in one call.
+3. **Bogus-key calls on both nodes plus method 9 generated the post-cutoff rows**, to show chaining right away. DEV waited for organic traffic and had one row.
+
+## Still open on #107
+
+- **Maintenance login:** deferred by the owner ("later"). `AuditLog:MaintenanceConnectionString` stays unset on both tiers.
+- **Head export:** off on both tiers until the owner names a UNC share. The pack §7 snippet uses the `<UNC-SHARE-TBD>` placeholder.