ScadaBridge/docs/deployment/deployment-records/2026-11-13-qa-audit-hash-chain.md

2026-11-13: QA (wonder-app-vq03 + vq04) audit hash chain enabled (#107 item 2, #139 cutoff)

Owner go: #107, 2026-11-13: "AuditLog:HashChain:Enabled on DEV now, QA after its window. No backlog chaining. Head export off. Maintenance login later." The QA window ran earlier today (2026-11-13-wonder-app-vq03-vq04-8e32f353.md). This record follows the DEV enablement (2026-11-13-wonder-app-vd03-8e32f353.md, "Audit hash chain") and the QA pack §7.

Build: unchanged. 8e32f353 (1.0.0+8e32f3538ce9…, which contains the #139 cutoff 52da02a7) is on both centrals, re-checked on the box before the change.

Scope:

• One service-env entry was added to both ScadaBridge-Central services: AuditLog__HashChain__Enabled=true. Both centrals were then stopped together and started together.
• ChainBacklog is unset, so the backlog is not chained. HeadExport is unset, so the export is off. MaintenanceConnectionString is unset, so the single SQL login stays.
• Not touched:
  • the Site services (not restarted), appsettings, and every other env entry;
  • OtOpcUa (Warm: vq04 Primary, vq03 Standby), mxaccessgw, HistorianGateway, Overview, the OTel collector and the ScadaBridge-LDAP service;
  • DEV vd03 and the docker rigs;
  • Z28061: nothing was deployed, written or called.

Transport:

• winrm (deploy scope) ran one rendered script per step on the box: qa-chain-{inspect,set-env,stop-central,start-central}.ps1.
• SMB was used for logs and appsettings, and to stage the scripts in E:\psrem-stage\sb-8e32f353\.
• The backup, backfill checks and chain checks ran from the Mac over direct SQL to wonder-sql-vq03. The CLI (8e32f353 build) and HTTP also ran from the Mac.
• On-box notes are in E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-hashchain-20261113T090524.txt.

Preflight (read-only, 09:04–09:06 EST)

Check	Result
Live build	Host 1.0.0+8e32f353… on both boxes
Schema	64 migrations, last 20261110120000_AuditMaintenanceGrants
AuditLog	3 453 rows, 0 chained. AuditLogChainHead 0 and AuditLogChainTombstone 0. 843 SourceNode IS NULL (2026-05-29 → 2026-07-07)
Service env	Central 15 entries, Site 8. Digests are identical on both boxes (key name + SHA-256 prefix only). No AuditLog__* key
appsettings	Unchanged since the window (C0C653D6… on both; Central vq03 EF01B129…, vq04 F333D48F…; Site vq03 D0E202F9…, vq04 E9E6336F…). Neither file contains HashChain
Per-node values	JwtExpiryMinutes / IdleTimeoutMinutes are 25 / 60 on vq03 and 15 / 30 on vq04
Health	/health/ready 200 ×4. /health/active: vq03 200 / vq04 503 on both pairs
Functional baseline	Site MxGateway Connected, 2196/2196 Good. WarmCanary "Connected to backup", 2 Good. LocalDb replicationConnected true, oplog backlog 0, site audit backlog 0. Z28062–Z28069: 244 Good / 0 Bad each, 10 alarms each. 10 deployed instances

Steps

Step	Local time (EST)	Result
1. Backup from the Mac	09:06:22 → 09:06:26	BACKUP DATABASE [ScadaBridge] TO DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-hashchain_20261113T090524.bak' WITH COPY_ONLY, CHECKSUM, COMPRESSION: 154 268 127 B compressed. RESTORE VERIFYONLY … WITH CHECKSUM reports the set valid; msdb records is_copy_only 1 and has_backup_checksums 1
2. SourceNode backfill (CLI → active central vq03)	09:06:33	audit backfill-source-node --before 2026-07-08T00:00:00Z --sentinel unknown → rows updated : 843. SQL afterwards: SourceNode IS NULL = 0, 843 rows unknown, still 0 chained
3. set-env on vq03, then vq04 (no restart)	09:06:40 → 09:06:57	reg export → E:\psrem-stage\sb-8e32f353\_envbak-ScadaBridge-Central-hashchain-set-env-20261113T090524.reg on each box. Central env went from 15 to 16 entries; every existing entry was asserted byte-identical
4. Stop Central vq04 (standby)	09:07:05 → 09:07:15	Graceful (CoordinatedShutdown). Site service still Running
5. Stop Central vq03 (active; central API down)	09:07:22 → 09:07:33	Graceful. Site service still Running
6. Start Central vq03, then vq04 2 s later	09:07:39 → 09:07:52	vq03 ready 09:07:51.7, vq04 ready 09:07:52.7

Downtime:

• Central API (active node): about 29 s (09:07:22.9 → 09:07:51.7).
• Whole central pair: about 48 s, from the vq04 stop at 09:07:05 to vq04 ready.
• The site data path was not interrupted, because the Site services were not stopped. Site audit rows buffer in the site backlog while the central is down, and the backlog was 0 afterwards.

Gates

Gate	Result
Health	/health/ready 200 ×4. /health/active: central vq03 200 / vq04 503, site vq03 200 / vq04 503, so exactly one active central. Central checks on both nodes are Healthy: database, akka-cluster, required-singletons, bundle-signing-policy, audit-chain-head-export
Cluster	health summary: central nodes 10.220.158.2 Primary and .3 Standby, both online. Akka log: [leader-auto-down] started with shut-out restart is ON on both centrals (14:07:43Z and 14:07:49Z), and vq03 gained leadership at 14:07:48Z
Anchor cutoff	Exactly 1 AuditLogChainTombstone row: ChainMonth 2026-11-01, ChainSeq 0, EventId 5CADAB1D-C4A1-4C0F-A000-000000000139, PurgedAtUtc (cutoff) 2026-11-13T14:07:43.3208169Z. vq03 Central log, 09:07:43.325 EST: AuditLog hash chain started at 2026-11-13T14:07:43.3208169Z: only rows dated from then on are chained; earlier rows stay unchained and out of scope (ScadaBridge#139). vq04 wrote no second anchor
New rows chained, old rows not	Post-cutoff traffic: 4 bogus/missing-key inbound calls (2 per node; vq04's reach the active node through the standby relay) and one flow-debug run-method --method-id 9 (SimpleAlarmStatusRequest, SAPID=000000; InboundRequest → DbWrite → ScriptLog ×14 → completed). SQL: 20 post-cutoff rows, ChainSeq 1–20, all central-a: InboundAuthFailure ×4, InboundRequest, DbWrite, ScriptLog ×14. Pre-cutoff rows chained: 0; pre-cutoff rows unchained: 3 453 (all of them). Unchained post-cutoff rows: 0. AuditLogChainHead 2026-11 LastSeq 20
verify-chain	Run against both centrals. 2026-11: Intact, rows verified 20, head sequence 20, unchained 0, chain from 2026-11-13T14:07:43.3208169Z, pre-cutoff rows 662. 2026-10: NotChained, pre-cutoff rows 1 466. Exit 0 on all four runs. Direct SQL gives the same counts: 662 November rows before the cutoff and 1 466 October rows
Head export	audit-chain-head-export: "Chain-head export disabled." on both centrals
Inbound API	A bogus X-API-Key and a missing key both get 401 UNAUTHORIZED on vq03 and on vq04 (relay). Method 9 completed: InboundRequest → BTDB → Ok
MxGateway	Site health: MxGateway Connected, 2196/2196 resolved, 2196 Good / 0 Bad (= baseline). Z28062–Z28069: 244 Good each, 10 alarms each (= baseline)
WarmCanary	"Connected to backup" (vq04 OtOpcUa, the Warm Primary), 2 Good / 0 Bad, instance 16 snapshot 2 Good
Site sync	localDbReplicationConnected true, oplog backlog 0, site audit backlog 0. Site role Active on vq03, both site nodes online. Site services were not restarted
Logs	Since 09:06 the only ERR/FTL lines are the usual first-probe akka-cluster … Removed health ERRs at boot: 7 on vq03 (09:07:44–48) and 4 on vq04 (09:07:49–51). No FTL

Env and appsettings, before vs after (key name + SHA-256 prefix only):

• The only change on each box is one added Central entry, AuditLog__HashChain__Enabled sha256:b5bea41b6c62 len 4. This is the same digest as DEV and equals true.
• The 14 other Central entries and all 8 Site entries are unchanged.
• The env digests are identical on vq03 and vq04.
• All three appsettings files have the same hashes as before on both boxes, so the per-node JwtExpiryMinutes / IdleTimeoutMinutes (25 / 60 on vq03, 15 / 30 on vq04) are preserved.

Rollback

Every on-box script is in E:\psrem-stage\sb-8e32f353\ on both boxes. Run each with winrm --timeout 900 ps <box> -f <script>.

Chain only (keep the build and the DB):

1. Run qa-chain-unset-env.ps1 on vq03 and on vq04. It exports the key to .reg, removes AuditLog__HashChain__Enabled from the Central env and asserts the other entries unchanged. It does not restart anything.
2. Run qa-chain-stop-central.ps1 on vq04, then on vq03.
3. Run qa-chain-start-central.ps1 on vq03, then on vq04.

The anchor and the chained rows stay. They are harmless, and chaining resumes from them if the chain is re-enabled. Do not delete the anchor row: verify-chain reports that as tampering. The .reg exports from step 3 hold the pre-change key if the env ever needs restoring by hand.

DB restore to before the enablement (loses every central DB write since 09:06:25):

1. Do the chain-only steps 1–2 above. Both centrals are now stopped. The sites can stay up, because they don't connect to this DB.
2. From the Mac:

   ALTER DATABASE [ScadaBridge] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
   RESTORE DATABASE [ScadaBridge] FROM DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-hashchain_20261113T090524.bak' WITH REPLACE, CHECKSUM;
   ALTER DATABASE [ScadaBridge] SET MULTI_USER;
   

   This removes the backfill, the anchor and the chained rows.
3. Start vq03 Central, then vq04. Confirm 64 migrations, 0 chained rows, 0 tombstones and 843 NULL SourceNode rows.

Full rollback of the 8e32f353 window: follow the QA window record (qa-rb-swap.ps1 on both boxes, then restore ScadaBridge_pre-8e32f353_20261113T080106.bak, then start the centrals together and the sites together). First run qa-chain-unset-env.ps1 on both boxes. 0e540291 has no chain code, but no binary may come up with the key set. Never roll back past 0e540291.

Deviations from the pack (§7) and DEV

1. Both centrals instead of DEV's single Central. As the pack says, both centrals got the setting and the pair was stopped together and started together. The stop order was standby vq04 first, then vq03. The start order was vq03 first, then vq04 2 s later, which matches the window's pattern.
2. The env edit was split from the restart. The setting was written on both boxes before either restart, so both nodes came up with the same configuration. DEV's enable-chain.enable.ps1 edits and restarts in one call.
3. Bogus-key calls on both nodes plus method 9 generated the post-cutoff rows, to show chaining right away. DEV waited for organic traffic and had one row.

Still open on #107

• Maintenance login: deferred by the owner ("later"). AuditLog:MaintenanceConnectionString stays unset on both tiers.
• Head export: off on both tiers until the owner names a UNC share. The pack §7 snippet uses the <UNC-SHARE-TBD> placeholder.