Audit hash chain + two-login hardening REMOVED (IT security, 2026-11-15) — deploy the drop migration to DEV + QA #143

Closed
opened 2026-11-15 17:51:35 -05:00 by dohertj2 · 1 comment
Owner

IT security asked (2026-11-15) for the AuditLog hash chain to be removed until a general, family-wide solution for tamper evidence exists. The two-login maintenance hardening goes with it.

Removed in code: main 3dea146c (commits ed2ed0a3 + b418408c).

  • Hash chain (#41, #139): chaining service, hasher, verify-chain, chain entities, ChainSeq/RowHash, and the AuditLog:HashChain options.
  • Head export (#107, #115): the File and Syslog sinks, the audit-chain-head-export health check, and verify-heads.
  • Two-login hardening (#113): AuditLog:MaintenanceConnectionString, AuditMaintenanceContext, and the keyed audit-maintenance registrations. Purge, archive, partition roll-forward and backfill-source-node run on the runtime login again.
  • Archive (#42): kept, rewritten to format v2 with no chain columns. v1 archives are still readable.
  • Leftover settings: a retired AuditLog:HashChain* / MaintenanceConnectionString key is ignored and logs one warning at central startup.

Schema: a new migration, RemoveAuditLogHashChain, drops the two chain indexes, the ChainSeq/RowHash columns, the AuditLogChainHead and AuditLogChainTombstone tables, and the scadabridge_audit_chainer role. It also revokes the two grants AuditMaintenanceGrants gave scadabridge_audit_purger; the role itself predates the chain and is kept. Down recreates all of it. The idempotent delta script is docs/plans/sql/RemoveAuditLogHashChain.sql.

Deploy gate (not done yet): DEV vd03 and QA vq03/vq04 still run 8e32f353 with AuditLog__HashChain__Enabled=true, so the chain is live on both tiers until this ships. The owner chose to leave it until the next deploy.

  • DEV vd03 (2026-11-15, 061432c8, record 2026-11-15-wonder-app-vd03-061432c8.md, 50a5291b): deploy main with the migration, then remove AuditLog__HashChain__Enabled from the Central service env.
  • QA vq03/vq04 (2026-11-15, 061432c8, record 2026-11-15-wonder-app-vq03-vq04-061432c8.md, 50a5291b): the same. Stop both centrals and start them together: the old binaries map the chain columns, so they can't run on the migrated schema.
  • Verify after each (all passed; the purge/partition switch was skipped: there is no read-only way to exercise it and nothing is due): the columns, tables and role are gone; the AuditLog keeps writing; a purge or partition switch works; the retired-key warning disappears once the env key is removed.

Open (needs a decision): a family-wide approach to tamper evidence for audit data, owned by IT security and the owner, before anything like this comes back.

Tests on the merged branch: AuditLog 455, ConfigurationDatabase 500 (MSSQL), ManagementService 778, CLI 588, Host 733, Commons 986. All green. Two NotifyDispatcher timing flakes passed on re-run and also pass on main.

**IT security asked (2026-11-15) for the AuditLog hash chain to be removed until a general, family-wide solution for tamper evidence exists.** The two-login maintenance hardening goes with it. **Removed in code:** main `3dea146c` (commits `ed2ed0a3` + `b418408c`). - **Hash chain (#41, #139):** chaining service, hasher, `verify-chain`, chain entities, `ChainSeq`/`RowHash`, and the `AuditLog:HashChain` options. - **Head export (#107, #115):** the File and Syslog sinks, the `audit-chain-head-export` health check, and `verify-heads`. - **Two-login hardening (#113):** `AuditLog:MaintenanceConnectionString`, `AuditMaintenanceContext`, and the keyed `audit-maintenance` registrations. Purge, archive, partition roll-forward and `backfill-source-node` run on the runtime login again. - **Archive (#42):** kept, rewritten to format v2 with no chain columns. v1 archives are still readable. - **Leftover settings:** a retired `AuditLog:HashChain*` / `MaintenanceConnectionString` key is ignored and logs one warning at central startup. **Schema:** a new migration, `RemoveAuditLogHashChain`, drops the two chain indexes, the `ChainSeq`/`RowHash` columns, the `AuditLogChainHead` and `AuditLogChainTombstone` tables, and the `scadabridge_audit_chainer` role. It also revokes the two grants `AuditMaintenanceGrants` gave `scadabridge_audit_purger`; the role itself predates the chain and is kept. Down recreates all of it. The idempotent delta script is `docs/plans/sql/RemoveAuditLogHashChain.sql`. **Deploy gate (not done yet):** DEV vd03 and QA vq03/vq04 still run `8e32f353` with `AuditLog__HashChain__Enabled=true`, so the chain is live on both tiers until this ships. The owner chose to leave it until the next deploy. - [x] DEV vd03 (2026-11-15, `061432c8`, record `2026-11-15-wonder-app-vd03-061432c8.md`, `50a5291b`): deploy main with the migration, then remove `AuditLog__HashChain__Enabled` from the Central service env. - [x] QA vq03/vq04 (2026-11-15, `061432c8`, record `2026-11-15-wonder-app-vq03-vq04-061432c8.md`, `50a5291b`): the same. **Stop both centrals and start them together:** the old binaries map the chain columns, so they can't run on the migrated schema. - [x] Verify after each (all passed; the purge/partition switch was skipped: there is no read-only way to exercise it and nothing is due): the columns, tables and role are gone; the AuditLog keeps writing; a purge or partition switch works; the retired-key warning disappears once the env key is removed. **Open (needs a decision):** a family-wide approach to tamper evidence for audit data, owned by IT security and the owner, before anything like this comes back. Tests on the merged branch: AuditLog 455, ConfigurationDatabase 500 (MSSQL), ManagementService 778, CLI 588, Host 733, Commons 986. All green. Two NotifyDispatcher timing flakes passed on re-run and also pass on main.
dohertj2 added the type:deploy-gatepriority:p2 labels 2026-11-15 17:51:35 -05:00
Author
Owner

Deployed and verified on DEV and QA (2026-11-15), build 061432c8. Records are in 50a5291b.

DEV vd03 QA vq03/vq04
Backup E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-061432c8_20261115T191258.bak F:\MSSQL\BACKUP\ScadaBridge_pre-061432c8_20261115T191851.bak (wonder-sql-vq03)
Migration 64 → 65 (RemoveAuditLogHashChain, < 1 s) 64 → 65 (< 1 s; all four services stopped)
Outage site ~48 s, central ~29 s site ~2 min 40 s, central ~1 min 37 s
Env AuditLog__HashChain__Enabled removed (13 → 12 entries; the rest byte-identical) removed on both Centrals (16 → 15; identical on both boxes)

Gates, all passing on both tiers:

  • the chain schema is gone: ChainSeq/RowHash, both tables, both indexes and scadabridge_audit_chainer;
  • _purger and _writer are kept, and the AuditLog rows are kept;
  • health ready, with one active per pair;
  • 0 retired-key warnings;
  • new AuditLog rows are written; a bogus key gets 401;
  • MxGateway and the instances are at baseline (QA: 2196 good, Z28061–69 at 244 Good, WarmCanary connected, LocalDb connected);
  • the CLI has no verify-chain / verify-heads;
  • OTLP service.version is 061432c8; Overview is Up.

The purge/partition switch was not exercised: there is no read-only way to run it and nothing is due.

Rollback is binaries plus the DB restore above, plus reg import of the pre-window Central key, with the QA pair stopped together. Details are in the records. The hash chain is now gone from every tier. The family-wide tamper-evidence question stays with the owner and IT security.

**Deployed and verified on DEV and QA (2026-11-15), build `061432c8`. Records are in `50a5291b`.** | | DEV vd03 | QA vq03/vq04 | |---|---|---| | Backup | `E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-061432c8_20261115T191258.bak` | `F:\MSSQL\BACKUP\ScadaBridge_pre-061432c8_20261115T191851.bak` (wonder-sql-vq03) | | Migration | 64 → **65** (`RemoveAuditLogHashChain`, < 1 s) | 64 → **65** (< 1 s; all four services stopped) | | Outage | site ~48 s, central ~29 s | site ~2 min 40 s, central ~1 min 37 s | | Env | `AuditLog__HashChain__Enabled` removed (13 → 12 entries; the rest byte-identical) | removed on both Centrals (16 → 15; identical on both boxes) | **Gates, all passing on both tiers:** - the chain schema is **gone**: `ChainSeq`/`RowHash`, both tables, both indexes and `scadabridge_audit_chainer`; - `_purger` and `_writer` are kept, and the AuditLog rows are kept; - health ready, with one active per pair; - 0 retired-key warnings; - new AuditLog rows are written; a bogus key gets 401; - MxGateway and the instances are at baseline (QA: 2196 good, Z28061–69 at 244 Good, WarmCanary connected, LocalDb connected); - the CLI has no `verify-chain` / `verify-heads`; - OTLP `service.version` is `061432c8`; Overview is Up. The purge/partition switch was not exercised: there is no read-only way to run it and nothing is due. **Rollback** is binaries plus the DB restore above, plus `reg import` of the pre-window Central key, with the QA pair stopped together. Details are in the records. The hash chain is now gone from every tier. The family-wide tamper-evidence question stays with the owner and IT security.
Sign in to join this conversation.