docs(audit): record the removal of the hash chain and the two-login hardening
Component-AuditLog (Security & Tamper-Evidence now a removal note, archive format v2, config), Component-CLI, Component-ManagementService, HighLevelReqs AL-11/AL-12, docs/components/CLI, the CLI README, alog.md §11.4/§15 and CLAUDE.md. New production script docs/plans/sql/RemoveAuditLogHashChain.sql (idempotent, from migration 64); AddAuditLogHashChain.sql marked superseded. Historical plans, deployment records and deploy-prep packs are left as written. Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
This commit is contained in:
@@ -184,8 +184,8 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- Append-only enforced via DB roles (writer role has INSERT only, no UPDATE/DELETE).
|
||||
- **The set-based ingest probe is `WITH (FORCESEEK)` (#124).** Left alone, `InsertManyIfNotExistsAsync`'s `NOT EXISTS` anti-semi-join scanned the EventId index, and concurrent writers (ingest + reconciliation singletons, dual-write, direct writes) deadlocked on S/IX page locks — 4 writers lost ~45 % of packets to 1205. With the hint: 0 deadlocks, 8 writers 8.6 k rows/s. Reproduce with `ResidualBench audit-contention`.
|
||||
- **Clustered key `(OccurredAtUtc, EventId)` + aligned unique `UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc)` (migration `FlipAuditLogClusteredKey`, #111, B2 of the #78 benchmark).** The Audit Log page's newest-first page/keyset now reads an ordered range instead of the whole table (300k rows: 90,532 → 34 reads). Offline whole-table rebuild — script `docs/plans/sql/FlipAuditLogClusteredKey.sql` (+ `.rollback.sql`), both centrals stopped: the partition-switch staging table now reads the live key order, but binaries older than the flip hard-code the old one. DEV/QA not applied (sizing + window in #111).
|
||||
- **Parquet cold archive (T2, ScadaBridge#42) — default OFF (`AuditLog:Archive:Enabled` + absolute `Directory`, shared by both centrals).** The purge actor writes each expired partition to `{Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet` (10 stored columns + `ChainSeq`/`RowHash`, tick-exact `OccurredAtUtcTicks`) plus `.tombstones.parquet` and a `.manifest.json` (written last; chain head at archive time, SHA-256 per file), read under the chain lock SHARED; then the switch runs with a guard (under the EXCLUSIVE lock: partition row count + each month's `LastSeq` must still equal the archive's, else THROW 51002/51003 and nothing is purged). Fail closed: a partition that cannot be archived is not purged. `audit archive list|verify` = `ListAuditArchives`/`VerifyAuditArchive` (verify-chain's gate): file hashes, chain recomputed from the archive, chain end vs the DB purge anchor (`Matches`/`Superseded`/`NotPurged`/`Mismatch`), CLI exit 3. `audit export --format parquet` streams the 24 export columns (was 501). No migration. **Owner decisions (#109):** destination stays a directory/UNC (no object-storage client; share + permissions named at enablement), parquet-tools-only reading (no re-import), unsigned like the chain. **Retention** `AuditLog:Archive:RetentionDays` (default 0 = never; else > `AuditLog:RetentionDays`, counted from the month's end) and **superseded stale-archive cleanup** (`DeleteSupersededStaleArchives`, default true: a retried purge's `Archived` archive is deleted only when a later `Purged`, verifying archive provably holds every row — same anchor + chain-end hash, unchained rows by EventId; else `KeptStale`) run after every purge tick; deletion touches only the archive name's own three files, manifest first; `audit archive cleanup [--dry-run]` = `CleanupAuditArchives` (Administrator).
|
||||
- **Hash-chain tamper evidence (T1, ScadaBridge#41) — default OFF (`AuditLog:HashChain:Enabled`).** One SHA-256 chain per UTC calendar month: `ChainSeq` + `RowHash = SHA-256(prev || canonical(row))` on `AuditLog` (migration `AddAuditLogHashChain`), linked by the central `AuditLogHashChainService` (both nodes, serialised by `sp_getapplock`) — never by the ingest path. `AuditLogChainHead` catches tail deletion and carries the partition-switch checkpoint anchor; the per-channel DELETE writes an `AuditLogChainTombstone` per chained row in the same transaction. `audit verify-chain --month` = management command `VerifyAuditChain` (OperationalAudit, unscoped callers only), first break with partition/ChainSeq/EventId, CLI exit 3. A chained row's `SourceNode` is frozen (backfill skips it). **Unsigned by owner decision (#107); external anchor = off-box head export** (`AuditLog:HashChain:HeadExport`, default off: both centrals append changed per-month heads — month, `LastSeq`/`LastHash`, purge anchor, row + tombstone counts — to append-only JSONL files `{Directory}/{yyyy}/audit-chain-heads-{date}-{node}.jsonl` or RFC 5424 syslog UDP/TCP); `audit verify-heads --path` = `VerifyAuditChainHeads` checks them against the DB (a re-hashed forgery passes verify-chain, fails verify-heads; exit 3). **Two-login hardening (#107 item 3):** `AuditLog:MaintenanceConnectionString` (`${secret:}`-capable, unset = runtime login) runs the chain job, verify/head export, partition switch-out + archive and per-channel purge on `AuditMaintenanceContext` (keyed `audit-maintenance` repository/archive); that login gets `scadabridge_audit_chainer` + `scadabridge_audit_purger`, the runtime login `scadabridge_audit_writer` (whose DENYs block chaining/purging) — pinned by `AuditMaintenanceLoginTests`. Since #113 the monthly roll-forward and `backfill-source-node` use it too (migration `AuditMaintenanceGrants`: purger gets `ALTER ANY DATASPACE` + `UPDATE (SourceNode)`). Head-export failures (#115) bump `scadabridge.audit.chain_head_export.failures` and Degrade `/health/ready` entry `audit-chain-head-export`; archive cleanup (#114) also deletes writer-named orphan temp / manifest-less files older than 24 h (`DeletedOrphan`).
|
||||
- **Parquet cold archive (T2, ScadaBridge#42) — default OFF (`AuditLog:Archive:Enabled` + absolute `Directory`, shared by both centrals).** The purge actor writes each expired partition to `{Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet` (the 10 stored columns, tick-exact `OccurredAtUtcTicks`) plus a `.manifest.json` (written last; row count + SHA-256/size of the data file) — layout **format v2** since 2026-11-15; a format-v1 archive (written with the hash chain: `ChainSeq`/`RowHash` columns, a `.tombstones.parquet`, chain fields in the manifest) still lists, verifies and is cleaned up. The switch then runs with a guard (the partition row count must still equal the archive's, else THROW 51002 and nothing is purged). Fail closed: a partition that cannot be archived is not purged. `audit archive list|verify` = `ListAuditArchives`/`VerifyAuditArchive` (OperationalAudit, unscoped callers): data-file size, SHA-256 and row count against the manifest — file integrity only, **no tamper evidence**; CLI exit 3. `audit export --format parquet` streams the 24 export columns. **Owner decisions (#109):** destination stays a directory/UNC (no object-storage client), parquet-tools-only reading (no re-import), unsigned. **Retention** `AuditLog:Archive:RetentionDays` (default 0 = never; else > `AuditLog:RetentionDays`, counted from the month's end) and **superseded stale-archive cleanup** (`DeleteSupersededStaleArchives`, default true: a retried purge's `Archived` archive is deleted only when a later `Purged`, verifying archive holds every one of its rows by EventId; else `KeptStale`) run after every purge tick; deletion touches only the archive name's own files, manifest first; `audit archive cleanup [--dry-run]` = `CleanupAuditArchives` (Administrator); it also deletes writer-named orphan temp / manifest-less files older than 24 h (`DeletedOrphan`, #114).
|
||||
- **Hash-chain tamper evidence and the two-login maintenance hardening: REMOVED 2026-11-15.** Tamper-evidence (hash chain, off-box head export) and the two-login maintenance hardening were removed on 2026-11-15 at IT security's request, pending a family-wide general solution; migration `RemoveAuditLogHashChain` drops their schema (`ChainSeq`/`RowHash`, `AuditLogChainHead`/`AuditLogChainTombstone`, the two chain indexes, `scadabridge_audit_chainer`, and the purger's `AuditMaintenanceGrants` grants; `scadabridge_audit_writer`/`_purger` stay). **Deploy gate:** binaries before this change map the chain columns, so the migration ships with the new binaries, both centrals stopped; the new binaries purge either schema (the staging table copies the chain columns only when the live table has them). A leftover `AuditLog:HashChain:*` or `AuditLog:MaintenanceConnectionString` is ignored with one startup warning (`RetiredAuditSettingsWarning`) — DEV vd03 and QA vq03/vq04 still carry `AuditLog__HashChain__Enabled=true` in the Central service env from the 2026-11-13 enablement; remove it at the next deploy. Everything runs on `ScadaBridge:Database:ConfigurationDb` again. History: ScadaBridge#41/#107/#113/#115/#139 and `docs/plans/`.
|
||||
- Node-of-origin is captured alongside site-of-origin: `SourceNode` (`varchar(64)` NULL) on `AuditLog`, `Notifications`, and `SiteCalls` — `node-a`/`node-b` for site rows (qualified by `SourceSiteId`/`SourceSite`), `central-a`/`central-b` for central direct-write rows. Stamped at the writing node, carried verbatim through telemetry + reconciliation, and indexed via `IX_AuditLog_Node_Occurred (SourceNode, OccurredAtUtc)` on `AuditLog`.
|
||||
- Per-node stuck KPIs (M5.3 T6): Notification Outbox and Site Call Audit expose `PerNodeNotificationKpiRequest`/`PerNodeSiteCallKpiRequest` messages that group stuck/parked/delivered counts by `SourceNode`, surfacing per-node breakdowns on the Health dashboard.
|
||||
- `audit tree --execution-id <guid>` CLI command (M5.3 T8) + `GET /api/audit/tree` endpoint — resolves any node to its chain root and renders the full execution tree; backed by `IAuditLogRepository.GetExecutionTreeAsync`.
|
||||
|
||||
Reference in New Issue
Block a user