feat(audit): archive retention and superseded stale-archive cleanup (ScadaBridge#109)

Owner decisions recorded: the archive destination stays a directory/UNC
(no object-storage client; share + permissions named at enablement),
parquet-tools-only reading (no re-import), unsigned like the chain (#107).

AuditLog:Archive:RetentionDays (default 0 = never delete; else must exceed
AuditLog:RetentionDays, counted from the archived month's end) and
DeleteSupersededStaleArchives (default true) run after every purge tick.
A retried purge's stale Archived archive is deleted only when a later,
verifying, Purged archive of the same partition provably holds every row:
same prior anchor and the stale chain-end hash at its chain-end sequence
(row or tombstone), every unchained row present by EventId; otherwise it is
reported KeptStale. A sole copy left at Archived is never touched. Deletion
removes only the archive name's own three files in its folder, manifest
first, never a manifest-supplied path. `audit archive cleanup [--dry-run]`
= CleanupAuditArchives (Administrator).

AuditLogArchiveCleanupTests (MSSQL) cover retried purge, kept stale copy,
sole copy, retention and the path guard; each safety rule mutation-checked.
Follow-ups filed: #113, #114, #115.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 23:32:53 -05:00
parent c4d7ec9b96
commit b95ee5a658
26 changed files with 1099 additions and 26 deletions
+1 -1
View File
@@ -183,7 +183,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- 365-day central retention with monthly partition-switch purge; per-channel retention overrides (`AuditLog:PerChannelRetentionDays`) expire rows earlier than the global window via a bounded, batched row DELETE on the purge actor's maintenance path — values must be shorter than the global window (M5.5 T3); 7-day site SQLite retention with a hard `ForwardState` invariant (no row purged until forwarded or reconciled). - 365-day central retention with monthly partition-switch purge; per-channel retention overrides (`AuditLog:PerChannelRetentionDays`) expire rows earlier than the global window via a bounded, batched row DELETE on the purge actor's maintenance path — values must be shorter than the global window (M5.5 T3); 7-day site SQLite retention with a hard `ForwardState` invariant (no row purged until forwarded or reconciled).
- Append-only enforced via DB roles (writer role has INSERT only, no UPDATE/DELETE). - Append-only enforced via DB roles (writer role has INSERT only, no UPDATE/DELETE).
- **Clustered key `(OccurredAtUtc, EventId)` + aligned unique `UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc)` (migration `FlipAuditLogClusteredKey`, #111, B2 of the #78 benchmark).** The Audit Log page's newest-first page/keyset now reads an ordered range instead of the whole table (300k rows: 90,532 → 34 reads). Offline whole-table rebuild — script `docs/plans/sql/FlipAuditLogClusteredKey.sql` (+ `.rollback.sql`), both centrals stopped: the partition-switch staging table now reads the live key order, but binaries older than the flip hard-code the old one. DEV/QA not applied (sizing + window in #111). - **Clustered key `(OccurredAtUtc, EventId)` + aligned unique `UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc)` (migration `FlipAuditLogClusteredKey`, #111, B2 of the #78 benchmark).** The Audit Log page's newest-first page/keyset now reads an ordered range instead of the whole table (300k rows: 90,532 → 34 reads). Offline whole-table rebuild — script `docs/plans/sql/FlipAuditLogClusteredKey.sql` (+ `.rollback.sql`), both centrals stopped: the partition-switch staging table now reads the live key order, but binaries older than the flip hard-code the old one. DEV/QA not applied (sizing + window in #111).
- **Parquet cold archive (T2, ScadaBridge#42) — default OFF (`AuditLog:Archive:Enabled` + absolute `Directory`, shared by both centrals).** The purge actor writes each expired partition to `{Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet` (10 stored columns + `ChainSeq`/`RowHash`, tick-exact `OccurredAtUtcTicks`) plus `.tombstones.parquet` and a `.manifest.json` (written last; chain head at archive time, SHA-256 per file), read under the chain lock SHARED; then the switch runs with a guard (under the EXCLUSIVE lock: partition row count + each month's `LastSeq` must still equal the archive's, else THROW 51002/51003 and nothing is purged). Fail closed: a partition that cannot be archived is not purged. `audit archive list|verify` = `ListAuditArchives`/`VerifyAuditArchive` (verify-chain's gate): file hashes, chain recomputed from the archive, chain end vs the DB purge anchor (`Matches`/`Superseded`/`NotPurged`/`Mismatch`), CLI exit 3. `audit export --format parquet` streams the 24 export columns (was 501). No migration; no re-import path (open decision), object storage not wired (mount it). - **Parquet cold archive (T2, ScadaBridge#42) — default OFF (`AuditLog:Archive:Enabled` + absolute `Directory`, shared by both centrals).** The purge actor writes each expired partition to `{Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet` (10 stored columns + `ChainSeq`/`RowHash`, tick-exact `OccurredAtUtcTicks`) plus `.tombstones.parquet` and a `.manifest.json` (written last; chain head at archive time, SHA-256 per file), read under the chain lock SHARED; then the switch runs with a guard (under the EXCLUSIVE lock: partition row count + each month's `LastSeq` must still equal the archive's, else THROW 51002/51003 and nothing is purged). Fail closed: a partition that cannot be archived is not purged. `audit archive list|verify` = `ListAuditArchives`/`VerifyAuditArchive` (verify-chain's gate): file hashes, chain recomputed from the archive, chain end vs the DB purge anchor (`Matches`/`Superseded`/`NotPurged`/`Mismatch`), CLI exit 3. `audit export --format parquet` streams the 24 export columns (was 501). No migration. **Owner decisions (#109):** destination stays a directory/UNC (no object-storage client; share + permissions named at enablement), parquet-tools-only reading (no re-import), unsigned like the chain. **Retention** `AuditLog:Archive:RetentionDays` (default 0 = never; else > `AuditLog:RetentionDays`, counted from the month's end) and **superseded stale-archive cleanup** (`DeleteSupersededStaleArchives`, default true: a retried purge's `Archived` archive is deleted only when a later `Purged`, verifying archive provably holds every row — same anchor + chain-end hash, unchained rows by EventId; else `KeptStale`) run after every purge tick; deletion touches only the archive name's own three files, manifest first; `audit archive cleanup [--dry-run]` = `CleanupAuditArchives` (Administrator).
- **Hash-chain tamper evidence (T1, ScadaBridge#41) — default OFF (`AuditLog:HashChain:Enabled`).** One SHA-256 chain per UTC calendar month: `ChainSeq` + `RowHash = SHA-256(prev || canonical(row))` on `AuditLog` (migration `AddAuditLogHashChain`), linked by the central `AuditLogHashChainService` (both nodes, serialised by `sp_getapplock`) — never by the ingest path. `AuditLogChainHead` catches tail deletion and carries the partition-switch checkpoint anchor; the per-channel DELETE writes an `AuditLogChainTombstone` per chained row in the same transaction. `audit verify-chain --month` = management command `VerifyAuditChain` (OperationalAudit, unscoped callers only), first break with partition/ChainSeq/EventId, CLI exit 3. A chained row's `SourceNode` is frozen (backfill skips it). **Unsigned by owner decision (#107); external anchor = off-box head export** (`AuditLog:HashChain:HeadExport`, default off: both centrals append changed per-month heads — month, `LastSeq`/`LastHash`, purge anchor, row + tombstone counts — to append-only JSONL files `{Directory}/{yyyy}/audit-chain-heads-{date}-{node}.jsonl` or RFC 5424 syslog UDP/TCP); `audit verify-heads --path` = `VerifyAuditChainHeads` checks them against the DB (a re-hashed forgery passes verify-chain, fails verify-heads; exit 3). **Two-login hardening (#107 item 3):** `AuditLog:MaintenanceConnectionString` (`${secret:}`-capable, unset = runtime login) runs the chain job, verify/head export, partition switch-out + archive and per-channel purge on `AuditMaintenanceContext` (keyed `audit-maintenance` repository/archive); that login gets `scadabridge_audit_chainer` + `scadabridge_audit_purger`, the runtime login `scadabridge_audit_writer` (whose DENYs block chaining/purging) — pinned by `AuditMaintenanceLoginTests`. - **Hash-chain tamper evidence (T1, ScadaBridge#41) — default OFF (`AuditLog:HashChain:Enabled`).** One SHA-256 chain per UTC calendar month: `ChainSeq` + `RowHash = SHA-256(prev || canonical(row))` on `AuditLog` (migration `AddAuditLogHashChain`), linked by the central `AuditLogHashChainService` (both nodes, serialised by `sp_getapplock`) — never by the ingest path. `AuditLogChainHead` catches tail deletion and carries the partition-switch checkpoint anchor; the per-channel DELETE writes an `AuditLogChainTombstone` per chained row in the same transaction. `audit verify-chain --month` = management command `VerifyAuditChain` (OperationalAudit, unscoped callers only), first break with partition/ChainSeq/EventId, CLI exit 3. A chained row's `SourceNode` is frozen (backfill skips it). **Unsigned by owner decision (#107); external anchor = off-box head export** (`AuditLog:HashChain:HeadExport`, default off: both centrals append changed per-month heads — month, `LastSeq`/`LastHash`, purge anchor, row + tombstone counts — to append-only JSONL files `{Directory}/{yyyy}/audit-chain-heads-{date}-{node}.jsonl` or RFC 5424 syslog UDP/TCP); `audit verify-heads --path` = `VerifyAuditChainHeads` checks them against the DB (a re-hashed forgery passes verify-chain, fails verify-heads; exit 3). **Two-login hardening (#107 item 3):** `AuditLog:MaintenanceConnectionString` (`${secret:}`-capable, unset = runtime login) runs the chain job, verify/head export, partition switch-out + archive and per-channel purge on `AuditMaintenanceContext` (keyed `audit-maintenance` repository/archive); that login gets `scadabridge_audit_chainer` + `scadabridge_audit_purger`, the runtime login `scadabridge_audit_writer` (whose DENYs block chaining/purging) — pinned by `AuditMaintenanceLoginTests`.
- Node-of-origin is captured alongside site-of-origin: `SourceNode` (`varchar(64)` NULL) on `AuditLog`, `Notifications`, and `SiteCalls` — `node-a`/`node-b` for site rows (qualified by `SourceSiteId`/`SourceSite`), `central-a`/`central-b` for central direct-write rows. Stamped at the writing node, carried verbatim through telemetry + reconciliation, and indexed via `IX_AuditLog_Node_Occurred (SourceNode, OccurredAtUtc)` on `AuditLog`. - Node-of-origin is captured alongside site-of-origin: `SourceNode` (`varchar(64)` NULL) on `AuditLog`, `Notifications`, and `SiteCalls` — `node-a`/`node-b` for site rows (qualified by `SourceSiteId`/`SourceSite`), `central-a`/`central-b` for central direct-write rows. Stamped at the writing node, carried verbatim through telemetry + reconciliation, and indexed via `IX_AuditLog_Node_Occurred (SourceNode, OccurredAtUtc)` on `AuditLog`.
- Per-node stuck KPIs (M5.3 T6): Notification Outbox and Site Call Audit expose `PerNodeNotificationKpiRequest`/`PerNodeSiteCallKpiRequest` messages that group stuck/parked/delivered counts by `SourceNode`, surfacing per-node breakdowns on the Health dashboard. - Per-node stuck KPIs (M5.3 T6): Notification Outbox and Site Call Audit expose `PerNodeNotificationKpiRequest`/`PerNodeSiteCallKpiRequest` messages that group stuck/parked/delivered counts by `SourceNode`, surfacing per-node breakdowns on the Health dashboard.
+3 -3
View File
@@ -24,8 +24,8 @@ Management commands use `POST /management`. Audit query, export, and tree use
`GET /api/audit/query`, `/api/audit/export`, and `/api/audit/tree`; source-node `GET /api/audit/query`, `/api/audit/export`, and `/api/audit/tree`; source-node
backfill uses `POST /api/audit/backfill-source-node`. `audit verify-chain` sends the backfill uses `POST /api/audit/backfill-source-node`. `audit verify-chain` sends the
`VerifyAuditChain` management command (ScadaBridge#41); `audit verify-heads` reads off-box `VerifyAuditChain` management command (ScadaBridge#41); `audit verify-heads` reads off-box
exported chain heads and sends `VerifyAuditChainHeads` (ScadaBridge#107); `audit archive list|verify` send exported chain heads and sends `VerifyAuditChainHeads` (ScadaBridge#107); `audit archive list|verify|cleanup` send
`ListAuditArchives` / `VerifyAuditArchive` (ScadaBridge#42). `debug stream` uses a persistent `ListAuditArchives` / `VerifyAuditArchive` (ScadaBridge#42) / `CleanupAuditArchives` (ScadaBridge#109). `debug stream` uses a persistent
SignalR connection. Audit export streams the response without buffering it all. SignalR connection. Audit export streams the response without buffering it all.
### Authentication and roles ### Authentication and roles
@@ -122,7 +122,7 @@ scadabridge instance deploy --id 7
| `db-connection` | `list`, `get`, `create`, `update`, `delete` | `Designer` | | `db-connection` | `list`, `get`, `create`, `update`, `delete` | `Designer` |
| `api-method` | `list`, `get`, `create`, `update`, `delete`, `validate`, `recompile` | `Designer` | | `api-method` | `list`, `get`, `create`, `update`, `delete`, `validate`, `recompile` | `Designer` |
| `bundle` | `export`, `preview`, `import` | Designer export; Administrator preview/import | | `bundle` | `export`, `preview`, `import` | Designer export; Administrator preview/import |
| `audit` | `query`, `export`, `tree`, `backfill-source-node`, `verify-chain`, `verify-heads`, `archive list`, `archive verify` | Audit roles; Administrator backfill; verify-chain, verify-heads and archive need an unscoped audit role | | `audit` | `query`, `export`, `tree`, `backfill-source-node`, `verify-chain`, `verify-heads`, `archive list`, `archive verify`, `archive cleanup` | Audit roles; Administrator backfill and archive cleanup; verify-chain, verify-heads and archive need an unscoped audit role |
| `audit-config` | `query` (config-change audit trail; was `audit-log` pre-M8) | `Admin` | | `audit-config` | `query` (config-change audit trail; was `audit-log` pre-M8) | `Admin` |
| `health` | `summary`, `pending-updates`, `site`, `event-log`, `parked-messages` | Authenticated reads with server-side scope checks | | `health` | `summary`, `pending-updates`, `site`, `event-log`, `parked-messages` | Authenticated reads with server-side scope checks |
| `debug` | `snapshot`, `stream` | `Deployer` | | `debug` | `snapshot`, `stream` | `Deployer` |
+45 -12
View File
@@ -770,8 +770,12 @@ MS SQL for direct-write events). Unredacted secrets never persist.
- **Where.** `AuditLog:Archive:Directory` — an absolute path (validated at startup) - **Where.** `AuditLog:Archive:Directory` — an absolute path (validated at startup)
that both central nodes reach, because the purge runs on the active node and that both central nodes reach, because the purge runs on the active node and
`audit archive list|verify` on whichever node serves the call. Files land in `audit archive list|verify` on whichever node serves the call. Files land in
`{Directory}/{YYYY}/`. Object storage (S3 / Azure Blob) is not called directly — `{Directory}/{YYYY}/`. **Owner decision (ScadaBridge#109): the destination stays a
mount it as a file system or copy the files on. directory / UNC share — no native object-storage client.** Object storage (S3 / Azure
Blob) is reached, if ever, by mounting it as a file system or copying the files on. The
share (UNC path) and its permissions are chosen when the archive is enabled on DEV/QA:
both central nodes' service account needs modify rights on it (write, rename, and — for
cleanup — delete), auditors read-only; the SQL Server login needs none.
- **What.** Per partition, three files sharing the stem - **What.** Per partition, three files sharing the stem
`auditlog-{YYYY-MM}-{archivedAt:yyyyMMddTHHmmssfff}Z`: `.parquet` (every row of the `auditlog-{YYYY-MM}-{archivedAt:yyyyMMddTHHmmssfff}Z`: `.parquet` (every row of the
partition — the ten stored columns, `OccurredAtUtc` both as a UTC microsecond partition — the ten stored columns, `OccurredAtUtc` both as a UTC microsecond
@@ -798,8 +802,33 @@ MS SQL for direct-write events). Unredacted secrets never persist.
- **Retention interplay.** Archiving changes nothing about *when* a partition is - **Retention interplay.** Archiving changes nothing about *when* a partition is
purged (`RetentionDays`). Per-channel overrides still delete their rows earlier, purged (`RetentionDays`). Per-channel overrides still delete their rows earlier,
so those rows are never archived — the archive carries their tombstones instead, so those rows are never archived — the archive carries their tombstones instead,
which is what keeps the archived chain verifiable. Archive files have no retention which is what keeps the archived chain verifiable.
of their own. - **Archive retention (ScadaBridge#109) — default never delete.**
`AuditLog:Archive:RetentionDays` (default `0` = archives are never deleted, until compliance
names a period) is counted from the END of the archived month and, when set, must exceed
`AuditLog:RetentionDays` (validated at startup) — an archive always outlives the rows' time
in the database. After each purge tick the purge actor deletes an expired archive that is
`Purged`, or an `Archived` one whose partition no longer holds rows.
- **Superseded stale archives (ScadaBridge#109).** A purge whose guard finds the partition
changed (51002/51003) leaves its archive at `Archived` (verify reports `NotPurged`, or
`Superseded` once a later purge anchored past it) and the next tick archives again. With
`AuditLog:Archive:DeleteSupersededStaleArchives` (default `true`) the purge actor deletes
such a stale archive only when a LATER archive of the same partition is `Purged`, verifies,
and provably holds every row of it: for every chain month, the same earlier anchor and the
stale archive's chain-end hash at its chain-end sequence (row or tombstone), and every
unchained row present by EventId. Otherwise it is kept and reported `KeptStale` (e.g. an
unchained row a per-channel purge removed in between — no tombstone — so the stale archive
is that row's only copy). An archive left at `Archived` because the node stopped between the
switch and the manifest update has no later purged archive and is never touched.
- **Safe deletion.** Cleanup deletes only the three files a well-formed archive name implies
(`{name}.manifest.json`, `.parquet`, `.tombstones.parquet`) in that archive's own
`{Directory}/{YYYY}` folder — never a path read from a manifest — manifest first, so an
interrupted deletion leaves leftovers that are no longer an archive (orphans of an interrupted
archive WRITE are not cleaned yet — #114). A cleanup failure is a
purge failure on the health surface. `scadabridge audit archive cleanup [--dry-run]`
(`CleanupAuditArchives`, Administrator) runs the same pass on demand or reports what it
would delete. Pinned by `AuditLogArchiveCleanupTests` (mutation-checked: each safety rule
removed fails its test).
- **Verify / read.** `audit archive verify --name` (`VerifyAuditArchive`) checks both - **Verify / read.** `audit archive verify --name` (`VerifyAuditArchive`) checks both
files' SHA-256 and counts against the manifest, recomputes each archived month's files' SHA-256 and counts against the manifest, recomputes each archived month's
chain from the archived rows and tombstones (`AuditChainVerifier`, starting from chain from the archived rows and tombstones (`AuditChainVerifier`, starting from
@@ -807,8 +836,9 @@ MS SQL for direct-write events). Unredacted secrets never persist.
`Matches`; `Superseded` (late rows for the month were chained, purged and archived `Matches`; `Superseded` (late rows for the month were chained, purged and archived
again afterwards — that later archive resumes from this one's end); `NotPurged`; again afterwards — that later archive resumes from this one's end); `NotPurged`;
or `Mismatch` (archive or head altered). The files are plain Parquet — DuckDB, or `Mismatch` (archive or head altered). The files are plain Parquet — DuckDB,
pyarrow or Spark read them directly. There is no re-import into `dbo.AuditLog`: pyarrow, Spark or parquet-tools read them directly. **Owner decision (ScadaBridge#109):
a re-inserted expired row would be purged again by the next tick. reading with Parquet tools is the supported access path; there is no re-import into
`dbo.AuditLog`** — a re-inserted expired row would be purged again by the next tick.
- **Sites:** `SiteAuditRetentionService` (site-only `IHostedService`) runs the - **Sites:** `SiteAuditRetentionService` (site-only `IHostedService`) runs the
site SQLite retention purge on a timer — first tick after site SQLite retention purge on a timer — first tick after
`AuditLog:SiteRetention:InitialDelay` (default 5 min, short so a daily-recycled `AuditLog:SiteRetention:InitialDelay` (default 5 min, short so a daily-recycled
@@ -880,7 +910,7 @@ MS SQL for direct-write events). Unredacted secrets never persist.
to the maintenance login: the monthly partition roll-forward (`ALTER PARTITION FUNCTION`, to the maintenance login: the monthly partition roll-forward (`ALTER PARTITION FUNCTION`,
needs `ALTER ANY DATASPACE` on the runtime login, or run it as a DBA job) and the one-time needs `ALTER ANY DATASPACE` on the runtime login, or run it as a DBA job) and the one-time
`audit backfill-source-node` (needs `UPDATE` on `SourceNode` — a DBA-run step in a hardened `audit backfill-source-node` (needs `UPDATE` on `SourceNode` — a DBA-run step in a hardened
setup). Migrations themselves are applied by a DBA with the production SQL scripts. setup) — both tracked in #113. Migrations themselves are applied by a DBA with the production SQL scripts.
- **Authorization.** Reading the Audit Log requires the existing **Audit** role - **Authorization.** Reading the Audit Log requires the existing **Audit** role
extended with a new **OperationalAudit** permission. Per-site row scoping extended with a new **OperationalAudit** permission. Per-site row scoping
reuses the existing site-permission model; bulk export requires an additional reuses the existing site-permission model; bulk export requires an additional
@@ -943,7 +973,7 @@ MS SQL for direct-write events). Unredacted secrets never persist.
`FileMode.Append` only, one file per node per day; give the service account create + `FileMode.Append` only, one file per node per day; give the service account create +
append-data rights and nothing else), or `Sink: Syslog` — RFC 5424 (`<PRI>1 … chainhead - append-data rights and nothing else), or `Sink: Syslog` — RFC 5424 (`<PRI>1 … chainhead -
{json}`, facility 13 "log audit", severity Notice) over UDP (RFC 5426) or TCP (RFC 6587 {json}`, facility 13 "log audit", severity Notice) over UDP (RFC 5426) or TCP (RFC 6587
octet counting). A failed write is logged and retried with the same heads next tick. octet counting). A failed write is logged and retried with the same heads next tick (no health counter yet — #115).
`scadabridge audit verify-heads --path <file|dir>` reads those records back (file-sink `scadabridge audit verify-heads --path <file|dir>` reads those records back (file-sink
lines or a syslog collector's log) and the `VerifyAuditChainHeads` command checks each lines or a syslog collector's log) and the `VerifyAuditChainHeads` command checks each
against the database: the link stored at the exported sequence (the head, a live row's against the database: the link stored at the exported sequence (the head, a live row's
@@ -961,9 +991,10 @@ MS SQL for direct-write events). Unredacted secrets never persist.
- **Parquet archival (T2) — shipped (ScadaBridge#42), default OFF.** See Retention & - **Parquet archival (T2) — shipped (ScadaBridge#42), default OFF.** See Retention &
Purge. The archive carries the hash-chain columns and the chain head, so tamper Purge. The archive carries the hash-chain columns and the chain head, so tamper
evidence survives the purge: `audit archive verify` ties an archive to the checkpoint evidence survives the purge: `audit archive verify` ties an archive to the checkpoint
anchor still in `AuditLogChainHead`. Like the chain itself, archives are unsigned; anchor still in `AuditLogChainHead`. Like the chain itself, archives are unsigned by owner
a principal able to rewrite both the archive files and the head row can forge a decision (ScadaBridge#107/#109); a principal able to rewrite both the archive files and the
consistent pair. head row can forge a consistent pair — unless the month's head was exported off-box first
(`AuditLog:HashChain:HeadExport`): `audit verify-heads` then exposes the rewritten anchor.
- **Site SQLite security.** File permissions: read/write by the ScadaBridge - **Site SQLite security.** File permissions: read/write by the ScadaBridge
service account only. Not backed up off-machine — site SQLite is a buffer, service account only. Not backed up off-machine — site SQLite is a buffer,
not a record. not a record.
@@ -1062,7 +1093,9 @@ archive (see Retention & Purge):
"Directory": "D:\\AuditArchive", // absolute; required when Enabled; reachable by both centrals "Directory": "D:\\AuditArchive", // absolute; required when Enabled; reachable by both centrals
"RowGroupSize": 50000, // 1,000 – 1,000,000 "RowGroupSize": 50000, // 1,000 – 1,000,000
"ReadCommandTimeoutMinutes": 60, // reading one partition out of SQL Server "ReadCommandTimeoutMinutes": 60, // reading one partition out of SQL Server
"ChainLockTimeoutSeconds": 300 // wait for a chaining pass before skipping the tick "ChainLockTimeoutSeconds": 300, // wait for a chaining pass before skipping the tick
"RetentionDays": 0, // 0 = never delete; else > AuditLog:RetentionDays, from the archived month's end (#109)
"DeleteSupersededStaleArchives": true // delete stale archives a later purged archive provably covers (#109)
} }
} }
``` ```
+7
View File
@@ -319,6 +319,7 @@ scadabridge audit verify-chain --month <YYYY-MM>
scadabridge audit verify-heads --path <file-or-directory> scadabridge audit verify-heads --path <file-or-directory>
scadabridge audit archive list scadabridge audit archive list
scadabridge audit archive verify --name <archive> scadabridge audit archive verify --name <archive>
scadabridge audit archive cleanup [--dry-run]
``` ```
- `audit query` — filtered query against the central `AuditLog` table, matching the - `audit query` — filtered query against the central `AuditLog` table, matching the
@@ -370,6 +371,12 @@ scadabridge audit archive verify --name <archive>
archived hash chain and compares its end with the purge anchor; exit `0` verified, archived hash chain and compares its end with the purge anchor; exit `0` verified,
`3` not verified, `1` error, `2` authorization failure. Same gate as `verify-chain`. `3` not verified, `1` error, `2` authorization failure. Same gate as `verify-chain`.
See Component-AuditLog.md, Retention & Purge. See Component-AuditLog.md, Retention & Purge.
- `audit archive cleanup [--dry-run]` (ScadaBridge#109) — sends `CleanupAuditArchives`:
deletes superseded stale archives (a retried purge's, once a later purged archive provably
holds all their rows) and archives past `AuditLog:Archive:RetentionDays` (0 = never), the
same pass the purge actor runs after every tick; `--dry-run` only reports. Prints each
`DeletedSuperseded` / `DeletedExpired` / `KeptStale` with its reason; exit `0`. Administrator
only (it deletes files).
### Health Commands ### Health Commands
``` ```
@@ -202,6 +202,7 @@ The two-person authorization workflow for writes through the MxAccess Gateway. B
- **VerifyAuditChain** (ScadaBridge#41): verifies one UTC calendar month (`Month`, `YYYY-MM`) of the central AuditLog hash chain and returns an `AuditChainVerificationResult` (status, rows verified, tombstones applied, purge anchor, head sequence, unchained rows, first break with partition / `ChainSeq` / `EventId` / kind). Gated on the OperationalAudit roles (`Administrator`, `Viewer`); a site-scoped non-Administrator caller is refused (`ManagementUnauthorized`) because a month's chain spans every site. Long-running Ask timeout. Errors: `INVALID_ARGUMENT` (bad month), `CHAIN_BUSY` (a chaining/purge pass held the chain lock past 60 s). Backs `scadabridge audit verify-chain`. - **VerifyAuditChain** (ScadaBridge#41): verifies one UTC calendar month (`Month`, `YYYY-MM`) of the central AuditLog hash chain and returns an `AuditChainVerificationResult` (status, rows verified, tombstones applied, purge anchor, head sequence, unchained rows, first break with partition / `ChainSeq` / `EventId` / kind). Gated on the OperationalAudit roles (`Administrator`, `Viewer`); a site-scoped non-Administrator caller is refused (`ManagementUnauthorized`) because a month's chain spans every site. Long-running Ask timeout. Errors: `INVALID_ARGUMENT` (bad month), `CHAIN_BUSY` (a chaining/purge pass held the chain lock past 60 s). Backs `scadabridge audit verify-chain`.
- **VerifyAuditChainHeads** (ScadaBridge#107): checks chain heads exported off-box (`Records`, at most 50,000 `AuditChainHeadRecord`s; duplicates collapsed) against the database chain and returns an `AuditChainHeadVerificationResult` (overall `Verified`, one `AuditChainHeadCheck` per distinct exported head: `Matches` / `Extended` / `Purged`, or the failures `Mismatch` / `Regressed` / `MissingHead` / `MissingLink` / `CountMismatch`). Same gate as VerifyAuditChain; long-running Ask timeout. Errors: `INVALID_ARGUMENT` (no records / over the cap), `CHAIN_BUSY`. Backs `scadabridge audit verify-heads`. - **VerifyAuditChainHeads** (ScadaBridge#107): checks chain heads exported off-box (`Records`, at most 50,000 `AuditChainHeadRecord`s; duplicates collapsed) against the database chain and returns an `AuditChainHeadVerificationResult` (overall `Verified`, one `AuditChainHeadCheck` per distinct exported head: `Matches` / `Extended` / `Purged`, or the failures `Mismatch` / `Regressed` / `MissingHead` / `MissingLink` / `CountMismatch`). Same gate as VerifyAuditChain; long-running Ask timeout. Errors: `INVALID_ARGUMENT` (no records / over the cap), `CHAIN_BUSY`. Backs `scadabridge audit verify-heads`.
- **ListAuditArchives** / **VerifyAuditArchive** (ScadaBridge#42): list the AuditLog Parquet cold archives under `AuditLog:Archive:Directory` (`AuditArchiveListResult`: enabled flag + every manifest), and verify one by `Name` (`AuditArchiveVerificationResult`: file integrity, per-month chain recomputed from the archived rows, anchor `Matches`/`Superseded`/`NotPurged`/`Mismatch`, overall `Verified`). Same gate as VerifyAuditChain (OperationalAudit roles, site-scoped non-Administrator refused). VerifyAuditArchive uses the long-running Ask timeout. Errors: `ARCHIVE_NOT_CONFIGURED`, `INVALID_ARGUMENT` (not an archive name), `NOT_FOUND`. Back `scadabridge audit archive list|verify`. - **ListAuditArchives** / **VerifyAuditArchive** (ScadaBridge#42): list the AuditLog Parquet cold archives under `AuditLog:Archive:Directory` (`AuditArchiveListResult`: enabled flag + every manifest), and verify one by `Name` (`AuditArchiveVerificationResult`: file integrity, per-month chain recomputed from the archived rows, anchor `Matches`/`Superseded`/`NotPurged`/`Mismatch`, overall `Verified`). Same gate as VerifyAuditChain (OperationalAudit roles, site-scoped non-Administrator refused). VerifyAuditArchive uses the long-running Ask timeout. Errors: `ARCHIVE_NOT_CONFIGURED`, `INVALID_ARGUMENT` (not an archive name), `NOT_FOUND`. Back `scadabridge audit archive list|verify`.
- **CleanupAuditArchives** (ScadaBridge#109): runs the archive cleanup now (`DryRun` = report only) — superseded stale archives a later purged archive provably covers, and archives past `AuditLog:Archive:RetentionDays` — and returns an `AuditArchiveCleanupResult` (`DryRun`, `RetentionDays`, one item per archive deleted / kept stale, with the action `DeletedExpired` / `DeletedSuperseded` / `KeptStale` and reason). Administrator only; runs on the maintenance-connection archive when registered; long-running Ask timeout. Errors: `ARCHIVE_NOT_CONFIGURED`. Backs `scadabridge audit archive cleanup`.
### Transport (Bundle Import / Export) ### Transport (Bundle Import / Export)
@@ -32,5 +32,34 @@ public sealed class AuditLogArchiveOptionsValidator : OptionsValidatorBase<Audit
$"AuditLog:Archive:{nameof(AuditLogArchiveOptions.ReadCommandTimeoutMinutes)} ({options.ReadCommandTimeoutMinutes}) must be > 0."); $"AuditLog:Archive:{nameof(AuditLogArchiveOptions.ReadCommandTimeoutMinutes)} ({options.ReadCommandTimeoutMinutes}) must be > 0.");
builder.RequireThat(options.ChainLockTimeoutSeconds > 0, builder.RequireThat(options.ChainLockTimeoutSeconds > 0,
$"AuditLog:Archive:{nameof(AuditLogArchiveOptions.ChainLockTimeoutSeconds)} ({options.ChainLockTimeoutSeconds}) must be > 0."); $"AuditLog:Archive:{nameof(AuditLogArchiveOptions.ChainLockTimeoutSeconds)} ({options.ChainLockTimeoutSeconds}) must be > 0.");
builder.RequireThat(options.RetentionDays is >= 0 and <= 36_500,
$"AuditLog:Archive:{nameof(AuditLogArchiveOptions.RetentionDays)} ({options.RetentionDays}) must be between 0 (never delete) and 36500.");
}
}
/// <summary>
/// Cross-section check for <see cref="AuditLogArchiveOptions.RetentionDays"/> (ScadaBridge#109):
/// when set, it must exceed <c>AuditLog:RetentionDays</c>, so an archive is never deleted before
/// (or as soon as) its rows leave the database — the archive exists to outlive them.
/// </summary>
public sealed class AuditLogArchiveRetentionValidator : Microsoft.Extensions.Options.IValidateOptions<AuditLogArchiveOptions>
{
private readonly Microsoft.Extensions.Options.IOptions<Configuration.AuditLogOptions> _auditOptions;
/// <summary>Creates the validator.</summary>
/// <param name="auditOptions">The <c>AuditLog</c> options (global retention).</param>
public AuditLogArchiveRetentionValidator(Microsoft.Extensions.Options.IOptions<Configuration.AuditLogOptions> auditOptions)
=> _auditOptions = auditOptions ?? throw new ArgumentNullException(nameof(auditOptions));
/// <inheritdoc />
public Microsoft.Extensions.Options.ValidateOptionsResult Validate(string? name, AuditLogArchiveOptions options)
{
ArgumentNullException.ThrowIfNull(options);
var global = _auditOptions.Value.RetentionDays;
return options.RetentionDays == 0 || options.RetentionDays > global
? Microsoft.Extensions.Options.ValidateOptionsResult.Success
: Microsoft.Extensions.Options.ValidateOptionsResult.Fail(
$"AuditLog:Archive:RetentionDays ({options.RetentionDays}) must be 0 (never delete) or greater than "
+ $"AuditLog:RetentionDays ({global}): archives must outlive the rows' time in the database.");
} }
} }
@@ -6,6 +6,7 @@ using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.AuditLog.Configuration; using ZB.MOM.WW.ScadaBridge.AuditLog.Configuration;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Types; using ZB.MOM.WW.ScadaBridge.Commons.Types;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance; using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance;
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central; namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
@@ -195,6 +196,7 @@ public class AuditLogPurgeActor : ReceiveActor
// No whole-month partitions are eligible, but per-channel overrides may // No whole-month partitions are eligible, but per-channel overrides may
// still expire rows earlier than the global window — run them below. // still expire rows earlier than the global window — run them below.
await RunPerChannelOverridesAsync(repository, eventStream, purgeFailureCounter).ConfigureAwait(false); await RunPerChannelOverridesAsync(repository, eventStream, purgeFailureCounter).ConfigureAwait(false);
await RunArchiveCleanupAsync(scope.ServiceProvider, eventStream, purgeFailureCounter).ConfigureAwait(false);
return; return;
} }
@@ -274,6 +276,52 @@ public class AuditLogPurgeActor : ReceiveActor
// deletes only ever expire rows EARLIER than that, so they run last and are a // deletes only ever expire rows EARLIER than that, so they run last and are a
// strict tightening. // strict tightening.
await RunPerChannelOverridesAsync(repository, eventStream, purgeFailureCounter).ConfigureAwait(false); await RunPerChannelOverridesAsync(repository, eventStream, purgeFailureCounter).ConfigureAwait(false);
// Archive housekeeping last (ScadaBridge#109): superseded stale archives this or an
// earlier tick's retried purge left behind, and archives past AuditLog:Archive:RetentionDays.
if (archive is not null)
{
await RunArchiveCleanupAsync(scope.ServiceProvider, eventStream, purgeFailureCounter, archive).ConfigureAwait(false);
}
}
/// <summary>
/// Runs the archive cleanup pass (ScadaBridge#109) when archiving is enabled. Contained: a
/// cleanup failure is a purge failure on the health surface, never a crash of the tick.
/// </summary>
private async Task RunArchiveCleanupAsync(
IServiceProvider services,
Akka.Event.EventStream eventStream,
IAuditPurgeFailureCounter? purgeFailureCounter,
IAuditLogArchive? archive = null)
{
var sw = Stopwatch.StartNew();
try
{
if (archive is null)
{
if (services.GetService<IOptions<AuditLogArchiveOptions>>()?.Value.Enabled != true)
{
return;
}
archive = services.GetKeyedService<IAuditLogArchive>(AuditMaintenanceContext.ServiceKey)
?? services.GetRequiredService<IAuditLogArchive>();
}
var result = await archive.CleanupAsync(dryRun: false).ConfigureAwait(false);
var deleted = result.Items.Count(i => i.Action != AuditArchiveCleanupAction.KeptStale);
if (deleted > 0)
{
_logger.LogInformation("AuditLog archive cleanup deleted {Count} archive(s) in {DurationMs} ms.", deleted, sw.ElapsedMilliseconds);
}
}
catch (Exception ex)
{
sw.Stop();
_logger.LogError(ex, "AuditLog archive cleanup failed; archives are kept and the next tick retries.");
PublishPurgeFailure(eventStream, purgeFailureCounter, DateTime.MinValue, ex, sw.ElapsedMilliseconds);
}
} }
/// <summary> /// <summary>
@@ -442,6 +442,10 @@ public static class ServiceCollectionExtensions
// registered by AddConfigurationDatabase (central only) and reads these options. // registered by AddConfigurationDatabase (central only) and reads these options.
services.AddValidatedOptions<ConfigurationDatabase.Maintenance.AuditLogArchiveOptions, AuditLogArchiveOptionsValidator>( services.AddValidatedOptions<ConfigurationDatabase.Maintenance.AuditLogArchiveOptions, AuditLogArchiveOptionsValidator>(
config, ArchiveSectionName); config, ArchiveSectionName);
// Archive retention must exceed the global AuditLog:RetentionDays (ScadaBridge#109).
services.TryAddEnumerable(ServiceDescriptor.Singleton<
Microsoft.Extensions.Options.IValidateOptions<ConfigurationDatabase.Maintenance.AuditLogArchiveOptions>,
AuditLogArchiveRetentionValidator>());
// I1 (review): bind the two central-singleton options HERE rather than in // I1 (review): bind the two central-singleton options HERE rather than in
// AddAuditLogCentralReconciliationClient. AkkaHostedService.RegisterCentralActors // AddAuditLogCentralReconciliationClient. AkkaHostedService.RegisterCentralActors
@@ -109,6 +109,43 @@ public static class AuditArchiveHelpers
return verified ? 0 : NotVerifiedExitCode; return verified ? 0 : NotVerifiedExitCode;
} }
/// <summary>
/// Renders a <c>CleanupAuditArchives</c> result (ScadaBridge#109). Always exit 0 — kept
/// stale archives are reported, not an error.
/// </summary>
/// <param name="json">The success body (an <c>AuditArchiveCleanupResult</c>, camelCase).</param>
/// <param name="format">Resolved output format.</param>
/// <param name="output">Writer for the report.</param>
/// <returns><c>0</c>.</returns>
public static int RenderCleanup(string json, string format, TextWriter output)
{
if (!string.Equals(format, "table", StringComparison.OrdinalIgnoreCase))
{
output.WriteLine(json);
output.Flush();
return 0;
}
using var doc = JsonDocument.Parse(json);
var root = doc.RootElement;
var dryRun = root.TryGetProperty("dryRun", out var d) && d.ValueKind == JsonValueKind.True;
var retention = Num(root, "retentionDays");
output.WriteLine($"{(dryRun ? "Dry run — nothing deleted. " : "")}Archive retention: {(retention == "0" ? "never delete" : retention + " days")}");
var items = root.TryGetProperty("items", out var i) && i.ValueKind == JsonValueKind.Array ? i.EnumerateArray().ToList() : [];
if (items.Count == 0)
{
output.WriteLine("Nothing to clean up.");
}
foreach (var item in items)
{
output.WriteLine($"{Str(item, "action"),-18} {Str(item, "name")} {Str(item, "detail")}");
}
output.Flush();
return 0;
}
private static string Str(JsonElement e, string name, string missing = "") private static string Str(JsonElement e, string name, string missing = "")
=> e.TryGetProperty(name, out var v) && v.ValueKind == JsonValueKind.String ? v.GetString() ?? missing : missing; => e.TryGetProperty(name, out var v) && v.ValueKind == JsonValueKind.String ? v.GetString() ?? missing : missing;
@@ -11,9 +11,9 @@ namespace ZB.MOM.WW.ScadaBridge.CLI.Commands;
/// <c>GET /api/audit/tree</c>), plus <c>verify-chain</c>, which sends the /// <c>GET /api/audit/tree</c>), plus <c>verify-chain</c>, which sends the
/// <c>VerifyAuditChain</c> management command (hash-chain tamper evidence, ScadaBridge#41), /// <c>VerifyAuditChain</c> management command (hash-chain tamper evidence, ScadaBridge#41),
/// <c>verify-heads</c>, which checks off-box exported chain heads with /// <c>verify-heads</c>, which checks off-box exported chain heads with
/// <c>VerifyAuditChainHeads</c> (ScadaBridge#107), and <c>archive list</c> / <c>archive verify</c>, /// <c>VerifyAuditChainHeads</c> (ScadaBridge#107), and <c>archive list</c> / <c>archive verify</c>
/// which send <c>ListAuditArchives</c> / <c>VerifyAuditArchive</c> (Parquet cold archive, /// / <c>archive cleanup</c>, which send <c>ListAuditArchives</c> / <c>VerifyAuditArchive</c>
/// ScadaBridge#42). /// (Parquet cold archive, ScadaBridge#42) / <c>CleanupAuditArchives</c> (ScadaBridge#109).
/// </summary> /// </summary>
public static class AuditCommands public static class AuditCommands
{ {
@@ -376,7 +376,7 @@ public static class AuditCommands
private static Command BuildArchive(Option<string> urlOption, Option<string> formatOption, Option<string> usernameOption, Option<string> passwordOption) private static Command BuildArchive(Option<string> urlOption, Option<string> formatOption, Option<string> usernameOption, Option<string> passwordOption)
{ {
var group = new Command("archive") { Description = "List and verify the AuditLog Parquet cold archives" }; var group = new Command("archive") { Description = "List, verify and clean up the AuditLog Parquet cold archives" };
var list = new Command("list") { Description = "List the AuditLog Parquet archives written by the retention purge" }; var list = new Command("list") { Description = "List the AuditLog Parquet archives written by the retention purge" };
list.SetAction(async (ParseResult result) => list.SetAction(async (ParseResult result) =>
@@ -410,6 +410,23 @@ public static class AuditCommands
}); });
group.Add(verify); group.Add(verify);
var dryRunOption = new Option<bool>("--dry-run") { Description = "Report what would be deleted; delete nothing" };
var cleanup = new Command("cleanup")
{
Description = "Delete superseded stale archives and archives past AuditLog:Archive:RetentionDays (Administrator)",
};
cleanup.Add(dryRunOption);
cleanup.SetAction(async (ParseResult result) =>
{
var format = AuditCommandHelpers.ResolveFormat(result, formatOption);
return await CommandHelpers.ExecuteCommandAsync(
result, urlOption, formatOption, usernameOption, passwordOption,
new CleanupAuditArchivesCommand(result.GetValue(dryRunOption)),
timeout: AuditArchiveHelpers.VerifyRequestTimeout,
onSuccess: json => AuditArchiveHelpers.RenderCleanup(json, format, Console.Out));
});
group.Add(cleanup);
return group; return group;
} }
+7
View File
@@ -2136,6 +2136,13 @@ tombstones, and compares the chain end with the checkpoint anchor the purge left
| `1` | Request error (`ARCHIVE_NOT_CONFIGURED`, `NOT_FOUND`, `INVALID_ARGUMENT`, server error) | | `1` | Request error (`ARCHIVE_NOT_CONFIGURED`, `NOT_FOUND`, `INVALID_ARGUMENT`, server error) |
| `2` | Authorization failure (needs `Administrator` or `Viewer`, not site-scoped) | | `2` | Authorization failure (needs `Administrator` or `Viewer`, not site-scoped) |
`audit archive cleanup [--dry-run]` (ScadaBridge#109, Administrator) sends
`CleanupAuditArchives`: it deletes stale archives a retried purge left behind once a later
purged archive provably holds all their rows (`DeletedSuperseded`), and archives whose month
ended more than `AuditLog:Archive:RetentionDays` ago (`DeletedExpired`; default `0` = never),
and lists stale archives it kept (`KeptStale`) with the reason. `--dry-run` deletes nothing.
The purge actor runs the same pass after every purge tick. Exit code `0`.
The archive files are plain Parquet; read them with any Parquet tool, e.g. DuckDB: The archive files are plain Parquet; read them with any Parquet tool, e.g. DuckDB:
`SELECT * FROM '<dir>/2025/auditlog-2025-10-….parquet'`. See `Component-AuditLog.md`, `SELECT * FROM '<dir>/2025/auditlog-2025-10-….parquet'`. See `Component-AuditLog.md`,
Retention & Purge. Retention & Purge.
@@ -66,6 +66,17 @@ public interface IAuditLogArchive
/// <returns>The verification result.</returns> /// <returns>The verification result.</returns>
/// <exception cref="FileNotFoundException">No archive has that name.</exception> /// <exception cref="FileNotFoundException">No archive has that name.</exception>
Task<AuditArchiveVerificationResult> VerifyAsync(string name, CancellationToken ct = default); Task<AuditArchiveVerificationResult> VerifyAsync(string name, CancellationToken ct = default);
/// <summary>
/// Archive cleanup (ScadaBridge#109): deletes superseded stale archives a retried purge left
/// behind (when <c>AuditLog:Archive:DeleteSupersededStaleArchives</c>), and archives past
/// <c>AuditLog:Archive:RetentionDays</c> (0 = never). Only ever deletes the three files of a
/// well-formed archive inside the configured directory, manifest first.
/// </summary>
/// <param name="dryRun">Report only; delete nothing.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>What was (or would be) deleted, and the stale archives kept.</returns>
Task<AuditArchiveCleanupResult> CleanupAsync(bool dryRun, CancellationToken ct = default);
} }
/// <summary> /// <summary>
@@ -57,3 +57,12 @@ public record ListAuditArchivesCommand;
/// </summary> /// </summary>
/// <param name="Name">The archive name, as <c>audit archive list</c> reports it.</param> /// <param name="Name">The archive name, as <c>audit archive list</c> reports it.</param>
public record VerifyAuditArchiveCommand(string Name); public record VerifyAuditArchiveCommand(string Name);
/// <summary>
/// Runs the AuditLog archive cleanup (ScadaBridge#109) now: deletes superseded stale archives
/// a retried purge left behind and archives past <c>AuditLog:Archive:RetentionDays</c> — or,
/// with <paramref name="DryRun"/>, only reports what it would delete. The purge actor runs the
/// same pass after every purge tick. Administrator only (it deletes files).
/// </summary>
/// <param name="DryRun">Report only; delete nothing.</param>
public record CleanupAuditArchivesCommand(bool DryRun);
@@ -145,3 +145,41 @@ public sealed record AuditArchiveVerificationResult(
/// <param name="Enabled">Whether the retention purge is archiving partitions (<c>AuditLog:Archive:Enabled</c>).</param> /// <param name="Enabled">Whether the retention purge is archiving partitions (<c>AuditLog:Archive:Enabled</c>).</param>
/// <param name="Archives">Every archive manifest, oldest partition first.</param> /// <param name="Archives">Every archive manifest, oldest partition first.</param>
public sealed record AuditArchiveListResult(bool Enabled, IReadOnlyList<AuditArchiveManifest> Archives); public sealed record AuditArchiveListResult(bool Enabled, IReadOnlyList<AuditArchiveManifest> Archives);
/// <summary>What archive cleanup did (or, on a dry run, would do) with one archive (ScadaBridge#109).</summary>
[JsonConverter(typeof(JsonStringEnumConverter<AuditArchiveCleanupAction>))]
public enum AuditArchiveCleanupAction
{
/// <summary>
/// Deleted by retention: the archived month ended more than <c>AuditLog:Archive:RetentionDays</c>
/// ago, and the archive is purged (or its partition no longer holds rows).
/// </summary>
DeletedExpired,
/// <summary>
/// Deleted as a superseded stale archive: status <c>Archived</c> (its partition switch never
/// completed — a retried purge), and a LATER, verified, <c>Purged</c> archive of the same
/// partition provably holds every row it holds (same chain through its chain end, every
/// unchained row present by EventId).
/// </summary>
DeletedSuperseded,
/// <summary>
/// A stale <c>Archived</c> archive that was kept: no later purged archive provably covers it,
/// so it may be the only copy of some rows (see Detail).
/// </summary>
KeptStale,
}
/// <summary>One archive considered by cleanup.</summary>
/// <param name="Name">Archive name.</param>
/// <param name="Partition">Archived partition, <c>YYYY-MM</c>.</param>
/// <param name="Action">What was (or would be) done.</param>
/// <param name="Detail">Why.</param>
public sealed record AuditArchiveCleanupItem(string Name, string Partition, AuditArchiveCleanupAction Action, string Detail);
/// <summary>Result of an archive cleanup pass (ScadaBridge#109).</summary>
/// <param name="DryRun">True when nothing was deleted (the items say what would be).</param>
/// <param name="RetentionDays">The archive retention in force (0 = archives never expire).</param>
/// <param name="Items">The archives deleted (or that would be) and the stale ones kept.</param>
public sealed record AuditArchiveCleanupResult(bool DryRun, int RetentionDays, IReadOnlyList<AuditArchiveCleanupItem> Items);
@@ -43,4 +43,19 @@ public sealed class AuditLogArchiveOptions
/// the hash-chain lock before giving up for this tick (default 300). /// the hash-chain lock before giving up for this tick (default 300).
/// </summary> /// </summary>
public int ChainLockTimeoutSeconds { get; set; } = 300; public int ChainLockTimeoutSeconds { get; set; } = 300;
/// <summary>
/// Archive retention in days, counted from the END of the archived month (ScadaBridge#109).
/// <c>0</c> (the default) = archives are never deleted — until compliance names a period.
/// When set it must exceed <c>AuditLog:RetentionDays</c>, so an archive always outlives the
/// rows' time in the database. Applied by the purge actor after each tick's partition purge.
/// </summary>
public int RetentionDays { get; set; }
/// <summary>
/// Delete stale archives a retried purge left behind (status <c>Archived</c>) once a later,
/// verified, purged archive of the same partition provably holds all their rows (default
/// true; ScadaBridge#109). A stale archive that cannot be proven redundant is kept.
/// </summary>
public bool DeleteSupersededStaleArchives { get; set; } = true;
} }
@@ -0,0 +1,290 @@
using System.Data;
using Microsoft.Data.SqlClient;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance;
/// <summary>
/// Archive cleanup (ScadaBridge#109): retention and superseded-stale-archive deletion.
/// </summary>
/// <remarks>
/// <para>
/// <b>Superseded stale archives.</b> A purge whose guard finds the partition changed after it
/// was archived (51002/51003) purges nothing and leaves its archive at status
/// <see cref="AuditArchiveStatus.Archived"/>; the next tick writes a fresh archive and purges
/// with it. The stale one is redundant only if a LATER archive B of the same partition, which
/// is <see cref="AuditArchiveStatus.Purged"/> and verifies, holds everything the stale archive A
/// holds. That is proven, not assumed: for every chain month of A, B starts from the same
/// earlier purge anchor and carries A's chain-end hash at A's chain-end sequence (as a row or a
/// tombstone) — so, the chain being a hash chain B itself verifies, every chained row of A is
/// in B; and every UNCHAINED row of A is found in B (row or tombstone) by EventId. Anything
/// else — notably an unchained row a per-channel purge removed in between, which leaves no
/// tombstone — keeps A (<see cref="AuditArchiveCleanupAction.KeptStale"/>). An archive left at
/// <c>Archived</c> because the node stopped between the switch and the manifest update has no
/// later purged archive and is kept: it is the only copy.
/// </para>
/// <para>
/// <b>Retention.</b> With <see cref="AuditLogArchiveOptions.RetentionDays"/> &gt; 0, an archive
/// whose month ended more than that many days ago is deleted when it is <c>Purged</c>, or —
/// an <c>Archived</c> one — when its partition no longer holds rows in <c>dbo.AuditLog</c>.
/// </para>
/// <para>
/// <b>Safe deletion.</b> Only the three files a well-formed archive name implies
/// (<c>{name}.manifest.json</c>, <c>{name}.parquet</c>, <c>{name}.tombstones.parquet</c>) in the
/// archive's own <c>{Directory}/{YYYY}</c> folder are deleted — never a path read from the
/// manifest — and the manifest goes FIRST, so an interrupted deletion leaves leftovers that are
/// no longer an archive rather than a manifest naming missing files.
/// </para>
/// </remarks>
public sealed partial class AuditLogPartitionArchive
{
/// <summary>
/// Upper bound on unchained rows compared by EventId when proving a stale archive redundant;
/// a larger set keeps the archive rather than spend the memory.
/// </summary>
internal const long MaxUnchainedRowsCompared = 5_000_000;
/// <inheritdoc />
public async Task<AuditArchiveCleanupResult> CleanupAsync(bool dryRun, CancellationToken ct = default)
{
RequireConfigured();
var items = new List<AuditArchiveCleanupItem>();
var manifests = await ListAsync(ct).ConfigureAwait(false);
var deleted = new HashSet<string>(StringComparer.Ordinal);
if (_options.DeleteSupersededStaleArchives)
{
foreach (var stale in manifests.Where(m => m.Status == AuditArchiveStatus.Archived))
{
var later = manifests
.Where(m => m.PartitionBoundaryUtc == stale.PartitionBoundaryUtc
&& m.Status == AuditArchiveStatus.Purged
&& m.ArchivedAtUtc > stale.ArchivedAtUtc)
.OrderBy(m => m.ArchivedAtUtc)
.ToList();
if (later.Count == 0)
{
continue; // not superseded (e.g. the node stopped before marking it Purged) — keep
}
string? reason = null;
AuditArchiveManifest? coveredBy = null;
foreach (var candidate in later)
{
reason = await ProveCoveredAsync(stale, candidate, ct).ConfigureAwait(false);
if (reason is null)
{
coveredBy = candidate;
break;
}
}
if (coveredBy is null)
{
items.Add(new AuditArchiveCleanupItem(stale.Name, stale.Partition, AuditArchiveCleanupAction.KeptStale,
$"No later purged archive provably covers it: {reason}"));
continue;
}
if (!dryRun)
{
DeleteArchiveFiles(stale);
}
deleted.Add(stale.Name);
items.Add(new AuditArchiveCleanupItem(stale.Name, stale.Partition, AuditArchiveCleanupAction.DeletedSuperseded,
$"Superseded by {coveredBy.Name}, which is purged, verifies and holds every row of this archive."));
}
}
if (_options.RetentionDays > 0)
{
var now = _time.GetUtcNow().UtcDateTime;
foreach (var manifest in manifests.Where(m => !deleted.Contains(m.Name)))
{
var monthEnd = AuditChainHasher.MonthOf(manifest.PartitionBoundaryUtc).AddMonths(1);
if (monthEnd.AddDays(_options.RetentionDays) > now)
{
continue;
}
if (manifest.Status != AuditArchiveStatus.Purged
&& await PartitionHasRowsAsync(manifest.PartitionBoundaryUtc, ct).ConfigureAwait(false))
{
continue; // the partition was never switched out; this archive is not a sole copy, nor expired data gone
}
if (!dryRun)
{
DeleteArchiveFiles(manifest);
}
deleted.Add(manifest.Name);
items.Add(new AuditArchiveCleanupItem(manifest.Name, manifest.Partition, AuditArchiveCleanupAction.DeletedExpired,
$"{manifest.Partition} ended {monthEnd:yyyy-MM-dd}, more than {_options.RetentionDays} days ago (AuditLog:Archive:RetentionDays)."));
}
}
foreach (var item in items.Where(i => i.Action != AuditArchiveCleanupAction.KeptStale))
{
_logger.LogInformation("AuditLog archive cleanup{DryRun}: {Action} {Archive} — {Detail}",
dryRun ? " (dry run)" : "", item.Action, item.Name, item.Detail);
}
return new AuditArchiveCleanupResult(dryRun, _options.RetentionDays, items);
}
/// <summary>
/// Proves <paramref name="later"/> holds every row of <paramref name="stale"/>; returns null
/// when it does, else why not.
/// </summary>
private async Task<string?> ProveCoveredAsync(AuditArchiveManifest stale, AuditArchiveManifest later, CancellationToken ct)
{
// Same chain generation and at least as far, for every chain month the stale archive has.
var laterChains = later.Chains.ToDictionary(c => c.Partition, StringComparer.Ordinal);
var targets = new Dictionary<(DateTime Month, long Seq), string>();
foreach (var chain in stale.Chains)
{
if (!laterChains.TryGetValue(chain.Partition, out var next))
{
return $"{later.Name} has no chain for {chain.Partition}.";
}
if (next.PriorPurgedThroughSeq != chain.PriorPurgedThroughSeq || next.LastSeq < chain.LastSeq)
{
return $"{later.Name}'s chain for {chain.Partition} does not extend this archive's (prior anchor or end differs).";
}
if (chain.LastSeq > (chain.PriorPurgedThroughSeq ?? 0))
{
targets[(AuditArchiveParquet.ParseMonth(chain.Partition), chain.LastSeq)] = chain.LastHash.ToLowerInvariant();
}
}
if (stale.UnchainedRowCount > MaxUnchainedRowsCompared)
{
return $"{stale.UnchainedRowCount} unchained rows exceed the {MaxUnchainedRowsCompared} compared by EventId.";
}
// The later archive must itself be intact and tied to the database.
var verification = await VerifyAsync(later.Name, ct).ConfigureAwait(false);
if (!verification.Verified)
{
return $"{later.Name} does not verify ({verification.FileProblem ?? "chain or anchor problem"}).";
}
var directory = ArchiveDirectory(stale);
var unchained = new HashSet<Guid>();
await using (var input = File.OpenRead(Path.Combine(directory, stale.Name + ".parquet")))
{
await foreach (var row in AuditArchiveParquet.ReadRowsAsync(input, ct).ConfigureAwait(false))
{
if (row.ChainSeq is null)
{
unchained.Add(row.EventId);
}
}
}
var laterDirectory = ArchiveDirectory(later);
await using (var input = File.OpenRead(Path.Combine(laterDirectory, later.Name + ".parquet")))
{
await foreach (var row in AuditArchiveParquet.ReadRowsAsync(input, ct).ConfigureAwait(false))
{
unchained.Remove(row.EventId);
if (row.ChainSeq is { } seq
&& targets.TryGetValue((AuditChainHasher.MonthOf(row.OccurredAtUtc), seq), out var expected))
{
if (row.RowHash is null || !string.Equals(Convert.ToHexStringLower(row.RowHash), expected, StringComparison.Ordinal))
{
return $"{later.Name} carries a different hash at the stale archive's chain end.";
}
targets.Remove((AuditChainHasher.MonthOf(row.OccurredAtUtc), seq));
}
}
}
await using (var input = File.OpenRead(Path.Combine(laterDirectory, later.Name + ".tombstones.parquet")))
{
await foreach (var tombstone in AuditArchiveParquet.ReadTombstonesAsync(input, ct).ConfigureAwait(false))
{
unchained.Remove(tombstone.EventId);
if (targets.TryGetValue((tombstone.ChainMonth, tombstone.ChainSeq), out var expected))
{
if (!string.Equals(Convert.ToHexStringLower(tombstone.RowHash), expected, StringComparison.Ordinal))
{
return $"{later.Name} carries a different hash at the stale archive's chain end.";
}
targets.Remove((tombstone.ChainMonth, tombstone.ChainSeq));
}
}
}
if (targets.Count > 0)
{
return $"{later.Name} does not contain the stale archive's chain end.";
}
return unchained.Count == 0
? null
: $"{unchained.Count} unchained row(s) of this archive are not in {later.Name} (e.g. removed by a per-channel purge, which leaves no tombstone for an unchained row).";
}
private async Task<bool> PartitionHasRowsAsync(DateTime boundary, CancellationToken ct)
{
var conn = _context.Database.GetDbConnection();
var openedHere = await AuditLogHashChain.OpenAsync(conn, ct).ConfigureAwait(false);
try
{
await using var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT CASE WHEN EXISTS (SELECT 1 FROM dbo.AuditLog " +
"WHERE $partition.pf_AuditLog_Month(OccurredAtUtc) = $partition.pf_AuditLog_Month(@boundary)) THEN 1 ELSE 0 END;";
cmd.Parameters.Add(new SqlParameter("@boundary", SqlDbType.DateTime2) { Scale = 7, Value = boundary });
return Convert.ToInt32(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false)) == 1;
}
finally
{
if (openedHere)
{
await conn.CloseAsync().ConfigureAwait(false);
}
}
}
private string ArchiveDirectory(AuditArchiveManifest manifest)
{
if (!IsValidName(manifest.Name))
{
throw new InvalidOperationException($"'{manifest.Name}' is not an archive name.");
}
return Path.Combine(_options.Directory!, manifest.Name.Substring("auditlog-".Length, 4));
}
/// <summary>
/// Deletes exactly the archive's three files, manifest first. File names are derived from
/// the validated archive name, never taken from the manifest.
/// </summary>
private void DeleteArchiveFiles(AuditArchiveManifest manifest)
{
var directory = Path.GetFullPath(ArchiveDirectory(manifest));
var root = Path.GetFullPath(_options.Directory!);
if (!directory.StartsWith(root, StringComparison.Ordinal))
{
throw new InvalidOperationException($"Archive {manifest.Name} resolves outside {root}.");
}
foreach (var file in new[] { ".manifest.json", ".parquet", ".tombstones.parquet" })
{
var path = Path.Combine(directory, manifest.Name + file);
if (File.Exists(path))
{
File.Delete(path);
}
}
}
}
@@ -398,6 +398,8 @@ public class ManagementActor : ReceiveActor
// AuditLog Parquet archives (ScadaBridge#42): audit reads with verify-chain's gate — // AuditLog Parquet archives (ScadaBridge#42): audit reads with verify-chain's gate —
// OperationalAudit roles, site-scoped callers refused in the handler. // OperationalAudit roles, site-scoped callers refused in the handler.
ListAuditArchivesCommand => OperationalAuditReaders, ListAuditArchivesCommand => OperationalAuditReaders,
// Archive cleanup deletes archive files (ScadaBridge#109): Administrator only.
CleanupAuditArchivesCommand => AdminOnly,
VerifyAuditArchiveCommand => OperationalAuditReaders, VerifyAuditArchiveCommand => OperationalAuditReaders,
// Read-only queries -- any authenticated user // Read-only queries -- any authenticated user
@@ -574,6 +576,7 @@ public class ManagementActor : ReceiveActor
VerifyAuditChainHeadsCommand cmd => await HandleVerifyAuditChainHeads(sp, cmd, user), VerifyAuditChainHeadsCommand cmd => await HandleVerifyAuditChainHeads(sp, cmd, user),
ListAuditArchivesCommand => await HandleListAuditArchives(sp, user), ListAuditArchivesCommand => await HandleListAuditArchives(sp, user),
VerifyAuditArchiveCommand cmd => await HandleVerifyAuditArchive(sp, cmd, user), VerifyAuditArchiveCommand cmd => await HandleVerifyAuditArchive(sp, cmd, user),
CleanupAuditArchivesCommand cmd => await HandleCleanupAuditArchives(sp, cmd, user),
// Health // Health
GetHealthSummaryCommand => HandleGetHealthSummary(sp), GetHealthSummaryCommand => HandleGetHealthSummary(sp),
@@ -3340,6 +3343,20 @@ public class ManagementActor : ReceiveActor
} }
} }
/// <summary>
/// Runs the archive cleanup now (ScadaBridge#109), on the maintenance-connection archive when
/// registered. Dry run reports only.
/// </summary>
private static async Task<object?> HandleCleanupAuditArchives(
IServiceProvider sp, CleanupAuditArchivesCommand cmd, AuthenticatedUser user)
{
RequireAuditArchive(sp, user);
var archive = sp.GetKeyedService<IAuditLogArchive>(
ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance.AuditMaintenanceContext.ServiceKey)
?? sp.GetRequiredService<IAuditLogArchive>();
return await archive.CleanupAsync(cmd.DryRun);
}
private static IAuditLogArchive RequireAuditArchive(IServiceProvider sp, AuthenticatedUser user) private static IAuditLogArchive RequireAuditArchive(IServiceProvider sp, AuthenticatedUser user)
{ {
if (user.PermittedSiteIds.Length > 0 if (user.PermittedSiteIds.Length > 0
@@ -69,6 +69,8 @@ public static class ManagementEndpoints
// AuditLog archive verification (ScadaBridge#42) hashes and re-chains a whole month's // AuditLog archive verification (ScadaBridge#42) hashes and re-chains a whole month's
// Parquet archive. // Parquet archive.
typeof(VerifyAuditArchiveCommand), typeof(VerifyAuditArchiveCommand),
// Archive cleanup proves stale archives redundant by reading Parquet files (ScadaBridge#109).
typeof(CleanupAuditArchivesCommand),
]; ];
/// <summary> /// <summary>
@@ -0,0 +1,82 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.AuditLog.Central;
using ZB.MOM.WW.ScadaBridge.AuditLog.Configuration;
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance;
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Central;
/// <summary>
/// <c>AuditLog:Archive:RetentionDays</c> (ScadaBridge#109): 0 = never delete; otherwise in range
/// and greater than the global <c>AuditLog:RetentionDays</c>.
/// </summary>
public sealed class AuditLogArchiveRetentionValidatorTests
{
/// <summary>Range check in the section validator.</summary>
/// <param name="days">RetentionDays.</param>
/// <param name="valid">Expected outcome.</param>
[Theory]
[InlineData(0, true)]
[InlineData(400, true)]
[InlineData(36_500, true)]
[InlineData(-1, false)]
[InlineData(36_501, false)]
public void SectionValidator_ChecksTheRange(int days, bool valid)
{
var result = new AuditLogArchiveOptionsValidator().Validate(null, new AuditLogArchiveOptions { RetentionDays = days });
Assert.Equal(valid, result.Succeeded);
}
/// <summary>Cross-section check against the global retention (365 here).</summary>
/// <param name="days">RetentionDays.</param>
/// <param name="valid">Expected outcome.</param>
[Theory]
[InlineData(0, true)]
[InlineData(366, true)]
[InlineData(365, false)]
[InlineData(30, false)]
public void RetentionValidator_RequiresMoreThanTheGlobalRetention(int days, bool valid)
{
var validator = new AuditLogArchiveRetentionValidator(Options.Create(new AuditLogOptions { RetentionDays = 365 }));
var result = validator.Validate(null, new AuditLogArchiveOptions { RetentionDays = days });
Assert.Equal(valid, result.Succeeded);
if (!valid)
{
Assert.Contains("AuditLog:RetentionDays (365)", result.FailureMessage);
}
}
/// <summary>The cross-section check is wired into the central registration.</summary>
/// <param name="days">Archive RetentionDays.</param>
/// <param name="valid">Whether binding succeeds.</param>
[Theory]
[InlineData("0", true)]
[InlineData("400", true)]
[InlineData("100", false)]
public void CentralRegistration_RejectsAnArchiveRetentionBelowTheGlobalOne(string days, bool valid)
{
var config = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?>
{
["AuditLog:SiteWriter:DatabasePath"] = ":memory:",
["AuditLog:RetentionDays"] = "365",
["AuditLog:Archive:RetentionDays"] = days,
})
.Build();
var services = new ServiceCollection();
services.AddAuditLog(config);
services.AddAuditLogCentralMaintenance(config);
using var provider = services.BuildServiceProvider();
var resolve = () => provider.GetRequiredService<IOptions<AuditLogArchiveOptions>>().Value;
if (valid)
{
Assert.Equal(int.Parse(days), resolve().RetentionDays);
}
else
{
Assert.Contains("AuditLog:Archive:RetentionDays (100)", Assert.Throws<OptionsValidationException>(resolve).Message);
}
}
}
@@ -822,4 +822,65 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
StopPurgeActor(actor); StopPurgeActor(actor);
} }
/// <summary>
/// ScadaBridge#109: with archiving enabled every tick ends with the archive cleanup pass —
/// also on a tick with no expired partition — and never runs it when archiving is off.
/// </summary>
/// <param name="withBoundary">Whether the tick has an expired partition to purge.</param>
[Theory]
[InlineData(true)]
[InlineData(false)]
public void ArchiveEnabled_RunsTheCleanupPass_EveryTick(bool withBoundary)
{
var boundary = new DateTime(2025, 10, 1, 0, 0, 0, DateTimeKind.Utc);
var repo = new RecordingRepo { Boundaries = withBoundary ? [boundary] : [] };
var archive = Substitute.For<IAuditLogArchive>();
var manifest = ManifestFor(boundary);
archive.ArchivePartitionAsync(boundary, Arg.Any<CancellationToken>()).Returns(manifest);
archive.CleanupAsync(false, Arg.Any<CancellationToken>()).Returns(new AuditArchiveCleanupResult(false, 0,
[new AuditArchiveCleanupItem("auditlog-2025-10-20261109T120000000Z", "2025-10", AuditArchiveCleanupAction.DeletedSuperseded, "x")]));
var actor = CreateArchivingActor(repo, archive, enabled: true);
actor.Tell(AuditLogPurgeActor.PurgeTick.Instance);
AwaitAssert(() => archive.Received().CleanupAsync(false, Arg.Any<CancellationToken>()),
duration: TimeSpan.FromSeconds(5), interval: TimeSpan.FromMilliseconds(50));
StopPurgeActor(actor);
}
/// <summary>A cleanup failure is contained and surfaces as a purge failure.</summary>
[Fact]
public void ArchiveCleanupFailure_IsAPurgeFailure()
{
var repo = new RecordingRepo();
var archive = Substitute.For<IAuditLogArchive>();
archive.CleanupAsync(false, Arg.Any<CancellationToken>())
.Returns<Task<AuditArchiveCleanupResult>>(_ => throw new IOException("share offline"));
var failed = CreateTestProbe();
Sys.EventStream.Subscribe(failed.Ref, typeof(AuditLogPurgeFailedEvent));
var actor = CreateArchivingActor(repo, archive, enabled: true);
actor.Tell(AuditLogPurgeActor.PurgeTick.Instance);
var evt = failed.ExpectMsg<AuditLogPurgeFailedEvent>(TimeSpan.FromSeconds(5));
Assert.Contains("share offline", evt.Error);
Assert.Equal(DateTime.MinValue, evt.MonthBoundary);
StopPurgeActor(actor);
}
/// <summary>Archiving off: no cleanup pass.</summary>
[Fact]
public void ArchiveDisabled_RunsNoCleanup()
{
var repo = new RecordingRepo();
var archive = Substitute.For<IAuditLogArchive>();
var actor = CreateArchivingActor(repo, archive, enabled: false);
actor.Tell(AuditLogPurgeActor.PurgeTick.Instance);
AwaitAssert(() => Assert.NotEmpty(repo.ThresholdQueries),
duration: TimeSpan.FromSeconds(5), interval: TimeSpan.FromMilliseconds(50));
archive.DidNotReceiveWithAnyArgs().CleanupAsync(default, default);
StopPurgeActor(actor);
}
} }
@@ -30,13 +30,13 @@ public class AuditArchiveCommandTests
"\"rowsRead\":11,\"chains\":[{\"chain\":{\"partition\":\"2025-10\",\"status\":\"Intact\",\"rowsVerified\":10," + "\"rowsRead\":11,\"chains\":[{\"chain\":{\"partition\":\"2025-10\",\"status\":\"Intact\",\"rowsVerified\":10," +
"\"tombstonesApplied\":0,\"unchainedRows\":0},\"anchor\":\"Mismatch\",\"anchorDetail\":\"hash differs\"}]}"; "\"tombstonesApplied\":0,\"unchainedRows\":0},\"anchor\":\"Mismatch\",\"anchorDetail\":\"hash differs\"}]}";
/// <summary>The audit group carries the archive group with list + verify.</summary> /// <summary>The audit group carries the archive group with list + verify + cleanup (ScadaBridge#109).</summary>
[Fact] [Fact]
public void AuditGroup_HasArchiveListAndVerify() public void AuditGroup_HasArchiveListVerifyAndCleanup()
{ {
var root = AuditCommandTestHarness.BuildRoot(); var root = AuditCommandTestHarness.BuildRoot();
var archive = root.Subcommands.Single(c => c.Name == "audit").Subcommands.Single(c => c.Name == "archive"); var archive = root.Subcommands.Single(c => c.Name == "audit").Subcommands.Single(c => c.Name == "archive");
Assert.Equal(["list", "verify"], archive.Subcommands.Select(c => c.Name).Order()); Assert.Equal(["cleanup", "list", "verify"], archive.Subcommands.Select(c => c.Name).Order());
} }
/// <summary><c>list</c> sends ListAuditArchives and prints the archives.</summary> /// <summary><c>list</c> sends ListAuditArchives and prints the archives.</summary>
@@ -78,6 +78,34 @@ public class AuditArchiveCommandTests
Assert.Equal(Name, name); Assert.Equal(Name, name);
} }
/// <summary>
/// <c>cleanup</c> (ScadaBridge#109) sends CleanupAuditArchives with the dry-run flag and
/// reports each action; it always exits 0.
/// </summary>
/// <param name="dryRun">Whether <c>--dry-run</c> is passed.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
[Theory]
[InlineData(true)]
[InlineData(false)]
public async Task Cleanup_SendsCleanupAuditArchives_WithTheDryRunFlag(bool dryRun)
{
var body = "{\"dryRun\":" + (dryRun ? "true" : "false") + ",\"retentionDays\":0,\"items\":[{\"name\":\"" + Name
+ "\",\"partition\":\"2025-10\",\"action\":\"DeletedSuperseded\",\"detail\":\"superseded by x\"}]}";
string[] args = dryRun
? ["audit", "archive", "cleanup", "--dry-run", "--format", "table"]
: ["audit", "archive", "cleanup", "--format", "table"];
var (exit, output, request) = await RunAsync(body, args);
Assert.Equal(0, exit);
Assert.Contains("DeletedSuperseded", output);
Assert.Contains("never delete", output);
Assert.Equal(dryRun, output.Contains("Dry run", StringComparison.Ordinal));
using var payload = JsonDocument.Parse(request);
Assert.Equal("CleanupAuditArchives", payload.RootElement.GetProperty("command").GetString());
Assert.Equal(dryRun, payload.RootElement.GetProperty("payload").EnumerateObject()
.First(p => string.Equals(p.Name, "dryRun", StringComparison.OrdinalIgnoreCase)).Value.GetBoolean());
}
/// <summary><c>verify</c> without <c>--name</c> is a usage error.</summary> /// <summary><c>verify</c> without <c>--name</c> is a usage error.</summary>
[Fact] [Fact]
public void Verify_MissingName_ProducesRequiredFlagError() public void Verify_MissingName_ProducesRequiredFlagError()
@@ -62,10 +62,10 @@ public class AuditCommandsScaffoldTests
[Fact] [Fact]
public void Audit_EveryLeafCommand_HasAnAction() public void Audit_EveryLeafCommand_HasAnAction()
{ {
// `archive` is a group (list / verify, ScadaBridge#42): check its leaves, not it. // `archive` is a group (list / verify, ScadaBridge#42; cleanup, ScadaBridge#109): check its leaves, not it.
var audit = BuildAudit(); var audit = BuildAudit();
var leaves = audit.Subcommands.SelectMany(c => c.Subcommands.Count == 0 ? [c] : c.Subcommands).ToList(); var leaves = audit.Subcommands.SelectMany(c => c.Subcommands.Count == 0 ? [c] : c.Subcommands).ToList();
Assert.Equal(8, leaves.Count); // + verify-heads (ScadaBridge#107) Assert.Equal(9, leaves.Count); // + verify-heads (ScadaBridge#107), archive cleanup (ScadaBridge#109)
Assert.All(leaves, sub => Assert.All(leaves, sub =>
Assert.True(sub.Action != null, $"Leaf command '{sub.Name}' has no action.")); Assert.True(sub.Action != null, $"Leaf command '{sub.Name}' has no action."));
} }
@@ -45,7 +45,7 @@ public class ManagementSurfaceParityTests
} }
private static readonly HashSet<string> AuditManagementLeaves = private static readonly HashSet<string> AuditManagementLeaves =
["audit verify-chain", "audit verify-heads", "audit archive list", "audit archive verify"]; ["audit verify-chain", "audit verify-heads", "audit archive list", "audit archive verify", "audit archive cleanup"];
private static async Task<ConcurrentDictionary<string, string>> ProbeProductionTree(string? omitGroup = null) private static async Task<ConcurrentDictionary<string, string>> ProbeProductionTree(string? omitGroup = null)
{ {
@@ -0,0 +1,296 @@
using System.Text.Json;
using Microsoft.EntityFrameworkCore;
using ZB.MOM.WW.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance;
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories;
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests.Migrations;
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests.Maintenance;
/// <summary>
/// SQL Server tests for AuditLog archive cleanup (ScadaBridge#109): a stale archive a retried
/// purge left behind is deleted only once a later purged archive provably holds all its rows;
/// a stale archive with a row no later archive holds, or a sole copy left at <c>Archived</c>, is
/// kept; retention deletes only expired archives whose partition is gone; deletion never
/// follows a path out of the archive folder.
/// </summary>
/// <remarks>Each test owns one month of the shared fixture database and its own archive directory.</remarks>
public sealed class AuditLogArchiveCleanupTests : IClassFixture<MsSqlMigrationFixture>, IDisposable
{
private readonly MsSqlMigrationFixture _fixture;
private readonly DirectoryInfo _root = Directory.CreateTempSubdirectory("auditlog-cleanup-");
/// <summary>Creates the test class with the shared migrated SQL Server fixture.</summary>
/// <param name="fixture">The shared migration fixture.</param>
public AuditLogArchiveCleanupTests(MsSqlMigrationFixture fixture) => _fixture = fixture;
/// <inheritdoc />
public void Dispose() => _root.Delete(recursive: true);
/// <summary>
/// The retried purge: archive A goes stale (a row arrives and the chain advances), B purges.
/// Dry run reports A as superseded and deletes nothing; the real pass deletes exactly A's
/// three files and B still verifies.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[SkippableFact]
public async Task RetriedPurge_StaleArchiveIsDeleted_OnceALaterPurgedArchiveCoversIt()
{
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
var dir = Dir("retried");
var month = Month(2027, 1);
await InsertAsync(Enumerable.Range(1, 5).Select(i => NewEvent(month.AddDays(i))));
await DrainAsync();
await InsertAsync([NewEvent(month.AddDays(7))]); // unchained in A
var stale = await ArchiveAsync(dir, month);
Assert.Equal(1, stale.UnchainedRowCount);
await InsertAsync([NewEvent(month.AddDays(9))]);
await DrainAsync(); // A's unchained row is now chained; the chain moved past A's end
await using (var ctx = CreateContext())
{
await Assert.ThrowsAsync<AuditArchiveStaleException>(() => Archive(ctx, dir).PurgeArchivedPartitionAsync(stale));
}
var fresh = await ArchiveAsync(dir, month);
await using (var ctx = CreateContext())
{
Assert.Equal(7, await Archive(ctx, dir).PurgeArchivedPartitionAsync(fresh));
}
var dry = await CleanupAsync(dir, dryRun: true);
var item = Assert.Single(dry.Items);
Assert.Equal((stale.Name, AuditArchiveCleanupAction.DeletedSuperseded), (item.Name, item.Action));
Assert.True(File.Exists(ManifestPath(dir, stale)));
var real = await CleanupAsync(dir, dryRun: false);
Assert.Equal(AuditArchiveCleanupAction.DeletedSuperseded, Assert.Single(real.Items).Action);
Assert.False(File.Exists(ManifestPath(dir, stale)));
Assert.False(File.Exists(Path.Combine(dir, "2027", stale.Name + ".parquet")));
Assert.False(File.Exists(Path.Combine(dir, "2027", stale.Name + ".tombstones.parquet")));
await using (var ctx = CreateContext())
{
var listed = Assert.Single(await Archive(ctx, dir).ListAsync());
Assert.Equal(fresh.Name, listed.Name);
Assert.True((await Archive(ctx, dir).VerifyAsync(fresh.Name)).Verified);
}
Assert.Empty((await CleanupAsync(dir, dryRun: false)).Items);
}
/// <summary>
/// A stale archive holding an unchained row that a per-channel purge removed before the
/// later archive (no tombstone for an unchained row) is the only copy of that row: kept.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[SkippableFact]
public async Task StaleArchive_WithARowNoLaterArchiveHolds_IsKept()
{
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
var dir = Dir("kept");
var month = Month(2027, 3);
await InsertAsync(Enumerable.Range(1, 3).Select(i => NewEvent(month.AddDays(i))));
await DrainAsync();
await InsertAsync([NewEvent(month.AddHours(1), AuditChannel.DbOutbound)]); // unchained
var stale = await ArchiveAsync(dir, month);
await using (var ctx = CreateContext())
{
Assert.Equal(1, await new AuditLogRepository(ctx).PurgeChannelOlderThanAsync("DbOutbound", month.AddDays(1), 100));
await Assert.ThrowsAsync<AuditArchiveStaleException>(() => Archive(ctx, dir).PurgeArchivedPartitionAsync(stale));
}
var fresh = await ArchiveAsync(dir, month);
await using (var ctx = CreateContext())
{
await Archive(ctx, dir).PurgeArchivedPartitionAsync(fresh);
}
var result = await CleanupAsync(dir, dryRun: false);
var kept = Assert.Single(result.Items);
Assert.Equal((stale.Name, AuditArchiveCleanupAction.KeptStale), (kept.Name, kept.Action));
Assert.Contains("unchained row", kept.Detail);
Assert.True(File.Exists(ManifestPath(dir, stale)));
}
/// <summary>
/// An archive left at <c>Archived</c> although its partition WAS switched out (the node
/// stopped before the manifest update) has no later purged archive: it is the sole copy, kept.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[SkippableFact]
public async Task SoleCopyLeftAtArchived_IsNotTouched()
{
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
var dir = Dir("sole");
var month = Month(2027, 4);
await InsertAsync(Enumerable.Range(1, 3).Select(i => NewEvent(month.AddDays(i))));
await DrainAsync();
var manifest = await ArchiveAsync(dir, month);
await using (var ctx = CreateContext())
{
await Archive(ctx, dir).PurgeArchivedPartitionAsync(manifest);
}
await SetStatusAsync(dir, manifest, AuditArchiveStatus.Archived);
Assert.Empty((await CleanupAsync(dir, dryRun: false)).Items);
Assert.True(File.Exists(ManifestPath(dir, manifest)));
}
/// <summary>
/// Retention: 0 never deletes; a purged archive is kept until its month ended more than
/// RetentionDays ago, then deleted; an expired archive whose partition still holds rows is kept.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[SkippableFact]
public async Task Retention_DeletesOnlyExpiredArchivesWhosePartitionIsGone()
{
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
var dir = Dir("retention");
var purgedMonth = Month(2027, 5);
await InsertAsync(Enumerable.Range(1, 2).Select(i => NewEvent(purgedMonth.AddDays(i))));
var purged = await ArchiveAsync(dir, purgedMonth);
await using (var ctx = CreateContext())
{
await Archive(ctx, dir).PurgeArchivedPartitionAsync(purged);
}
var liveMonth = Month(2027, 6);
await InsertAsync([NewEvent(liveMonth.AddDays(1))]);
var live = await ArchiveAsync(dir, liveMonth); // never purged: rows still in the table
var monthEnd = purgedMonth.AddMonths(1);
var farFuture = liveMonth.AddMonths(1).AddDays(1000);
Assert.Empty((await CleanupAsync(dir, dryRun: false, retentionDays: 0, now: farFuture)).Items);
Assert.Empty((await CleanupAsync(dir, dryRun: false, retentionDays: 400, now: monthEnd.AddDays(399))).Items);
var result = await CleanupAsync(dir, dryRun: false, retentionDays: 400, now: farFuture);
var expired = Assert.Single(result.Items);
Assert.Equal((purged.Name, AuditArchiveCleanupAction.DeletedExpired), (expired.Name, expired.Action));
Assert.False(File.Exists(ManifestPath(dir, purged)));
Assert.True(File.Exists(ManifestPath(dir, live)));
}
/// <summary>Deletion uses the archive name's own three files — never a path from the manifest.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[SkippableFact]
public async Task Deletion_NeverFollowsAManifestPath_OutOfTheArchiveFolder()
{
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
var dir = Dir("paths");
var month = Month(2027, 7);
await InsertAsync([NewEvent(month.AddDays(1))]);
var manifest = await ArchiveAsync(dir, month);
await using (var ctx = CreateContext())
{
await Archive(ctx, dir).PurgeArchivedPartitionAsync(manifest);
}
var outside = Path.Combine(_root.FullName, "outside.parquet");
await File.WriteAllTextAsync(outside, "not an archive");
var path = ManifestPath(dir, manifest);
var tampered = JsonSerializer.Deserialize<AuditArchiveManifest>(await File.ReadAllTextAsync(path), Json)! with
{
DataFile = "../../outside.parquet",
TombstoneFile = "../../outside.parquet",
};
await File.WriteAllTextAsync(path, JsonSerializer.Serialize(tampered, Json));
var result = await CleanupAsync(dir, dryRun: false, retentionDays: 30, now: month.AddYears(5));
Assert.Equal(AuditArchiveCleanupAction.DeletedExpired, Assert.Single(result.Items).Action);
Assert.True(File.Exists(outside));
Assert.Empty(Directory.EnumerateFiles(Path.Combine(dir, "2027")));
}
private static readonly JsonSerializerOptions Json = new() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase };
private sealed class FixedClock(DateTime now) : TimeProvider
{
public override DateTimeOffset GetUtcNow() => new(DateTime.SpecifyKind(now, DateTimeKind.Utc));
}
private string Dir(string name) => Directory.CreateDirectory(Path.Combine(_root.FullName, name)).FullName;
private static string ManifestPath(string dir, AuditArchiveManifest m) => Path.Combine(dir, m.Partition[..4], m.Name + ".manifest.json");
private static async Task SetStatusAsync(string dir, AuditArchiveManifest manifest, AuditArchiveStatus status)
{
var path = ManifestPath(dir, manifest);
var current = JsonSerializer.Deserialize<AuditArchiveManifest>(await File.ReadAllTextAsync(path), Json)!;
await File.WriteAllTextAsync(path, JsonSerializer.Serialize(current with { Status = status, PurgedAtUtc = null }, Json));
}
private static AuditLogPartitionArchive Archive(
ScadaBridgeDbContext ctx, string dir, int retentionDays = 0, DateTime? now = null) => new(ctx, new AuditLogArchiveOptions
{
Enabled = true,
Directory = dir,
RowGroupSize = 1_000,
RetentionDays = retentionDays,
}, time: now is null ? null : new FixedClock(now.Value));
private async Task<AuditArchiveManifest> ArchiveAsync(string dir, DateTime month)
{
await using var ctx = CreateContext();
var manifest = await Archive(ctx, dir).ArchivePartitionAsync(month);
await Task.Delay(5); // distinct millisecond archive names
return manifest;
}
private async Task<AuditArchiveCleanupResult> CleanupAsync(string dir, bool dryRun, int retentionDays = 0, DateTime? now = null)
{
await using var ctx = CreateContext();
return await Archive(ctx, dir, retentionDays, now).CleanupAsync(dryRun);
}
private async Task InsertAsync(IEnumerable<AuditEvent> events)
{
await using var ctx = CreateContext();
var repo = new AuditLogRepository(ctx);
foreach (var e in events)
{
await repo.InsertIfNotExistsAsync(e);
}
}
private async Task DrainAsync()
{
await using var ctx = CreateContext();
var chain = new AuditLogHashChain(ctx);
for (var i = 0; i < 1000; i++)
{
var r = await chain.ChainPendingAsync(500);
if (r.RowsChained == 0 && !r.LockBusy)
{
return;
}
}
throw new InvalidOperationException("Chain did not drain.");
}
private static DateTime Month(int year, int month) => new(year, month, 1, 0, 0, 0, DateTimeKind.Utc);
private ScadaBridgeDbContext CreateContext()
=> new(new DbContextOptionsBuilder<ScadaBridgeDbContext>().UseSqlServer(_fixture.ConnectionString).Options);
private static AuditEvent NewEvent(DateTime occurredAtUtc, AuditChannel channel = AuditChannel.ApiOutbound) =>
ScadaBridgeAuditEventFactory.Create(
channel: channel,
kind: channel == AuditChannel.DbOutbound ? AuditKind.DbWrite : AuditKind.ApiCall,
status: AuditStatus.Delivered,
occurredAtUtc: occurredAtUtc,
actor: "cleanup-test",
target: "erp",
sourceNode: "central-a",
sourceSiteId: "cleanup-test",
correlationId: Guid.NewGuid());
}
@@ -144,4 +144,43 @@ public class AuditArchiveCommandTests : TestKit
Assert.Equal(TimeSpan.FromMinutes(5), Assert.Equal(TimeSpan.FromMinutes(5),
ManagementEndpoints.ResolveAskTimeout(new ManagementServiceOptions(), typeof(VerifyAuditArchiveCommand))); ManagementEndpoints.ResolveAskTimeout(new ManagementServiceOptions(), typeof(VerifyAuditArchiveCommand)));
} }
/// <summary>ScadaBridge#109: an Administrator runs the cleanup; the dry-run flag reaches the archive.</summary>
/// <param name="dryRun">The flag.</param>
[Theory]
[InlineData(true)]
[InlineData(false)]
public void Administrator_RunsCleanup_WithTheDryRunFlag(bool dryRun)
{
_archive.CleanupAsync(dryRun, Arg.Any<CancellationToken>()).Returns(new AuditArchiveCleanupResult(dryRun, 0,
[new AuditArchiveCleanupItem(Name, "2025-10", AuditArchiveCleanupAction.DeletedSuperseded, "superseded")]));
CreateActor().Tell(Envelope(new CleanupAuditArchivesCommand(dryRun), [], "Administrator"));
var response = ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
using var json = JsonDocument.Parse(response.JsonData);
Assert.Equal(dryRun, json.RootElement.GetProperty("dryRun").GetBoolean());
Assert.Equal("DeletedSuperseded", json.RootElement.GetProperty("items")[0].GetProperty("action").GetString());
_archive.Received(1).CleanupAsync(dryRun, Arg.Any<CancellationToken>());
}
/// <summary>Cleanup deletes files: a Viewer (audit reader) is refused.</summary>
[Fact]
public void Viewer_CannotRunCleanup()
{
CreateActor().Tell(Envelope(new CleanupAuditArchivesCommand(true), [], "Viewer"));
ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
_archive.DidNotReceiveWithAnyArgs().CleanupAsync(default, default);
}
/// <summary>No archive directory configured: ARCHIVE_NOT_CONFIGURED, as for list/verify.</summary>
[Fact]
public void Cleanup_WithoutAnArchiveDirectory_IsNotConfigured()
{
_archive.IsConfigured.Returns(false);
CreateActor().Tell(Envelope(new CleanupAuditArchivesCommand(true), [], "Administrator"));
Assert.Equal("ARCHIVE_NOT_CONFIGURED", ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5)).ErrorCode);
}
} }
@@ -176,6 +176,8 @@ public class RequiredRoleMatrixTests
// AuditLog Parquet archives (ScadaBridge#42): same audit-read gate as verify-chain. // AuditLog Parquet archives (ScadaBridge#42): same audit-read gate as verify-chain.
["ListAuditArchives"] = [Roles.Administrator, Roles.Viewer], ["ListAuditArchives"] = [Roles.Administrator, Roles.Viewer],
["VerifyAuditArchive"] = [Roles.Administrator, Roles.Viewer], ["VerifyAuditArchive"] = [Roles.Administrator, Roles.Viewer],
// Archive cleanup deletes files (ScadaBridge#109): Administrator only.
["CleanupAuditArchives"] = [Roles.Administrator],
// ---- Two-person secured write ---------------------------------------------- // ---- Two-person secured write ----------------------------------------------
["SubmitSecuredWrite"] = [Roles.Operator], ["SubmitSecuredWrite"] = [Roles.Operator],