feat(audit): archive retention and superseded stale-archive cleanup (ScadaBridge#109)
Owner decisions recorded: the archive destination stays a directory/UNC (no object-storage client; share + permissions named at enablement), parquet-tools-only reading (no re-import), unsigned like the chain (#107). AuditLog:Archive:RetentionDays (default 0 = never delete; else must exceed AuditLog:RetentionDays, counted from the archived month's end) and DeleteSupersededStaleArchives (default true) run after every purge tick. A retried purge's stale Archived archive is deleted only when a later, verifying, Purged archive of the same partition provably holds every row: same prior anchor and the stale chain-end hash at its chain-end sequence (row or tombstone), every unchained row present by EventId; otherwise it is reported KeptStale. A sole copy left at Archived is never touched. Deletion removes only the archive name's own three files in its folder, manifest first, never a manifest-supplied path. `audit archive cleanup [--dry-run]` = CleanupAuditArchives (Administrator). AuditLogArchiveCleanupTests (MSSQL) cover retried purge, kept stale copy, sole copy, retention and the path guard; each safety rule mutation-checked. Follow-ups filed: #113, #114, #115. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
@@ -183,7 +183,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- 365-day central retention with monthly partition-switch purge; per-channel retention overrides (`AuditLog:PerChannelRetentionDays`) expire rows earlier than the global window via a bounded, batched row DELETE on the purge actor's maintenance path — values must be shorter than the global window (M5.5 T3); 7-day site SQLite retention with a hard `ForwardState` invariant (no row purged until forwarded or reconciled).
|
||||
- Append-only enforced via DB roles (writer role has INSERT only, no UPDATE/DELETE).
|
||||
- **Clustered key `(OccurredAtUtc, EventId)` + aligned unique `UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc)` (migration `FlipAuditLogClusteredKey`, #111, B2 of the #78 benchmark).** The Audit Log page's newest-first page/keyset now reads an ordered range instead of the whole table (300k rows: 90,532 → 34 reads). Offline whole-table rebuild — script `docs/plans/sql/FlipAuditLogClusteredKey.sql` (+ `.rollback.sql`), both centrals stopped: the partition-switch staging table now reads the live key order, but binaries older than the flip hard-code the old one. DEV/QA not applied (sizing + window in #111).
|
||||
- **Parquet cold archive (T2, ScadaBridge#42) — default OFF (`AuditLog:Archive:Enabled` + absolute `Directory`, shared by both centrals).** The purge actor writes each expired partition to `{Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet` (10 stored columns + `ChainSeq`/`RowHash`, tick-exact `OccurredAtUtcTicks`) plus `.tombstones.parquet` and a `.manifest.json` (written last; chain head at archive time, SHA-256 per file), read under the chain lock SHARED; then the switch runs with a guard (under the EXCLUSIVE lock: partition row count + each month's `LastSeq` must still equal the archive's, else THROW 51002/51003 and nothing is purged). Fail closed: a partition that cannot be archived is not purged. `audit archive list|verify` = `ListAuditArchives`/`VerifyAuditArchive` (verify-chain's gate): file hashes, chain recomputed from the archive, chain end vs the DB purge anchor (`Matches`/`Superseded`/`NotPurged`/`Mismatch`), CLI exit 3. `audit export --format parquet` streams the 24 export columns (was 501). No migration; no re-import path (open decision), object storage not wired (mount it).
|
||||
- **Parquet cold archive (T2, ScadaBridge#42) — default OFF (`AuditLog:Archive:Enabled` + absolute `Directory`, shared by both centrals).** The purge actor writes each expired partition to `{Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet` (10 stored columns + `ChainSeq`/`RowHash`, tick-exact `OccurredAtUtcTicks`) plus `.tombstones.parquet` and a `.manifest.json` (written last; chain head at archive time, SHA-256 per file), read under the chain lock SHARED; then the switch runs with a guard (under the EXCLUSIVE lock: partition row count + each month's `LastSeq` must still equal the archive's, else THROW 51002/51003 and nothing is purged). Fail closed: a partition that cannot be archived is not purged. `audit archive list|verify` = `ListAuditArchives`/`VerifyAuditArchive` (verify-chain's gate): file hashes, chain recomputed from the archive, chain end vs the DB purge anchor (`Matches`/`Superseded`/`NotPurged`/`Mismatch`), CLI exit 3. `audit export --format parquet` streams the 24 export columns (was 501). No migration. **Owner decisions (#109):** destination stays a directory/UNC (no object-storage client; share + permissions named at enablement), parquet-tools-only reading (no re-import), unsigned like the chain. **Retention** `AuditLog:Archive:RetentionDays` (default 0 = never; else > `AuditLog:RetentionDays`, counted from the month's end) and **superseded stale-archive cleanup** (`DeleteSupersededStaleArchives`, default true: a retried purge's `Archived` archive is deleted only when a later `Purged`, verifying archive provably holds every row — same anchor + chain-end hash, unchained rows by EventId; else `KeptStale`) run after every purge tick; deletion touches only the archive name's own three files, manifest first; `audit archive cleanup [--dry-run]` = `CleanupAuditArchives` (Administrator).
|
||||
- **Hash-chain tamper evidence (T1, ScadaBridge#41) — default OFF (`AuditLog:HashChain:Enabled`).** One SHA-256 chain per UTC calendar month: `ChainSeq` + `RowHash = SHA-256(prev || canonical(row))` on `AuditLog` (migration `AddAuditLogHashChain`), linked by the central `AuditLogHashChainService` (both nodes, serialised by `sp_getapplock`) — never by the ingest path. `AuditLogChainHead` catches tail deletion and carries the partition-switch checkpoint anchor; the per-channel DELETE writes an `AuditLogChainTombstone` per chained row in the same transaction. `audit verify-chain --month` = management command `VerifyAuditChain` (OperationalAudit, unscoped callers only), first break with partition/ChainSeq/EventId, CLI exit 3. A chained row's `SourceNode` is frozen (backfill skips it). **Unsigned by owner decision (#107); external anchor = off-box head export** (`AuditLog:HashChain:HeadExport`, default off: both centrals append changed per-month heads — month, `LastSeq`/`LastHash`, purge anchor, row + tombstone counts — to append-only JSONL files `{Directory}/{yyyy}/audit-chain-heads-{date}-{node}.jsonl` or RFC 5424 syslog UDP/TCP); `audit verify-heads --path` = `VerifyAuditChainHeads` checks them against the DB (a re-hashed forgery passes verify-chain, fails verify-heads; exit 3). **Two-login hardening (#107 item 3):** `AuditLog:MaintenanceConnectionString` (`${secret:}`-capable, unset = runtime login) runs the chain job, verify/head export, partition switch-out + archive and per-channel purge on `AuditMaintenanceContext` (keyed `audit-maintenance` repository/archive); that login gets `scadabridge_audit_chainer` + `scadabridge_audit_purger`, the runtime login `scadabridge_audit_writer` (whose DENYs block chaining/purging) — pinned by `AuditMaintenanceLoginTests`.
|
||||
- Node-of-origin is captured alongside site-of-origin: `SourceNode` (`varchar(64)` NULL) on `AuditLog`, `Notifications`, and `SiteCalls` — `node-a`/`node-b` for site rows (qualified by `SourceSiteId`/`SourceSite`), `central-a`/`central-b` for central direct-write rows. Stamped at the writing node, carried verbatim through telemetry + reconciliation, and indexed via `IX_AuditLog_Node_Occurred (SourceNode, OccurredAtUtc)` on `AuditLog`.
|
||||
- Per-node stuck KPIs (M5.3 T6): Notification Outbox and Site Call Audit expose `PerNodeNotificationKpiRequest`/`PerNodeSiteCallKpiRequest` messages that group stuck/parked/delivered counts by `SourceNode`, surfacing per-node breakdowns on the Health dashboard.
|
||||
|
||||
@@ -24,8 +24,8 @@ Management commands use `POST /management`. Audit query, export, and tree use
|
||||
`GET /api/audit/query`, `/api/audit/export`, and `/api/audit/tree`; source-node
|
||||
backfill uses `POST /api/audit/backfill-source-node`. `audit verify-chain` sends the
|
||||
`VerifyAuditChain` management command (ScadaBridge#41); `audit verify-heads` reads off-box
|
||||
exported chain heads and sends `VerifyAuditChainHeads` (ScadaBridge#107); `audit archive list|verify` send
|
||||
`ListAuditArchives` / `VerifyAuditArchive` (ScadaBridge#42). `debug stream` uses a persistent
|
||||
exported chain heads and sends `VerifyAuditChainHeads` (ScadaBridge#107); `audit archive list|verify|cleanup` send
|
||||
`ListAuditArchives` / `VerifyAuditArchive` (ScadaBridge#42) / `CleanupAuditArchives` (ScadaBridge#109). `debug stream` uses a persistent
|
||||
SignalR connection. Audit export streams the response without buffering it all.
|
||||
|
||||
### Authentication and roles
|
||||
@@ -122,7 +122,7 @@ scadabridge instance deploy --id 7
|
||||
| `db-connection` | `list`, `get`, `create`, `update`, `delete` | `Designer` |
|
||||
| `api-method` | `list`, `get`, `create`, `update`, `delete`, `validate`, `recompile` | `Designer` |
|
||||
| `bundle` | `export`, `preview`, `import` | Designer export; Administrator preview/import |
|
||||
| `audit` | `query`, `export`, `tree`, `backfill-source-node`, `verify-chain`, `verify-heads`, `archive list`, `archive verify` | Audit roles; Administrator backfill; verify-chain, verify-heads and archive need an unscoped audit role |
|
||||
| `audit` | `query`, `export`, `tree`, `backfill-source-node`, `verify-chain`, `verify-heads`, `archive list`, `archive verify`, `archive cleanup` | Audit roles; Administrator backfill and archive cleanup; verify-chain, verify-heads and archive need an unscoped audit role |
|
||||
| `audit-config` | `query` (config-change audit trail; was `audit-log` pre-M8) | `Admin` |
|
||||
| `health` | `summary`, `pending-updates`, `site`, `event-log`, `parked-messages` | Authenticated reads with server-side scope checks |
|
||||
| `debug` | `snapshot`, `stream` | `Deployer` |
|
||||
|
||||
@@ -770,8 +770,12 @@ MS SQL for direct-write events). Unredacted secrets never persist.
|
||||
- **Where.** `AuditLog:Archive:Directory` — an absolute path (validated at startup)
|
||||
that both central nodes reach, because the purge runs on the active node and
|
||||
`audit archive list|verify` on whichever node serves the call. Files land in
|
||||
`{Directory}/{YYYY}/`. Object storage (S3 / Azure Blob) is not called directly —
|
||||
mount it as a file system or copy the files on.
|
||||
`{Directory}/{YYYY}/`. **Owner decision (ScadaBridge#109): the destination stays a
|
||||
directory / UNC share — no native object-storage client.** Object storage (S3 / Azure
|
||||
Blob) is reached, if ever, by mounting it as a file system or copying the files on. The
|
||||
share (UNC path) and its permissions are chosen when the archive is enabled on DEV/QA:
|
||||
both central nodes' service account needs modify rights on it (write, rename, and — for
|
||||
cleanup — delete), auditors read-only; the SQL Server login needs none.
|
||||
- **What.** Per partition, three files sharing the stem
|
||||
`auditlog-{YYYY-MM}-{archivedAt:yyyyMMddTHHmmssfff}Z`: `.parquet` (every row of the
|
||||
partition — the ten stored columns, `OccurredAtUtc` both as a UTC microsecond
|
||||
@@ -798,8 +802,33 @@ MS SQL for direct-write events). Unredacted secrets never persist.
|
||||
- **Retention interplay.** Archiving changes nothing about *when* a partition is
|
||||
purged (`RetentionDays`). Per-channel overrides still delete their rows earlier,
|
||||
so those rows are never archived — the archive carries their tombstones instead,
|
||||
which is what keeps the archived chain verifiable. Archive files have no retention
|
||||
of their own.
|
||||
which is what keeps the archived chain verifiable.
|
||||
- **Archive retention (ScadaBridge#109) — default never delete.**
|
||||
`AuditLog:Archive:RetentionDays` (default `0` = archives are never deleted, until compliance
|
||||
names a period) is counted from the END of the archived month and, when set, must exceed
|
||||
`AuditLog:RetentionDays` (validated at startup) — an archive always outlives the rows' time
|
||||
in the database. After each purge tick the purge actor deletes an expired archive that is
|
||||
`Purged`, or an `Archived` one whose partition no longer holds rows.
|
||||
- **Superseded stale archives (ScadaBridge#109).** A purge whose guard finds the partition
|
||||
changed (51002/51003) leaves its archive at `Archived` (verify reports `NotPurged`, or
|
||||
`Superseded` once a later purge anchored past it) and the next tick archives again. With
|
||||
`AuditLog:Archive:DeleteSupersededStaleArchives` (default `true`) the purge actor deletes
|
||||
such a stale archive only when a LATER archive of the same partition is `Purged`, verifies,
|
||||
and provably holds every row of it: for every chain month, the same earlier anchor and the
|
||||
stale archive's chain-end hash at its chain-end sequence (row or tombstone), and every
|
||||
unchained row present by EventId. Otherwise it is kept and reported `KeptStale` (e.g. an
|
||||
unchained row a per-channel purge removed in between — no tombstone — so the stale archive
|
||||
is that row's only copy). An archive left at `Archived` because the node stopped between the
|
||||
switch and the manifest update has no later purged archive and is never touched.
|
||||
- **Safe deletion.** Cleanup deletes only the three files a well-formed archive name implies
|
||||
(`{name}.manifest.json`, `.parquet`, `.tombstones.parquet`) in that archive's own
|
||||
`{Directory}/{YYYY}` folder — never a path read from a manifest — manifest first, so an
|
||||
interrupted deletion leaves leftovers that are no longer an archive (orphans of an interrupted
|
||||
archive WRITE are not cleaned yet — #114). A cleanup failure is a
|
||||
purge failure on the health surface. `scadabridge audit archive cleanup [--dry-run]`
|
||||
(`CleanupAuditArchives`, Administrator) runs the same pass on demand or reports what it
|
||||
would delete. Pinned by `AuditLogArchiveCleanupTests` (mutation-checked: each safety rule
|
||||
removed fails its test).
|
||||
- **Verify / read.** `audit archive verify --name` (`VerifyAuditArchive`) checks both
|
||||
files' SHA-256 and counts against the manifest, recomputes each archived month's
|
||||
chain from the archived rows and tombstones (`AuditChainVerifier`, starting from
|
||||
@@ -807,8 +836,9 @@ MS SQL for direct-write events). Unredacted secrets never persist.
|
||||
`Matches`; `Superseded` (late rows for the month were chained, purged and archived
|
||||
again afterwards — that later archive resumes from this one's end); `NotPurged`;
|
||||
or `Mismatch` (archive or head altered). The files are plain Parquet — DuckDB,
|
||||
pyarrow or Spark read them directly. There is no re-import into `dbo.AuditLog`:
|
||||
a re-inserted expired row would be purged again by the next tick.
|
||||
pyarrow, Spark or parquet-tools read them directly. **Owner decision (ScadaBridge#109):
|
||||
reading with Parquet tools is the supported access path; there is no re-import into
|
||||
`dbo.AuditLog`** — a re-inserted expired row would be purged again by the next tick.
|
||||
- **Sites:** `SiteAuditRetentionService` (site-only `IHostedService`) runs the
|
||||
site SQLite retention purge on a timer — first tick after
|
||||
`AuditLog:SiteRetention:InitialDelay` (default 5 min, short so a daily-recycled
|
||||
@@ -880,7 +910,7 @@ MS SQL for direct-write events). Unredacted secrets never persist.
|
||||
to the maintenance login: the monthly partition roll-forward (`ALTER PARTITION FUNCTION`,
|
||||
needs `ALTER ANY DATASPACE` on the runtime login, or run it as a DBA job) and the one-time
|
||||
`audit backfill-source-node` (needs `UPDATE` on `SourceNode` — a DBA-run step in a hardened
|
||||
setup). Migrations themselves are applied by a DBA with the production SQL scripts.
|
||||
setup) — both tracked in #113. Migrations themselves are applied by a DBA with the production SQL scripts.
|
||||
- **Authorization.** Reading the Audit Log requires the existing **Audit** role
|
||||
extended with a new **OperationalAudit** permission. Per-site row scoping
|
||||
reuses the existing site-permission model; bulk export requires an additional
|
||||
@@ -943,7 +973,7 @@ MS SQL for direct-write events). Unredacted secrets never persist.
|
||||
`FileMode.Append` only, one file per node per day; give the service account create +
|
||||
append-data rights and nothing else), or `Sink: Syslog` — RFC 5424 (`<PRI>1 … chainhead -
|
||||
{json}`, facility 13 "log audit", severity Notice) over UDP (RFC 5426) or TCP (RFC 6587
|
||||
octet counting). A failed write is logged and retried with the same heads next tick.
|
||||
octet counting). A failed write is logged and retried with the same heads next tick (no health counter yet — #115).
|
||||
`scadabridge audit verify-heads --path <file|dir>` reads those records back (file-sink
|
||||
lines or a syslog collector's log) and the `VerifyAuditChainHeads` command checks each
|
||||
against the database: the link stored at the exported sequence (the head, a live row's
|
||||
@@ -961,9 +991,10 @@ MS SQL for direct-write events). Unredacted secrets never persist.
|
||||
- **Parquet archival (T2) — shipped (ScadaBridge#42), default OFF.** See Retention &
|
||||
Purge. The archive carries the hash-chain columns and the chain head, so tamper
|
||||
evidence survives the purge: `audit archive verify` ties an archive to the checkpoint
|
||||
anchor still in `AuditLogChainHead`. Like the chain itself, archives are unsigned;
|
||||
a principal able to rewrite both the archive files and the head row can forge a
|
||||
consistent pair.
|
||||
anchor still in `AuditLogChainHead`. Like the chain itself, archives are unsigned by owner
|
||||
decision (ScadaBridge#107/#109); a principal able to rewrite both the archive files and the
|
||||
head row can forge a consistent pair — unless the month's head was exported off-box first
|
||||
(`AuditLog:HashChain:HeadExport`): `audit verify-heads` then exposes the rewritten anchor.
|
||||
- **Site SQLite security.** File permissions: read/write by the ScadaBridge
|
||||
service account only. Not backed up off-machine — site SQLite is a buffer,
|
||||
not a record.
|
||||
@@ -1062,7 +1093,9 @@ archive (see Retention & Purge):
|
||||
"Directory": "D:\\AuditArchive", // absolute; required when Enabled; reachable by both centrals
|
||||
"RowGroupSize": 50000, // 1,000 – 1,000,000
|
||||
"ReadCommandTimeoutMinutes": 60, // reading one partition out of SQL Server
|
||||
"ChainLockTimeoutSeconds": 300 // wait for a chaining pass before skipping the tick
|
||||
"ChainLockTimeoutSeconds": 300, // wait for a chaining pass before skipping the tick
|
||||
"RetentionDays": 0, // 0 = never delete; else > AuditLog:RetentionDays, from the archived month's end (#109)
|
||||
"DeleteSupersededStaleArchives": true // delete stale archives a later purged archive provably covers (#109)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
@@ -319,6 +319,7 @@ scadabridge audit verify-chain --month <YYYY-MM>
|
||||
scadabridge audit verify-heads --path <file-or-directory>
|
||||
scadabridge audit archive list
|
||||
scadabridge audit archive verify --name <archive>
|
||||
scadabridge audit archive cleanup [--dry-run]
|
||||
```
|
||||
|
||||
- `audit query` — filtered query against the central `AuditLog` table, matching the
|
||||
@@ -370,6 +371,12 @@ scadabridge audit archive verify --name <archive>
|
||||
archived hash chain and compares its end with the purge anchor; exit `0` verified,
|
||||
`3` not verified, `1` error, `2` authorization failure. Same gate as `verify-chain`.
|
||||
See Component-AuditLog.md, Retention & Purge.
|
||||
- `audit archive cleanup [--dry-run]` (ScadaBridge#109) — sends `CleanupAuditArchives`:
|
||||
deletes superseded stale archives (a retried purge's, once a later purged archive provably
|
||||
holds all their rows) and archives past `AuditLog:Archive:RetentionDays` (0 = never), the
|
||||
same pass the purge actor runs after every tick; `--dry-run` only reports. Prints each
|
||||
`DeletedSuperseded` / `DeletedExpired` / `KeptStale` with its reason; exit `0`. Administrator
|
||||
only (it deletes files).
|
||||
|
||||
### Health Commands
|
||||
```
|
||||
|
||||
@@ -202,6 +202,7 @@ The two-person authorization workflow for writes through the MxAccess Gateway. B
|
||||
- **VerifyAuditChain** (ScadaBridge#41): verifies one UTC calendar month (`Month`, `YYYY-MM`) of the central AuditLog hash chain and returns an `AuditChainVerificationResult` (status, rows verified, tombstones applied, purge anchor, head sequence, unchained rows, first break with partition / `ChainSeq` / `EventId` / kind). Gated on the OperationalAudit roles (`Administrator`, `Viewer`); a site-scoped non-Administrator caller is refused (`ManagementUnauthorized`) because a month's chain spans every site. Long-running Ask timeout. Errors: `INVALID_ARGUMENT` (bad month), `CHAIN_BUSY` (a chaining/purge pass held the chain lock past 60 s). Backs `scadabridge audit verify-chain`.
|
||||
- **VerifyAuditChainHeads** (ScadaBridge#107): checks chain heads exported off-box (`Records`, at most 50,000 `AuditChainHeadRecord`s; duplicates collapsed) against the database chain and returns an `AuditChainHeadVerificationResult` (overall `Verified`, one `AuditChainHeadCheck` per distinct exported head: `Matches` / `Extended` / `Purged`, or the failures `Mismatch` / `Regressed` / `MissingHead` / `MissingLink` / `CountMismatch`). Same gate as VerifyAuditChain; long-running Ask timeout. Errors: `INVALID_ARGUMENT` (no records / over the cap), `CHAIN_BUSY`. Backs `scadabridge audit verify-heads`.
|
||||
- **ListAuditArchives** / **VerifyAuditArchive** (ScadaBridge#42): list the AuditLog Parquet cold archives under `AuditLog:Archive:Directory` (`AuditArchiveListResult`: enabled flag + every manifest), and verify one by `Name` (`AuditArchiveVerificationResult`: file integrity, per-month chain recomputed from the archived rows, anchor `Matches`/`Superseded`/`NotPurged`/`Mismatch`, overall `Verified`). Same gate as VerifyAuditChain (OperationalAudit roles, site-scoped non-Administrator refused). VerifyAuditArchive uses the long-running Ask timeout. Errors: `ARCHIVE_NOT_CONFIGURED`, `INVALID_ARGUMENT` (not an archive name), `NOT_FOUND`. Back `scadabridge audit archive list|verify`.
|
||||
- **CleanupAuditArchives** (ScadaBridge#109): runs the archive cleanup now (`DryRun` = report only) — superseded stale archives a later purged archive provably covers, and archives past `AuditLog:Archive:RetentionDays` — and returns an `AuditArchiveCleanupResult` (`DryRun`, `RetentionDays`, one item per archive deleted / kept stale, with the action `DeletedExpired` / `DeletedSuperseded` / `KeptStale` and reason). Administrator only; runs on the maintenance-connection archive when registered; long-running Ask timeout. Errors: `ARCHIVE_NOT_CONFIGURED`. Backs `scadabridge audit archive cleanup`.
|
||||
|
||||
### Transport (Bundle Import / Export)
|
||||
|
||||
|
||||
@@ -32,5 +32,34 @@ public sealed class AuditLogArchiveOptionsValidator : OptionsValidatorBase<Audit
|
||||
$"AuditLog:Archive:{nameof(AuditLogArchiveOptions.ReadCommandTimeoutMinutes)} ({options.ReadCommandTimeoutMinutes}) must be > 0.");
|
||||
builder.RequireThat(options.ChainLockTimeoutSeconds > 0,
|
||||
$"AuditLog:Archive:{nameof(AuditLogArchiveOptions.ChainLockTimeoutSeconds)} ({options.ChainLockTimeoutSeconds}) must be > 0.");
|
||||
builder.RequireThat(options.RetentionDays is >= 0 and <= 36_500,
|
||||
$"AuditLog:Archive:{nameof(AuditLogArchiveOptions.RetentionDays)} ({options.RetentionDays}) must be between 0 (never delete) and 36500.");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Cross-section check for <see cref="AuditLogArchiveOptions.RetentionDays"/> (ScadaBridge#109):
|
||||
/// when set, it must exceed <c>AuditLog:RetentionDays</c>, so an archive is never deleted before
|
||||
/// (or as soon as) its rows leave the database — the archive exists to outlive them.
|
||||
/// </summary>
|
||||
public sealed class AuditLogArchiveRetentionValidator : Microsoft.Extensions.Options.IValidateOptions<AuditLogArchiveOptions>
|
||||
{
|
||||
private readonly Microsoft.Extensions.Options.IOptions<Configuration.AuditLogOptions> _auditOptions;
|
||||
|
||||
/// <summary>Creates the validator.</summary>
|
||||
/// <param name="auditOptions">The <c>AuditLog</c> options (global retention).</param>
|
||||
public AuditLogArchiveRetentionValidator(Microsoft.Extensions.Options.IOptions<Configuration.AuditLogOptions> auditOptions)
|
||||
=> _auditOptions = auditOptions ?? throw new ArgumentNullException(nameof(auditOptions));
|
||||
|
||||
/// <inheritdoc />
|
||||
public Microsoft.Extensions.Options.ValidateOptionsResult Validate(string? name, AuditLogArchiveOptions options)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
var global = _auditOptions.Value.RetentionDays;
|
||||
return options.RetentionDays == 0 || options.RetentionDays > global
|
||||
? Microsoft.Extensions.Options.ValidateOptionsResult.Success
|
||||
: Microsoft.Extensions.Options.ValidateOptionsResult.Fail(
|
||||
$"AuditLog:Archive:RetentionDays ({options.RetentionDays}) must be 0 (never delete) or greater than "
|
||||
+ $"AuditLog:RetentionDays ({global}): archives must outlive the rows' time in the database.");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,6 +6,7 @@ using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.AuditLog.Configuration;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
|
||||
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
|
||||
@@ -195,6 +196,7 @@ public class AuditLogPurgeActor : ReceiveActor
|
||||
// No whole-month partitions are eligible, but per-channel overrides may
|
||||
// still expire rows earlier than the global window — run them below.
|
||||
await RunPerChannelOverridesAsync(repository, eventStream, purgeFailureCounter).ConfigureAwait(false);
|
||||
await RunArchiveCleanupAsync(scope.ServiceProvider, eventStream, purgeFailureCounter).ConfigureAwait(false);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -274,6 +276,52 @@ public class AuditLogPurgeActor : ReceiveActor
|
||||
// deletes only ever expire rows EARLIER than that, so they run last and are a
|
||||
// strict tightening.
|
||||
await RunPerChannelOverridesAsync(repository, eventStream, purgeFailureCounter).ConfigureAwait(false);
|
||||
|
||||
// Archive housekeeping last (ScadaBridge#109): superseded stale archives this or an
|
||||
// earlier tick's retried purge left behind, and archives past AuditLog:Archive:RetentionDays.
|
||||
if (archive is not null)
|
||||
{
|
||||
await RunArchiveCleanupAsync(scope.ServiceProvider, eventStream, purgeFailureCounter, archive).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Runs the archive cleanup pass (ScadaBridge#109) when archiving is enabled. Contained: a
|
||||
/// cleanup failure is a purge failure on the health surface, never a crash of the tick.
|
||||
/// </summary>
|
||||
private async Task RunArchiveCleanupAsync(
|
||||
IServiceProvider services,
|
||||
Akka.Event.EventStream eventStream,
|
||||
IAuditPurgeFailureCounter? purgeFailureCounter,
|
||||
IAuditLogArchive? archive = null)
|
||||
{
|
||||
var sw = Stopwatch.StartNew();
|
||||
try
|
||||
{
|
||||
if (archive is null)
|
||||
{
|
||||
if (services.GetService<IOptions<AuditLogArchiveOptions>>()?.Value.Enabled != true)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
archive = services.GetKeyedService<IAuditLogArchive>(AuditMaintenanceContext.ServiceKey)
|
||||
?? services.GetRequiredService<IAuditLogArchive>();
|
||||
}
|
||||
|
||||
var result = await archive.CleanupAsync(dryRun: false).ConfigureAwait(false);
|
||||
var deleted = result.Items.Count(i => i.Action != AuditArchiveCleanupAction.KeptStale);
|
||||
if (deleted > 0)
|
||||
{
|
||||
_logger.LogInformation("AuditLog archive cleanup deleted {Count} archive(s) in {DurationMs} ms.", deleted, sw.ElapsedMilliseconds);
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
sw.Stop();
|
||||
_logger.LogError(ex, "AuditLog archive cleanup failed; archives are kept and the next tick retries.");
|
||||
PublishPurgeFailure(eventStream, purgeFailureCounter, DateTime.MinValue, ex, sw.ElapsedMilliseconds);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -442,6 +442,10 @@ public static class ServiceCollectionExtensions
|
||||
// registered by AddConfigurationDatabase (central only) and reads these options.
|
||||
services.AddValidatedOptions<ConfigurationDatabase.Maintenance.AuditLogArchiveOptions, AuditLogArchiveOptionsValidator>(
|
||||
config, ArchiveSectionName);
|
||||
// Archive retention must exceed the global AuditLog:RetentionDays (ScadaBridge#109).
|
||||
services.TryAddEnumerable(ServiceDescriptor.Singleton<
|
||||
Microsoft.Extensions.Options.IValidateOptions<ConfigurationDatabase.Maintenance.AuditLogArchiveOptions>,
|
||||
AuditLogArchiveRetentionValidator>());
|
||||
|
||||
// I1 (review): bind the two central-singleton options HERE rather than in
|
||||
// AddAuditLogCentralReconciliationClient. AkkaHostedService.RegisterCentralActors
|
||||
|
||||
@@ -109,6 +109,43 @@ public static class AuditArchiveHelpers
|
||||
return verified ? 0 : NotVerifiedExitCode;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Renders a <c>CleanupAuditArchives</c> result (ScadaBridge#109). Always exit 0 — kept
|
||||
/// stale archives are reported, not an error.
|
||||
/// </summary>
|
||||
/// <param name="json">The success body (an <c>AuditArchiveCleanupResult</c>, camelCase).</param>
|
||||
/// <param name="format">Resolved output format.</param>
|
||||
/// <param name="output">Writer for the report.</param>
|
||||
/// <returns><c>0</c>.</returns>
|
||||
public static int RenderCleanup(string json, string format, TextWriter output)
|
||||
{
|
||||
if (!string.Equals(format, "table", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
output.WriteLine(json);
|
||||
output.Flush();
|
||||
return 0;
|
||||
}
|
||||
|
||||
using var doc = JsonDocument.Parse(json);
|
||||
var root = doc.RootElement;
|
||||
var dryRun = root.TryGetProperty("dryRun", out var d) && d.ValueKind == JsonValueKind.True;
|
||||
var retention = Num(root, "retentionDays");
|
||||
output.WriteLine($"{(dryRun ? "Dry run — nothing deleted. " : "")}Archive retention: {(retention == "0" ? "never delete" : retention + " days")}");
|
||||
var items = root.TryGetProperty("items", out var i) && i.ValueKind == JsonValueKind.Array ? i.EnumerateArray().ToList() : [];
|
||||
if (items.Count == 0)
|
||||
{
|
||||
output.WriteLine("Nothing to clean up.");
|
||||
}
|
||||
|
||||
foreach (var item in items)
|
||||
{
|
||||
output.WriteLine($"{Str(item, "action"),-18} {Str(item, "name")} {Str(item, "detail")}");
|
||||
}
|
||||
|
||||
output.Flush();
|
||||
return 0;
|
||||
}
|
||||
|
||||
private static string Str(JsonElement e, string name, string missing = "")
|
||||
=> e.TryGetProperty(name, out var v) && v.ValueKind == JsonValueKind.String ? v.GetString() ?? missing : missing;
|
||||
|
||||
|
||||
@@ -11,9 +11,9 @@ namespace ZB.MOM.WW.ScadaBridge.CLI.Commands;
|
||||
/// <c>GET /api/audit/tree</c>), plus <c>verify-chain</c>, which sends the
|
||||
/// <c>VerifyAuditChain</c> management command (hash-chain tamper evidence, ScadaBridge#41),
|
||||
/// <c>verify-heads</c>, which checks off-box exported chain heads with
|
||||
/// <c>VerifyAuditChainHeads</c> (ScadaBridge#107), and <c>archive list</c> / <c>archive verify</c>,
|
||||
/// which send <c>ListAuditArchives</c> / <c>VerifyAuditArchive</c> (Parquet cold archive,
|
||||
/// ScadaBridge#42).
|
||||
/// <c>VerifyAuditChainHeads</c> (ScadaBridge#107), and <c>archive list</c> / <c>archive verify</c>
|
||||
/// / <c>archive cleanup</c>, which send <c>ListAuditArchives</c> / <c>VerifyAuditArchive</c>
|
||||
/// (Parquet cold archive, ScadaBridge#42) / <c>CleanupAuditArchives</c> (ScadaBridge#109).
|
||||
/// </summary>
|
||||
public static class AuditCommands
|
||||
{
|
||||
@@ -376,7 +376,7 @@ public static class AuditCommands
|
||||
|
||||
private static Command BuildArchive(Option<string> urlOption, Option<string> formatOption, Option<string> usernameOption, Option<string> passwordOption)
|
||||
{
|
||||
var group = new Command("archive") { Description = "List and verify the AuditLog Parquet cold archives" };
|
||||
var group = new Command("archive") { Description = "List, verify and clean up the AuditLog Parquet cold archives" };
|
||||
|
||||
var list = new Command("list") { Description = "List the AuditLog Parquet archives written by the retention purge" };
|
||||
list.SetAction(async (ParseResult result) =>
|
||||
@@ -410,6 +410,23 @@ public static class AuditCommands
|
||||
});
|
||||
group.Add(verify);
|
||||
|
||||
var dryRunOption = new Option<bool>("--dry-run") { Description = "Report what would be deleted; delete nothing" };
|
||||
var cleanup = new Command("cleanup")
|
||||
{
|
||||
Description = "Delete superseded stale archives and archives past AuditLog:Archive:RetentionDays (Administrator)",
|
||||
};
|
||||
cleanup.Add(dryRunOption);
|
||||
cleanup.SetAction(async (ParseResult result) =>
|
||||
{
|
||||
var format = AuditCommandHelpers.ResolveFormat(result, formatOption);
|
||||
return await CommandHelpers.ExecuteCommandAsync(
|
||||
result, urlOption, formatOption, usernameOption, passwordOption,
|
||||
new CleanupAuditArchivesCommand(result.GetValue(dryRunOption)),
|
||||
timeout: AuditArchiveHelpers.VerifyRequestTimeout,
|
||||
onSuccess: json => AuditArchiveHelpers.RenderCleanup(json, format, Console.Out));
|
||||
});
|
||||
group.Add(cleanup);
|
||||
|
||||
return group;
|
||||
}
|
||||
|
||||
|
||||
@@ -2136,6 +2136,13 @@ tombstones, and compares the chain end with the checkpoint anchor the purge left
|
||||
| `1` | Request error (`ARCHIVE_NOT_CONFIGURED`, `NOT_FOUND`, `INVALID_ARGUMENT`, server error) |
|
||||
| `2` | Authorization failure (needs `Administrator` or `Viewer`, not site-scoped) |
|
||||
|
||||
`audit archive cleanup [--dry-run]` (ScadaBridge#109, Administrator) sends
|
||||
`CleanupAuditArchives`: it deletes stale archives a retried purge left behind once a later
|
||||
purged archive provably holds all their rows (`DeletedSuperseded`), and archives whose month
|
||||
ended more than `AuditLog:Archive:RetentionDays` ago (`DeletedExpired`; default `0` = never),
|
||||
and lists stale archives it kept (`KeptStale`) with the reason. `--dry-run` deletes nothing.
|
||||
The purge actor runs the same pass after every purge tick. Exit code `0`.
|
||||
|
||||
The archive files are plain Parquet; read them with any Parquet tool, e.g. DuckDB:
|
||||
`SELECT * FROM '<dir>/2025/auditlog-2025-10-….parquet'`. See `Component-AuditLog.md`,
|
||||
Retention & Purge.
|
||||
|
||||
@@ -66,6 +66,17 @@ public interface IAuditLogArchive
|
||||
/// <returns>The verification result.</returns>
|
||||
/// <exception cref="FileNotFoundException">No archive has that name.</exception>
|
||||
Task<AuditArchiveVerificationResult> VerifyAsync(string name, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Archive cleanup (ScadaBridge#109): deletes superseded stale archives a retried purge left
|
||||
/// behind (when <c>AuditLog:Archive:DeleteSupersededStaleArchives</c>), and archives past
|
||||
/// <c>AuditLog:Archive:RetentionDays</c> (0 = never). Only ever deletes the three files of a
|
||||
/// well-formed archive inside the configured directory, manifest first.
|
||||
/// </summary>
|
||||
/// <param name="dryRun">Report only; delete nothing.</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>What was (or would be) deleted, and the stale archives kept.</returns>
|
||||
Task<AuditArchiveCleanupResult> CleanupAsync(bool dryRun, CancellationToken ct = default);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -57,3 +57,12 @@ public record ListAuditArchivesCommand;
|
||||
/// </summary>
|
||||
/// <param name="Name">The archive name, as <c>audit archive list</c> reports it.</param>
|
||||
public record VerifyAuditArchiveCommand(string Name);
|
||||
|
||||
/// <summary>
|
||||
/// Runs the AuditLog archive cleanup (ScadaBridge#109) now: deletes superseded stale archives
|
||||
/// a retried purge left behind and archives past <c>AuditLog:Archive:RetentionDays</c> — or,
|
||||
/// with <paramref name="DryRun"/>, only reports what it would delete. The purge actor runs the
|
||||
/// same pass after every purge tick. Administrator only (it deletes files).
|
||||
/// </summary>
|
||||
/// <param name="DryRun">Report only; delete nothing.</param>
|
||||
public record CleanupAuditArchivesCommand(bool DryRun);
|
||||
|
||||
@@ -145,3 +145,41 @@ public sealed record AuditArchiveVerificationResult(
|
||||
/// <param name="Enabled">Whether the retention purge is archiving partitions (<c>AuditLog:Archive:Enabled</c>).</param>
|
||||
/// <param name="Archives">Every archive manifest, oldest partition first.</param>
|
||||
public sealed record AuditArchiveListResult(bool Enabled, IReadOnlyList<AuditArchiveManifest> Archives);
|
||||
|
||||
/// <summary>What archive cleanup did (or, on a dry run, would do) with one archive (ScadaBridge#109).</summary>
|
||||
[JsonConverter(typeof(JsonStringEnumConverter<AuditArchiveCleanupAction>))]
|
||||
public enum AuditArchiveCleanupAction
|
||||
{
|
||||
/// <summary>
|
||||
/// Deleted by retention: the archived month ended more than <c>AuditLog:Archive:RetentionDays</c>
|
||||
/// ago, and the archive is purged (or its partition no longer holds rows).
|
||||
/// </summary>
|
||||
DeletedExpired,
|
||||
|
||||
/// <summary>
|
||||
/// Deleted as a superseded stale archive: status <c>Archived</c> (its partition switch never
|
||||
/// completed — a retried purge), and a LATER, verified, <c>Purged</c> archive of the same
|
||||
/// partition provably holds every row it holds (same chain through its chain end, every
|
||||
/// unchained row present by EventId).
|
||||
/// </summary>
|
||||
DeletedSuperseded,
|
||||
|
||||
/// <summary>
|
||||
/// A stale <c>Archived</c> archive that was kept: no later purged archive provably covers it,
|
||||
/// so it may be the only copy of some rows (see Detail).
|
||||
/// </summary>
|
||||
KeptStale,
|
||||
}
|
||||
|
||||
/// <summary>One archive considered by cleanup.</summary>
|
||||
/// <param name="Name">Archive name.</param>
|
||||
/// <param name="Partition">Archived partition, <c>YYYY-MM</c>.</param>
|
||||
/// <param name="Action">What was (or would be) done.</param>
|
||||
/// <param name="Detail">Why.</param>
|
||||
public sealed record AuditArchiveCleanupItem(string Name, string Partition, AuditArchiveCleanupAction Action, string Detail);
|
||||
|
||||
/// <summary>Result of an archive cleanup pass (ScadaBridge#109).</summary>
|
||||
/// <param name="DryRun">True when nothing was deleted (the items say what would be).</param>
|
||||
/// <param name="RetentionDays">The archive retention in force (0 = archives never expire).</param>
|
||||
/// <param name="Items">The archives deleted (or that would be) and the stale ones kept.</param>
|
||||
public sealed record AuditArchiveCleanupResult(bool DryRun, int RetentionDays, IReadOnlyList<AuditArchiveCleanupItem> Items);
|
||||
|
||||
@@ -43,4 +43,19 @@ public sealed class AuditLogArchiveOptions
|
||||
/// the hash-chain lock before giving up for this tick (default 300).
|
||||
/// </summary>
|
||||
public int ChainLockTimeoutSeconds { get; set; } = 300;
|
||||
|
||||
/// <summary>
|
||||
/// Archive retention in days, counted from the END of the archived month (ScadaBridge#109).
|
||||
/// <c>0</c> (the default) = archives are never deleted — until compliance names a period.
|
||||
/// When set it must exceed <c>AuditLog:RetentionDays</c>, so an archive always outlives the
|
||||
/// rows' time in the database. Applied by the purge actor after each tick's partition purge.
|
||||
/// </summary>
|
||||
public int RetentionDays { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Delete stale archives a retried purge left behind (status <c>Archived</c>) once a later,
|
||||
/// verified, purged archive of the same partition provably holds all their rows (default
|
||||
/// true; ScadaBridge#109). A stale archive that cannot be proven redundant is kept.
|
||||
/// </summary>
|
||||
public bool DeleteSupersededStaleArchives { get; set; } = true;
|
||||
}
|
||||
|
||||
+290
@@ -0,0 +1,290 @@
|
||||
using System.Data;
|
||||
using Microsoft.Data.SqlClient;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance;
|
||||
|
||||
/// <summary>
|
||||
/// Archive cleanup (ScadaBridge#109): retention and superseded-stale-archive deletion.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Superseded stale archives.</b> A purge whose guard finds the partition changed after it
|
||||
/// was archived (51002/51003) purges nothing and leaves its archive at status
|
||||
/// <see cref="AuditArchiveStatus.Archived"/>; the next tick writes a fresh archive and purges
|
||||
/// with it. The stale one is redundant only if a LATER archive B of the same partition, which
|
||||
/// is <see cref="AuditArchiveStatus.Purged"/> and verifies, holds everything the stale archive A
|
||||
/// holds. That is proven, not assumed: for every chain month of A, B starts from the same
|
||||
/// earlier purge anchor and carries A's chain-end hash at A's chain-end sequence (as a row or a
|
||||
/// tombstone) — so, the chain being a hash chain B itself verifies, every chained row of A is
|
||||
/// in B; and every UNCHAINED row of A is found in B (row or tombstone) by EventId. Anything
|
||||
/// else — notably an unchained row a per-channel purge removed in between, which leaves no
|
||||
/// tombstone — keeps A (<see cref="AuditArchiveCleanupAction.KeptStale"/>). An archive left at
|
||||
/// <c>Archived</c> because the node stopped between the switch and the manifest update has no
|
||||
/// later purged archive and is kept: it is the only copy.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Retention.</b> With <see cref="AuditLogArchiveOptions.RetentionDays"/> > 0, an archive
|
||||
/// whose month ended more than that many days ago is deleted when it is <c>Purged</c>, or —
|
||||
/// an <c>Archived</c> one — when its partition no longer holds rows in <c>dbo.AuditLog</c>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Safe deletion.</b> Only the three files a well-formed archive name implies
|
||||
/// (<c>{name}.manifest.json</c>, <c>{name}.parquet</c>, <c>{name}.tombstones.parquet</c>) in the
|
||||
/// archive's own <c>{Directory}/{YYYY}</c> folder are deleted — never a path read from the
|
||||
/// manifest — and the manifest goes FIRST, so an interrupted deletion leaves leftovers that are
|
||||
/// no longer an archive rather than a manifest naming missing files.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed partial class AuditLogPartitionArchive
|
||||
{
|
||||
/// <summary>
|
||||
/// Upper bound on unchained rows compared by EventId when proving a stale archive redundant;
|
||||
/// a larger set keeps the archive rather than spend the memory.
|
||||
/// </summary>
|
||||
internal const long MaxUnchainedRowsCompared = 5_000_000;
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<AuditArchiveCleanupResult> CleanupAsync(bool dryRun, CancellationToken ct = default)
|
||||
{
|
||||
RequireConfigured();
|
||||
var items = new List<AuditArchiveCleanupItem>();
|
||||
var manifests = await ListAsync(ct).ConfigureAwait(false);
|
||||
var deleted = new HashSet<string>(StringComparer.Ordinal);
|
||||
|
||||
if (_options.DeleteSupersededStaleArchives)
|
||||
{
|
||||
foreach (var stale in manifests.Where(m => m.Status == AuditArchiveStatus.Archived))
|
||||
{
|
||||
var later = manifests
|
||||
.Where(m => m.PartitionBoundaryUtc == stale.PartitionBoundaryUtc
|
||||
&& m.Status == AuditArchiveStatus.Purged
|
||||
&& m.ArchivedAtUtc > stale.ArchivedAtUtc)
|
||||
.OrderBy(m => m.ArchivedAtUtc)
|
||||
.ToList();
|
||||
if (later.Count == 0)
|
||||
{
|
||||
continue; // not superseded (e.g. the node stopped before marking it Purged) — keep
|
||||
}
|
||||
|
||||
string? reason = null;
|
||||
AuditArchiveManifest? coveredBy = null;
|
||||
foreach (var candidate in later)
|
||||
{
|
||||
reason = await ProveCoveredAsync(stale, candidate, ct).ConfigureAwait(false);
|
||||
if (reason is null)
|
||||
{
|
||||
coveredBy = candidate;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (coveredBy is null)
|
||||
{
|
||||
items.Add(new AuditArchiveCleanupItem(stale.Name, stale.Partition, AuditArchiveCleanupAction.KeptStale,
|
||||
$"No later purged archive provably covers it: {reason}"));
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!dryRun)
|
||||
{
|
||||
DeleteArchiveFiles(stale);
|
||||
}
|
||||
|
||||
deleted.Add(stale.Name);
|
||||
items.Add(new AuditArchiveCleanupItem(stale.Name, stale.Partition, AuditArchiveCleanupAction.DeletedSuperseded,
|
||||
$"Superseded by {coveredBy.Name}, which is purged, verifies and holds every row of this archive."));
|
||||
}
|
||||
}
|
||||
|
||||
if (_options.RetentionDays > 0)
|
||||
{
|
||||
var now = _time.GetUtcNow().UtcDateTime;
|
||||
foreach (var manifest in manifests.Where(m => !deleted.Contains(m.Name)))
|
||||
{
|
||||
var monthEnd = AuditChainHasher.MonthOf(manifest.PartitionBoundaryUtc).AddMonths(1);
|
||||
if (monthEnd.AddDays(_options.RetentionDays) > now)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (manifest.Status != AuditArchiveStatus.Purged
|
||||
&& await PartitionHasRowsAsync(manifest.PartitionBoundaryUtc, ct).ConfigureAwait(false))
|
||||
{
|
||||
continue; // the partition was never switched out; this archive is not a sole copy, nor expired data gone
|
||||
}
|
||||
|
||||
if (!dryRun)
|
||||
{
|
||||
DeleteArchiveFiles(manifest);
|
||||
}
|
||||
|
||||
deleted.Add(manifest.Name);
|
||||
items.Add(new AuditArchiveCleanupItem(manifest.Name, manifest.Partition, AuditArchiveCleanupAction.DeletedExpired,
|
||||
$"{manifest.Partition} ended {monthEnd:yyyy-MM-dd}, more than {_options.RetentionDays} days ago (AuditLog:Archive:RetentionDays)."));
|
||||
}
|
||||
}
|
||||
|
||||
foreach (var item in items.Where(i => i.Action != AuditArchiveCleanupAction.KeptStale))
|
||||
{
|
||||
_logger.LogInformation("AuditLog archive cleanup{DryRun}: {Action} {Archive} — {Detail}",
|
||||
dryRun ? " (dry run)" : "", item.Action, item.Name, item.Detail);
|
||||
}
|
||||
|
||||
return new AuditArchiveCleanupResult(dryRun, _options.RetentionDays, items);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Proves <paramref name="later"/> holds every row of <paramref name="stale"/>; returns null
|
||||
/// when it does, else why not.
|
||||
/// </summary>
|
||||
private async Task<string?> ProveCoveredAsync(AuditArchiveManifest stale, AuditArchiveManifest later, CancellationToken ct)
|
||||
{
|
||||
// Same chain generation and at least as far, for every chain month the stale archive has.
|
||||
var laterChains = later.Chains.ToDictionary(c => c.Partition, StringComparer.Ordinal);
|
||||
var targets = new Dictionary<(DateTime Month, long Seq), string>();
|
||||
foreach (var chain in stale.Chains)
|
||||
{
|
||||
if (!laterChains.TryGetValue(chain.Partition, out var next))
|
||||
{
|
||||
return $"{later.Name} has no chain for {chain.Partition}.";
|
||||
}
|
||||
|
||||
if (next.PriorPurgedThroughSeq != chain.PriorPurgedThroughSeq || next.LastSeq < chain.LastSeq)
|
||||
{
|
||||
return $"{later.Name}'s chain for {chain.Partition} does not extend this archive's (prior anchor or end differs).";
|
||||
}
|
||||
|
||||
if (chain.LastSeq > (chain.PriorPurgedThroughSeq ?? 0))
|
||||
{
|
||||
targets[(AuditArchiveParquet.ParseMonth(chain.Partition), chain.LastSeq)] = chain.LastHash.ToLowerInvariant();
|
||||
}
|
||||
}
|
||||
|
||||
if (stale.UnchainedRowCount > MaxUnchainedRowsCompared)
|
||||
{
|
||||
return $"{stale.UnchainedRowCount} unchained rows exceed the {MaxUnchainedRowsCompared} compared by EventId.";
|
||||
}
|
||||
|
||||
// The later archive must itself be intact and tied to the database.
|
||||
var verification = await VerifyAsync(later.Name, ct).ConfigureAwait(false);
|
||||
if (!verification.Verified)
|
||||
{
|
||||
return $"{later.Name} does not verify ({verification.FileProblem ?? "chain or anchor problem"}).";
|
||||
}
|
||||
|
||||
var directory = ArchiveDirectory(stale);
|
||||
var unchained = new HashSet<Guid>();
|
||||
await using (var input = File.OpenRead(Path.Combine(directory, stale.Name + ".parquet")))
|
||||
{
|
||||
await foreach (var row in AuditArchiveParquet.ReadRowsAsync(input, ct).ConfigureAwait(false))
|
||||
{
|
||||
if (row.ChainSeq is null)
|
||||
{
|
||||
unchained.Add(row.EventId);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var laterDirectory = ArchiveDirectory(later);
|
||||
await using (var input = File.OpenRead(Path.Combine(laterDirectory, later.Name + ".parquet")))
|
||||
{
|
||||
await foreach (var row in AuditArchiveParquet.ReadRowsAsync(input, ct).ConfigureAwait(false))
|
||||
{
|
||||
unchained.Remove(row.EventId);
|
||||
if (row.ChainSeq is { } seq
|
||||
&& targets.TryGetValue((AuditChainHasher.MonthOf(row.OccurredAtUtc), seq), out var expected))
|
||||
{
|
||||
if (row.RowHash is null || !string.Equals(Convert.ToHexStringLower(row.RowHash), expected, StringComparison.Ordinal))
|
||||
{
|
||||
return $"{later.Name} carries a different hash at the stale archive's chain end.";
|
||||
}
|
||||
|
||||
targets.Remove((AuditChainHasher.MonthOf(row.OccurredAtUtc), seq));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
await using (var input = File.OpenRead(Path.Combine(laterDirectory, later.Name + ".tombstones.parquet")))
|
||||
{
|
||||
await foreach (var tombstone in AuditArchiveParquet.ReadTombstonesAsync(input, ct).ConfigureAwait(false))
|
||||
{
|
||||
unchained.Remove(tombstone.EventId);
|
||||
if (targets.TryGetValue((tombstone.ChainMonth, tombstone.ChainSeq), out var expected))
|
||||
{
|
||||
if (!string.Equals(Convert.ToHexStringLower(tombstone.RowHash), expected, StringComparison.Ordinal))
|
||||
{
|
||||
return $"{later.Name} carries a different hash at the stale archive's chain end.";
|
||||
}
|
||||
|
||||
targets.Remove((tombstone.ChainMonth, tombstone.ChainSeq));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (targets.Count > 0)
|
||||
{
|
||||
return $"{later.Name} does not contain the stale archive's chain end.";
|
||||
}
|
||||
|
||||
return unchained.Count == 0
|
||||
? null
|
||||
: $"{unchained.Count} unchained row(s) of this archive are not in {later.Name} (e.g. removed by a per-channel purge, which leaves no tombstone for an unchained row).";
|
||||
}
|
||||
|
||||
private async Task<bool> PartitionHasRowsAsync(DateTime boundary, CancellationToken ct)
|
||||
{
|
||||
var conn = _context.Database.GetDbConnection();
|
||||
var openedHere = await AuditLogHashChain.OpenAsync(conn, ct).ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
await using var cmd = conn.CreateCommand();
|
||||
cmd.CommandText = "SELECT CASE WHEN EXISTS (SELECT 1 FROM dbo.AuditLog " +
|
||||
"WHERE $partition.pf_AuditLog_Month(OccurredAtUtc) = $partition.pf_AuditLog_Month(@boundary)) THEN 1 ELSE 0 END;";
|
||||
cmd.Parameters.Add(new SqlParameter("@boundary", SqlDbType.DateTime2) { Scale = 7, Value = boundary });
|
||||
return Convert.ToInt32(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false)) == 1;
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (openedHere)
|
||||
{
|
||||
await conn.CloseAsync().ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private string ArchiveDirectory(AuditArchiveManifest manifest)
|
||||
{
|
||||
if (!IsValidName(manifest.Name))
|
||||
{
|
||||
throw new InvalidOperationException($"'{manifest.Name}' is not an archive name.");
|
||||
}
|
||||
|
||||
return Path.Combine(_options.Directory!, manifest.Name.Substring("auditlog-".Length, 4));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Deletes exactly the archive's three files, manifest first. File names are derived from
|
||||
/// the validated archive name, never taken from the manifest.
|
||||
/// </summary>
|
||||
private void DeleteArchiveFiles(AuditArchiveManifest manifest)
|
||||
{
|
||||
var directory = Path.GetFullPath(ArchiveDirectory(manifest));
|
||||
var root = Path.GetFullPath(_options.Directory!);
|
||||
if (!directory.StartsWith(root, StringComparison.Ordinal))
|
||||
{
|
||||
throw new InvalidOperationException($"Archive {manifest.Name} resolves outside {root}.");
|
||||
}
|
||||
|
||||
foreach (var file in new[] { ".manifest.json", ".parquet", ".tombstones.parquet" })
|
||||
{
|
||||
var path = Path.Combine(directory, manifest.Name + file);
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -398,6 +398,8 @@ public class ManagementActor : ReceiveActor
|
||||
// AuditLog Parquet archives (ScadaBridge#42): audit reads with verify-chain's gate —
|
||||
// OperationalAudit roles, site-scoped callers refused in the handler.
|
||||
ListAuditArchivesCommand => OperationalAuditReaders,
|
||||
// Archive cleanup deletes archive files (ScadaBridge#109): Administrator only.
|
||||
CleanupAuditArchivesCommand => AdminOnly,
|
||||
VerifyAuditArchiveCommand => OperationalAuditReaders,
|
||||
|
||||
// Read-only queries -- any authenticated user
|
||||
@@ -574,6 +576,7 @@ public class ManagementActor : ReceiveActor
|
||||
VerifyAuditChainHeadsCommand cmd => await HandleVerifyAuditChainHeads(sp, cmd, user),
|
||||
ListAuditArchivesCommand => await HandleListAuditArchives(sp, user),
|
||||
VerifyAuditArchiveCommand cmd => await HandleVerifyAuditArchive(sp, cmd, user),
|
||||
CleanupAuditArchivesCommand cmd => await HandleCleanupAuditArchives(sp, cmd, user),
|
||||
|
||||
// Health
|
||||
GetHealthSummaryCommand => HandleGetHealthSummary(sp),
|
||||
@@ -3340,6 +3343,20 @@ public class ManagementActor : ReceiveActor
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Runs the archive cleanup now (ScadaBridge#109), on the maintenance-connection archive when
|
||||
/// registered. Dry run reports only.
|
||||
/// </summary>
|
||||
private static async Task<object?> HandleCleanupAuditArchives(
|
||||
IServiceProvider sp, CleanupAuditArchivesCommand cmd, AuthenticatedUser user)
|
||||
{
|
||||
RequireAuditArchive(sp, user);
|
||||
var archive = sp.GetKeyedService<IAuditLogArchive>(
|
||||
ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance.AuditMaintenanceContext.ServiceKey)
|
||||
?? sp.GetRequiredService<IAuditLogArchive>();
|
||||
return await archive.CleanupAsync(cmd.DryRun);
|
||||
}
|
||||
|
||||
private static IAuditLogArchive RequireAuditArchive(IServiceProvider sp, AuthenticatedUser user)
|
||||
{
|
||||
if (user.PermittedSiteIds.Length > 0
|
||||
|
||||
@@ -69,6 +69,8 @@ public static class ManagementEndpoints
|
||||
// AuditLog archive verification (ScadaBridge#42) hashes and re-chains a whole month's
|
||||
// Parquet archive.
|
||||
typeof(VerifyAuditArchiveCommand),
|
||||
// Archive cleanup proves stale archives redundant by reading Parquet files (ScadaBridge#109).
|
||||
typeof(CleanupAuditArchivesCommand),
|
||||
];
|
||||
|
||||
/// <summary>
|
||||
|
||||
+82
@@ -0,0 +1,82 @@
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.AuditLog.Central;
|
||||
using ZB.MOM.WW.ScadaBridge.AuditLog.Configuration;
|
||||
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Central;
|
||||
|
||||
/// <summary>
|
||||
/// <c>AuditLog:Archive:RetentionDays</c> (ScadaBridge#109): 0 = never delete; otherwise in range
|
||||
/// and greater than the global <c>AuditLog:RetentionDays</c>.
|
||||
/// </summary>
|
||||
public sealed class AuditLogArchiveRetentionValidatorTests
|
||||
{
|
||||
/// <summary>Range check in the section validator.</summary>
|
||||
/// <param name="days">RetentionDays.</param>
|
||||
/// <param name="valid">Expected outcome.</param>
|
||||
[Theory]
|
||||
[InlineData(0, true)]
|
||||
[InlineData(400, true)]
|
||||
[InlineData(36_500, true)]
|
||||
[InlineData(-1, false)]
|
||||
[InlineData(36_501, false)]
|
||||
public void SectionValidator_ChecksTheRange(int days, bool valid)
|
||||
{
|
||||
var result = new AuditLogArchiveOptionsValidator().Validate(null, new AuditLogArchiveOptions { RetentionDays = days });
|
||||
Assert.Equal(valid, result.Succeeded);
|
||||
}
|
||||
|
||||
/// <summary>Cross-section check against the global retention (365 here).</summary>
|
||||
/// <param name="days">RetentionDays.</param>
|
||||
/// <param name="valid">Expected outcome.</param>
|
||||
[Theory]
|
||||
[InlineData(0, true)]
|
||||
[InlineData(366, true)]
|
||||
[InlineData(365, false)]
|
||||
[InlineData(30, false)]
|
||||
public void RetentionValidator_RequiresMoreThanTheGlobalRetention(int days, bool valid)
|
||||
{
|
||||
var validator = new AuditLogArchiveRetentionValidator(Options.Create(new AuditLogOptions { RetentionDays = 365 }));
|
||||
var result = validator.Validate(null, new AuditLogArchiveOptions { RetentionDays = days });
|
||||
Assert.Equal(valid, result.Succeeded);
|
||||
if (!valid)
|
||||
{
|
||||
Assert.Contains("AuditLog:RetentionDays (365)", result.FailureMessage);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The cross-section check is wired into the central registration.</summary>
|
||||
/// <param name="days">Archive RetentionDays.</param>
|
||||
/// <param name="valid">Whether binding succeeds.</param>
|
||||
[Theory]
|
||||
[InlineData("0", true)]
|
||||
[InlineData("400", true)]
|
||||
[InlineData("100", false)]
|
||||
public void CentralRegistration_RejectsAnArchiveRetentionBelowTheGlobalOne(string days, bool valid)
|
||||
{
|
||||
var config = new ConfigurationBuilder()
|
||||
.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["AuditLog:SiteWriter:DatabasePath"] = ":memory:",
|
||||
["AuditLog:RetentionDays"] = "365",
|
||||
["AuditLog:Archive:RetentionDays"] = days,
|
||||
})
|
||||
.Build();
|
||||
var services = new ServiceCollection();
|
||||
services.AddAuditLog(config);
|
||||
services.AddAuditLogCentralMaintenance(config);
|
||||
using var provider = services.BuildServiceProvider();
|
||||
|
||||
var resolve = () => provider.GetRequiredService<IOptions<AuditLogArchiveOptions>>().Value;
|
||||
if (valid)
|
||||
{
|
||||
Assert.Equal(int.Parse(days), resolve().RetentionDays);
|
||||
}
|
||||
else
|
||||
{
|
||||
Assert.Contains("AuditLog:Archive:RetentionDays (100)", Assert.Throws<OptionsValidationException>(resolve).Message);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -822,4 +822,65 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture<MsSqlMigrationFixt
|
||||
|
||||
StopPurgeActor(actor);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#109: with archiving enabled every tick ends with the archive cleanup pass —
|
||||
/// also on a tick with no expired partition — and never runs it when archiving is off.
|
||||
/// </summary>
|
||||
/// <param name="withBoundary">Whether the tick has an expired partition to purge.</param>
|
||||
[Theory]
|
||||
[InlineData(true)]
|
||||
[InlineData(false)]
|
||||
public void ArchiveEnabled_RunsTheCleanupPass_EveryTick(bool withBoundary)
|
||||
{
|
||||
var boundary = new DateTime(2025, 10, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
var repo = new RecordingRepo { Boundaries = withBoundary ? [boundary] : [] };
|
||||
var archive = Substitute.For<IAuditLogArchive>();
|
||||
var manifest = ManifestFor(boundary);
|
||||
archive.ArchivePartitionAsync(boundary, Arg.Any<CancellationToken>()).Returns(manifest);
|
||||
archive.CleanupAsync(false, Arg.Any<CancellationToken>()).Returns(new AuditArchiveCleanupResult(false, 0,
|
||||
[new AuditArchiveCleanupItem("auditlog-2025-10-20261109T120000000Z", "2025-10", AuditArchiveCleanupAction.DeletedSuperseded, "x")]));
|
||||
|
||||
var actor = CreateArchivingActor(repo, archive, enabled: true);
|
||||
actor.Tell(AuditLogPurgeActor.PurgeTick.Instance);
|
||||
|
||||
AwaitAssert(() => archive.Received().CleanupAsync(false, Arg.Any<CancellationToken>()),
|
||||
duration: TimeSpan.FromSeconds(5), interval: TimeSpan.FromMilliseconds(50));
|
||||
StopPurgeActor(actor);
|
||||
}
|
||||
|
||||
/// <summary>A cleanup failure is contained and surfaces as a purge failure.</summary>
|
||||
[Fact]
|
||||
public void ArchiveCleanupFailure_IsAPurgeFailure()
|
||||
{
|
||||
var repo = new RecordingRepo();
|
||||
var archive = Substitute.For<IAuditLogArchive>();
|
||||
archive.CleanupAsync(false, Arg.Any<CancellationToken>())
|
||||
.Returns<Task<AuditArchiveCleanupResult>>(_ => throw new IOException("share offline"));
|
||||
var failed = CreateTestProbe();
|
||||
Sys.EventStream.Subscribe(failed.Ref, typeof(AuditLogPurgeFailedEvent));
|
||||
|
||||
var actor = CreateArchivingActor(repo, archive, enabled: true);
|
||||
actor.Tell(AuditLogPurgeActor.PurgeTick.Instance);
|
||||
|
||||
var evt = failed.ExpectMsg<AuditLogPurgeFailedEvent>(TimeSpan.FromSeconds(5));
|
||||
Assert.Contains("share offline", evt.Error);
|
||||
Assert.Equal(DateTime.MinValue, evt.MonthBoundary);
|
||||
StopPurgeActor(actor);
|
||||
}
|
||||
|
||||
/// <summary>Archiving off: no cleanup pass.</summary>
|
||||
[Fact]
|
||||
public void ArchiveDisabled_RunsNoCleanup()
|
||||
{
|
||||
var repo = new RecordingRepo();
|
||||
var archive = Substitute.For<IAuditLogArchive>();
|
||||
var actor = CreateArchivingActor(repo, archive, enabled: false);
|
||||
actor.Tell(AuditLogPurgeActor.PurgeTick.Instance);
|
||||
|
||||
AwaitAssert(() => Assert.NotEmpty(repo.ThresholdQueries),
|
||||
duration: TimeSpan.FromSeconds(5), interval: TimeSpan.FromMilliseconds(50));
|
||||
archive.DidNotReceiveWithAnyArgs().CleanupAsync(default, default);
|
||||
StopPurgeActor(actor);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -30,13 +30,13 @@ public class AuditArchiveCommandTests
|
||||
"\"rowsRead\":11,\"chains\":[{\"chain\":{\"partition\":\"2025-10\",\"status\":\"Intact\",\"rowsVerified\":10," +
|
||||
"\"tombstonesApplied\":0,\"unchainedRows\":0},\"anchor\":\"Mismatch\",\"anchorDetail\":\"hash differs\"}]}";
|
||||
|
||||
/// <summary>The audit group carries the archive group with list + verify.</summary>
|
||||
/// <summary>The audit group carries the archive group with list + verify + cleanup (ScadaBridge#109).</summary>
|
||||
[Fact]
|
||||
public void AuditGroup_HasArchiveListAndVerify()
|
||||
public void AuditGroup_HasArchiveListVerifyAndCleanup()
|
||||
{
|
||||
var root = AuditCommandTestHarness.BuildRoot();
|
||||
var archive = root.Subcommands.Single(c => c.Name == "audit").Subcommands.Single(c => c.Name == "archive");
|
||||
Assert.Equal(["list", "verify"], archive.Subcommands.Select(c => c.Name).Order());
|
||||
Assert.Equal(["cleanup", "list", "verify"], archive.Subcommands.Select(c => c.Name).Order());
|
||||
}
|
||||
|
||||
/// <summary><c>list</c> sends ListAuditArchives and prints the archives.</summary>
|
||||
@@ -78,6 +78,34 @@ public class AuditArchiveCommandTests
|
||||
Assert.Equal(Name, name);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// <c>cleanup</c> (ScadaBridge#109) sends CleanupAuditArchives with the dry-run flag and
|
||||
/// reports each action; it always exits 0.
|
||||
/// </summary>
|
||||
/// <param name="dryRun">Whether <c>--dry-run</c> is passed.</param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Theory]
|
||||
[InlineData(true)]
|
||||
[InlineData(false)]
|
||||
public async Task Cleanup_SendsCleanupAuditArchives_WithTheDryRunFlag(bool dryRun)
|
||||
{
|
||||
var body = "{\"dryRun\":" + (dryRun ? "true" : "false") + ",\"retentionDays\":0,\"items\":[{\"name\":\"" + Name
|
||||
+ "\",\"partition\":\"2025-10\",\"action\":\"DeletedSuperseded\",\"detail\":\"superseded by x\"}]}";
|
||||
string[] args = dryRun
|
||||
? ["audit", "archive", "cleanup", "--dry-run", "--format", "table"]
|
||||
: ["audit", "archive", "cleanup", "--format", "table"];
|
||||
var (exit, output, request) = await RunAsync(body, args);
|
||||
|
||||
Assert.Equal(0, exit);
|
||||
Assert.Contains("DeletedSuperseded", output);
|
||||
Assert.Contains("never delete", output);
|
||||
Assert.Equal(dryRun, output.Contains("Dry run", StringComparison.Ordinal));
|
||||
using var payload = JsonDocument.Parse(request);
|
||||
Assert.Equal("CleanupAuditArchives", payload.RootElement.GetProperty("command").GetString());
|
||||
Assert.Equal(dryRun, payload.RootElement.GetProperty("payload").EnumerateObject()
|
||||
.First(p => string.Equals(p.Name, "dryRun", StringComparison.OrdinalIgnoreCase)).Value.GetBoolean());
|
||||
}
|
||||
|
||||
/// <summary><c>verify</c> without <c>--name</c> is a usage error.</summary>
|
||||
[Fact]
|
||||
public void Verify_MissingName_ProducesRequiredFlagError()
|
||||
|
||||
@@ -62,10 +62,10 @@ public class AuditCommandsScaffoldTests
|
||||
[Fact]
|
||||
public void Audit_EveryLeafCommand_HasAnAction()
|
||||
{
|
||||
// `archive` is a group (list / verify, ScadaBridge#42): check its leaves, not it.
|
||||
// `archive` is a group (list / verify, ScadaBridge#42; cleanup, ScadaBridge#109): check its leaves, not it.
|
||||
var audit = BuildAudit();
|
||||
var leaves = audit.Subcommands.SelectMany(c => c.Subcommands.Count == 0 ? [c] : c.Subcommands).ToList();
|
||||
Assert.Equal(8, leaves.Count); // + verify-heads (ScadaBridge#107)
|
||||
Assert.Equal(9, leaves.Count); // + verify-heads (ScadaBridge#107), archive cleanup (ScadaBridge#109)
|
||||
Assert.All(leaves, sub =>
|
||||
Assert.True(sub.Action != null, $"Leaf command '{sub.Name}' has no action."));
|
||||
}
|
||||
|
||||
@@ -45,7 +45,7 @@ public class ManagementSurfaceParityTests
|
||||
}
|
||||
|
||||
private static readonly HashSet<string> AuditManagementLeaves =
|
||||
["audit verify-chain", "audit verify-heads", "audit archive list", "audit archive verify"];
|
||||
["audit verify-chain", "audit verify-heads", "audit archive list", "audit archive verify", "audit archive cleanup"];
|
||||
|
||||
private static async Task<ConcurrentDictionary<string, string>> ProbeProductionTree(string? omitGroup = null)
|
||||
{
|
||||
|
||||
+296
@@ -0,0 +1,296 @@
|
||||
using System.Text.Json;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using ZB.MOM.WW.Audit;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance;
|
||||
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests.Migrations;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests.Maintenance;
|
||||
|
||||
/// <summary>
|
||||
/// SQL Server tests for AuditLog archive cleanup (ScadaBridge#109): a stale archive a retried
|
||||
/// purge left behind is deleted only once a later purged archive provably holds all its rows;
|
||||
/// a stale archive with a row no later archive holds, or a sole copy left at <c>Archived</c>, is
|
||||
/// kept; retention deletes only expired archives whose partition is gone; deletion never
|
||||
/// follows a path out of the archive folder.
|
||||
/// </summary>
|
||||
/// <remarks>Each test owns one month of the shared fixture database and its own archive directory.</remarks>
|
||||
public sealed class AuditLogArchiveCleanupTests : IClassFixture<MsSqlMigrationFixture>, IDisposable
|
||||
{
|
||||
private readonly MsSqlMigrationFixture _fixture;
|
||||
private readonly DirectoryInfo _root = Directory.CreateTempSubdirectory("auditlog-cleanup-");
|
||||
|
||||
/// <summary>Creates the test class with the shared migrated SQL Server fixture.</summary>
|
||||
/// <param name="fixture">The shared migration fixture.</param>
|
||||
public AuditLogArchiveCleanupTests(MsSqlMigrationFixture fixture) => _fixture = fixture;
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Dispose() => _root.Delete(recursive: true);
|
||||
|
||||
/// <summary>
|
||||
/// The retried purge: archive A goes stale (a row arrives and the chain advances), B purges.
|
||||
/// Dry run reports A as superseded and deletes nothing; the real pass deletes exactly A's
|
||||
/// three files and B still verifies.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[SkippableFact]
|
||||
public async Task RetriedPurge_StaleArchiveIsDeleted_OnceALaterPurgedArchiveCoversIt()
|
||||
{
|
||||
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
|
||||
var dir = Dir("retried");
|
||||
var month = Month(2027, 1);
|
||||
await InsertAsync(Enumerable.Range(1, 5).Select(i => NewEvent(month.AddDays(i))));
|
||||
await DrainAsync();
|
||||
await InsertAsync([NewEvent(month.AddDays(7))]); // unchained in A
|
||||
var stale = await ArchiveAsync(dir, month);
|
||||
Assert.Equal(1, stale.UnchainedRowCount);
|
||||
|
||||
await InsertAsync([NewEvent(month.AddDays(9))]);
|
||||
await DrainAsync(); // A's unchained row is now chained; the chain moved past A's end
|
||||
await using (var ctx = CreateContext())
|
||||
{
|
||||
await Assert.ThrowsAsync<AuditArchiveStaleException>(() => Archive(ctx, dir).PurgeArchivedPartitionAsync(stale));
|
||||
}
|
||||
|
||||
var fresh = await ArchiveAsync(dir, month);
|
||||
await using (var ctx = CreateContext())
|
||||
{
|
||||
Assert.Equal(7, await Archive(ctx, dir).PurgeArchivedPartitionAsync(fresh));
|
||||
}
|
||||
|
||||
var dry = await CleanupAsync(dir, dryRun: true);
|
||||
var item = Assert.Single(dry.Items);
|
||||
Assert.Equal((stale.Name, AuditArchiveCleanupAction.DeletedSuperseded), (item.Name, item.Action));
|
||||
Assert.True(File.Exists(ManifestPath(dir, stale)));
|
||||
|
||||
var real = await CleanupAsync(dir, dryRun: false);
|
||||
Assert.Equal(AuditArchiveCleanupAction.DeletedSuperseded, Assert.Single(real.Items).Action);
|
||||
Assert.False(File.Exists(ManifestPath(dir, stale)));
|
||||
Assert.False(File.Exists(Path.Combine(dir, "2027", stale.Name + ".parquet")));
|
||||
Assert.False(File.Exists(Path.Combine(dir, "2027", stale.Name + ".tombstones.parquet")));
|
||||
|
||||
await using (var ctx = CreateContext())
|
||||
{
|
||||
var listed = Assert.Single(await Archive(ctx, dir).ListAsync());
|
||||
Assert.Equal(fresh.Name, listed.Name);
|
||||
Assert.True((await Archive(ctx, dir).VerifyAsync(fresh.Name)).Verified);
|
||||
}
|
||||
|
||||
Assert.Empty((await CleanupAsync(dir, dryRun: false)).Items);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A stale archive holding an unchained row that a per-channel purge removed before the
|
||||
/// later archive (no tombstone for an unchained row) is the only copy of that row: kept.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[SkippableFact]
|
||||
public async Task StaleArchive_WithARowNoLaterArchiveHolds_IsKept()
|
||||
{
|
||||
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
|
||||
var dir = Dir("kept");
|
||||
var month = Month(2027, 3);
|
||||
await InsertAsync(Enumerable.Range(1, 3).Select(i => NewEvent(month.AddDays(i))));
|
||||
await DrainAsync();
|
||||
await InsertAsync([NewEvent(month.AddHours(1), AuditChannel.DbOutbound)]); // unchained
|
||||
var stale = await ArchiveAsync(dir, month);
|
||||
|
||||
await using (var ctx = CreateContext())
|
||||
{
|
||||
Assert.Equal(1, await new AuditLogRepository(ctx).PurgeChannelOlderThanAsync("DbOutbound", month.AddDays(1), 100));
|
||||
await Assert.ThrowsAsync<AuditArchiveStaleException>(() => Archive(ctx, dir).PurgeArchivedPartitionAsync(stale));
|
||||
}
|
||||
|
||||
var fresh = await ArchiveAsync(dir, month);
|
||||
await using (var ctx = CreateContext())
|
||||
{
|
||||
await Archive(ctx, dir).PurgeArchivedPartitionAsync(fresh);
|
||||
}
|
||||
|
||||
var result = await CleanupAsync(dir, dryRun: false);
|
||||
|
||||
var kept = Assert.Single(result.Items);
|
||||
Assert.Equal((stale.Name, AuditArchiveCleanupAction.KeptStale), (kept.Name, kept.Action));
|
||||
Assert.Contains("unchained row", kept.Detail);
|
||||
Assert.True(File.Exists(ManifestPath(dir, stale)));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// An archive left at <c>Archived</c> although its partition WAS switched out (the node
|
||||
/// stopped before the manifest update) has no later purged archive: it is the sole copy, kept.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[SkippableFact]
|
||||
public async Task SoleCopyLeftAtArchived_IsNotTouched()
|
||||
{
|
||||
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
|
||||
var dir = Dir("sole");
|
||||
var month = Month(2027, 4);
|
||||
await InsertAsync(Enumerable.Range(1, 3).Select(i => NewEvent(month.AddDays(i))));
|
||||
await DrainAsync();
|
||||
var manifest = await ArchiveAsync(dir, month);
|
||||
await using (var ctx = CreateContext())
|
||||
{
|
||||
await Archive(ctx, dir).PurgeArchivedPartitionAsync(manifest);
|
||||
}
|
||||
|
||||
await SetStatusAsync(dir, manifest, AuditArchiveStatus.Archived);
|
||||
|
||||
Assert.Empty((await CleanupAsync(dir, dryRun: false)).Items);
|
||||
Assert.True(File.Exists(ManifestPath(dir, manifest)));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Retention: 0 never deletes; a purged archive is kept until its month ended more than
|
||||
/// RetentionDays ago, then deleted; an expired archive whose partition still holds rows is kept.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[SkippableFact]
|
||||
public async Task Retention_DeletesOnlyExpiredArchivesWhosePartitionIsGone()
|
||||
{
|
||||
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
|
||||
var dir = Dir("retention");
|
||||
var purgedMonth = Month(2027, 5);
|
||||
await InsertAsync(Enumerable.Range(1, 2).Select(i => NewEvent(purgedMonth.AddDays(i))));
|
||||
var purged = await ArchiveAsync(dir, purgedMonth);
|
||||
await using (var ctx = CreateContext())
|
||||
{
|
||||
await Archive(ctx, dir).PurgeArchivedPartitionAsync(purged);
|
||||
}
|
||||
|
||||
var liveMonth = Month(2027, 6);
|
||||
await InsertAsync([NewEvent(liveMonth.AddDays(1))]);
|
||||
var live = await ArchiveAsync(dir, liveMonth); // never purged: rows still in the table
|
||||
|
||||
var monthEnd = purgedMonth.AddMonths(1);
|
||||
var farFuture = liveMonth.AddMonths(1).AddDays(1000);
|
||||
Assert.Empty((await CleanupAsync(dir, dryRun: false, retentionDays: 0, now: farFuture)).Items);
|
||||
Assert.Empty((await CleanupAsync(dir, dryRun: false, retentionDays: 400, now: monthEnd.AddDays(399))).Items);
|
||||
|
||||
var result = await CleanupAsync(dir, dryRun: false, retentionDays: 400, now: farFuture);
|
||||
|
||||
var expired = Assert.Single(result.Items);
|
||||
Assert.Equal((purged.Name, AuditArchiveCleanupAction.DeletedExpired), (expired.Name, expired.Action));
|
||||
Assert.False(File.Exists(ManifestPath(dir, purged)));
|
||||
Assert.True(File.Exists(ManifestPath(dir, live)));
|
||||
}
|
||||
|
||||
/// <summary>Deletion uses the archive name's own three files — never a path from the manifest.</summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[SkippableFact]
|
||||
public async Task Deletion_NeverFollowsAManifestPath_OutOfTheArchiveFolder()
|
||||
{
|
||||
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
|
||||
var dir = Dir("paths");
|
||||
var month = Month(2027, 7);
|
||||
await InsertAsync([NewEvent(month.AddDays(1))]);
|
||||
var manifest = await ArchiveAsync(dir, month);
|
||||
await using (var ctx = CreateContext())
|
||||
{
|
||||
await Archive(ctx, dir).PurgeArchivedPartitionAsync(manifest);
|
||||
}
|
||||
|
||||
var outside = Path.Combine(_root.FullName, "outside.parquet");
|
||||
await File.WriteAllTextAsync(outside, "not an archive");
|
||||
var path = ManifestPath(dir, manifest);
|
||||
var tampered = JsonSerializer.Deserialize<AuditArchiveManifest>(await File.ReadAllTextAsync(path), Json)! with
|
||||
{
|
||||
DataFile = "../../outside.parquet",
|
||||
TombstoneFile = "../../outside.parquet",
|
||||
};
|
||||
await File.WriteAllTextAsync(path, JsonSerializer.Serialize(tampered, Json));
|
||||
|
||||
var result = await CleanupAsync(dir, dryRun: false, retentionDays: 30, now: month.AddYears(5));
|
||||
|
||||
Assert.Equal(AuditArchiveCleanupAction.DeletedExpired, Assert.Single(result.Items).Action);
|
||||
Assert.True(File.Exists(outside));
|
||||
Assert.Empty(Directory.EnumerateFiles(Path.Combine(dir, "2027")));
|
||||
}
|
||||
|
||||
private static readonly JsonSerializerOptions Json = new() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase };
|
||||
|
||||
private sealed class FixedClock(DateTime now) : TimeProvider
|
||||
{
|
||||
public override DateTimeOffset GetUtcNow() => new(DateTime.SpecifyKind(now, DateTimeKind.Utc));
|
||||
}
|
||||
|
||||
private string Dir(string name) => Directory.CreateDirectory(Path.Combine(_root.FullName, name)).FullName;
|
||||
|
||||
private static string ManifestPath(string dir, AuditArchiveManifest m) => Path.Combine(dir, m.Partition[..4], m.Name + ".manifest.json");
|
||||
|
||||
private static async Task SetStatusAsync(string dir, AuditArchiveManifest manifest, AuditArchiveStatus status)
|
||||
{
|
||||
var path = ManifestPath(dir, manifest);
|
||||
var current = JsonSerializer.Deserialize<AuditArchiveManifest>(await File.ReadAllTextAsync(path), Json)!;
|
||||
await File.WriteAllTextAsync(path, JsonSerializer.Serialize(current with { Status = status, PurgedAtUtc = null }, Json));
|
||||
}
|
||||
|
||||
private static AuditLogPartitionArchive Archive(
|
||||
ScadaBridgeDbContext ctx, string dir, int retentionDays = 0, DateTime? now = null) => new(ctx, new AuditLogArchiveOptions
|
||||
{
|
||||
Enabled = true,
|
||||
Directory = dir,
|
||||
RowGroupSize = 1_000,
|
||||
RetentionDays = retentionDays,
|
||||
}, time: now is null ? null : new FixedClock(now.Value));
|
||||
|
||||
private async Task<AuditArchiveManifest> ArchiveAsync(string dir, DateTime month)
|
||||
{
|
||||
await using var ctx = CreateContext();
|
||||
var manifest = await Archive(ctx, dir).ArchivePartitionAsync(month);
|
||||
await Task.Delay(5); // distinct millisecond archive names
|
||||
return manifest;
|
||||
}
|
||||
|
||||
private async Task<AuditArchiveCleanupResult> CleanupAsync(string dir, bool dryRun, int retentionDays = 0, DateTime? now = null)
|
||||
{
|
||||
await using var ctx = CreateContext();
|
||||
return await Archive(ctx, dir, retentionDays, now).CleanupAsync(dryRun);
|
||||
}
|
||||
|
||||
private async Task InsertAsync(IEnumerable<AuditEvent> events)
|
||||
{
|
||||
await using var ctx = CreateContext();
|
||||
var repo = new AuditLogRepository(ctx);
|
||||
foreach (var e in events)
|
||||
{
|
||||
await repo.InsertIfNotExistsAsync(e);
|
||||
}
|
||||
}
|
||||
|
||||
private async Task DrainAsync()
|
||||
{
|
||||
await using var ctx = CreateContext();
|
||||
var chain = new AuditLogHashChain(ctx);
|
||||
for (var i = 0; i < 1000; i++)
|
||||
{
|
||||
var r = await chain.ChainPendingAsync(500);
|
||||
if (r.RowsChained == 0 && !r.LockBusy)
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
throw new InvalidOperationException("Chain did not drain.");
|
||||
}
|
||||
|
||||
private static DateTime Month(int year, int month) => new(year, month, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
private ScadaBridgeDbContext CreateContext()
|
||||
=> new(new DbContextOptionsBuilder<ScadaBridgeDbContext>().UseSqlServer(_fixture.ConnectionString).Options);
|
||||
|
||||
private static AuditEvent NewEvent(DateTime occurredAtUtc, AuditChannel channel = AuditChannel.ApiOutbound) =>
|
||||
ScadaBridgeAuditEventFactory.Create(
|
||||
channel: channel,
|
||||
kind: channel == AuditChannel.DbOutbound ? AuditKind.DbWrite : AuditKind.ApiCall,
|
||||
status: AuditStatus.Delivered,
|
||||
occurredAtUtc: occurredAtUtc,
|
||||
actor: "cleanup-test",
|
||||
target: "erp",
|
||||
sourceNode: "central-a",
|
||||
sourceSiteId: "cleanup-test",
|
||||
correlationId: Guid.NewGuid());
|
||||
}
|
||||
@@ -144,4 +144,43 @@ public class AuditArchiveCommandTests : TestKit
|
||||
Assert.Equal(TimeSpan.FromMinutes(5),
|
||||
ManagementEndpoints.ResolveAskTimeout(new ManagementServiceOptions(), typeof(VerifyAuditArchiveCommand)));
|
||||
}
|
||||
|
||||
/// <summary>ScadaBridge#109: an Administrator runs the cleanup; the dry-run flag reaches the archive.</summary>
|
||||
/// <param name="dryRun">The flag.</param>
|
||||
[Theory]
|
||||
[InlineData(true)]
|
||||
[InlineData(false)]
|
||||
public void Administrator_RunsCleanup_WithTheDryRunFlag(bool dryRun)
|
||||
{
|
||||
_archive.CleanupAsync(dryRun, Arg.Any<CancellationToken>()).Returns(new AuditArchiveCleanupResult(dryRun, 0,
|
||||
[new AuditArchiveCleanupItem(Name, "2025-10", AuditArchiveCleanupAction.DeletedSuperseded, "superseded")]));
|
||||
|
||||
CreateActor().Tell(Envelope(new CleanupAuditArchivesCommand(dryRun), [], "Administrator"));
|
||||
|
||||
var response = ExpectMsg<ManagementSuccess>(TimeSpan.FromSeconds(5));
|
||||
using var json = JsonDocument.Parse(response.JsonData);
|
||||
Assert.Equal(dryRun, json.RootElement.GetProperty("dryRun").GetBoolean());
|
||||
Assert.Equal("DeletedSuperseded", json.RootElement.GetProperty("items")[0].GetProperty("action").GetString());
|
||||
_archive.Received(1).CleanupAsync(dryRun, Arg.Any<CancellationToken>());
|
||||
}
|
||||
|
||||
/// <summary>Cleanup deletes files: a Viewer (audit reader) is refused.</summary>
|
||||
[Fact]
|
||||
public void Viewer_CannotRunCleanup()
|
||||
{
|
||||
CreateActor().Tell(Envelope(new CleanupAuditArchivesCommand(true), [], "Viewer"));
|
||||
|
||||
ExpectMsg<ManagementUnauthorized>(TimeSpan.FromSeconds(5));
|
||||
_archive.DidNotReceiveWithAnyArgs().CleanupAsync(default, default);
|
||||
}
|
||||
|
||||
/// <summary>No archive directory configured: ARCHIVE_NOT_CONFIGURED, as for list/verify.</summary>
|
||||
[Fact]
|
||||
public void Cleanup_WithoutAnArchiveDirectory_IsNotConfigured()
|
||||
{
|
||||
_archive.IsConfigured.Returns(false);
|
||||
CreateActor().Tell(Envelope(new CleanupAuditArchivesCommand(true), [], "Administrator"));
|
||||
|
||||
Assert.Equal("ARCHIVE_NOT_CONFIGURED", ExpectMsg<ManagementError>(TimeSpan.FromSeconds(5)).ErrorCode);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -176,6 +176,8 @@ public class RequiredRoleMatrixTests
|
||||
// AuditLog Parquet archives (ScadaBridge#42): same audit-read gate as verify-chain.
|
||||
["ListAuditArchives"] = [Roles.Administrator, Roles.Viewer],
|
||||
["VerifyAuditArchive"] = [Roles.Administrator, Roles.Viewer],
|
||||
// Archive cleanup deletes files (ScadaBridge#109): Administrator only.
|
||||
["CleanupAuditArchives"] = [Roles.Administrator],
|
||||
|
||||
// ---- Two-person secured write ----------------------------------------------
|
||||
["SubmitSecuredWrite"] = [Roles.Operator],
|
||||
|
||||
Reference in New Issue
Block a user