diff --git a/CLAUDE.md b/CLAUDE.md index 1d0dcaad..e71befef 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -183,7 +183,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - 365-day central retention with monthly partition-switch purge; per-channel retention overrides (`AuditLog:PerChannelRetentionDays`) expire rows earlier than the global window via a bounded, batched row DELETE on the purge actor's maintenance path — values must be shorter than the global window (M5.5 T3); 7-day site SQLite retention with a hard `ForwardState` invariant (no row purged until forwarded or reconciled). - Append-only enforced via DB roles (writer role has INSERT only, no UPDATE/DELETE). - **Clustered key `(OccurredAtUtc, EventId)` + aligned unique `UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc)` (migration `FlipAuditLogClusteredKey`, #111, B2 of the #78 benchmark).** The Audit Log page's newest-first page/keyset now reads an ordered range instead of the whole table (300k rows: 90,532 → 34 reads). Offline whole-table rebuild — script `docs/plans/sql/FlipAuditLogClusteredKey.sql` (+ `.rollback.sql`), both centrals stopped: the partition-switch staging table now reads the live key order, but binaries older than the flip hard-code the old one. DEV/QA not applied (sizing + window in #111). -- **Parquet cold archive (T2, ScadaBridge#42) — default OFF (`AuditLog:Archive:Enabled` + absolute `Directory`, shared by both centrals).** The purge actor writes each expired partition to `{Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet` (10 stored columns + `ChainSeq`/`RowHash`, tick-exact `OccurredAtUtcTicks`) plus `.tombstones.parquet` and a `.manifest.json` (written last; chain head at archive time, SHA-256 per file), read under the chain lock SHARED; then the switch runs with a guard (under the EXCLUSIVE lock: partition row count + each month's `LastSeq` must still equal the archive's, else THROW 51002/51003 and nothing is purged). Fail closed: a partition that cannot be archived is not purged. `audit archive list|verify` = `ListAuditArchives`/`VerifyAuditArchive` (verify-chain's gate): file hashes, chain recomputed from the archive, chain end vs the DB purge anchor (`Matches`/`Superseded`/`NotPurged`/`Mismatch`), CLI exit 3. `audit export --format parquet` streams the 24 export columns (was 501). No migration; no re-import path (open decision), object storage not wired (mount it). +- **Parquet cold archive (T2, ScadaBridge#42) — default OFF (`AuditLog:Archive:Enabled` + absolute `Directory`, shared by both centrals).** The purge actor writes each expired partition to `{Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet` (10 stored columns + `ChainSeq`/`RowHash`, tick-exact `OccurredAtUtcTicks`) plus `.tombstones.parquet` and a `.manifest.json` (written last; chain head at archive time, SHA-256 per file), read under the chain lock SHARED; then the switch runs with a guard (under the EXCLUSIVE lock: partition row count + each month's `LastSeq` must still equal the archive's, else THROW 51002/51003 and nothing is purged). Fail closed: a partition that cannot be archived is not purged. `audit archive list|verify` = `ListAuditArchives`/`VerifyAuditArchive` (verify-chain's gate): file hashes, chain recomputed from the archive, chain end vs the DB purge anchor (`Matches`/`Superseded`/`NotPurged`/`Mismatch`), CLI exit 3. `audit export --format parquet` streams the 24 export columns (was 501). No migration. **Owner decisions (#109):** destination stays a directory/UNC (no object-storage client; share + permissions named at enablement), parquet-tools-only reading (no re-import), unsigned like the chain. **Retention** `AuditLog:Archive:RetentionDays` (default 0 = never; else > `AuditLog:RetentionDays`, counted from the month's end) and **superseded stale-archive cleanup** (`DeleteSupersededStaleArchives`, default true: a retried purge's `Archived` archive is deleted only when a later `Purged`, verifying archive provably holds every row — same anchor + chain-end hash, unchained rows by EventId; else `KeptStale`) run after every purge tick; deletion touches only the archive name's own three files, manifest first; `audit archive cleanup [--dry-run]` = `CleanupAuditArchives` (Administrator). - **Hash-chain tamper evidence (T1, ScadaBridge#41) — default OFF (`AuditLog:HashChain:Enabled`).** One SHA-256 chain per UTC calendar month: `ChainSeq` + `RowHash = SHA-256(prev || canonical(row))` on `AuditLog` (migration `AddAuditLogHashChain`), linked by the central `AuditLogHashChainService` (both nodes, serialised by `sp_getapplock`) — never by the ingest path. `AuditLogChainHead` catches tail deletion and carries the partition-switch checkpoint anchor; the per-channel DELETE writes an `AuditLogChainTombstone` per chained row in the same transaction. `audit verify-chain --month` = management command `VerifyAuditChain` (OperationalAudit, unscoped callers only), first break with partition/ChainSeq/EventId, CLI exit 3. A chained row's `SourceNode` is frozen (backfill skips it). **Unsigned by owner decision (#107); external anchor = off-box head export** (`AuditLog:HashChain:HeadExport`, default off: both centrals append changed per-month heads — month, `LastSeq`/`LastHash`, purge anchor, row + tombstone counts — to append-only JSONL files `{Directory}/{yyyy}/audit-chain-heads-{date}-{node}.jsonl` or RFC 5424 syslog UDP/TCP); `audit verify-heads --path` = `VerifyAuditChainHeads` checks them against the DB (a re-hashed forgery passes verify-chain, fails verify-heads; exit 3). **Two-login hardening (#107 item 3):** `AuditLog:MaintenanceConnectionString` (`${secret:}`-capable, unset = runtime login) runs the chain job, verify/head export, partition switch-out + archive and per-channel purge on `AuditMaintenanceContext` (keyed `audit-maintenance` repository/archive); that login gets `scadabridge_audit_chainer` + `scadabridge_audit_purger`, the runtime login `scadabridge_audit_writer` (whose DENYs block chaining/purging) — pinned by `AuditMaintenanceLoginTests`. - Node-of-origin is captured alongside site-of-origin: `SourceNode` (`varchar(64)` NULL) on `AuditLog`, `Notifications`, and `SiteCalls` — `node-a`/`node-b` for site rows (qualified by `SourceSiteId`/`SourceSite`), `central-a`/`central-b` for central direct-write rows. Stamped at the writing node, carried verbatim through telemetry + reconciliation, and indexed via `IX_AuditLog_Node_Occurred (SourceNode, OccurredAtUtc)` on `AuditLog`. - Per-node stuck KPIs (M5.3 T6): Notification Outbox and Site Call Audit expose `PerNodeNotificationKpiRequest`/`PerNodeSiteCallKpiRequest` messages that group stuck/parked/delivered counts by `SourceNode`, surfacing per-node breakdowns on the Health dashboard. diff --git a/docs/components/CLI.md b/docs/components/CLI.md index 18e2e870..1954b9bf 100644 --- a/docs/components/CLI.md +++ b/docs/components/CLI.md @@ -24,8 +24,8 @@ Management commands use `POST /management`. Audit query, export, and tree use `GET /api/audit/query`, `/api/audit/export`, and `/api/audit/tree`; source-node backfill uses `POST /api/audit/backfill-source-node`. `audit verify-chain` sends the `VerifyAuditChain` management command (ScadaBridge#41); `audit verify-heads` reads off-box -exported chain heads and sends `VerifyAuditChainHeads` (ScadaBridge#107); `audit archive list|verify` send -`ListAuditArchives` / `VerifyAuditArchive` (ScadaBridge#42). `debug stream` uses a persistent +exported chain heads and sends `VerifyAuditChainHeads` (ScadaBridge#107); `audit archive list|verify|cleanup` send +`ListAuditArchives` / `VerifyAuditArchive` (ScadaBridge#42) / `CleanupAuditArchives` (ScadaBridge#109). `debug stream` uses a persistent SignalR connection. Audit export streams the response without buffering it all. ### Authentication and roles @@ -122,7 +122,7 @@ scadabridge instance deploy --id 7 | `db-connection` | `list`, `get`, `create`, `update`, `delete` | `Designer` | | `api-method` | `list`, `get`, `create`, `update`, `delete`, `validate`, `recompile` | `Designer` | | `bundle` | `export`, `preview`, `import` | Designer export; Administrator preview/import | -| `audit` | `query`, `export`, `tree`, `backfill-source-node`, `verify-chain`, `verify-heads`, `archive list`, `archive verify` | Audit roles; Administrator backfill; verify-chain, verify-heads and archive need an unscoped audit role | +| `audit` | `query`, `export`, `tree`, `backfill-source-node`, `verify-chain`, `verify-heads`, `archive list`, `archive verify`, `archive cleanup` | Audit roles; Administrator backfill and archive cleanup; verify-chain, verify-heads and archive need an unscoped audit role | | `audit-config` | `query` (config-change audit trail; was `audit-log` pre-M8) | `Admin` | | `health` | `summary`, `pending-updates`, `site`, `event-log`, `parked-messages` | Authenticated reads with server-side scope checks | | `debug` | `snapshot`, `stream` | `Deployer` | diff --git a/docs/requirements/Component-AuditLog.md b/docs/requirements/Component-AuditLog.md index cab7e42e..f3753d51 100644 --- a/docs/requirements/Component-AuditLog.md +++ b/docs/requirements/Component-AuditLog.md @@ -770,8 +770,12 @@ MS SQL for direct-write events). Unredacted secrets never persist. - **Where.** `AuditLog:Archive:Directory` — an absolute path (validated at startup) that both central nodes reach, because the purge runs on the active node and `audit archive list|verify` on whichever node serves the call. Files land in - `{Directory}/{YYYY}/`. Object storage (S3 / Azure Blob) is not called directly — - mount it as a file system or copy the files on. + `{Directory}/{YYYY}/`. **Owner decision (ScadaBridge#109): the destination stays a + directory / UNC share — no native object-storage client.** Object storage (S3 / Azure + Blob) is reached, if ever, by mounting it as a file system or copying the files on. The + share (UNC path) and its permissions are chosen when the archive is enabled on DEV/QA: + both central nodes' service account needs modify rights on it (write, rename, and — for + cleanup — delete), auditors read-only; the SQL Server login needs none. - **What.** Per partition, three files sharing the stem `auditlog-{YYYY-MM}-{archivedAt:yyyyMMddTHHmmssfff}Z`: `.parquet` (every row of the partition — the ten stored columns, `OccurredAtUtc` both as a UTC microsecond @@ -798,8 +802,33 @@ MS SQL for direct-write events). Unredacted secrets never persist. - **Retention interplay.** Archiving changes nothing about *when* a partition is purged (`RetentionDays`). Per-channel overrides still delete their rows earlier, so those rows are never archived — the archive carries their tombstones instead, - which is what keeps the archived chain verifiable. Archive files have no retention - of their own. + which is what keeps the archived chain verifiable. + - **Archive retention (ScadaBridge#109) — default never delete.** + `AuditLog:Archive:RetentionDays` (default `0` = archives are never deleted, until compliance + names a period) is counted from the END of the archived month and, when set, must exceed + `AuditLog:RetentionDays` (validated at startup) — an archive always outlives the rows' time + in the database. After each purge tick the purge actor deletes an expired archive that is + `Purged`, or an `Archived` one whose partition no longer holds rows. + - **Superseded stale archives (ScadaBridge#109).** A purge whose guard finds the partition + changed (51002/51003) leaves its archive at `Archived` (verify reports `NotPurged`, or + `Superseded` once a later purge anchored past it) and the next tick archives again. With + `AuditLog:Archive:DeleteSupersededStaleArchives` (default `true`) the purge actor deletes + such a stale archive only when a LATER archive of the same partition is `Purged`, verifies, + and provably holds every row of it: for every chain month, the same earlier anchor and the + stale archive's chain-end hash at its chain-end sequence (row or tombstone), and every + unchained row present by EventId. Otherwise it is kept and reported `KeptStale` (e.g. an + unchained row a per-channel purge removed in between — no tombstone — so the stale archive + is that row's only copy). An archive left at `Archived` because the node stopped between the + switch and the manifest update has no later purged archive and is never touched. + - **Safe deletion.** Cleanup deletes only the three files a well-formed archive name implies + (`{name}.manifest.json`, `.parquet`, `.tombstones.parquet`) in that archive's own + `{Directory}/{YYYY}` folder — never a path read from a manifest — manifest first, so an + interrupted deletion leaves leftovers that are no longer an archive (orphans of an interrupted + archive WRITE are not cleaned yet — #114). A cleanup failure is a + purge failure on the health surface. `scadabridge audit archive cleanup [--dry-run]` + (`CleanupAuditArchives`, Administrator) runs the same pass on demand or reports what it + would delete. Pinned by `AuditLogArchiveCleanupTests` (mutation-checked: each safety rule + removed fails its test). - **Verify / read.** `audit archive verify --name` (`VerifyAuditArchive`) checks both files' SHA-256 and counts against the manifest, recomputes each archived month's chain from the archived rows and tombstones (`AuditChainVerifier`, starting from @@ -807,8 +836,9 @@ MS SQL for direct-write events). Unredacted secrets never persist. `Matches`; `Superseded` (late rows for the month were chained, purged and archived again afterwards — that later archive resumes from this one's end); `NotPurged`; or `Mismatch` (archive or head altered). The files are plain Parquet — DuckDB, - pyarrow or Spark read them directly. There is no re-import into `dbo.AuditLog`: - a re-inserted expired row would be purged again by the next tick. + pyarrow, Spark or parquet-tools read them directly. **Owner decision (ScadaBridge#109): + reading with Parquet tools is the supported access path; there is no re-import into + `dbo.AuditLog`** — a re-inserted expired row would be purged again by the next tick. - **Sites:** `SiteAuditRetentionService` (site-only `IHostedService`) runs the site SQLite retention purge on a timer — first tick after `AuditLog:SiteRetention:InitialDelay` (default 5 min, short so a daily-recycled @@ -880,7 +910,7 @@ MS SQL for direct-write events). Unredacted secrets never persist. to the maintenance login: the monthly partition roll-forward (`ALTER PARTITION FUNCTION`, needs `ALTER ANY DATASPACE` on the runtime login, or run it as a DBA job) and the one-time `audit backfill-source-node` (needs `UPDATE` on `SourceNode` — a DBA-run step in a hardened - setup). Migrations themselves are applied by a DBA with the production SQL scripts. + setup) — both tracked in #113. Migrations themselves are applied by a DBA with the production SQL scripts. - **Authorization.** Reading the Audit Log requires the existing **Audit** role extended with a new **OperationalAudit** permission. Per-site row scoping reuses the existing site-permission model; bulk export requires an additional @@ -943,7 +973,7 @@ MS SQL for direct-write events). Unredacted secrets never persist. `FileMode.Append` only, one file per node per day; give the service account create + append-data rights and nothing else), or `Sink: Syslog` — RFC 5424 (`1 … chainhead - {json}`, facility 13 "log audit", severity Notice) over UDP (RFC 5426) or TCP (RFC 6587 - octet counting). A failed write is logged and retried with the same heads next tick. + octet counting). A failed write is logged and retried with the same heads next tick (no health counter yet — #115). `scadabridge audit verify-heads --path ` reads those records back (file-sink lines or a syslog collector's log) and the `VerifyAuditChainHeads` command checks each against the database: the link stored at the exported sequence (the head, a live row's @@ -961,9 +991,10 @@ MS SQL for direct-write events). Unredacted secrets never persist. - **Parquet archival (T2) — shipped (ScadaBridge#42), default OFF.** See Retention & Purge. The archive carries the hash-chain columns and the chain head, so tamper evidence survives the purge: `audit archive verify` ties an archive to the checkpoint - anchor still in `AuditLogChainHead`. Like the chain itself, archives are unsigned; - a principal able to rewrite both the archive files and the head row can forge a - consistent pair. + anchor still in `AuditLogChainHead`. Like the chain itself, archives are unsigned by owner + decision (ScadaBridge#107/#109); a principal able to rewrite both the archive files and the + head row can forge a consistent pair — unless the month's head was exported off-box first + (`AuditLog:HashChain:HeadExport`): `audit verify-heads` then exposes the rewritten anchor. - **Site SQLite security.** File permissions: read/write by the ScadaBridge service account only. Not backed up off-machine — site SQLite is a buffer, not a record. @@ -1062,7 +1093,9 @@ archive (see Retention & Purge): "Directory": "D:\\AuditArchive", // absolute; required when Enabled; reachable by both centrals "RowGroupSize": 50000, // 1,000 – 1,000,000 "ReadCommandTimeoutMinutes": 60, // reading one partition out of SQL Server - "ChainLockTimeoutSeconds": 300 // wait for a chaining pass before skipping the tick + "ChainLockTimeoutSeconds": 300, // wait for a chaining pass before skipping the tick + "RetentionDays": 0, // 0 = never delete; else > AuditLog:RetentionDays, from the archived month's end (#109) + "DeleteSupersededStaleArchives": true // delete stale archives a later purged archive provably covers (#109) } } ``` diff --git a/docs/requirements/Component-CLI.md b/docs/requirements/Component-CLI.md index f91685fb..e7dcf9bb 100644 --- a/docs/requirements/Component-CLI.md +++ b/docs/requirements/Component-CLI.md @@ -319,6 +319,7 @@ scadabridge audit verify-chain --month scadabridge audit verify-heads --path scadabridge audit archive list scadabridge audit archive verify --name +scadabridge audit archive cleanup [--dry-run] ``` - `audit query` — filtered query against the central `AuditLog` table, matching the @@ -370,6 +371,12 @@ scadabridge audit archive verify --name archived hash chain and compares its end with the purge anchor; exit `0` verified, `3` not verified, `1` error, `2` authorization failure. Same gate as `verify-chain`. See Component-AuditLog.md, Retention & Purge. +- `audit archive cleanup [--dry-run]` (ScadaBridge#109) — sends `CleanupAuditArchives`: + deletes superseded stale archives (a retried purge's, once a later purged archive provably + holds all their rows) and archives past `AuditLog:Archive:RetentionDays` (0 = never), the + same pass the purge actor runs after every tick; `--dry-run` only reports. Prints each + `DeletedSuperseded` / `DeletedExpired` / `KeptStale` with its reason; exit `0`. Administrator + only (it deletes files). ### Health Commands ``` diff --git a/docs/requirements/Component-ManagementService.md b/docs/requirements/Component-ManagementService.md index 380f486e..1379b35a 100644 --- a/docs/requirements/Component-ManagementService.md +++ b/docs/requirements/Component-ManagementService.md @@ -202,6 +202,7 @@ The two-person authorization workflow for writes through the MxAccess Gateway. B - **VerifyAuditChain** (ScadaBridge#41): verifies one UTC calendar month (`Month`, `YYYY-MM`) of the central AuditLog hash chain and returns an `AuditChainVerificationResult` (status, rows verified, tombstones applied, purge anchor, head sequence, unchained rows, first break with partition / `ChainSeq` / `EventId` / kind). Gated on the OperationalAudit roles (`Administrator`, `Viewer`); a site-scoped non-Administrator caller is refused (`ManagementUnauthorized`) because a month's chain spans every site. Long-running Ask timeout. Errors: `INVALID_ARGUMENT` (bad month), `CHAIN_BUSY` (a chaining/purge pass held the chain lock past 60 s). Backs `scadabridge audit verify-chain`. - **VerifyAuditChainHeads** (ScadaBridge#107): checks chain heads exported off-box (`Records`, at most 50,000 `AuditChainHeadRecord`s; duplicates collapsed) against the database chain and returns an `AuditChainHeadVerificationResult` (overall `Verified`, one `AuditChainHeadCheck` per distinct exported head: `Matches` / `Extended` / `Purged`, or the failures `Mismatch` / `Regressed` / `MissingHead` / `MissingLink` / `CountMismatch`). Same gate as VerifyAuditChain; long-running Ask timeout. Errors: `INVALID_ARGUMENT` (no records / over the cap), `CHAIN_BUSY`. Backs `scadabridge audit verify-heads`. - **ListAuditArchives** / **VerifyAuditArchive** (ScadaBridge#42): list the AuditLog Parquet cold archives under `AuditLog:Archive:Directory` (`AuditArchiveListResult`: enabled flag + every manifest), and verify one by `Name` (`AuditArchiveVerificationResult`: file integrity, per-month chain recomputed from the archived rows, anchor `Matches`/`Superseded`/`NotPurged`/`Mismatch`, overall `Verified`). Same gate as VerifyAuditChain (OperationalAudit roles, site-scoped non-Administrator refused). VerifyAuditArchive uses the long-running Ask timeout. Errors: `ARCHIVE_NOT_CONFIGURED`, `INVALID_ARGUMENT` (not an archive name), `NOT_FOUND`. Back `scadabridge audit archive list|verify`. +- **CleanupAuditArchives** (ScadaBridge#109): runs the archive cleanup now (`DryRun` = report only) — superseded stale archives a later purged archive provably covers, and archives past `AuditLog:Archive:RetentionDays` — and returns an `AuditArchiveCleanupResult` (`DryRun`, `RetentionDays`, one item per archive deleted / kept stale, with the action `DeletedExpired` / `DeletedSuperseded` / `KeptStale` and reason). Administrator only; runs on the maintenance-connection archive when registered; long-running Ask timeout. Errors: `ARCHIVE_NOT_CONFIGURED`. Backs `scadabridge audit archive cleanup`. ### Transport (Bundle Import / Export) diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogArchiveOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogArchiveOptionsValidator.cs index fa069df6..abe0b567 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogArchiveOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogArchiveOptionsValidator.cs @@ -32,5 +32,34 @@ public sealed class AuditLogArchiveOptionsValidator : OptionsValidatorBase 0."); builder.RequireThat(options.ChainLockTimeoutSeconds > 0, $"AuditLog:Archive:{nameof(AuditLogArchiveOptions.ChainLockTimeoutSeconds)} ({options.ChainLockTimeoutSeconds}) must be > 0."); + builder.RequireThat(options.RetentionDays is >= 0 and <= 36_500, + $"AuditLog:Archive:{nameof(AuditLogArchiveOptions.RetentionDays)} ({options.RetentionDays}) must be between 0 (never delete) and 36500."); + } +} + +/// +/// Cross-section check for (ScadaBridge#109): +/// when set, it must exceed AuditLog:RetentionDays, so an archive is never deleted before +/// (or as soon as) its rows leave the database — the archive exists to outlive them. +/// +public sealed class AuditLogArchiveRetentionValidator : Microsoft.Extensions.Options.IValidateOptions +{ + private readonly Microsoft.Extensions.Options.IOptions _auditOptions; + + /// Creates the validator. + /// The AuditLog options (global retention). + public AuditLogArchiveRetentionValidator(Microsoft.Extensions.Options.IOptions auditOptions) + => _auditOptions = auditOptions ?? throw new ArgumentNullException(nameof(auditOptions)); + + /// + public Microsoft.Extensions.Options.ValidateOptionsResult Validate(string? name, AuditLogArchiveOptions options) + { + ArgumentNullException.ThrowIfNull(options); + var global = _auditOptions.Value.RetentionDays; + return options.RetentionDays == 0 || options.RetentionDays > global + ? Microsoft.Extensions.Options.ValidateOptionsResult.Success + : Microsoft.Extensions.Options.ValidateOptionsResult.Fail( + $"AuditLog:Archive:RetentionDays ({options.RetentionDays}) must be 0 (never delete) or greater than " + + $"AuditLog:RetentionDays ({global}): archives must outlive the rows' time in the database."); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs index 33240372..bdfec5e2 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPurgeActor.cs @@ -6,6 +6,7 @@ using Microsoft.Extensions.Options; using ZB.MOM.WW.ScadaBridge.AuditLog.Configuration; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.Commons.Types; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit; using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance; namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central; @@ -195,6 +196,7 @@ public class AuditLogPurgeActor : ReceiveActor // No whole-month partitions are eligible, but per-channel overrides may // still expire rows earlier than the global window — run them below. await RunPerChannelOverridesAsync(repository, eventStream, purgeFailureCounter).ConfigureAwait(false); + await RunArchiveCleanupAsync(scope.ServiceProvider, eventStream, purgeFailureCounter).ConfigureAwait(false); return; } @@ -274,6 +276,52 @@ public class AuditLogPurgeActor : ReceiveActor // deletes only ever expire rows EARLIER than that, so they run last and are a // strict tightening. await RunPerChannelOverridesAsync(repository, eventStream, purgeFailureCounter).ConfigureAwait(false); + + // Archive housekeeping last (ScadaBridge#109): superseded stale archives this or an + // earlier tick's retried purge left behind, and archives past AuditLog:Archive:RetentionDays. + if (archive is not null) + { + await RunArchiveCleanupAsync(scope.ServiceProvider, eventStream, purgeFailureCounter, archive).ConfigureAwait(false); + } + } + + /// + /// Runs the archive cleanup pass (ScadaBridge#109) when archiving is enabled. Contained: a + /// cleanup failure is a purge failure on the health surface, never a crash of the tick. + /// + private async Task RunArchiveCleanupAsync( + IServiceProvider services, + Akka.Event.EventStream eventStream, + IAuditPurgeFailureCounter? purgeFailureCounter, + IAuditLogArchive? archive = null) + { + var sw = Stopwatch.StartNew(); + try + { + if (archive is null) + { + if (services.GetService>()?.Value.Enabled != true) + { + return; + } + + archive = services.GetKeyedService(AuditMaintenanceContext.ServiceKey) + ?? services.GetRequiredService(); + } + + var result = await archive.CleanupAsync(dryRun: false).ConfigureAwait(false); + var deleted = result.Items.Count(i => i.Action != AuditArchiveCleanupAction.KeptStale); + if (deleted > 0) + { + _logger.LogInformation("AuditLog archive cleanup deleted {Count} archive(s) in {DurationMs} ms.", deleted, sw.ElapsedMilliseconds); + } + } + catch (Exception ex) + { + sw.Stop(); + _logger.LogError(ex, "AuditLog archive cleanup failed; archives are kept and the next tick retries."); + PublishPurgeFailure(eventStream, purgeFailureCounter, DateTime.MinValue, ex, sw.ElapsedMilliseconds); + } } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/ServiceCollectionExtensions.cs index 0bc2c98b..7c1bbc12 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/ServiceCollectionExtensions.cs @@ -442,6 +442,10 @@ public static class ServiceCollectionExtensions // registered by AddConfigurationDatabase (central only) and reads these options. services.AddValidatedOptions( config, ArchiveSectionName); + // Archive retention must exceed the global AuditLog:RetentionDays (ScadaBridge#109). + services.TryAddEnumerable(ServiceDescriptor.Singleton< + Microsoft.Extensions.Options.IValidateOptions, + AuditLogArchiveRetentionValidator>()); // I1 (review): bind the two central-singleton options HERE rather than in // AddAuditLogCentralReconciliationClient. AkkaHostedService.RegisterCentralActors diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditArchiveHelpers.cs b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditArchiveHelpers.cs index 20f053ef..4f52679c 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditArchiveHelpers.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditArchiveHelpers.cs @@ -109,6 +109,43 @@ public static class AuditArchiveHelpers return verified ? 0 : NotVerifiedExitCode; } + /// + /// Renders a CleanupAuditArchives result (ScadaBridge#109). Always exit 0 — kept + /// stale archives are reported, not an error. + /// + /// The success body (an AuditArchiveCleanupResult, camelCase). + /// Resolved output format. + /// Writer for the report. + /// 0. + public static int RenderCleanup(string json, string format, TextWriter output) + { + if (!string.Equals(format, "table", StringComparison.OrdinalIgnoreCase)) + { + output.WriteLine(json); + output.Flush(); + return 0; + } + + using var doc = JsonDocument.Parse(json); + var root = doc.RootElement; + var dryRun = root.TryGetProperty("dryRun", out var d) && d.ValueKind == JsonValueKind.True; + var retention = Num(root, "retentionDays"); + output.WriteLine($"{(dryRun ? "Dry run — nothing deleted. " : "")}Archive retention: {(retention == "0" ? "never delete" : retention + " days")}"); + var items = root.TryGetProperty("items", out var i) && i.ValueKind == JsonValueKind.Array ? i.EnumerateArray().ToList() : []; + if (items.Count == 0) + { + output.WriteLine("Nothing to clean up."); + } + + foreach (var item in items) + { + output.WriteLine($"{Str(item, "action"),-18} {Str(item, "name")} {Str(item, "detail")}"); + } + + output.Flush(); + return 0; + } + private static string Str(JsonElement e, string name, string missing = "") => e.TryGetProperty(name, out var v) && v.ValueKind == JsonValueKind.String ? v.GetString() ?? missing : missing; diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditCommands.cs b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditCommands.cs index 43528803..e693b399 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditCommands.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/Commands/AuditCommands.cs @@ -11,9 +11,9 @@ namespace ZB.MOM.WW.ScadaBridge.CLI.Commands; /// GET /api/audit/tree), plus verify-chain, which sends the /// VerifyAuditChain management command (hash-chain tamper evidence, ScadaBridge#41), /// verify-heads, which checks off-box exported chain heads with -/// VerifyAuditChainHeads (ScadaBridge#107), and archive list / archive verify, -/// which send ListAuditArchives / VerifyAuditArchive (Parquet cold archive, -/// ScadaBridge#42). +/// VerifyAuditChainHeads (ScadaBridge#107), and archive list / archive verify +/// / archive cleanup, which send ListAuditArchives / VerifyAuditArchive +/// (Parquet cold archive, ScadaBridge#42) / CleanupAuditArchives (ScadaBridge#109). /// public static class AuditCommands { @@ -376,7 +376,7 @@ public static class AuditCommands private static Command BuildArchive(Option urlOption, Option formatOption, Option usernameOption, Option passwordOption) { - var group = new Command("archive") { Description = "List and verify the AuditLog Parquet cold archives" }; + var group = new Command("archive") { Description = "List, verify and clean up the AuditLog Parquet cold archives" }; var list = new Command("list") { Description = "List the AuditLog Parquet archives written by the retention purge" }; list.SetAction(async (ParseResult result) => @@ -410,6 +410,23 @@ public static class AuditCommands }); group.Add(verify); + var dryRunOption = new Option("--dry-run") { Description = "Report what would be deleted; delete nothing" }; + var cleanup = new Command("cleanup") + { + Description = "Delete superseded stale archives and archives past AuditLog:Archive:RetentionDays (Administrator)", + }; + cleanup.Add(dryRunOption); + cleanup.SetAction(async (ParseResult result) => + { + var format = AuditCommandHelpers.ResolveFormat(result, formatOption); + return await CommandHelpers.ExecuteCommandAsync( + result, urlOption, formatOption, usernameOption, passwordOption, + new CleanupAuditArchivesCommand(result.GetValue(dryRunOption)), + timeout: AuditArchiveHelpers.VerifyRequestTimeout, + onSuccess: json => AuditArchiveHelpers.RenderCleanup(json, format, Console.Out)); + }); + group.Add(cleanup); + return group; } diff --git a/src/ZB.MOM.WW.ScadaBridge.CLI/README.md b/src/ZB.MOM.WW.ScadaBridge.CLI/README.md index e499bd83..db501b6d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CLI/README.md +++ b/src/ZB.MOM.WW.ScadaBridge.CLI/README.md @@ -2136,6 +2136,13 @@ tombstones, and compares the chain end with the checkpoint anchor the purge left | `1` | Request error (`ARCHIVE_NOT_CONFIGURED`, `NOT_FOUND`, `INVALID_ARGUMENT`, server error) | | `2` | Authorization failure (needs `Administrator` or `Viewer`, not site-scoped) | +`audit archive cleanup [--dry-run]` (ScadaBridge#109, Administrator) sends +`CleanupAuditArchives`: it deletes stale archives a retried purge left behind once a later +purged archive provably holds all their rows (`DeletedSuperseded`), and archives whose month +ended more than `AuditLog:Archive:RetentionDays` ago (`DeletedExpired`; default `0` = never), +and lists stale archives it kept (`KeptStale`) with the reason. `--dry-run` deletes nothing. +The purge actor runs the same pass after every purge tick. Exit code `0`. + The archive files are plain Parquet; read them with any Parquet tool, e.g. DuckDB: `SELECT * FROM '/2025/auditlog-2025-10-….parquet'`. See `Component-AuditLog.md`, Retention & Purge. diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogArchive.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogArchive.cs index 3fff5186..8bb2714f 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogArchive.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IAuditLogArchive.cs @@ -66,6 +66,17 @@ public interface IAuditLogArchive /// The verification result. /// No archive has that name. Task VerifyAsync(string name, CancellationToken ct = default); + + /// + /// Archive cleanup (ScadaBridge#109): deletes superseded stale archives a retried purge left + /// behind (when AuditLog:Archive:DeleteSupersededStaleArchives), and archives past + /// AuditLog:Archive:RetentionDays (0 = never). Only ever deletes the three files of a + /// well-formed archive inside the configured directory, manifest first. + /// + /// Report only; delete nothing. + /// Cancellation token. + /// What was (or would be) deleted, and the stale archives kept. + Task CleanupAsync(bool dryRun, CancellationToken ct = default); } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/AuditLogCommands.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/AuditLogCommands.cs index 4f0be926..6dd9564e 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/AuditLogCommands.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/AuditLogCommands.cs @@ -57,3 +57,12 @@ public record ListAuditArchivesCommand; /// /// The archive name, as audit archive list reports it. public record VerifyAuditArchiveCommand(string Name); + +/// +/// Runs the AuditLog archive cleanup (ScadaBridge#109) now: deletes superseded stale archives +/// a retried purge left behind and archives past AuditLog:Archive:RetentionDays — or, +/// with , only reports what it would delete. The purge actor runs the +/// same pass after every purge tick. Administrator only (it deletes files). +/// +/// Report only; delete nothing. +public record CleanupAuditArchivesCommand(bool DryRun); diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Audit/AuditArchive.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Audit/AuditArchive.cs index e3a762d8..36ede4fa 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Audit/AuditArchive.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Types/Audit/AuditArchive.cs @@ -145,3 +145,41 @@ public sealed record AuditArchiveVerificationResult( /// Whether the retention purge is archiving partitions (AuditLog:Archive:Enabled). /// Every archive manifest, oldest partition first. public sealed record AuditArchiveListResult(bool Enabled, IReadOnlyList Archives); + +/// What archive cleanup did (or, on a dry run, would do) with one archive (ScadaBridge#109). +[JsonConverter(typeof(JsonStringEnumConverter))] +public enum AuditArchiveCleanupAction +{ + /// + /// Deleted by retention: the archived month ended more than AuditLog:Archive:RetentionDays + /// ago, and the archive is purged (or its partition no longer holds rows). + /// + DeletedExpired, + + /// + /// Deleted as a superseded stale archive: status Archived (its partition switch never + /// completed — a retried purge), and a LATER, verified, Purged archive of the same + /// partition provably holds every row it holds (same chain through its chain end, every + /// unchained row present by EventId). + /// + DeletedSuperseded, + + /// + /// A stale Archived archive that was kept: no later purged archive provably covers it, + /// so it may be the only copy of some rows (see Detail). + /// + KeptStale, +} + +/// One archive considered by cleanup. +/// Archive name. +/// Archived partition, YYYY-MM. +/// What was (or would be) done. +/// Why. +public sealed record AuditArchiveCleanupItem(string Name, string Partition, AuditArchiveCleanupAction Action, string Detail); + +/// Result of an archive cleanup pass (ScadaBridge#109). +/// True when nothing was deleted (the items say what would be). +/// The archive retention in force (0 = archives never expire). +/// The archives deleted (or that would be) and the stale ones kept. +public sealed record AuditArchiveCleanupResult(bool DryRun, int RetentionDays, IReadOnlyList Items); diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Maintenance/AuditLogArchiveOptions.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Maintenance/AuditLogArchiveOptions.cs index 8042c72d..839de381 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Maintenance/AuditLogArchiveOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Maintenance/AuditLogArchiveOptions.cs @@ -43,4 +43,19 @@ public sealed class AuditLogArchiveOptions /// the hash-chain lock before giving up for this tick (default 300). /// public int ChainLockTimeoutSeconds { get; set; } = 300; + + /// + /// Archive retention in days, counted from the END of the archived month (ScadaBridge#109). + /// 0 (the default) = archives are never deleted — until compliance names a period. + /// When set it must exceed AuditLog:RetentionDays, so an archive always outlives the + /// rows' time in the database. Applied by the purge actor after each tick's partition purge. + /// + public int RetentionDays { get; set; } + + /// + /// Delete stale archives a retried purge left behind (status Archived) once a later, + /// verified, purged archive of the same partition provably holds all their rows (default + /// true; ScadaBridge#109). A stale archive that cannot be proven redundant is kept. + /// + public bool DeleteSupersededStaleArchives { get; set; } = true; } diff --git a/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Maintenance/AuditLogPartitionArchive.Cleanup.cs b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Maintenance/AuditLogPartitionArchive.Cleanup.cs new file mode 100644 index 00000000..9a438bba --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Maintenance/AuditLogPartitionArchive.Cleanup.cs @@ -0,0 +1,290 @@ +using System.Data; +using Microsoft.Data.SqlClient; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.Logging; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit; + +namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance; + +/// +/// Archive cleanup (ScadaBridge#109): retention and superseded-stale-archive deletion. +/// +/// +/// +/// Superseded stale archives. A purge whose guard finds the partition changed after it +/// was archived (51002/51003) purges nothing and leaves its archive at status +/// ; the next tick writes a fresh archive and purges +/// with it. The stale one is redundant only if a LATER archive B of the same partition, which +/// is and verifies, holds everything the stale archive A +/// holds. That is proven, not assumed: for every chain month of A, B starts from the same +/// earlier purge anchor and carries A's chain-end hash at A's chain-end sequence (as a row or a +/// tombstone) — so, the chain being a hash chain B itself verifies, every chained row of A is +/// in B; and every UNCHAINED row of A is found in B (row or tombstone) by EventId. Anything +/// else — notably an unchained row a per-channel purge removed in between, which leaves no +/// tombstone — keeps A (). An archive left at +/// Archived because the node stopped between the switch and the manifest update has no +/// later purged archive and is kept: it is the only copy. +/// +/// +/// Retention. With > 0, an archive +/// whose month ended more than that many days ago is deleted when it is Purged, or — +/// an Archived one — when its partition no longer holds rows in dbo.AuditLog. +/// +/// +/// Safe deletion. Only the three files a well-formed archive name implies +/// ({name}.manifest.json, {name}.parquet, {name}.tombstones.parquet) in the +/// archive's own {Directory}/{YYYY} folder are deleted — never a path read from the +/// manifest — and the manifest goes FIRST, so an interrupted deletion leaves leftovers that are +/// no longer an archive rather than a manifest naming missing files. +/// +/// +public sealed partial class AuditLogPartitionArchive +{ + /// + /// Upper bound on unchained rows compared by EventId when proving a stale archive redundant; + /// a larger set keeps the archive rather than spend the memory. + /// + internal const long MaxUnchainedRowsCompared = 5_000_000; + + /// + public async Task CleanupAsync(bool dryRun, CancellationToken ct = default) + { + RequireConfigured(); + var items = new List(); + var manifests = await ListAsync(ct).ConfigureAwait(false); + var deleted = new HashSet(StringComparer.Ordinal); + + if (_options.DeleteSupersededStaleArchives) + { + foreach (var stale in manifests.Where(m => m.Status == AuditArchiveStatus.Archived)) + { + var later = manifests + .Where(m => m.PartitionBoundaryUtc == stale.PartitionBoundaryUtc + && m.Status == AuditArchiveStatus.Purged + && m.ArchivedAtUtc > stale.ArchivedAtUtc) + .OrderBy(m => m.ArchivedAtUtc) + .ToList(); + if (later.Count == 0) + { + continue; // not superseded (e.g. the node stopped before marking it Purged) — keep + } + + string? reason = null; + AuditArchiveManifest? coveredBy = null; + foreach (var candidate in later) + { + reason = await ProveCoveredAsync(stale, candidate, ct).ConfigureAwait(false); + if (reason is null) + { + coveredBy = candidate; + break; + } + } + + if (coveredBy is null) + { + items.Add(new AuditArchiveCleanupItem(stale.Name, stale.Partition, AuditArchiveCleanupAction.KeptStale, + $"No later purged archive provably covers it: {reason}")); + continue; + } + + if (!dryRun) + { + DeleteArchiveFiles(stale); + } + + deleted.Add(stale.Name); + items.Add(new AuditArchiveCleanupItem(stale.Name, stale.Partition, AuditArchiveCleanupAction.DeletedSuperseded, + $"Superseded by {coveredBy.Name}, which is purged, verifies and holds every row of this archive.")); + } + } + + if (_options.RetentionDays > 0) + { + var now = _time.GetUtcNow().UtcDateTime; + foreach (var manifest in manifests.Where(m => !deleted.Contains(m.Name))) + { + var monthEnd = AuditChainHasher.MonthOf(manifest.PartitionBoundaryUtc).AddMonths(1); + if (monthEnd.AddDays(_options.RetentionDays) > now) + { + continue; + } + + if (manifest.Status != AuditArchiveStatus.Purged + && await PartitionHasRowsAsync(manifest.PartitionBoundaryUtc, ct).ConfigureAwait(false)) + { + continue; // the partition was never switched out; this archive is not a sole copy, nor expired data gone + } + + if (!dryRun) + { + DeleteArchiveFiles(manifest); + } + + deleted.Add(manifest.Name); + items.Add(new AuditArchiveCleanupItem(manifest.Name, manifest.Partition, AuditArchiveCleanupAction.DeletedExpired, + $"{manifest.Partition} ended {monthEnd:yyyy-MM-dd}, more than {_options.RetentionDays} days ago (AuditLog:Archive:RetentionDays).")); + } + } + + foreach (var item in items.Where(i => i.Action != AuditArchiveCleanupAction.KeptStale)) + { + _logger.LogInformation("AuditLog archive cleanup{DryRun}: {Action} {Archive} — {Detail}", + dryRun ? " (dry run)" : "", item.Action, item.Name, item.Detail); + } + + return new AuditArchiveCleanupResult(dryRun, _options.RetentionDays, items); + } + + /// + /// Proves holds every row of ; returns null + /// when it does, else why not. + /// + private async Task ProveCoveredAsync(AuditArchiveManifest stale, AuditArchiveManifest later, CancellationToken ct) + { + // Same chain generation and at least as far, for every chain month the stale archive has. + var laterChains = later.Chains.ToDictionary(c => c.Partition, StringComparer.Ordinal); + var targets = new Dictionary<(DateTime Month, long Seq), string>(); + foreach (var chain in stale.Chains) + { + if (!laterChains.TryGetValue(chain.Partition, out var next)) + { + return $"{later.Name} has no chain for {chain.Partition}."; + } + + if (next.PriorPurgedThroughSeq != chain.PriorPurgedThroughSeq || next.LastSeq < chain.LastSeq) + { + return $"{later.Name}'s chain for {chain.Partition} does not extend this archive's (prior anchor or end differs)."; + } + + if (chain.LastSeq > (chain.PriorPurgedThroughSeq ?? 0)) + { + targets[(AuditArchiveParquet.ParseMonth(chain.Partition), chain.LastSeq)] = chain.LastHash.ToLowerInvariant(); + } + } + + if (stale.UnchainedRowCount > MaxUnchainedRowsCompared) + { + return $"{stale.UnchainedRowCount} unchained rows exceed the {MaxUnchainedRowsCompared} compared by EventId."; + } + + // The later archive must itself be intact and tied to the database. + var verification = await VerifyAsync(later.Name, ct).ConfigureAwait(false); + if (!verification.Verified) + { + return $"{later.Name} does not verify ({verification.FileProblem ?? "chain or anchor problem"})."; + } + + var directory = ArchiveDirectory(stale); + var unchained = new HashSet(); + await using (var input = File.OpenRead(Path.Combine(directory, stale.Name + ".parquet"))) + { + await foreach (var row in AuditArchiveParquet.ReadRowsAsync(input, ct).ConfigureAwait(false)) + { + if (row.ChainSeq is null) + { + unchained.Add(row.EventId); + } + } + } + + var laterDirectory = ArchiveDirectory(later); + await using (var input = File.OpenRead(Path.Combine(laterDirectory, later.Name + ".parquet"))) + { + await foreach (var row in AuditArchiveParquet.ReadRowsAsync(input, ct).ConfigureAwait(false)) + { + unchained.Remove(row.EventId); + if (row.ChainSeq is { } seq + && targets.TryGetValue((AuditChainHasher.MonthOf(row.OccurredAtUtc), seq), out var expected)) + { + if (row.RowHash is null || !string.Equals(Convert.ToHexStringLower(row.RowHash), expected, StringComparison.Ordinal)) + { + return $"{later.Name} carries a different hash at the stale archive's chain end."; + } + + targets.Remove((AuditChainHasher.MonthOf(row.OccurredAtUtc), seq)); + } + } + } + + await using (var input = File.OpenRead(Path.Combine(laterDirectory, later.Name + ".tombstones.parquet"))) + { + await foreach (var tombstone in AuditArchiveParquet.ReadTombstonesAsync(input, ct).ConfigureAwait(false)) + { + unchained.Remove(tombstone.EventId); + if (targets.TryGetValue((tombstone.ChainMonth, tombstone.ChainSeq), out var expected)) + { + if (!string.Equals(Convert.ToHexStringLower(tombstone.RowHash), expected, StringComparison.Ordinal)) + { + return $"{later.Name} carries a different hash at the stale archive's chain end."; + } + + targets.Remove((tombstone.ChainMonth, tombstone.ChainSeq)); + } + } + } + + if (targets.Count > 0) + { + return $"{later.Name} does not contain the stale archive's chain end."; + } + + return unchained.Count == 0 + ? null + : $"{unchained.Count} unchained row(s) of this archive are not in {later.Name} (e.g. removed by a per-channel purge, which leaves no tombstone for an unchained row)."; + } + + private async Task PartitionHasRowsAsync(DateTime boundary, CancellationToken ct) + { + var conn = _context.Database.GetDbConnection(); + var openedHere = await AuditLogHashChain.OpenAsync(conn, ct).ConfigureAwait(false); + try + { + await using var cmd = conn.CreateCommand(); + cmd.CommandText = "SELECT CASE WHEN EXISTS (SELECT 1 FROM dbo.AuditLog " + + "WHERE $partition.pf_AuditLog_Month(OccurredAtUtc) = $partition.pf_AuditLog_Month(@boundary)) THEN 1 ELSE 0 END;"; + cmd.Parameters.Add(new SqlParameter("@boundary", SqlDbType.DateTime2) { Scale = 7, Value = boundary }); + return Convert.ToInt32(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false)) == 1; + } + finally + { + if (openedHere) + { + await conn.CloseAsync().ConfigureAwait(false); + } + } + } + + private string ArchiveDirectory(AuditArchiveManifest manifest) + { + if (!IsValidName(manifest.Name)) + { + throw new InvalidOperationException($"'{manifest.Name}' is not an archive name."); + } + + return Path.Combine(_options.Directory!, manifest.Name.Substring("auditlog-".Length, 4)); + } + + /// + /// Deletes exactly the archive's three files, manifest first. File names are derived from + /// the validated archive name, never taken from the manifest. + /// + private void DeleteArchiveFiles(AuditArchiveManifest manifest) + { + var directory = Path.GetFullPath(ArchiveDirectory(manifest)); + var root = Path.GetFullPath(_options.Directory!); + if (!directory.StartsWith(root, StringComparison.Ordinal)) + { + throw new InvalidOperationException($"Archive {manifest.Name} resolves outside {root}."); + } + + foreach (var file in new[] { ".manifest.json", ".parquet", ".tombstones.parquet" }) + { + var path = Path.Combine(directory, manifest.Name + file); + if (File.Exists(path)) + { + File.Delete(path); + } + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs b/src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs index 6f18874a..783d6722 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs @@ -398,6 +398,8 @@ public class ManagementActor : ReceiveActor // AuditLog Parquet archives (ScadaBridge#42): audit reads with verify-chain's gate — // OperationalAudit roles, site-scoped callers refused in the handler. ListAuditArchivesCommand => OperationalAuditReaders, + // Archive cleanup deletes archive files (ScadaBridge#109): Administrator only. + CleanupAuditArchivesCommand => AdminOnly, VerifyAuditArchiveCommand => OperationalAuditReaders, // Read-only queries -- any authenticated user @@ -574,6 +576,7 @@ public class ManagementActor : ReceiveActor VerifyAuditChainHeadsCommand cmd => await HandleVerifyAuditChainHeads(sp, cmd, user), ListAuditArchivesCommand => await HandleListAuditArchives(sp, user), VerifyAuditArchiveCommand cmd => await HandleVerifyAuditArchive(sp, cmd, user), + CleanupAuditArchivesCommand cmd => await HandleCleanupAuditArchives(sp, cmd, user), // Health GetHealthSummaryCommand => HandleGetHealthSummary(sp), @@ -3340,6 +3343,20 @@ public class ManagementActor : ReceiveActor } } + /// + /// Runs the archive cleanup now (ScadaBridge#109), on the maintenance-connection archive when + /// registered. Dry run reports only. + /// + private static async Task HandleCleanupAuditArchives( + IServiceProvider sp, CleanupAuditArchivesCommand cmd, AuthenticatedUser user) + { + RequireAuditArchive(sp, user); + var archive = sp.GetKeyedService( + ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance.AuditMaintenanceContext.ServiceKey) + ?? sp.GetRequiredService(); + return await archive.CleanupAsync(cmd.DryRun); + } + private static IAuditLogArchive RequireAuditArchive(IServiceProvider sp, AuthenticatedUser user) { if (user.PermittedSiteIds.Length > 0 diff --git a/src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementEndpoints.cs b/src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementEndpoints.cs index cb3dff59..69e0b1d9 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementEndpoints.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementEndpoints.cs @@ -69,6 +69,8 @@ public static class ManagementEndpoints // AuditLog archive verification (ScadaBridge#42) hashes and re-chains a whole month's // Parquet archive. typeof(VerifyAuditArchiveCommand), + // Archive cleanup proves stale archives redundant by reading Parquet files (ScadaBridge#109). + typeof(CleanupAuditArchivesCommand), ]; /// diff --git a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogArchiveRetentionValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogArchiveRetentionValidatorTests.cs new file mode 100644 index 00000000..a29d4096 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogArchiveRetentionValidatorTests.cs @@ -0,0 +1,82 @@ +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.ScadaBridge.AuditLog.Central; +using ZB.MOM.WW.ScadaBridge.AuditLog.Configuration; +using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance; + +namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Central; + +/// +/// AuditLog:Archive:RetentionDays (ScadaBridge#109): 0 = never delete; otherwise in range +/// and greater than the global AuditLog:RetentionDays. +/// +public sealed class AuditLogArchiveRetentionValidatorTests +{ + /// Range check in the section validator. + /// RetentionDays. + /// Expected outcome. + [Theory] + [InlineData(0, true)] + [InlineData(400, true)] + [InlineData(36_500, true)] + [InlineData(-1, false)] + [InlineData(36_501, false)] + public void SectionValidator_ChecksTheRange(int days, bool valid) + { + var result = new AuditLogArchiveOptionsValidator().Validate(null, new AuditLogArchiveOptions { RetentionDays = days }); + Assert.Equal(valid, result.Succeeded); + } + + /// Cross-section check against the global retention (365 here). + /// RetentionDays. + /// Expected outcome. + [Theory] + [InlineData(0, true)] + [InlineData(366, true)] + [InlineData(365, false)] + [InlineData(30, false)] + public void RetentionValidator_RequiresMoreThanTheGlobalRetention(int days, bool valid) + { + var validator = new AuditLogArchiveRetentionValidator(Options.Create(new AuditLogOptions { RetentionDays = 365 })); + var result = validator.Validate(null, new AuditLogArchiveOptions { RetentionDays = days }); + Assert.Equal(valid, result.Succeeded); + if (!valid) + { + Assert.Contains("AuditLog:RetentionDays (365)", result.FailureMessage); + } + } + + /// The cross-section check is wired into the central registration. + /// Archive RetentionDays. + /// Whether binding succeeds. + [Theory] + [InlineData("0", true)] + [InlineData("400", true)] + [InlineData("100", false)] + public void CentralRegistration_RejectsAnArchiveRetentionBelowTheGlobalOne(string days, bool valid) + { + var config = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["AuditLog:SiteWriter:DatabasePath"] = ":memory:", + ["AuditLog:RetentionDays"] = "365", + ["AuditLog:Archive:RetentionDays"] = days, + }) + .Build(); + var services = new ServiceCollection(); + services.AddAuditLog(config); + services.AddAuditLogCentralMaintenance(config); + using var provider = services.BuildServiceProvider(); + + var resolve = () => provider.GetRequiredService>().Value; + if (valid) + { + Assert.Equal(int.Parse(days), resolve().RetentionDays); + } + else + { + Assert.Contains("AuditLog:Archive:RetentionDays (100)", Assert.Throws(resolve).Message); + } + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogPurgeActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogPurgeActorTests.cs index 23b9cc5e..b4bdb096 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogPurgeActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Central/AuditLogPurgeActorTests.cs @@ -822,4 +822,65 @@ public class AuditLogPurgeActorTests : TestKit, IClassFixture + /// ScadaBridge#109: with archiving enabled every tick ends with the archive cleanup pass — + /// also on a tick with no expired partition — and never runs it when archiving is off. + /// + /// Whether the tick has an expired partition to purge. + [Theory] + [InlineData(true)] + [InlineData(false)] + public void ArchiveEnabled_RunsTheCleanupPass_EveryTick(bool withBoundary) + { + var boundary = new DateTime(2025, 10, 1, 0, 0, 0, DateTimeKind.Utc); + var repo = new RecordingRepo { Boundaries = withBoundary ? [boundary] : [] }; + var archive = Substitute.For(); + var manifest = ManifestFor(boundary); + archive.ArchivePartitionAsync(boundary, Arg.Any()).Returns(manifest); + archive.CleanupAsync(false, Arg.Any()).Returns(new AuditArchiveCleanupResult(false, 0, + [new AuditArchiveCleanupItem("auditlog-2025-10-20261109T120000000Z", "2025-10", AuditArchiveCleanupAction.DeletedSuperseded, "x")])); + + var actor = CreateArchivingActor(repo, archive, enabled: true); + actor.Tell(AuditLogPurgeActor.PurgeTick.Instance); + + AwaitAssert(() => archive.Received().CleanupAsync(false, Arg.Any()), + duration: TimeSpan.FromSeconds(5), interval: TimeSpan.FromMilliseconds(50)); + StopPurgeActor(actor); + } + + /// A cleanup failure is contained and surfaces as a purge failure. + [Fact] + public void ArchiveCleanupFailure_IsAPurgeFailure() + { + var repo = new RecordingRepo(); + var archive = Substitute.For(); + archive.CleanupAsync(false, Arg.Any()) + .Returns>(_ => throw new IOException("share offline")); + var failed = CreateTestProbe(); + Sys.EventStream.Subscribe(failed.Ref, typeof(AuditLogPurgeFailedEvent)); + + var actor = CreateArchivingActor(repo, archive, enabled: true); + actor.Tell(AuditLogPurgeActor.PurgeTick.Instance); + + var evt = failed.ExpectMsg(TimeSpan.FromSeconds(5)); + Assert.Contains("share offline", evt.Error); + Assert.Equal(DateTime.MinValue, evt.MonthBoundary); + StopPurgeActor(actor); + } + + /// Archiving off: no cleanup pass. + [Fact] + public void ArchiveDisabled_RunsNoCleanup() + { + var repo = new RecordingRepo(); + var archive = Substitute.For(); + var actor = CreateArchivingActor(repo, archive, enabled: false); + actor.Tell(AuditLogPurgeActor.PurgeTick.Instance); + + AwaitAssert(() => Assert.NotEmpty(repo.ThresholdQueries), + duration: TimeSpan.FromSeconds(5), interval: TimeSpan.FromMilliseconds(50)); + archive.DidNotReceiveWithAnyArgs().CleanupAsync(default, default); + StopPurgeActor(actor); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/Commands/AuditArchiveCommandTests.cs b/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/Commands/AuditArchiveCommandTests.cs index 8c0b4651..0f283c6d 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/Commands/AuditArchiveCommandTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/Commands/AuditArchiveCommandTests.cs @@ -30,13 +30,13 @@ public class AuditArchiveCommandTests "\"rowsRead\":11,\"chains\":[{\"chain\":{\"partition\":\"2025-10\",\"status\":\"Intact\",\"rowsVerified\":10," + "\"tombstonesApplied\":0,\"unchainedRows\":0},\"anchor\":\"Mismatch\",\"anchorDetail\":\"hash differs\"}]}"; - /// The audit group carries the archive group with list + verify. + /// The audit group carries the archive group with list + verify + cleanup (ScadaBridge#109). [Fact] - public void AuditGroup_HasArchiveListAndVerify() + public void AuditGroup_HasArchiveListVerifyAndCleanup() { var root = AuditCommandTestHarness.BuildRoot(); var archive = root.Subcommands.Single(c => c.Name == "audit").Subcommands.Single(c => c.Name == "archive"); - Assert.Equal(["list", "verify"], archive.Subcommands.Select(c => c.Name).Order()); + Assert.Equal(["cleanup", "list", "verify"], archive.Subcommands.Select(c => c.Name).Order()); } /// list sends ListAuditArchives and prints the archives. @@ -78,6 +78,34 @@ public class AuditArchiveCommandTests Assert.Equal(Name, name); } + /// + /// cleanup (ScadaBridge#109) sends CleanupAuditArchives with the dry-run flag and + /// reports each action; it always exits 0. + /// + /// Whether --dry-run is passed. + /// A task that represents the asynchronous operation. + [Theory] + [InlineData(true)] + [InlineData(false)] + public async Task Cleanup_SendsCleanupAuditArchives_WithTheDryRunFlag(bool dryRun) + { + var body = "{\"dryRun\":" + (dryRun ? "true" : "false") + ",\"retentionDays\":0,\"items\":[{\"name\":\"" + Name + + "\",\"partition\":\"2025-10\",\"action\":\"DeletedSuperseded\",\"detail\":\"superseded by x\"}]}"; + string[] args = dryRun + ? ["audit", "archive", "cleanup", "--dry-run", "--format", "table"] + : ["audit", "archive", "cleanup", "--format", "table"]; + var (exit, output, request) = await RunAsync(body, args); + + Assert.Equal(0, exit); + Assert.Contains("DeletedSuperseded", output); + Assert.Contains("never delete", output); + Assert.Equal(dryRun, output.Contains("Dry run", StringComparison.Ordinal)); + using var payload = JsonDocument.Parse(request); + Assert.Equal("CleanupAuditArchives", payload.RootElement.GetProperty("command").GetString()); + Assert.Equal(dryRun, payload.RootElement.GetProperty("payload").EnumerateObject() + .First(p => string.Equals(p.Name, "dryRun", StringComparison.OrdinalIgnoreCase)).Value.GetBoolean()); + } + /// verify without --name is a usage error. [Fact] public void Verify_MissingName_ProducesRequiredFlagError() diff --git a/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/Commands/AuditCommandsScaffoldTests.cs b/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/Commands/AuditCommandsScaffoldTests.cs index c237c5ba..2bd680d7 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/Commands/AuditCommandsScaffoldTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/Commands/AuditCommandsScaffoldTests.cs @@ -62,10 +62,10 @@ public class AuditCommandsScaffoldTests [Fact] public void Audit_EveryLeafCommand_HasAnAction() { - // `archive` is a group (list / verify, ScadaBridge#42): check its leaves, not it. + // `archive` is a group (list / verify, ScadaBridge#42; cleanup, ScadaBridge#109): check its leaves, not it. var audit = BuildAudit(); var leaves = audit.Subcommands.SelectMany(c => c.Subcommands.Count == 0 ? [c] : c.Subcommands).ToList(); - Assert.Equal(8, leaves.Count); // + verify-heads (ScadaBridge#107) + Assert.Equal(9, leaves.Count); // + verify-heads (ScadaBridge#107), archive cleanup (ScadaBridge#109) Assert.All(leaves, sub => Assert.True(sub.Action != null, $"Leaf command '{sub.Name}' has no action.")); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/ManagementSurfaceParityTests.cs b/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/ManagementSurfaceParityTests.cs index 53ff80be..f50e9dcd 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/ManagementSurfaceParityTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/ManagementSurfaceParityTests.cs @@ -45,7 +45,7 @@ public class ManagementSurfaceParityTests } private static readonly HashSet AuditManagementLeaves = - ["audit verify-chain", "audit verify-heads", "audit archive list", "audit archive verify"]; + ["audit verify-chain", "audit verify-heads", "audit archive list", "audit archive verify", "audit archive cleanup"]; private static async Task> ProbeProductionTree(string? omitGroup = null) { diff --git a/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Maintenance/AuditLogArchiveCleanupTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Maintenance/AuditLogArchiveCleanupTests.cs new file mode 100644 index 00000000..2f6641be --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Maintenance/AuditLogArchiveCleanupTests.cs @@ -0,0 +1,296 @@ +using System.Text.Json; +using Microsoft.EntityFrameworkCore; +using ZB.MOM.WW.Audit; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance; +using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories; +using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests.Migrations; + +namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests.Maintenance; + +/// +/// SQL Server tests for AuditLog archive cleanup (ScadaBridge#109): a stale archive a retried +/// purge left behind is deleted only once a later purged archive provably holds all its rows; +/// a stale archive with a row no later archive holds, or a sole copy left at Archived, is +/// kept; retention deletes only expired archives whose partition is gone; deletion never +/// follows a path out of the archive folder. +/// +/// Each test owns one month of the shared fixture database and its own archive directory. +public sealed class AuditLogArchiveCleanupTests : IClassFixture, IDisposable +{ + private readonly MsSqlMigrationFixture _fixture; + private readonly DirectoryInfo _root = Directory.CreateTempSubdirectory("auditlog-cleanup-"); + + /// Creates the test class with the shared migrated SQL Server fixture. + /// The shared migration fixture. + public AuditLogArchiveCleanupTests(MsSqlMigrationFixture fixture) => _fixture = fixture; + + /// + public void Dispose() => _root.Delete(recursive: true); + + /// + /// The retried purge: archive A goes stale (a row arrives and the chain advances), B purges. + /// Dry run reports A as superseded and deletes nothing; the real pass deletes exactly A's + /// three files and B still verifies. + /// + /// A task that represents the asynchronous operation. + [SkippableFact] + public async Task RetriedPurge_StaleArchiveIsDeleted_OnceALaterPurgedArchiveCoversIt() + { + Skip.IfNot(_fixture.Available, _fixture.SkipReason); + var dir = Dir("retried"); + var month = Month(2027, 1); + await InsertAsync(Enumerable.Range(1, 5).Select(i => NewEvent(month.AddDays(i)))); + await DrainAsync(); + await InsertAsync([NewEvent(month.AddDays(7))]); // unchained in A + var stale = await ArchiveAsync(dir, month); + Assert.Equal(1, stale.UnchainedRowCount); + + await InsertAsync([NewEvent(month.AddDays(9))]); + await DrainAsync(); // A's unchained row is now chained; the chain moved past A's end + await using (var ctx = CreateContext()) + { + await Assert.ThrowsAsync(() => Archive(ctx, dir).PurgeArchivedPartitionAsync(stale)); + } + + var fresh = await ArchiveAsync(dir, month); + await using (var ctx = CreateContext()) + { + Assert.Equal(7, await Archive(ctx, dir).PurgeArchivedPartitionAsync(fresh)); + } + + var dry = await CleanupAsync(dir, dryRun: true); + var item = Assert.Single(dry.Items); + Assert.Equal((stale.Name, AuditArchiveCleanupAction.DeletedSuperseded), (item.Name, item.Action)); + Assert.True(File.Exists(ManifestPath(dir, stale))); + + var real = await CleanupAsync(dir, dryRun: false); + Assert.Equal(AuditArchiveCleanupAction.DeletedSuperseded, Assert.Single(real.Items).Action); + Assert.False(File.Exists(ManifestPath(dir, stale))); + Assert.False(File.Exists(Path.Combine(dir, "2027", stale.Name + ".parquet"))); + Assert.False(File.Exists(Path.Combine(dir, "2027", stale.Name + ".tombstones.parquet"))); + + await using (var ctx = CreateContext()) + { + var listed = Assert.Single(await Archive(ctx, dir).ListAsync()); + Assert.Equal(fresh.Name, listed.Name); + Assert.True((await Archive(ctx, dir).VerifyAsync(fresh.Name)).Verified); + } + + Assert.Empty((await CleanupAsync(dir, dryRun: false)).Items); + } + + /// + /// A stale archive holding an unchained row that a per-channel purge removed before the + /// later archive (no tombstone for an unchained row) is the only copy of that row: kept. + /// + /// A task that represents the asynchronous operation. + [SkippableFact] + public async Task StaleArchive_WithARowNoLaterArchiveHolds_IsKept() + { + Skip.IfNot(_fixture.Available, _fixture.SkipReason); + var dir = Dir("kept"); + var month = Month(2027, 3); + await InsertAsync(Enumerable.Range(1, 3).Select(i => NewEvent(month.AddDays(i)))); + await DrainAsync(); + await InsertAsync([NewEvent(month.AddHours(1), AuditChannel.DbOutbound)]); // unchained + var stale = await ArchiveAsync(dir, month); + + await using (var ctx = CreateContext()) + { + Assert.Equal(1, await new AuditLogRepository(ctx).PurgeChannelOlderThanAsync("DbOutbound", month.AddDays(1), 100)); + await Assert.ThrowsAsync(() => Archive(ctx, dir).PurgeArchivedPartitionAsync(stale)); + } + + var fresh = await ArchiveAsync(dir, month); + await using (var ctx = CreateContext()) + { + await Archive(ctx, dir).PurgeArchivedPartitionAsync(fresh); + } + + var result = await CleanupAsync(dir, dryRun: false); + + var kept = Assert.Single(result.Items); + Assert.Equal((stale.Name, AuditArchiveCleanupAction.KeptStale), (kept.Name, kept.Action)); + Assert.Contains("unchained row", kept.Detail); + Assert.True(File.Exists(ManifestPath(dir, stale))); + } + + /// + /// An archive left at Archived although its partition WAS switched out (the node + /// stopped before the manifest update) has no later purged archive: it is the sole copy, kept. + /// + /// A task that represents the asynchronous operation. + [SkippableFact] + public async Task SoleCopyLeftAtArchived_IsNotTouched() + { + Skip.IfNot(_fixture.Available, _fixture.SkipReason); + var dir = Dir("sole"); + var month = Month(2027, 4); + await InsertAsync(Enumerable.Range(1, 3).Select(i => NewEvent(month.AddDays(i)))); + await DrainAsync(); + var manifest = await ArchiveAsync(dir, month); + await using (var ctx = CreateContext()) + { + await Archive(ctx, dir).PurgeArchivedPartitionAsync(manifest); + } + + await SetStatusAsync(dir, manifest, AuditArchiveStatus.Archived); + + Assert.Empty((await CleanupAsync(dir, dryRun: false)).Items); + Assert.True(File.Exists(ManifestPath(dir, manifest))); + } + + /// + /// Retention: 0 never deletes; a purged archive is kept until its month ended more than + /// RetentionDays ago, then deleted; an expired archive whose partition still holds rows is kept. + /// + /// A task that represents the asynchronous operation. + [SkippableFact] + public async Task Retention_DeletesOnlyExpiredArchivesWhosePartitionIsGone() + { + Skip.IfNot(_fixture.Available, _fixture.SkipReason); + var dir = Dir("retention"); + var purgedMonth = Month(2027, 5); + await InsertAsync(Enumerable.Range(1, 2).Select(i => NewEvent(purgedMonth.AddDays(i)))); + var purged = await ArchiveAsync(dir, purgedMonth); + await using (var ctx = CreateContext()) + { + await Archive(ctx, dir).PurgeArchivedPartitionAsync(purged); + } + + var liveMonth = Month(2027, 6); + await InsertAsync([NewEvent(liveMonth.AddDays(1))]); + var live = await ArchiveAsync(dir, liveMonth); // never purged: rows still in the table + + var monthEnd = purgedMonth.AddMonths(1); + var farFuture = liveMonth.AddMonths(1).AddDays(1000); + Assert.Empty((await CleanupAsync(dir, dryRun: false, retentionDays: 0, now: farFuture)).Items); + Assert.Empty((await CleanupAsync(dir, dryRun: false, retentionDays: 400, now: monthEnd.AddDays(399))).Items); + + var result = await CleanupAsync(dir, dryRun: false, retentionDays: 400, now: farFuture); + + var expired = Assert.Single(result.Items); + Assert.Equal((purged.Name, AuditArchiveCleanupAction.DeletedExpired), (expired.Name, expired.Action)); + Assert.False(File.Exists(ManifestPath(dir, purged))); + Assert.True(File.Exists(ManifestPath(dir, live))); + } + + /// Deletion uses the archive name's own three files — never a path from the manifest. + /// A task that represents the asynchronous operation. + [SkippableFact] + public async Task Deletion_NeverFollowsAManifestPath_OutOfTheArchiveFolder() + { + Skip.IfNot(_fixture.Available, _fixture.SkipReason); + var dir = Dir("paths"); + var month = Month(2027, 7); + await InsertAsync([NewEvent(month.AddDays(1))]); + var manifest = await ArchiveAsync(dir, month); + await using (var ctx = CreateContext()) + { + await Archive(ctx, dir).PurgeArchivedPartitionAsync(manifest); + } + + var outside = Path.Combine(_root.FullName, "outside.parquet"); + await File.WriteAllTextAsync(outside, "not an archive"); + var path = ManifestPath(dir, manifest); + var tampered = JsonSerializer.Deserialize(await File.ReadAllTextAsync(path), Json)! with + { + DataFile = "../../outside.parquet", + TombstoneFile = "../../outside.parquet", + }; + await File.WriteAllTextAsync(path, JsonSerializer.Serialize(tampered, Json)); + + var result = await CleanupAsync(dir, dryRun: false, retentionDays: 30, now: month.AddYears(5)); + + Assert.Equal(AuditArchiveCleanupAction.DeletedExpired, Assert.Single(result.Items).Action); + Assert.True(File.Exists(outside)); + Assert.Empty(Directory.EnumerateFiles(Path.Combine(dir, "2027"))); + } + + private static readonly JsonSerializerOptions Json = new() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase }; + + private sealed class FixedClock(DateTime now) : TimeProvider + { + public override DateTimeOffset GetUtcNow() => new(DateTime.SpecifyKind(now, DateTimeKind.Utc)); + } + + private string Dir(string name) => Directory.CreateDirectory(Path.Combine(_root.FullName, name)).FullName; + + private static string ManifestPath(string dir, AuditArchiveManifest m) => Path.Combine(dir, m.Partition[..4], m.Name + ".manifest.json"); + + private static async Task SetStatusAsync(string dir, AuditArchiveManifest manifest, AuditArchiveStatus status) + { + var path = ManifestPath(dir, manifest); + var current = JsonSerializer.Deserialize(await File.ReadAllTextAsync(path), Json)!; + await File.WriteAllTextAsync(path, JsonSerializer.Serialize(current with { Status = status, PurgedAtUtc = null }, Json)); + } + + private static AuditLogPartitionArchive Archive( + ScadaBridgeDbContext ctx, string dir, int retentionDays = 0, DateTime? now = null) => new(ctx, new AuditLogArchiveOptions + { + Enabled = true, + Directory = dir, + RowGroupSize = 1_000, + RetentionDays = retentionDays, + }, time: now is null ? null : new FixedClock(now.Value)); + + private async Task ArchiveAsync(string dir, DateTime month) + { + await using var ctx = CreateContext(); + var manifest = await Archive(ctx, dir).ArchivePartitionAsync(month); + await Task.Delay(5); // distinct millisecond archive names + return manifest; + } + + private async Task CleanupAsync(string dir, bool dryRun, int retentionDays = 0, DateTime? now = null) + { + await using var ctx = CreateContext(); + return await Archive(ctx, dir, retentionDays, now).CleanupAsync(dryRun); + } + + private async Task InsertAsync(IEnumerable events) + { + await using var ctx = CreateContext(); + var repo = new AuditLogRepository(ctx); + foreach (var e in events) + { + await repo.InsertIfNotExistsAsync(e); + } + } + + private async Task DrainAsync() + { + await using var ctx = CreateContext(); + var chain = new AuditLogHashChain(ctx); + for (var i = 0; i < 1000; i++) + { + var r = await chain.ChainPendingAsync(500); + if (r.RowsChained == 0 && !r.LockBusy) + { + return; + } + } + + throw new InvalidOperationException("Chain did not drain."); + } + + private static DateTime Month(int year, int month) => new(year, month, 1, 0, 0, 0, DateTimeKind.Utc); + + private ScadaBridgeDbContext CreateContext() + => new(new DbContextOptionsBuilder().UseSqlServer(_fixture.ConnectionString).Options); + + private static AuditEvent NewEvent(DateTime occurredAtUtc, AuditChannel channel = AuditChannel.ApiOutbound) => + ScadaBridgeAuditEventFactory.Create( + channel: channel, + kind: channel == AuditChannel.DbOutbound ? AuditKind.DbWrite : AuditKind.ApiCall, + status: AuditStatus.Delivered, + occurredAtUtc: occurredAtUtc, + actor: "cleanup-test", + target: "erp", + sourceNode: "central-a", + sourceSiteId: "cleanup-test", + correlationId: Guid.NewGuid()); +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/AuditArchiveCommandTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/AuditArchiveCommandTests.cs index d69a04ec..9a706d7e 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/AuditArchiveCommandTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/AuditArchiveCommandTests.cs @@ -144,4 +144,43 @@ public class AuditArchiveCommandTests : TestKit Assert.Equal(TimeSpan.FromMinutes(5), ManagementEndpoints.ResolveAskTimeout(new ManagementServiceOptions(), typeof(VerifyAuditArchiveCommand))); } + + /// ScadaBridge#109: an Administrator runs the cleanup; the dry-run flag reaches the archive. + /// The flag. + [Theory] + [InlineData(true)] + [InlineData(false)] + public void Administrator_RunsCleanup_WithTheDryRunFlag(bool dryRun) + { + _archive.CleanupAsync(dryRun, Arg.Any()).Returns(new AuditArchiveCleanupResult(dryRun, 0, + [new AuditArchiveCleanupItem(Name, "2025-10", AuditArchiveCleanupAction.DeletedSuperseded, "superseded")])); + + CreateActor().Tell(Envelope(new CleanupAuditArchivesCommand(dryRun), [], "Administrator")); + + var response = ExpectMsg(TimeSpan.FromSeconds(5)); + using var json = JsonDocument.Parse(response.JsonData); + Assert.Equal(dryRun, json.RootElement.GetProperty("dryRun").GetBoolean()); + Assert.Equal("DeletedSuperseded", json.RootElement.GetProperty("items")[0].GetProperty("action").GetString()); + _archive.Received(1).CleanupAsync(dryRun, Arg.Any()); + } + + /// Cleanup deletes files: a Viewer (audit reader) is refused. + [Fact] + public void Viewer_CannotRunCleanup() + { + CreateActor().Tell(Envelope(new CleanupAuditArchivesCommand(true), [], "Viewer")); + + ExpectMsg(TimeSpan.FromSeconds(5)); + _archive.DidNotReceiveWithAnyArgs().CleanupAsync(default, default); + } + + /// No archive directory configured: ARCHIVE_NOT_CONFIGURED, as for list/verify. + [Fact] + public void Cleanup_WithoutAnArchiveDirectory_IsNotConfigured() + { + _archive.IsConfigured.Returns(false); + CreateActor().Tell(Envelope(new CleanupAuditArchivesCommand(true), [], "Administrator")); + + Assert.Equal("ARCHIVE_NOT_CONFIGURED", ExpectMsg(TimeSpan.FromSeconds(5)).ErrorCode); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/RequiredRoleMatrixTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/RequiredRoleMatrixTests.cs index 2999eec3..f4b9d3b4 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/RequiredRoleMatrixTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ManagementService.Tests/RequiredRoleMatrixTests.cs @@ -176,6 +176,8 @@ public class RequiredRoleMatrixTests // AuditLog Parquet archives (ScadaBridge#42): same audit-read gate as verify-chain. ["ListAuditArchives"] = [Roles.Administrator, Roles.Viewer], ["VerifyAuditArchive"] = [Roles.Administrator, Roles.Viewer], + // Archive cleanup deletes files (ScadaBridge#109): Administrator only. + ["CleanupAuditArchives"] = [Roles.Administrator], // ---- Two-person secured write ---------------------------------------------- ["SubmitSecuredWrite"] = [Roles.Operator],