feat(audit): archive retention and superseded stale-archive cleanup (ScadaBridge#109)

Owner decisions recorded: the archive destination stays a directory/UNC
(no object-storage client; share + permissions named at enablement),
parquet-tools-only reading (no re-import), unsigned like the chain (#107).

AuditLog:Archive:RetentionDays (default 0 = never delete; else must exceed
AuditLog:RetentionDays, counted from the archived month's end) and
DeleteSupersededStaleArchives (default true) run after every purge tick.
A retried purge's stale Archived archive is deleted only when a later,
verifying, Purged archive of the same partition provably holds every row:
same prior anchor and the stale chain-end hash at its chain-end sequence
(row or tombstone), every unchained row present by EventId; otherwise it is
reported KeptStale. A sole copy left at Archived is never touched. Deletion
removes only the archive name's own three files in its folder, manifest
first, never a manifest-supplied path. `audit archive cleanup [--dry-run]`
= CleanupAuditArchives (Administrator).

AuditLogArchiveCleanupTests (MSSQL) cover retried purge, kept stale copy,
sole copy, retention and the path guard; each safety rule mutation-checked.
Follow-ups filed: #113, #114, #115.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 23:32:53 -05:00
parent c4d7ec9b96
commit b95ee5a658
26 changed files with 1099 additions and 26 deletions
@@ -109,6 +109,43 @@ public static class AuditArchiveHelpers
return verified ? 0 : NotVerifiedExitCode;
}
/// <summary>
/// Renders a <c>CleanupAuditArchives</c> result (ScadaBridge#109). Always exit 0 — kept
/// stale archives are reported, not an error.
/// </summary>
/// <param name="json">The success body (an <c>AuditArchiveCleanupResult</c>, camelCase).</param>
/// <param name="format">Resolved output format.</param>
/// <param name="output">Writer for the report.</param>
/// <returns><c>0</c>.</returns>
public static int RenderCleanup(string json, string format, TextWriter output)
{
if (!string.Equals(format, "table", StringComparison.OrdinalIgnoreCase))
{
output.WriteLine(json);
output.Flush();
return 0;
}
using var doc = JsonDocument.Parse(json);
var root = doc.RootElement;
var dryRun = root.TryGetProperty("dryRun", out var d) && d.ValueKind == JsonValueKind.True;
var retention = Num(root, "retentionDays");
output.WriteLine($"{(dryRun ? "Dry run — nothing deleted. " : "")}Archive retention: {(retention == "0" ? "never delete" : retention + " days")}");
var items = root.TryGetProperty("items", out var i) && i.ValueKind == JsonValueKind.Array ? i.EnumerateArray().ToList() : [];
if (items.Count == 0)
{
output.WriteLine("Nothing to clean up.");
}
foreach (var item in items)
{
output.WriteLine($"{Str(item, "action"),-18} {Str(item, "name")} {Str(item, "detail")}");
}
output.Flush();
return 0;
}
private static string Str(JsonElement e, string name, string missing = "")
=> e.TryGetProperty(name, out var v) && v.ValueKind == JsonValueKind.String ? v.GetString() ?? missing : missing;
@@ -11,9 +11,9 @@ namespace ZB.MOM.WW.ScadaBridge.CLI.Commands;
/// <c>GET /api/audit/tree</c>), plus <c>verify-chain</c>, which sends the
/// <c>VerifyAuditChain</c> management command (hash-chain tamper evidence, ScadaBridge#41),
/// <c>verify-heads</c>, which checks off-box exported chain heads with
/// <c>VerifyAuditChainHeads</c> (ScadaBridge#107), and <c>archive list</c> / <c>archive verify</c>,
/// which send <c>ListAuditArchives</c> / <c>VerifyAuditArchive</c> (Parquet cold archive,
/// ScadaBridge#42).
/// <c>VerifyAuditChainHeads</c> (ScadaBridge#107), and <c>archive list</c> / <c>archive verify</c>
/// / <c>archive cleanup</c>, which send <c>ListAuditArchives</c> / <c>VerifyAuditArchive</c>
/// (Parquet cold archive, ScadaBridge#42) / <c>CleanupAuditArchives</c> (ScadaBridge#109).
/// </summary>
public static class AuditCommands
{
@@ -376,7 +376,7 @@ public static class AuditCommands
private static Command BuildArchive(Option<string> urlOption, Option<string> formatOption, Option<string> usernameOption, Option<string> passwordOption)
{
var group = new Command("archive") { Description = "List and verify the AuditLog Parquet cold archives" };
var group = new Command("archive") { Description = "List, verify and clean up the AuditLog Parquet cold archives" };
var list = new Command("list") { Description = "List the AuditLog Parquet archives written by the retention purge" };
list.SetAction(async (ParseResult result) =>
@@ -410,6 +410,23 @@ public static class AuditCommands
});
group.Add(verify);
var dryRunOption = new Option<bool>("--dry-run") { Description = "Report what would be deleted; delete nothing" };
var cleanup = new Command("cleanup")
{
Description = "Delete superseded stale archives and archives past AuditLog:Archive:RetentionDays (Administrator)",
};
cleanup.Add(dryRunOption);
cleanup.SetAction(async (ParseResult result) =>
{
var format = AuditCommandHelpers.ResolveFormat(result, formatOption);
return await CommandHelpers.ExecuteCommandAsync(
result, urlOption, formatOption, usernameOption, passwordOption,
new CleanupAuditArchivesCommand(result.GetValue(dryRunOption)),
timeout: AuditArchiveHelpers.VerifyRequestTimeout,
onSuccess: json => AuditArchiveHelpers.RenderCleanup(json, format, Console.Out));
});
group.Add(cleanup);
return group;
}
+7
View File
@@ -2136,6 +2136,13 @@ tombstones, and compares the chain end with the checkpoint anchor the purge left
| `1` | Request error (`ARCHIVE_NOT_CONFIGURED`, `NOT_FOUND`, `INVALID_ARGUMENT`, server error) |
| `2` | Authorization failure (needs `Administrator` or `Viewer`, not site-scoped) |
`audit archive cleanup [--dry-run]` (ScadaBridge#109, Administrator) sends
`CleanupAuditArchives`: it deletes stale archives a retried purge left behind once a later
purged archive provably holds all their rows (`DeletedSuperseded`), and archives whose month
ended more than `AuditLog:Archive:RetentionDays` ago (`DeletedExpired`; default `0` = never),
and lists stale archives it kept (`KeptStale`) with the reason. `--dry-run` deletes nothing.
The purge actor runs the same pass after every purge tick. Exit code `0`.
The archive files are plain Parquet; read them with any Parquet tool, e.g. DuckDB:
`SELECT * FROM '<dir>/2025/auditlog-2025-10-….parquet'`. See `Component-AuditLog.md`,
Retention & Purge.