feat(audit): Parquet cold archive of expired AuditLog partitions before the purge (ScadaBridge#42)

Audit roadmap T2 (alog.md §15.2): with AuditLog:Archive:Enabled (default off)
the purge actor writes each expired monthly partition to Parquet under an
absolute AuditLog:Archive:Directory before switching it out, so audit rows
outlive RetentionDays.

- Parquet.Net 6.1.0 (fully managed; NuGetAudit clean) in central package
  management.
- AuditArchiveParquet: data file = the ten stored columns, OccurredAtUtc as a
  UTC microsecond timestamp plus exact OccurredAtUtcTicks, and ChainSeq/RowHash;
  tombstone file = the month's per-channel purge tombstones. Buffered row
  groups; the files are plain Parquet (DuckDB/pyarrow read them).
- AuditLogPartitionArchive (IAuditLogArchive): reads the partition's rows,
  chain heads and tombstones in one transaction under the chain lock SHARED,
  writes {Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet,
  .tombstones.parquet and a .manifest.json (counts, SHA-256 per file, each
  month's head and prior anchor) via tmp + rename, manifest last. Archives are
  never overwritten.
- Archive before purge: SwitchOutPartitionAsync gains an internal guarded
  overload. Under the EXCLUSIVE chain lock it re-counts the partition and
  re-reads each month's LastSeq and THROWs 51002/51003 (nothing purged) unless
  both still equal the archive's, so the switch's checkpoint anchor is exactly
  the archive's chain end. The manifest is then marked Purged.
- Fail closed: a partition that cannot be archived is not purged; the failure
  takes the existing AuditLogPurgeFailedEvent / PurgeFailures path. Success
  publishes AuditLogArchivedEvent.
- Verify: file hashes and counts against the manifest, the chain recomputed
  from the archived rows and tombstones (AuditChainVerifier), and the chain end
  against the database anchor (Matches / Superseded / NotPurged / Mismatch).
- AuditLogArchiveOptionsValidator: Directory required when enabled and must be
  absolute; RowGroupSize, timeouts bounded.

No migration: the purger role already reads AuditLog, the head and tombstones.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 17:00:49 -05:00
parent 31cb2c79e3
commit 7be4f19861
18 changed files with 2232 additions and 8 deletions
@@ -0,0 +1,147 @@
using System.Text.Json.Serialization;
namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
/// <summary>
/// Lifecycle of one AuditLog Parquet archive (audit roadmap T2, ScadaBridge#42).
/// </summary>
[JsonConverter(typeof(JsonStringEnumConverter<AuditArchiveStatus>))]
public enum AuditArchiveStatus
{
/// <summary>
/// The archive files are durable but the partition switch that should follow has not
/// been confirmed: either it failed (the archive went stale and a later tick wrote a
/// fresh one) or the node stopped between the switch and the manifest update. Verify
/// tells the two apart: only the second one's chain end matches the database anchor.
/// </summary>
Archived,
/// <summary>The archived partition was switched out of <c>dbo.AuditLog</c>; the archive is the only copy.</summary>
Purged,
}
/// <summary>
/// One chain partition (UTC calendar month) as it stood when its rows were archived: the
/// chain head and the checkpoint anchor an earlier partition purge had left, so the archived
/// rows can be re-verified with <c>AuditChainVerifier</c> long after the database dropped them.
/// </summary>
/// <param name="Partition">The chain partition, <c>YYYY-MM</c>.</param>
/// <param name="LastSeq">The head's last sequence number at archive time.</param>
/// <param name="LastHash">The head's last hash at archive time (lowercase hex).</param>
/// <param name="PriorPurgedThroughSeq">The checkpoint anchor an earlier purge of this month left, or null.</param>
/// <param name="PriorPurgedThroughHash">The hash at <paramref name="PriorPurgedThroughSeq"/> (lowercase hex), or null.</param>
/// <param name="TombstoneCount">Per-channel purge tombstones of this month carried in the archive.</param>
public sealed record AuditArchiveChainState(
string Partition,
long LastSeq,
string LastHash,
long? PriorPurgedThroughSeq,
string? PriorPurgedThroughHash,
long TombstoneCount);
/// <summary>
/// The JSON manifest written beside every AuditLog Parquet archive (ScadaBridge#42). It is
/// written LAST, so its presence is the commit marker for the archive's two Parquet files.
/// </summary>
/// <param name="FormatVersion">Manifest/file layout version (currently 1).</param>
/// <param name="Name">Archive name, <c>auditlog-YYYY-MM-yyyyMMddTHHmmssfffZ</c>; the file stem of all three files.</param>
/// <param name="PartitionBoundaryUtc">The <c>pf_AuditLog_Month</c> lower boundary of the archived partition.</param>
/// <param name="Partition">The archived partition, <c>YYYY-MM</c>.</param>
/// <param name="ArchivedAtUtc">When the archive was written.</param>
/// <param name="Status">Whether the partition switch that follows the archive has been confirmed.</param>
/// <param name="PurgedAtUtc">When the partition switch completed, or null.</param>
/// <param name="RowCount">Rows in the data file.</param>
/// <param name="ChainedRowCount">Rows that carried a ChainSeq/RowHash.</param>
/// <param name="UnchainedRowCount">Rows the chaining job had not reached (not covered by the chain).</param>
/// <param name="MinOccurredAtUtc">Earliest OccurredAtUtc in the data file, or null when empty.</param>
/// <param name="MaxOccurredAtUtc">Latest OccurredAtUtc in the data file, or null when empty.</param>
/// <param name="DataFile">Data file name, relative to the manifest's directory.</param>
/// <param name="DataSha256">SHA-256 of the data file (lowercase hex).</param>
/// <param name="DataBytes">Size of the data file in bytes.</param>
/// <param name="TombstoneFile">Tombstone file name, relative to the manifest's directory.</param>
/// <param name="TombstoneSha256">SHA-256 of the tombstone file (lowercase hex).</param>
/// <param name="TombstoneCount">Tombstones in the tombstone file.</param>
/// <param name="Chains">One entry per chain partition that had a head when the rows were archived.</param>
public sealed record AuditArchiveManifest(
int FormatVersion,
string Name,
DateTime PartitionBoundaryUtc,
string Partition,
DateTime ArchivedAtUtc,
AuditArchiveStatus Status,
DateTime? PurgedAtUtc,
long RowCount,
long ChainedRowCount,
long UnchainedRowCount,
DateTime? MinOccurredAtUtc,
DateTime? MaxOccurredAtUtc,
string DataFile,
string DataSha256,
long DataBytes,
string TombstoneFile,
string TombstoneSha256,
long TombstoneCount,
IReadOnlyList<AuditArchiveChainState> Chains);
/// <summary>
/// How an archive's recorded chain end relates to the checkpoint anchor the partition purge
/// left on the database's chain head.
/// </summary>
[JsonConverter(typeof(JsonStringEnumConverter<AuditArchiveAnchorStatus>))]
public enum AuditArchiveAnchorStatus
{
/// <summary>The database anchor is exactly the archive's chain end (sequence and hash).</summary>
Matches,
/// <summary>
/// The database anchor is past the archive's chain end: rows that arrived late for this
/// month were chained and purged (and archived) again afterwards.
/// </summary>
Superseded,
/// <summary>The partition has not been purged: the archive's rows are still in <c>dbo.AuditLog</c>.</summary>
NotPurged,
/// <summary>
/// The anchor disagrees with the archive (an earlier sequence number, a different hash, or
/// no chain head at all): the archive or the head was altered.
/// </summary>
Mismatch,
}
/// <summary>Verification of one chain partition inside an archive.</summary>
/// <param name="Chain">The chain recomputed over the archived rows, the archived head and tombstones.</param>
/// <param name="Anchor">How the archived chain end relates to the database's checkpoint anchor.</param>
/// <param name="AnchorDetail">Human-readable explanation of <paramref name="Anchor"/>.</param>
public sealed record AuditArchiveChainVerification(
AuditChainVerificationResult Chain,
AuditArchiveAnchorStatus Anchor,
string AnchorDetail);
/// <summary>
/// Result of <c>audit archive verify</c> (ScadaBridge#42): the archive files match their
/// manifest, every archived chain recomputes to the archived head, and each chain end is the
/// anchor the partition purge left in the database.
/// </summary>
/// <param name="Name">The archive verified.</param>
/// <param name="Partition">The archived partition, <c>YYYY-MM</c>.</param>
/// <param name="ManifestStatus">The manifest's lifecycle status.</param>
/// <param name="Verified">True when the files are intact, no chain is broken and no anchor mismatches.</param>
/// <param name="FilesIntact">True when both files' size and SHA-256 and the row/tombstone counts match the manifest.</param>
/// <param name="FileProblem">Why <paramref name="FilesIntact"/> is false, or null.</param>
/// <param name="RowsRead">Rows read from the data file.</param>
/// <param name="Chains">Per chain partition verification.</param>
public sealed record AuditArchiveVerificationResult(
string Name,
string Partition,
AuditArchiveStatus ManifestStatus,
bool Verified,
bool FilesIntact,
string? FileProblem,
long RowsRead,
IReadOnlyList<AuditArchiveChainVerification> Chains);
/// <summary>Result of <c>audit archive list</c> (ScadaBridge#42).</summary>
/// <param name="Enabled">Whether the retention purge is archiving partitions (<c>AuditLog:Archive:Enabled</c>).</param>
/// <param name="Archives">Every archive manifest, oldest partition first.</param>
public sealed record AuditArchiveListResult(bool Enabled, IReadOnlyList<AuditArchiveManifest> Archives);