feat(audit): Parquet cold archive of expired AuditLog partitions before the purge (ScadaBridge#42)

Audit roadmap T2 (alog.md §15.2): with AuditLog:Archive:Enabled (default off)
the purge actor writes each expired monthly partition to Parquet under an
absolute AuditLog:Archive:Directory before switching it out, so audit rows
outlive RetentionDays.

- Parquet.Net 6.1.0 (fully managed; NuGetAudit clean) in central package
  management.
- AuditArchiveParquet: data file = the ten stored columns, OccurredAtUtc as a
  UTC microsecond timestamp plus exact OccurredAtUtcTicks, and ChainSeq/RowHash;
  tombstone file = the month's per-channel purge tombstones. Buffered row
  groups; the files are plain Parquet (DuckDB/pyarrow read them).
- AuditLogPartitionArchive (IAuditLogArchive): reads the partition's rows,
  chain heads and tombstones in one transaction under the chain lock SHARED,
  writes {Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet,
  .tombstones.parquet and a .manifest.json (counts, SHA-256 per file, each
  month's head and prior anchor) via tmp + rename, manifest last. Archives are
  never overwritten.
- Archive before purge: SwitchOutPartitionAsync gains an internal guarded
  overload. Under the EXCLUSIVE chain lock it re-counts the partition and
  re-reads each month's LastSeq and THROWs 51002/51003 (nothing purged) unless
  both still equal the archive's, so the switch's checkpoint anchor is exactly
  the archive's chain end. The manifest is then marked Purged.
- Fail closed: a partition that cannot be archived is not purged; the failure
  takes the existing AuditLogPurgeFailedEvent / PurgeFailures path. Success
  publishes AuditLogArchivedEvent.
- Verify: file hashes and counts against the manifest, the chain recomputed
  from the archived rows and tombstones (AuditChainVerifier), and the chain end
  against the database anchor (Matches / Superseded / NotPurged / Mismatch).
- AuditLogArchiveOptionsValidator: Directory required when enabled and must be
  absolute; RowGroupSize, timeouts bounded.

No migration: the purger role already reads AuditLog, the head and tombstones.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 17:00:49 -05:00
parent 31cb2c79e3
commit 7be4f19861
18 changed files with 2232 additions and 8 deletions
@@ -0,0 +1,82 @@
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
namespace ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
/// <summary>
/// Parquet cold archive of the central <c>AuditLog</c> (audit roadmap T2, alog.md §15.2,
/// ScadaBridge#42): a monthly partition is written to Parquet before the retention purge
/// switches it out, so audit rows outlive the SQL retention window.
/// </summary>
/// <remarks>
/// <para>
/// <b>Archive before purge.</b> <see cref="ArchivePartitionAsync"/> writes the partition's
/// rows (with their hash-chain columns), the month's chain head and its per-channel purge
/// tombstones; <see cref="PurgeArchivedPartitionAsync"/> then switches the partition out
/// inside a guard that refuses to drop it if the partition gained or lost a row, or its
/// chain advanced, since the archive was written. A partition is therefore never purged
/// without an archive that holds exactly the rows the purge drops.
/// </para>
/// <para>
/// <b>Hash chain.</b> The switch records each month's chain end as the checkpoint anchor
/// (ScadaBridge#41); the archive records the same chain end, so <see cref="VerifyAsync"/>
/// can recompute the archived chain and tie its end to the anchor still in the database.
/// </para>
/// </remarks>
public interface IAuditLogArchive
{
/// <summary>True when an archive directory is configured (listing and verifying need one; archiving needs <c>Enabled</c> too).</summary>
bool IsConfigured { get; }
/// <summary>True when the retention purge archives partitions before dropping them (<c>AuditLog:Archive:Enabled</c>).</summary>
bool IsEnabled { get; }
/// <summary>
/// Writes the partition whose lower boundary is <paramref name="partitionBoundaryUtc"/> to
/// a new archive (data file, tombstone file, then manifest with status
/// <see cref="AuditArchiveStatus.Archived"/>).
/// </summary>
/// <param name="partitionBoundaryUtc">The <c>pf_AuditLog_Month</c> lower boundary (day 1, midnight UTC).</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>The written manifest.</returns>
Task<AuditArchiveManifest> ArchivePartitionAsync(DateTime partitionBoundaryUtc, CancellationToken ct = default);
/// <summary>
/// Switches the archived partition out of <c>dbo.AuditLog</c>, guarded against the
/// partition having changed since <paramref name="manifest"/> was written, then marks the
/// manifest <see cref="AuditArchiveStatus.Purged"/>.
/// </summary>
/// <param name="manifest">The manifest <see cref="ArchivePartitionAsync"/> returned.</param>
/// <param name="commandTimeout">Maintenance command timeout for the switch.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>The approximate number of rows the switch discarded.</returns>
/// <exception cref="AuditArchiveStaleException">The partition changed after it was archived; nothing was purged.</exception>
Task<long> PurgeArchivedPartitionAsync(AuditArchiveManifest manifest, TimeSpan? commandTimeout = null, CancellationToken ct = default);
/// <summary>Lists every archive manifest under the archive directory, oldest partition first.</summary>
/// <param name="ct">Cancellation token.</param>
/// <returns>The manifests.</returns>
Task<IReadOnlyList<AuditArchiveManifest>> ListAsync(CancellationToken ct = default);
/// <summary>
/// Verifies one archive: file integrity against the manifest, the hash chain recomputed
/// over the archived rows, and the chain end against the database's checkpoint anchor.
/// </summary>
/// <param name="name">The archive name (as <see cref="ListAsync"/> reports it).</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>The verification result.</returns>
/// <exception cref="FileNotFoundException">No archive has that name.</exception>
Task<AuditArchiveVerificationResult> VerifyAsync(string name, CancellationToken ct = default);
}
/// <summary>
/// Thrown by <see cref="IAuditLogArchive.PurgeArchivedPartitionAsync"/> when the partition
/// gained or lost rows, or its hash chain advanced, after the archive was written. Nothing
/// is purged; the next purge tick writes a fresh archive.
/// </summary>
public sealed class AuditArchiveStaleException : Exception
{
/// <summary>Initializes a new instance with a message and the SQL error that carried it.</summary>
/// <param name="message">The message.</param>
/// <param name="inner">The underlying exception.</param>
public AuditArchiveStaleException(string message, Exception? inner = null) : base(message, inner) { }
}
@@ -0,0 +1,147 @@
using System.Text.Json.Serialization;
namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
/// <summary>
/// Lifecycle of one AuditLog Parquet archive (audit roadmap T2, ScadaBridge#42).
/// </summary>
[JsonConverter(typeof(JsonStringEnumConverter<AuditArchiveStatus>))]
public enum AuditArchiveStatus
{
/// <summary>
/// The archive files are durable but the partition switch that should follow has not
/// been confirmed: either it failed (the archive went stale and a later tick wrote a
/// fresh one) or the node stopped between the switch and the manifest update. Verify
/// tells the two apart: only the second one's chain end matches the database anchor.
/// </summary>
Archived,
/// <summary>The archived partition was switched out of <c>dbo.AuditLog</c>; the archive is the only copy.</summary>
Purged,
}
/// <summary>
/// One chain partition (UTC calendar month) as it stood when its rows were archived: the
/// chain head and the checkpoint anchor an earlier partition purge had left, so the archived
/// rows can be re-verified with <c>AuditChainVerifier</c> long after the database dropped them.
/// </summary>
/// <param name="Partition">The chain partition, <c>YYYY-MM</c>.</param>
/// <param name="LastSeq">The head's last sequence number at archive time.</param>
/// <param name="LastHash">The head's last hash at archive time (lowercase hex).</param>
/// <param name="PriorPurgedThroughSeq">The checkpoint anchor an earlier purge of this month left, or null.</param>
/// <param name="PriorPurgedThroughHash">The hash at <paramref name="PriorPurgedThroughSeq"/> (lowercase hex), or null.</param>
/// <param name="TombstoneCount">Per-channel purge tombstones of this month carried in the archive.</param>
public sealed record AuditArchiveChainState(
string Partition,
long LastSeq,
string LastHash,
long? PriorPurgedThroughSeq,
string? PriorPurgedThroughHash,
long TombstoneCount);
/// <summary>
/// The JSON manifest written beside every AuditLog Parquet archive (ScadaBridge#42). It is
/// written LAST, so its presence is the commit marker for the archive's two Parquet files.
/// </summary>
/// <param name="FormatVersion">Manifest/file layout version (currently 1).</param>
/// <param name="Name">Archive name, <c>auditlog-YYYY-MM-yyyyMMddTHHmmssfffZ</c>; the file stem of all three files.</param>
/// <param name="PartitionBoundaryUtc">The <c>pf_AuditLog_Month</c> lower boundary of the archived partition.</param>
/// <param name="Partition">The archived partition, <c>YYYY-MM</c>.</param>
/// <param name="ArchivedAtUtc">When the archive was written.</param>
/// <param name="Status">Whether the partition switch that follows the archive has been confirmed.</param>
/// <param name="PurgedAtUtc">When the partition switch completed, or null.</param>
/// <param name="RowCount">Rows in the data file.</param>
/// <param name="ChainedRowCount">Rows that carried a ChainSeq/RowHash.</param>
/// <param name="UnchainedRowCount">Rows the chaining job had not reached (not covered by the chain).</param>
/// <param name="MinOccurredAtUtc">Earliest OccurredAtUtc in the data file, or null when empty.</param>
/// <param name="MaxOccurredAtUtc">Latest OccurredAtUtc in the data file, or null when empty.</param>
/// <param name="DataFile">Data file name, relative to the manifest's directory.</param>
/// <param name="DataSha256">SHA-256 of the data file (lowercase hex).</param>
/// <param name="DataBytes">Size of the data file in bytes.</param>
/// <param name="TombstoneFile">Tombstone file name, relative to the manifest's directory.</param>
/// <param name="TombstoneSha256">SHA-256 of the tombstone file (lowercase hex).</param>
/// <param name="TombstoneCount">Tombstones in the tombstone file.</param>
/// <param name="Chains">One entry per chain partition that had a head when the rows were archived.</param>
public sealed record AuditArchiveManifest(
int FormatVersion,
string Name,
DateTime PartitionBoundaryUtc,
string Partition,
DateTime ArchivedAtUtc,
AuditArchiveStatus Status,
DateTime? PurgedAtUtc,
long RowCount,
long ChainedRowCount,
long UnchainedRowCount,
DateTime? MinOccurredAtUtc,
DateTime? MaxOccurredAtUtc,
string DataFile,
string DataSha256,
long DataBytes,
string TombstoneFile,
string TombstoneSha256,
long TombstoneCount,
IReadOnlyList<AuditArchiveChainState> Chains);
/// <summary>
/// How an archive's recorded chain end relates to the checkpoint anchor the partition purge
/// left on the database's chain head.
/// </summary>
[JsonConverter(typeof(JsonStringEnumConverter<AuditArchiveAnchorStatus>))]
public enum AuditArchiveAnchorStatus
{
/// <summary>The database anchor is exactly the archive's chain end (sequence and hash).</summary>
Matches,
/// <summary>
/// The database anchor is past the archive's chain end: rows that arrived late for this
/// month were chained and purged (and archived) again afterwards.
/// </summary>
Superseded,
/// <summary>The partition has not been purged: the archive's rows are still in <c>dbo.AuditLog</c>.</summary>
NotPurged,
/// <summary>
/// The anchor disagrees with the archive (an earlier sequence number, a different hash, or
/// no chain head at all): the archive or the head was altered.
/// </summary>
Mismatch,
}
/// <summary>Verification of one chain partition inside an archive.</summary>
/// <param name="Chain">The chain recomputed over the archived rows, the archived head and tombstones.</param>
/// <param name="Anchor">How the archived chain end relates to the database's checkpoint anchor.</param>
/// <param name="AnchorDetail">Human-readable explanation of <paramref name="Anchor"/>.</param>
public sealed record AuditArchiveChainVerification(
AuditChainVerificationResult Chain,
AuditArchiveAnchorStatus Anchor,
string AnchorDetail);
/// <summary>
/// Result of <c>audit archive verify</c> (ScadaBridge#42): the archive files match their
/// manifest, every archived chain recomputes to the archived head, and each chain end is the
/// anchor the partition purge left in the database.
/// </summary>
/// <param name="Name">The archive verified.</param>
/// <param name="Partition">The archived partition, <c>YYYY-MM</c>.</param>
/// <param name="ManifestStatus">The manifest's lifecycle status.</param>
/// <param name="Verified">True when the files are intact, no chain is broken and no anchor mismatches.</param>
/// <param name="FilesIntact">True when both files' size and SHA-256 and the row/tombstone counts match the manifest.</param>
/// <param name="FileProblem">Why <paramref name="FilesIntact"/> is false, or null.</param>
/// <param name="RowsRead">Rows read from the data file.</param>
/// <param name="Chains">Per chain partition verification.</param>
public sealed record AuditArchiveVerificationResult(
string Name,
string Partition,
AuditArchiveStatus ManifestStatus,
bool Verified,
bool FilesIntact,
string? FileProblem,
long RowsRead,
IReadOnlyList<AuditArchiveChainVerification> Chains);
/// <summary>Result of <c>audit archive list</c> (ScadaBridge#42).</summary>
/// <param name="Enabled">Whether the retention purge is archiving partitions (<c>AuditLog:Archive:Enabled</c>).</param>
/// <param name="Archives">Every archive manifest, oldest partition first.</param>
public sealed record AuditArchiveListResult(bool Enabled, IReadOnlyList<AuditArchiveManifest> Archives);