feat(audit): Parquet cold archive of expired AuditLog partitions before the purge (ScadaBridge#42)
Audit roadmap T2 (alog.md §15.2): with AuditLog:Archive:Enabled (default off)
the purge actor writes each expired monthly partition to Parquet under an
absolute AuditLog:Archive:Directory before switching it out, so audit rows
outlive RetentionDays.
- Parquet.Net 6.1.0 (fully managed; NuGetAudit clean) in central package
management.
- AuditArchiveParquet: data file = the ten stored columns, OccurredAtUtc as a
UTC microsecond timestamp plus exact OccurredAtUtcTicks, and ChainSeq/RowHash;
tombstone file = the month's per-channel purge tombstones. Buffered row
groups; the files are plain Parquet (DuckDB/pyarrow read them).
- AuditLogPartitionArchive (IAuditLogArchive): reads the partition's rows,
chain heads and tombstones in one transaction under the chain lock SHARED,
writes {Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet,
.tombstones.parquet and a .manifest.json (counts, SHA-256 per file, each
month's head and prior anchor) via tmp + rename, manifest last. Archives are
never overwritten.
- Archive before purge: SwitchOutPartitionAsync gains an internal guarded
overload. Under the EXCLUSIVE chain lock it re-counts the partition and
re-reads each month's LastSeq and THROWs 51002/51003 (nothing purged) unless
both still equal the archive's, so the switch's checkpoint anchor is exactly
the archive's chain end. The manifest is then marked Purged.
- Fail closed: a partition that cannot be archived is not purged; the failure
takes the existing AuditLogPurgeFailedEvent / PurgeFailures path. Success
publishes AuditLogArchivedEvent.
- Verify: file hashes and counts against the manifest, the chain recomputed
from the archived rows and tombstones (AuditChainVerifier), and the chain end
against the database anchor (Matches / Superseded / NotPurged / Mismatch).
- AuditLogArchiveOptionsValidator: Directory required when enabled and must be
absolute; RowGroupSize, timeouts bounded.
No migration: the purger role already reads AuditLog, the head and tombstones.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -477,7 +477,24 @@ VALUES
|
||||
};
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<long> SwitchOutPartitionAsync(DateTime monthBoundary, TimeSpan? commandTimeout = null, CancellationToken ct = default)
|
||||
public Task<long> SwitchOutPartitionAsync(DateTime monthBoundary, TimeSpan? commandTimeout = null, CancellationToken ct = default)
|
||||
=> SwitchOutPartitionAsync(monthBoundary, guard: null, commandTimeout, ct);
|
||||
|
||||
/// <summary>
|
||||
/// The partition switch-out, optionally guarded by an archive's expectations
|
||||
/// (ScadaBridge#42). With a <paramref name="guard"/>, the switch transaction — after it
|
||||
/// holds the exclusive chain lock — re-counts the partition and re-reads its months'
|
||||
/// chain heads, and THROWs (error 51002 / 51003, nothing purged) when either differs
|
||||
/// from what the archive captured. That is what makes "archive, then purge" safe
|
||||
/// against a row arriving, a per-channel purge or a chaining pass in between.
|
||||
/// </summary>
|
||||
/// <param name="monthBoundary">The partition's lower boundary.</param>
|
||||
/// <param name="guard">Archive expectations, or null for the unguarded purge.</param>
|
||||
/// <param name="commandTimeout">Maintenance command timeout.</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>The approximate number of rows discarded.</returns>
|
||||
internal async Task<long> SwitchOutPartitionAsync(
|
||||
DateTime monthBoundary, Maintenance.AuditPartitionSwitchGuard? guard, TimeSpan? commandTimeout, CancellationToken ct)
|
||||
{
|
||||
// The switch batch below runs via
|
||||
// ExecuteSqlRaw with NO EF user-transaction — it carries its own server-side
|
||||
@@ -526,6 +543,11 @@ VALUES
|
||||
WHERE $PARTITION.pf_AuditLog_Month(OccurredAtUtc) =
|
||||
$partition.pf_AuditLog_Month('{monthBoundaryStr}');";
|
||||
|
||||
// ScadaBridge#42 archive guard (empty for the unguarded purge). Runs under the chain
|
||||
// lock, so no chaining pass or per-channel purge can move the partition between the
|
||||
// check and the SWITCH; an INSERT racing it is caught by the next tick's archive.
|
||||
var guardSql = guard is null ? string.Empty : guard.ToSql(monthBoundaryStr);
|
||||
|
||||
var sql = $@"
|
||||
BEGIN TRY
|
||||
BEGIN TRANSACTION;
|
||||
@@ -536,7 +558,7 @@ VALUES
|
||||
DECLARE @chainLock int;
|
||||
EXEC @chainLock = sp_getapplock @Resource = N'{Maintenance.AuditChainLock.Resource}', @LockMode = 'Exclusive', @LockOwner = 'Transaction', @LockTimeout = -1;
|
||||
IF @chainLock < 0 THROW 51000, 'AuditLog hash-chain lock was not granted.', 1;
|
||||
|
||||
{guardSql}
|
||||
-- 1. Defensive cleanup for databases created before
|
||||
-- AlignAuditLogEventIdUniqueness: the migration drops the
|
||||
-- non-aligned UX_AuditLog_EventId, but a database restored from an
|
||||
|
||||
Reference in New Issue
Block a user