feat(audit): Parquet cold archive of expired AuditLog partitions before the purge (ScadaBridge#42)

Audit roadmap T2 (alog.md §15.2): with AuditLog:Archive:Enabled (default off)
the purge actor writes each expired monthly partition to Parquet under an
absolute AuditLog:Archive:Directory before switching it out, so audit rows
outlive RetentionDays.

- Parquet.Net 6.1.0 (fully managed; NuGetAudit clean) in central package
  management.
- AuditArchiveParquet: data file = the ten stored columns, OccurredAtUtc as a
  UTC microsecond timestamp plus exact OccurredAtUtcTicks, and ChainSeq/RowHash;
  tombstone file = the month's per-channel purge tombstones. Buffered row
  groups; the files are plain Parquet (DuckDB/pyarrow read them).
- AuditLogPartitionArchive (IAuditLogArchive): reads the partition's rows,
  chain heads and tombstones in one transaction under the chain lock SHARED,
  writes {Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet,
  .tombstones.parquet and a .manifest.json (counts, SHA-256 per file, each
  month's head and prior anchor) via tmp + rename, manifest last. Archives are
  never overwritten.
- Archive before purge: SwitchOutPartitionAsync gains an internal guarded
  overload. Under the EXCLUSIVE chain lock it re-counts the partition and
  re-reads each month's LastSeq and THROWs 51002/51003 (nothing purged) unless
  both still equal the archive's, so the switch's checkpoint anchor is exactly
  the archive's chain end. The manifest is then marked Purged.
- Fail closed: a partition that cannot be archived is not purged; the failure
  takes the existing AuditLogPurgeFailedEvent / PurgeFailures path. Success
  publishes AuditLogArchivedEvent.
- Verify: file hashes and counts against the manifest, the chain recomputed
  from the archived rows and tombstones (AuditChainVerifier), and the chain end
  against the database anchor (Matches / Superseded / NotPurged / Mismatch).
- AuditLogArchiveOptionsValidator: Directory required when enabled and must be
  absolute; RowGroupSize, timeouts bounded.

No migration: the purger role already reads AuditLog, the head and tombstones.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 17:00:49 -05:00
parent 31cb2c79e3
commit 7be4f19861
18 changed files with 2232 additions and 8 deletions
@@ -55,6 +55,12 @@ public static class ServiceCollectionExtensions
/// <summary>Configuration section bound to <see cref="AuditLogHashChainOptions"/> (ScadaBridge#41).</summary>
public const string HashChainSectionName = "AuditLog:HashChain";
/// <summary>
/// Configuration section bound to
/// <see cref="ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Maintenance.AuditLogArchiveOptions"/> (ScadaBridge#42).
/// </summary>
public const string ArchiveSectionName = "AuditLog:Archive";
/// <summary>Configuration section bound to <see cref="ZB.MOM.WW.ScadaBridge.AuditLog.Central.AuditLogPurgeOptions"/>.</summary>
public const string PurgeSectionName = "AuditLog:Purge";
@@ -405,6 +411,12 @@ public static class ServiceCollectionExtensions
config, HashChainSectionName);
services.AddHostedService<AuditLogHashChainService>();
// Parquet cold archive (ScadaBridge#42). Default OFF; when enabled the purge actor
// archives each expired partition before switching it out. IAuditLogArchive is
// registered by AddConfigurationDatabase (central only) and reads these options.
services.AddValidatedOptions<ConfigurationDatabase.Maintenance.AuditLogArchiveOptions, AuditLogArchiveOptionsValidator>(
config, ArchiveSectionName);
// I1 (review): bind the two central-singleton options HERE rather than in
// AddAuditLogCentralReconciliationClient. AkkaHostedService.RegisterCentralActors
// resolves IOptions<AuditLogPurgeOptions> / <SiteAuditReconciliationOptions>