AuditLog archive: orphaned files of an interrupted archive write are never cleaned up #114

New Issue

2026-11-09T23:32:37-05:00

dohertj2 commented

2026-11-09 23:32:37 -05:00

Follow-up from #109. Archive cleanup (AuditLog:Archive:RetentionDays, superseded stale archives) only deletes well-formed archives with a manifest. An archive write interrupted before its manifest (process killed, volume full mid-write) leaves {name}.parquet.tmp / {name}.tombstones.parquet.tmp or renamed data files without a .manifest.json in {Directory}/{YYYY}; nothing ever removes them (the archive path does best-effort TryDelete of its own .tmp files only when it catches the failure).

Proposal: in the cleanup pass, delete .tmp files and manifest-less auditlog-*.parquet files older than e.g. 24 h (never a file a running archive could still be writing), report them as a new cleanup action, dry-run aware. Low priority — disk only, no correctness impact.

Follow-up from #109. Archive cleanup (`AuditLog:Archive:RetentionDays`, superseded stale archives) only deletes **well-formed archives with a manifest**. An archive write interrupted before its manifest (process killed, volume full mid-write) leaves `{name}.parquet.tmp` / `{name}.tombstones.parquet.tmp` or renamed data files without a `.manifest.json` in `{Directory}/{YYYY}`; nothing ever removes them (the archive path does best-effort `TryDelete` of its own `.tmp` files only when it catches the failure). Proposal: in the cleanup pass, delete `.tmp` files and manifest-less `auditlog-*.parquet` files older than e.g. 24 h (never a file a running archive could still be writing), report them as a new cleanup action, dry-run aware. Low priority — disk only, no correctness impact.

dohertj2 added the type:followup priority:p3 labels 2026-11-09 23:32:37 -05:00

dohertj2 referenced this issue from a commit

2026-11-09 23:33:11 -05:00

feat(audit): archive retention and superseded stale-archive cleanup (ScadaBridge#109)

dohertj2 referenced this issue

2026-11-09 23:33:24 -05:00

Parquet audit archive: destination/object storage, share + permissions, retention, signing, stale-archive cleanup (#42) #109

dohertj2 referenced this issue from a commit

2026-11-09 23:47:59 -05:00

fix(audit): maintenance-login roll-forward/backfill, archive orphan cleanup, head-export health (ScadaBridge#113 #114 #115)

dohertj2 commented

2026-11-09 23:48:06 -05:00

Fixed in 533414a1 (main). Archive cleanup now also deletes orphans of an interrupted archive write (action DeletedOrphan, dry-run aware): the writer's own *.parquet.tmp / *.tombstones.parquet.tmp / *.manifest.json.tmp and manifest-less data files, only for well-formed archive names directly inside a 4-digit {Directory}/{YYYY} folder matching the name's year, not reparse points, last written > 24 h ago. Evidence: AuditLogArchiveCleanupTests.InterruptedWriteOrphans_AreRemoved_OnlyWhenOldAndWriterNamed (fresh files, committed archives, foreign/misnamed/wrong-year/wrong-folder files survive); mutation-proved (removing the age check or the manifest check fails it).

Fixed in 533414a1 (main). Archive cleanup now also deletes orphans of an interrupted archive write (action DeletedOrphan, dry-run aware): the writer's own *.parquet.tmp / *.tombstones.parquet.tmp / *.manifest.json.tmp and manifest-less data files, only for well-formed archive names directly inside a 4-digit {Directory}/{YYYY} folder matching the name's year, not reparse points, last written > 24 h ago. Evidence: AuditLogArchiveCleanupTests.InterruptedWriteOrphans_AreRemoved_OnlyWhenOldAndWriterNamed (fresh files, committed archives, foreign/misnamed/wrong-year/wrong-folder files survive); mutation-proved (removing the age check or the manifest check fails it).

dohertj2 closed this issue

2026-11-09 23:48:06 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/ScadaBridge#114