Parquet audit archive: destination/object storage, share + permissions, retention, signing, stale-archive cleanup (#42) #109

New Issue

2026-11-09T17:19:53-05:00

dohertj2 commented

2026-11-09 17:19:53 -05:00

Owner decisions for the Parquet cold archive (#42, off by default):

Destination: the spec says object storage; it's implemented as a configured directory (mount/copy for S3/Azure Blob) — want a native object-storage client + credentials (Secrets store)?
Which share both central nodes use (UNC) and its permissions; add an AuditLog:Archive section to DEV/QA configs when enabling.
Archive retention — files are never deleted today.
No re-import path (re-inserted expired rows would be purged again) — confirm Parquet-tools-only reading is acceptable.
Archives are unsigned, like the #41 chain (decision tracked in #107).
A retried purge leaves its first stale archive (status Archived; verify reports NotPurged/Superseded) — add cleanup?
Also: the deferred-work register / stillpending.md still describe Parquet as deferred (left untouched per the backlog rule).

Owner decisions for the Parquet cold archive (#42, off by default): 1. Destination: the spec says object storage; it's implemented as a configured directory (mount/copy for S3/Azure Blob) — want a native object-storage client + credentials (Secrets store)? 2. Which share both central nodes use (UNC) and its permissions; add an `AuditLog:Archive` section to DEV/QA configs when enabling. 3. Archive retention — files are never deleted today. 4. No re-import path (re-inserted expired rows would be purged again) — confirm Parquet-tools-only reading is acceptable. 5. Archives are unsigned, like the #41 chain (decision tracked in #107). 6. A retried purge leaves its first stale archive (status Archived; verify reports NotPurged/Superseded) — add cleanup? Also: the deferred-work register / stillpending.md still describe Parquet as deferred (left untouched per the backlog rule).

dohertj2 added the type:followup priority:p3 labels 2026-11-09 17:19:53 -05:00

dohertj2 commented

2026-11-09 22:13:26 -05:00

Owner decision 2026-11-09: keep the directory/UNC destination (no native object-storage client); configurable retention, default never-delete until compliance names a period; parquet-tools-only reading accepted (no re-import); add cleanup of superseded stale archives; signing follows #107 (head export, no signing). Share name + permissions to be named when it is enabled on DEV/QA.

**Owner decision 2026-11-09:** keep the **directory/UNC destination** (no native object-storage client); **configurable retention, default never-delete** until compliance names a period; parquet-tools-only reading accepted (no re-import); **add cleanup of superseded stale archives**; signing follows #107 (head export, no signing). Share name + permissions to be named when it is enabled on DEV/QA.

dohertj2 referenced this issue

2026-11-09 23:32:37 -05:00

AuditLog archive: orphaned files of an interrupted archive write are never cleaned up #114

dohertj2 referenced this issue

2026-11-09 23:33:24 -05:00

Audit hash chain: signing/anchoring decision, enablement plan, chainer login, migration deploy gate (#41) #107

dohertj2 commented

2026-11-09 23:33:24 -05:00

Done — b95ee5a6 (main). Closing; the share name + permissions are named when the archive is enabled on DEV/QA (#107 item 2 / enablement), no code needed for that.

Owner decisions implemented / recorded (Component-AuditLog.md, Retention & Purge; CLAUDE.md):

Destination stays a directory/UNC — no object-storage client. Doc now states the share is chosen at enablement and what rights it needs (both centrals' service account: write/rename/delete; auditors read-only; the SQL login none).
Retention AuditLog:Archive:RetentionDays — default 0 = never delete; when set must exceed AuditLog:RetentionDays (startup validation, AuditLogArchiveRetentionValidator), counted from the archived month's end. Expired archives are deleted when Purged, or Archived with no rows left in the partition.
Parquet-tools-only reading (DuckDB / pyarrow / Spark / parquet-tools), no re-import — documented as the supported path.
Unsigned, following #107 (off-box head export; verify-heads exposes a rewritten anchor for exported months).
Superseded stale-archive cleanup AuditLog:Archive:DeleteSupersededStaleArchives (default true), run by the purge actor after every tick: a retried purge's Archived archive is deleted only when a later, verifying, Purged archive of the same partition provably holds every row — same prior anchor and the stale chain-end hash at its chain-end sequence (row or tombstone), every unchained row present by EventId. Otherwise reported KeptStale (e.g. an unchained row a per-channel purge removed in between leaves no tombstone → the stale archive is its only copy). An archive left at Archived by a crash between switch and manifest update has no later purged archive and is never touched.

Safe deleter: only the archive name's own three files in its {Directory}/{YYYY} folder, manifest first, never a path from the manifest; failures surface as purge failures.
scadabridge audit archive cleanup [--dry-run] = new CleanupAuditArchives management command (Administrator only).

Evidence: AuditLogArchiveCleanupTests (MSSQL): retried purge → stale deleted / later kept + still verifies, dry run deletes nothing; stale with a missing row kept; sole copy untouched; retention 0 / not-yet-expired / expired / partition-still-live; tampered manifest path (../../outside.parquet) cannot delete outside. Mutations — dropping the unchained-row proof, following the manifest path, dropping the partition-live check — each fail their test. Plus purge-actor (cleanup every tick incl. no-boundary ticks, failure contained, off when archiving off), validator, management (role gate, dry-run flag, not-configured) and CLI tests.
Totals: AuditLog 479, ConfigurationDatabase 531, ManagementService 790, CLI 592, Host 671 — green; build 0 warnings. No migration.

Follow-ups filed: #114 (orphan files of an interrupted archive write), #113, #115.

**Done — `b95ee5a6` (main).** Closing; the share name + permissions are named when the archive is enabled on DEV/QA (#107 item 2 / enablement), no code needed for that. Owner decisions implemented / recorded (Component-AuditLog.md, Retention & Purge; CLAUDE.md): 1. Destination stays a directory/UNC — no object-storage client. Doc now states the share is chosen at enablement and what rights it needs (both centrals' service account: write/rename/delete; auditors read-only; the SQL login none). 2. **Retention** `AuditLog:Archive:RetentionDays` — default `0` = never delete; when set must exceed `AuditLog:RetentionDays` (startup validation, `AuditLogArchiveRetentionValidator`), counted from the archived month's end. Expired archives are deleted when `Purged`, or `Archived` with no rows left in the partition. 3. Parquet-tools-only reading (DuckDB / pyarrow / Spark / parquet-tools), no re-import — documented as the supported path. 4. Unsigned, following #107 (off-box head export; `verify-heads` exposes a rewritten anchor for exported months). 5. **Superseded stale-archive cleanup** `AuditLog:Archive:DeleteSupersededStaleArchives` (default true), run by the purge actor after every tick: a retried purge's `Archived` archive is deleted only when a later, verifying, `Purged` archive of the same partition **provably** holds every row — same prior anchor and the stale chain-end hash at its chain-end sequence (row or tombstone), every unchained row present by EventId. Otherwise reported `KeptStale` (e.g. an unchained row a per-channel purge removed in between leaves no tombstone → the stale archive is its only copy). An archive left at `Archived` by a crash between switch and manifest update has no later purged archive and is never touched. - Safe deleter: only the archive name's own three files in its `{Directory}/{YYYY}` folder, manifest first, never a path from the manifest; failures surface as purge failures. - `scadabridge audit archive cleanup [--dry-run]` = new `CleanupAuditArchives` management command (Administrator only). Evidence: `AuditLogArchiveCleanupTests` (MSSQL): retried purge → stale deleted / later kept + still verifies, dry run deletes nothing; stale with a missing row kept; sole copy untouched; retention 0 / not-yet-expired / expired / partition-still-live; tampered manifest path (`../../outside.parquet`) cannot delete outside. Mutations — dropping the unchained-row proof, following the manifest path, dropping the partition-live check — each fail their test. Plus purge-actor (cleanup every tick incl. no-boundary ticks, failure contained, off when archiving off), validator, management (role gate, dry-run flag, not-configured) and CLI tests. Totals: AuditLog 479, ConfigurationDatabase 531, ManagementService 790, CLI 592, Host 671 — green; build 0 warnings. No migration. Follow-ups filed: #114 (orphan files of an interrupted archive write), #113, #115.

dohertj2 closed this issue

2026-11-09 23:33:25 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/ScadaBridge#109