docs(audit): Parquet cold archive shipped, default off (ScadaBridge#42)
Component-AuditLog Retention & Purge describes the archive: directory and
layout, the three files, consistency with the hash chain (SHARED read,
guarded switch, anchor = archived chain end), fail-closed purge, retention
interplay (per-channel overrides are not archived, their tombstones are),
verify/read, and the AuditLog:Archive configuration. alog.md §13.6/§15.2/§16,
CLAUDE.md, HighLevelReqs AL-12, Component-CLI, Component-ManagementService,
docs/components/{CLI,ManagementService}.md and the CLI README drop the
"deferred / 501" wording and document `audit archive list|verify` and the
parquet export.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
@@ -759,6 +759,51 @@ MS SQL for direct-write events). Unredacted secrets never persist.
|
||||
partition switch-out loop. Values are validated to be in
|
||||
`[30, RetentionDays]`; keys that are not a recognized `AuditChannel` enum name
|
||||
are rejected at startup.
|
||||
- **Parquet cold archive (T2, ScadaBridge#42) — default OFF.** With
|
||||
`AuditLog:Archive:Enabled` the purge actor archives every expired partition before
|
||||
switching it out, so audit rows outlive `RetentionDays` as Parquet files:
|
||||
- **Where.** `AuditLog:Archive:Directory` — an absolute path (validated at startup)
|
||||
that both central nodes reach, because the purge runs on the active node and
|
||||
`audit archive list|verify` on whichever node serves the call. Files land in
|
||||
`{Directory}/{YYYY}/`. Object storage (S3 / Azure Blob) is not called directly —
|
||||
mount it as a file system or copy the files on.
|
||||
- **What.** Per partition, three files sharing the stem
|
||||
`auditlog-{YYYY-MM}-{archivedAt:yyyyMMddTHHmmssfff}Z`: `.parquet` (every row of the
|
||||
partition — the ten stored columns, `OccurredAtUtc` both as a UTC microsecond
|
||||
timestamp and as exact `OccurredAtUtcTicks`, and `ChainSeq`/`RowHash`, null for
|
||||
rows the chaining job had not reached; computed columns are not stored, they derive
|
||||
from `DetailsJson`), `.tombstones.parquet` (the month's per-channel purge
|
||||
tombstones) and `.manifest.json` (row / chained / unchained / tombstone counts,
|
||||
SHA-256 and size of each file, and each chain month's head — `LastSeq`, `LastHash`,
|
||||
and the anchor an earlier purge left). Each file is written to a `.tmp` sibling,
|
||||
flushed and renamed; the manifest goes last and is the commit marker. Archives are
|
||||
never overwritten. Row groups of `RowGroupSize` (default 50,000) bound memory.
|
||||
- **Consistent with the hash chain.** The partition's rows, heads and tombstones are
|
||||
read in one transaction holding the chain lock SHARED, so no chaining pass or purge
|
||||
moves the chain mid-read. The switch then runs with an archive guard: under the
|
||||
EXCLUSIVE chain lock it re-counts the partition and re-reads each month's
|
||||
`LastSeq`, and THROWs (51002 row count changed / 51003 chain advanced — nothing
|
||||
purged) unless both still equal the archive's. The switch's checkpoint anchor
|
||||
(`PurgedThroughSeq/Hash`) therefore equals the archive's recorded chain end. A
|
||||
stale archive keeps manifest status `Archived`; the next tick writes a fresh one.
|
||||
- **Fail closed.** A partition that cannot be archived (volume full, lock busy past
|
||||
`ChainLockTimeoutSeconds`, …) is not purged: the failure takes the usual
|
||||
`AuditLogPurgeFailedEvent` / `PurgeFailures` path and the table keeps the rows.
|
||||
A successful archive publishes `AuditLogArchivedEvent` before `AuditLogPurgedEvent`.
|
||||
- **Retention interplay.** Archiving changes nothing about *when* a partition is
|
||||
purged (`RetentionDays`). Per-channel overrides still delete their rows earlier,
|
||||
so those rows are never archived — the archive carries their tombstones instead,
|
||||
which is what keeps the archived chain verifiable. Archive files have no retention
|
||||
of their own.
|
||||
- **Verify / read.** `audit archive verify --name` (`VerifyAuditArchive`) checks both
|
||||
files' SHA-256 and counts against the manifest, recomputes each archived month's
|
||||
chain from the archived rows and tombstones (`AuditChainVerifier`, starting from
|
||||
any earlier anchor), and compares the chain end with the database's current anchor:
|
||||
`Matches`; `Superseded` (late rows for the month were chained, purged and archived
|
||||
again afterwards — that later archive resumes from this one's end); `NotPurged`;
|
||||
or `Mismatch` (archive or head altered). The files are plain Parquet — DuckDB,
|
||||
pyarrow or Spark read them directly. There is no re-import into `dbo.AuditLog`:
|
||||
a re-inserted expired row would be purged again by the next tick.
|
||||
- **Sites:** `SiteAuditRetentionService` (site-only `IHostedService`) runs the
|
||||
site SQLite retention purge on a timer — first tick after
|
||||
`AuditLog:SiteRetention:InitialDelay` (default 5 min, short so a daily-recycled
|
||||
@@ -868,9 +913,12 @@ MS SQL for direct-write events). Unredacted secrets never persist.
|
||||
(SELECT on `AuditLog`, column-level `UPDATE (ChainSeq, RowHash)`, read/write on the
|
||||
head), grants the purger the head/tombstone rights its purge paths need, and DENYs
|
||||
INSERT/UPDATE/DELETE on both chain tables to `scadabridge_audit_writer`.
|
||||
- **Parquet archival (T2) — deferred to v1.x.** Long-term cold storage of purged
|
||||
monthly partitions as Parquet files (suitable for offline analytics) will be
|
||||
added in a future milestone. (T1 shipped later — see above; T2 is still deferred.)
|
||||
- **Parquet archival (T2) — shipped (ScadaBridge#42), default OFF.** See Retention &
|
||||
Purge. The archive carries the hash-chain columns and the chain head, so tamper
|
||||
evidence survives the purge: `audit archive verify` ties an archive to the checkpoint
|
||||
anchor still in `AuditLogChainHead`. Like the chain itself, archives are unsigned;
|
||||
a principal able to rewrite both the archive files and the head row can forge a
|
||||
consistent pair.
|
||||
- **Site SQLite security.** File permissions: read/write by the ScadaBridge
|
||||
service account only. Not backed up off-machine — site SQLite is a buffer,
|
||||
not a record.
|
||||
@@ -934,6 +982,21 @@ the global window; `PerChannelRetentionDays` specifies per-channel windows that
|
||||
are strictly shorter — any channel whose override equals or exceeds the global
|
||||
value is silently ignored (the global partition switch-out already governs it).
|
||||
|
||||
The nested `AuditLog:Archive` section (ScadaBridge#42) turns on the Parquet cold
|
||||
archive (see Retention & Purge):
|
||||
|
||||
```jsonc
|
||||
"AuditLog": {
|
||||
"Archive": {
|
||||
"Enabled": false, // default off
|
||||
"Directory": "D:\\AuditArchive", // absolute; required when Enabled; reachable by both centrals
|
||||
"RowGroupSize": 50000, // 1,000 – 1,000,000
|
||||
"ReadCommandTimeoutMinutes": 60, // reading one partition out of SQL Server
|
||||
"ChainLockTimeoutSeconds": 300 // wait for a chaining pass before skipping the tick
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The nested `AuditLog:Purge` section controls the purge actor cadence and batch
|
||||
size:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user