dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 15 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(audit): Parquet cold archive shipped, default off (ScadaBridge#42)

Browse Source 
Component-AuditLog Retention & Purge describes the archive: directory and
layout, the three files, consistency with the hash chain (SHARED read,
guarded switch, anchor = archived chain end), fail-closed purge, retention
interplay (per-channel overrides are not archived, their tombstones are),
verify/read, and the AuditLog:Archive configuration. alog.md §13.6/§15.2/§16,
CLAUDE.md, HighLevelReqs AL-12, Component-CLI, Component-ManagementService,
docs/components/{CLI,ManagementService}.md and the CLI README drop the
"deferred / 501" wording and document `audit archive list|verify` and the
parquet export.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 17:01:03 -05:00
parent d1fc336e18
commit 3ff463b6fd
9 changed files with 136 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
alog.md
+9 -4
View File
@@ -571,7 +571,7 @@ MS SQL handles this with batched ingest and the time-aligned indexes.
### 13.6 Levers
- Reduce `DefaultCapBytes` per §8.1.
- Tighten per-channel retention per §12.1 (especially `DbOutbound.DbWrite` read traffic).
- Defer to v1.x: Parquet archival to object storage before purge (§15.2).
- Parquet archival before purge (§15.2) — shipped by ScadaBridge#42, default off.
---
@@ -601,8 +601,13 @@ New `scadabridge audit` command group:
Requires the same **OperationalAudit** / **AuditExport** permissions as the UI.
### 15.2 Object-storage archival (deferred, v1.x)
A monthly job dumps the closing partition to Parquet on operator-configured object storage before central purge — enabling indefinite cold retention without bloating MS SQL. Flag for v1.x; not in initial scope.
### 15.2 Parquet archival (shipped, ScadaBridge#42 — default off)
The retention purge dumps each expiring monthly partition to Parquet before switching it out — enabling indefinite cold retention without bloating MS SQL. Shipped as:
- `AuditLog:Archive:Enabled` (default **off**) + `Directory` (absolute path both central nodes reach — a shared volume or UNC share; object storage is mounted, not called).
- Per partition: `auditlog-YYYY-MM-<stamp>Z.parquet` (the stored columns + `ChainSeq`/`RowHash`), `.tombstones.parquet` (per-channel purge tombstones) and `.manifest.json` (counts, SHA-256 per file, the month's chain head at archive time), under `{Directory}/{YYYY}/`.
- Archive before purge, fail closed: the switch runs only after the archive is durable, and re-checks under the chain lock that the partition still holds exactly the archived rows and chain head; otherwise nothing is purged and the next tick re-archives.
- `scadabridge audit archive list` / `verify --name <archive>`: verify recomputes the archived chain and ties its end to the checkpoint anchor the purge left (§11.4).
- `audit export --format parquet` streams the export columns as Parquet.
---
@@ -613,7 +618,7 @@ A monthly job dumps the closing partition to Parquet on operator-configured obje
| 1 | Component number | **#23 Audit Log** (README matrix + HighLevelReqs). |
| 2 | Nav placement | New top-level **Audit** nav group in Central UI. |
| 3 | Hash-chain tamper evidence (§11.4) | Deferred to v1.x; shipped by ScadaBridge#41 (unsigned SHA-256 chain, default off). |
| 4 | Parquet archival to object storage (§15.2) | Deferred to v1.x. |
| 4 | Parquet archival to object storage (§15.2) | Deferred to v1.x; shipped by ScadaBridge#42 (Parquet on a configured directory, default off). |
| 5 | Per-channel retention overrides (§12.1) | Deferred to v1.x. v1 uses a single global `RetentionDays`. |
| 6 | Default payload cap | **8 KB** for `RequestSummary` / `ResponseSummary`; **64 KB** on error rows (`Status IN ('Failed', 'Parked', 'Discarded')`). |