docs(audit): Parquet cold archive shipped, default off (ScadaBridge#42)
Component-AuditLog Retention & Purge describes the archive: directory and
layout, the three files, consistency with the hash chain (SHARED read,
guarded switch, anchor = archived chain end), fail-closed purge, retention
interplay (per-channel overrides are not archived, their tombstones are),
verify/read, and the AuditLog:Archive configuration. alog.md §13.6/§15.2/§16,
CLAUDE.md, HighLevelReqs AL-12, Component-CLI, Component-ManagementService,
docs/components/{CLI,ManagementService}.md and the CLI README drop the
"deferred / 501" wording and document `audit archive list|verify` and the
parquet export.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -181,7 +181,8 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- Payload cap 8 KB by default / 64 KB on error rows; auth headers redacted by default; SQL parameter values captured by default; per-target redaction opt-in. Inbound API: full verbatim capture up to `InboundMaxBytes` (default 1 MiB); request headers stored in `Extra.requestHeaders` (post-redaction); per-method `SkipBodyCapture` flag suppresses bodies while still recording headers + metadata; `AuditInboundCeilingHits` counter surfaced on health snapshot. (M5.3 T7)
|
||||
- Audit-write failure NEVER aborts the user-facing action — audit is best-effort, the action's own success/failure path is authoritative.
|
||||
- 365-day central retention with monthly partition-switch purge; per-channel retention overrides (`AuditLog:PerChannelRetentionDays`) expire rows earlier than the global window via a bounded, batched row DELETE on the purge actor's maintenance path — values must be shorter than the global window (M5.5 T3); 7-day site SQLite retention with a hard `ForwardState` invariant (no row purged until forwarded or reconciled).
|
||||
- Append-only enforced via DB roles (writer role has INSERT only, no UPDATE/DELETE); Parquet archival (T2) is deferred to v1.x.
|
||||
- Append-only enforced via DB roles (writer role has INSERT only, no UPDATE/DELETE).
|
||||
- **Parquet cold archive (T2, ScadaBridge#42) — default OFF (`AuditLog:Archive:Enabled` + absolute `Directory`, shared by both centrals).** The purge actor writes each expired partition to `{Directory}/{YYYY}/auditlog-{YYYY-MM}-{stamp}Z.parquet` (10 stored columns + `ChainSeq`/`RowHash`, tick-exact `OccurredAtUtcTicks`) plus `.tombstones.parquet` and a `.manifest.json` (written last; chain head at archive time, SHA-256 per file), read under the chain lock SHARED; then the switch runs with a guard (under the EXCLUSIVE lock: partition row count + each month's `LastSeq` must still equal the archive's, else THROW 51002/51003 and nothing is purged). Fail closed: a partition that cannot be archived is not purged. `audit archive list|verify` = `ListAuditArchives`/`VerifyAuditArchive` (verify-chain's gate): file hashes, chain recomputed from the archive, chain end vs the DB purge anchor (`Matches`/`Superseded`/`NotPurged`/`Mismatch`), CLI exit 3. `audit export --format parquet` streams the 24 export columns (was 501). No migration; no re-import path (open decision), object storage not wired (mount it).
|
||||
- **Hash-chain tamper evidence (T1, ScadaBridge#41) — default OFF (`AuditLog:HashChain:Enabled`).** One SHA-256 chain per UTC calendar month: `ChainSeq` + `RowHash = SHA-256(prev || canonical(row))` on `AuditLog` (migration `AddAuditLogHashChain`), linked by the central `AuditLogHashChainService` (both nodes, serialised by `sp_getapplock`) — never by the ingest path. `AuditLogChainHead` catches tail deletion and carries the partition-switch checkpoint anchor; the per-channel DELETE writes an `AuditLogChainTombstone` per chained row in the same transaction. `audit verify-chain --month` = management command `VerifyAuditChain` (OperationalAudit, unscoped callers only), first break with partition/ChainSeq/EventId, CLI exit 3. A chained row's `SourceNode` is frozen (backfill skips it). Unsigned; signing/external anchoring are open.
|
||||
- Node-of-origin is captured alongside site-of-origin: `SourceNode` (`varchar(64)` NULL) on `AuditLog`, `Notifications`, and `SiteCalls` — `node-a`/`node-b` for site rows (qualified by `SourceSiteId`/`SourceSite`), `central-a`/`central-b` for central direct-write rows. Stamped at the writing node, carried verbatim through telemetry + reconciliation, and indexed via `IX_AuditLog_Node_Occurred (SourceNode, OccurredAtUtc)` on `AuditLog`.
|
||||
- Per-node stuck KPIs (M5.3 T6): Notification Outbox and Site Call Audit expose `PerNodeNotificationKpiRequest`/`PerNodeSiteCallKpiRequest` messages that group stuck/parked/delivered counts by `SourceNode`, surfacing per-node breakdowns on the Health dashboard.
|
||||
|
||||
@@ -571,7 +571,7 @@ MS SQL handles this with batched ingest and the time-aligned indexes.
|
||||
### 13.6 Levers
|
||||
- Reduce `DefaultCapBytes` per §8.1.
|
||||
- Tighten per-channel retention per §12.1 (especially `DbOutbound.DbWrite` read traffic).
|
||||
- Defer to v1.x: Parquet archival to object storage before purge (§15.2).
|
||||
- Parquet archival before purge (§15.2) — shipped by ScadaBridge#42, default off.
|
||||
|
||||
---
|
||||
|
||||
@@ -601,8 +601,13 @@ New `scadabridge audit` command group:
|
||||
|
||||
Requires the same **OperationalAudit** / **AuditExport** permissions as the UI.
|
||||
|
||||
### 15.2 Object-storage archival (deferred, v1.x)
|
||||
A monthly job dumps the closing partition to Parquet on operator-configured object storage before central purge — enabling indefinite cold retention without bloating MS SQL. Flag for v1.x; not in initial scope.
|
||||
### 15.2 Parquet archival (shipped, ScadaBridge#42 — default off)
|
||||
The retention purge dumps each expiring monthly partition to Parquet before switching it out — enabling indefinite cold retention without bloating MS SQL. Shipped as:
|
||||
- `AuditLog:Archive:Enabled` (default **off**) + `Directory` (absolute path both central nodes reach — a shared volume or UNC share; object storage is mounted, not called).
|
||||
- Per partition: `auditlog-YYYY-MM-<stamp>Z.parquet` (the stored columns + `ChainSeq`/`RowHash`), `.tombstones.parquet` (per-channel purge tombstones) and `.manifest.json` (counts, SHA-256 per file, the month's chain head at archive time), under `{Directory}/{YYYY}/`.
|
||||
- Archive before purge, fail closed: the switch runs only after the archive is durable, and re-checks under the chain lock that the partition still holds exactly the archived rows and chain head; otherwise nothing is purged and the next tick re-archives.
|
||||
- `scadabridge audit archive list` / `verify --name <archive>`: verify recomputes the archived chain and ties its end to the checkpoint anchor the purge left (§11.4).
|
||||
- `audit export --format parquet` streams the export columns as Parquet.
|
||||
|
||||
---
|
||||
|
||||
@@ -613,7 +618,7 @@ A monthly job dumps the closing partition to Parquet on operator-configured obje
|
||||
| 1 | Component number | **#23 Audit Log** (README matrix + HighLevelReqs). |
|
||||
| 2 | Nav placement | New top-level **Audit** nav group in Central UI. |
|
||||
| 3 | Hash-chain tamper evidence (§11.4) | Deferred to v1.x; shipped by ScadaBridge#41 (unsigned SHA-256 chain, default off). |
|
||||
| 4 | Parquet archival to object storage (§15.2) | Deferred to v1.x. |
|
||||
| 4 | Parquet archival to object storage (§15.2) | Deferred to v1.x; shipped by ScadaBridge#42 (Parquet on a configured directory, default off). |
|
||||
| 5 | Per-channel retention overrides (§12.1) | Deferred to v1.x. v1 uses a single global `RetentionDays`. |
|
||||
| 6 | Default payload cap | **8 KB** for `RequestSummary` / `ResponseSummary`; **64 KB** on error rows (`Status IN ('Failed', 'Parked', 'Discarded')`). |
|
||||
|
||||
|
||||
@@ -23,7 +23,8 @@ Direct-to-node access (`http://localhost:9001` for `central-a`, `http://localhos
|
||||
Management commands use `POST /management`. Audit query, export, and tree use
|
||||
`GET /api/audit/query`, `/api/audit/export`, and `/api/audit/tree`; source-node
|
||||
backfill uses `POST /api/audit/backfill-source-node`. `audit verify-chain` sends the
|
||||
`VerifyAuditChain` management command (ScadaBridge#41). `debug stream` uses a persistent
|
||||
`VerifyAuditChain` management command (ScadaBridge#41); `audit archive list|verify` send
|
||||
`ListAuditArchives` / `VerifyAuditArchive` (ScadaBridge#42). `debug stream` uses a persistent
|
||||
SignalR connection. Audit export streams the response without buffering it all.
|
||||
|
||||
### Authentication and roles
|
||||
@@ -120,7 +121,7 @@ scadabridge instance deploy --id 7
|
||||
| `db-connection` | `list`, `get`, `create`, `update`, `delete` | `Designer` |
|
||||
| `api-method` | `list`, `get`, `create`, `update`, `delete`, `validate`, `recompile` | `Designer` |
|
||||
| `bundle` | `export`, `preview`, `import` | Designer export; Administrator preview/import |
|
||||
| `audit` | `query`, `export`, `tree`, `backfill-source-node`, `verify-chain` | Audit roles; Administrator backfill; verify-chain needs an unscoped audit role |
|
||||
| `audit` | `query`, `export`, `tree`, `backfill-source-node`, `verify-chain`, `archive list`, `archive verify` | Audit roles; Administrator backfill; verify-chain and archive need an unscoped audit role |
|
||||
| `audit-config` | `query` (config-change audit trail; was `audit-log` pre-M8) | `Admin` |
|
||||
| `health` | `summary`, `pending-updates`, `site`, `event-log`, `parked-messages` | Authenticated reads with server-side scope checks |
|
||||
| `debug` | `snapshot`, `stream` | `Deployer` |
|
||||
@@ -195,7 +196,7 @@ The management URL is resolved in this order: `--url` flag → `SCADABRIDGE_MANA
|
||||
|
||||
- [Management Service (#18)](./ManagementService.md) — the server-side counterpart. Every CLI command (except the audit REST/placeholder and SignalR stream operations) translates to a named management command dispatched through `POST /management` to the `ManagementActor`. Role enforcement and LDAP authentication are applied there. The `ManagementCommandRegistry` in Commons maps command types to their names; both sides must stay in sync.
|
||||
- [Traefik Proxy (#20)](./TraefikProxy.md) — the recommended connection target. Pointing `--url` at the Traefik address ensures requests reach the active central node without per-command failover logic in the CLI. The `debug stream` command's WebSocket connection (SignalR `/hubs/debug-stream`) also traverses Traefik, which proxies the WebSocket upgrade natively.
|
||||
- [Audit Log (#23)](./AuditLog.md) — the `audit` command group targets the query, export, tree, and backfill REST endpoints exposed by the Audit Log component, bypassing the management command envelope; `verify-chain` is a local placeholder. The `audit-config` group (formerly `audit-log`) targets the configuration-change audit trail (`IAuditService`) via the standard management envelope.
|
||||
- [Audit Log (#23)](./AuditLog.md) — the `audit` command group targets the query, export, tree, and backfill REST endpoints exposed by the Audit Log component, bypassing the management command envelope; `verify-chain` and `archive list|verify` are management commands. The `audit-config` group (formerly `audit-log`) targets the configuration-change audit trail (`IAuditService`) via the standard management envelope.
|
||||
- [Security & Auth (#10)](./Security.md) — the server resolves LDAP credentials and maps group memberships to ScadaBridge roles (including `Administrator`, `Designer`, `Deployer`, `Operator`, `Verifier`, and audit roles). The CLI does not interact with LDAP directly.
|
||||
- [Commons (#16)](./Commons.md) — owns the management command record types and the `ManagementCommandRegistry` that maps each type to its wire name. The CLI project references Commons for these contracts.
|
||||
- [Transport (#24)](./Transport.md) — the `bundle` command group drives the Transport feature: `bundle export` requests a base64-encoded bundle from the server and streams it to a local `.scadabundle` file; `bundle preview` uploads a file and returns the diff manifest; `bundle import` uploads a file and applies it with a configurable conflict policy.
|
||||
|
||||
@@ -128,7 +128,7 @@ private void HandleEnvelope(ManagementEnvelope envelope)
|
||||
`AuditEndpoints.MapAuditAPI` registers two GET endpoints that go directly to `IAuditLogRepository`, bypassing the actor:
|
||||
|
||||
- `GET /api/audit/query` — keyset-paged JSON result. Requires `OperationalAudit` permission (Admin / Audit / AuditReadOnly roles). Accepts `channel`, `kind`, `status`, `sourceSiteId`, `correlationId`, `executionId`, `parentExecutionId`, `fromUtc`, `toUtc`, `pageSize`, and cursor params `afterOccurredAtUtc`/`afterEventId`. Returns `{ events, nextCursor }` where `nextCursor` is explicit `null` on the last page.
|
||||
- `GET /api/audit/export` — server-side streaming export (CSV or JSONL) of all matching rows, paging the repository internally at 1 000 rows per batch and flushing after each batch. Requires `AuditExport` permission (Admin / Audit roles). `format=parquet` returns HTTP 501 (deferred).
|
||||
- `GET /api/audit/export` — server-side streaming export (CSV, JSONL or Parquet) of all matching rows, paging the repository internally at 1 000 rows per batch and flushing after each batch (Parquet: after each 10 000-row row group; same 24 columns, content type `application/vnd.apache.parquet` — ScadaBridge#42). Requires `AuditExport` permission (Admin / Audit roles).
|
||||
|
||||
Both endpoints apply the same HTTP Basic Auth / LDAP / role flow as `/management`. Site-scoped callers have their `sourceSiteId` filter intersected with their `PermittedSiteIds`; an explicit out-of-scope filter returns HTTP 403 rather than silently empty results.
|
||||
|
||||
|
||||
@@ -759,6 +759,51 @@ MS SQL for direct-write events). Unredacted secrets never persist.
|
||||
partition switch-out loop. Values are validated to be in
|
||||
`[30, RetentionDays]`; keys that are not a recognized `AuditChannel` enum name
|
||||
are rejected at startup.
|
||||
- **Parquet cold archive (T2, ScadaBridge#42) — default OFF.** With
|
||||
`AuditLog:Archive:Enabled` the purge actor archives every expired partition before
|
||||
switching it out, so audit rows outlive `RetentionDays` as Parquet files:
|
||||
- **Where.** `AuditLog:Archive:Directory` — an absolute path (validated at startup)
|
||||
that both central nodes reach, because the purge runs on the active node and
|
||||
`audit archive list|verify` on whichever node serves the call. Files land in
|
||||
`{Directory}/{YYYY}/`. Object storage (S3 / Azure Blob) is not called directly —
|
||||
mount it as a file system or copy the files on.
|
||||
- **What.** Per partition, three files sharing the stem
|
||||
`auditlog-{YYYY-MM}-{archivedAt:yyyyMMddTHHmmssfff}Z`: `.parquet` (every row of the
|
||||
partition — the ten stored columns, `OccurredAtUtc` both as a UTC microsecond
|
||||
timestamp and as exact `OccurredAtUtcTicks`, and `ChainSeq`/`RowHash`, null for
|
||||
rows the chaining job had not reached; computed columns are not stored, they derive
|
||||
from `DetailsJson`), `.tombstones.parquet` (the month's per-channel purge
|
||||
tombstones) and `.manifest.json` (row / chained / unchained / tombstone counts,
|
||||
SHA-256 and size of each file, and each chain month's head — `LastSeq`, `LastHash`,
|
||||
and the anchor an earlier purge left). Each file is written to a `.tmp` sibling,
|
||||
flushed and renamed; the manifest goes last and is the commit marker. Archives are
|
||||
never overwritten. Row groups of `RowGroupSize` (default 50,000) bound memory.
|
||||
- **Consistent with the hash chain.** The partition's rows, heads and tombstones are
|
||||
read in one transaction holding the chain lock SHARED, so no chaining pass or purge
|
||||
moves the chain mid-read. The switch then runs with an archive guard: under the
|
||||
EXCLUSIVE chain lock it re-counts the partition and re-reads each month's
|
||||
`LastSeq`, and THROWs (51002 row count changed / 51003 chain advanced — nothing
|
||||
purged) unless both still equal the archive's. The switch's checkpoint anchor
|
||||
(`PurgedThroughSeq/Hash`) therefore equals the archive's recorded chain end. A
|
||||
stale archive keeps manifest status `Archived`; the next tick writes a fresh one.
|
||||
- **Fail closed.** A partition that cannot be archived (volume full, lock busy past
|
||||
`ChainLockTimeoutSeconds`, …) is not purged: the failure takes the usual
|
||||
`AuditLogPurgeFailedEvent` / `PurgeFailures` path and the table keeps the rows.
|
||||
A successful archive publishes `AuditLogArchivedEvent` before `AuditLogPurgedEvent`.
|
||||
- **Retention interplay.** Archiving changes nothing about *when* a partition is
|
||||
purged (`RetentionDays`). Per-channel overrides still delete their rows earlier,
|
||||
so those rows are never archived — the archive carries their tombstones instead,
|
||||
which is what keeps the archived chain verifiable. Archive files have no retention
|
||||
of their own.
|
||||
- **Verify / read.** `audit archive verify --name` (`VerifyAuditArchive`) checks both
|
||||
files' SHA-256 and counts against the manifest, recomputes each archived month's
|
||||
chain from the archived rows and tombstones (`AuditChainVerifier`, starting from
|
||||
any earlier anchor), and compares the chain end with the database's current anchor:
|
||||
`Matches`; `Superseded` (late rows for the month were chained, purged and archived
|
||||
again afterwards — that later archive resumes from this one's end); `NotPurged`;
|
||||
or `Mismatch` (archive or head altered). The files are plain Parquet — DuckDB,
|
||||
pyarrow or Spark read them directly. There is no re-import into `dbo.AuditLog`:
|
||||
a re-inserted expired row would be purged again by the next tick.
|
||||
- **Sites:** `SiteAuditRetentionService` (site-only `IHostedService`) runs the
|
||||
site SQLite retention purge on a timer — first tick after
|
||||
`AuditLog:SiteRetention:InitialDelay` (default 5 min, short so a daily-recycled
|
||||
@@ -868,9 +913,12 @@ MS SQL for direct-write events). Unredacted secrets never persist.
|
||||
(SELECT on `AuditLog`, column-level `UPDATE (ChainSeq, RowHash)`, read/write on the
|
||||
head), grants the purger the head/tombstone rights its purge paths need, and DENYs
|
||||
INSERT/UPDATE/DELETE on both chain tables to `scadabridge_audit_writer`.
|
||||
- **Parquet archival (T2) — deferred to v1.x.** Long-term cold storage of purged
|
||||
monthly partitions as Parquet files (suitable for offline analytics) will be
|
||||
added in a future milestone. (T1 shipped later — see above; T2 is still deferred.)
|
||||
- **Parquet archival (T2) — shipped (ScadaBridge#42), default OFF.** See Retention &
|
||||
Purge. The archive carries the hash-chain columns and the chain head, so tamper
|
||||
evidence survives the purge: `audit archive verify` ties an archive to the checkpoint
|
||||
anchor still in `AuditLogChainHead`. Like the chain itself, archives are unsigned;
|
||||
a principal able to rewrite both the archive files and the head row can forge a
|
||||
consistent pair.
|
||||
- **Site SQLite security.** File permissions: read/write by the ScadaBridge
|
||||
service account only. Not backed up off-machine — site SQLite is a buffer,
|
||||
not a record.
|
||||
@@ -934,6 +982,21 @@ the global window; `PerChannelRetentionDays` specifies per-channel windows that
|
||||
are strictly shorter — any channel whose override equals or exceeds the global
|
||||
value is silently ignored (the global partition switch-out already governs it).
|
||||
|
||||
The nested `AuditLog:Archive` section (ScadaBridge#42) turns on the Parquet cold
|
||||
archive (see Retention & Purge):
|
||||
|
||||
```jsonc
|
||||
"AuditLog": {
|
||||
"Archive": {
|
||||
"Enabled": false, // default off
|
||||
"Directory": "D:\\AuditArchive", // absolute; required when Enabled; reachable by both centrals
|
||||
"RowGroupSize": 50000, // 1,000 – 1,000,000
|
||||
"ReadCommandTimeoutMinutes": 60, // reading one partition out of SQL Server
|
||||
"ChainLockTimeoutSeconds": 300 // wait for a chaining pass before skipping the tick
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The nested `AuditLog:Purge` section controls the purge actor cadence and batch
|
||||
size:
|
||||
|
||||
|
||||
@@ -304,7 +304,8 @@ The `scadabridge audit` group targets the centralized Audit Log component (#23)
|
||||
exposes the UI-equivalent operational audit surface. Permissions follow the same
|
||||
read-vs-export split the Central UI uses (see Component-AuditLog.md, Security &
|
||||
Tamper-Evidence, and Security & Auth #10): `audit query`, `audit tree`, and
|
||||
`audit verify-chain` require the `OperationalAudit` permission; `audit export`
|
||||
`audit verify-chain` and `audit archive list|verify` require the `OperationalAudit`
|
||||
permission; `audit export`
|
||||
additionally requires `AuditExport`; `audit backfill-source-node` requires the
|
||||
`Admin` role (maintenance path only). The server enforces permission checks and
|
||||
returns HTTP 403 (CLI exit code 2) on denial.
|
||||
@@ -315,6 +316,8 @@ scadabridge audit export --since <t> --until <t> --format csv|jsonl|parquet --ou
|
||||
scadabridge audit tree --execution-id <guid> [--format table|json]
|
||||
scadabridge audit backfill-source-node --before <ISO-8601-UTC> [--sentinel <value>] [--batch <n>]
|
||||
scadabridge audit verify-chain --month <YYYY-MM>
|
||||
scadabridge audit archive list
|
||||
scadabridge audit archive verify --name <archive>
|
||||
```
|
||||
|
||||
- `audit query` — filtered query against the central `AuditLog` table, matching the
|
||||
@@ -323,8 +326,9 @@ scadabridge audit verify-chain --month <YYYY-MM>
|
||||
JSON (default) or table.
|
||||
- `audit export` — server-side streaming export of the central `AuditLog` to the
|
||||
requested format (`csv`, `jsonl`, `parquet`) written to `--output`. The server
|
||||
streams rows rather than materializing them in memory; the CLI writes bytes
|
||||
through to disk. Supports the same scoping filters as `audit query`.
|
||||
streams rows rather than materializing them in memory (Parquet: one row group of
|
||||
10,000 rows at a time, the same 24 columns as csv/jsonl — ScadaBridge#42); the CLI
|
||||
writes bytes through to disk. Supports the same scoping filters as `audit query`.
|
||||
- `audit tree --execution-id <guid>` (M5.3 T8) — renders the full execution-chain
|
||||
tree for the given `ExecutionId`. The server resolves the root from any node in
|
||||
the chain (walks `ParentExecutionId` to find the root, then traverses downward)
|
||||
@@ -346,6 +350,13 @@ scadabridge audit verify-chain --month <YYYY-MM>
|
||||
`0` intact or not chained, `3` broken, `1` error, `2` authorization failure. Requires
|
||||
`OperationalAudit` and a caller not scoped to specific sites. See
|
||||
Component-AuditLog.md, Security & Tamper-Evidence.
|
||||
- `audit archive list` / `audit archive verify --name <archive>` (ScadaBridge#42) —
|
||||
send the `ListAuditArchives` / `VerifyAuditArchive` management commands for the
|
||||
AuditLog Parquet cold archives the retention purge writes (`AuditLog:Archive`,
|
||||
default off). `verify` checks the files against their manifest, recomputes the
|
||||
archived hash chain and compares its end with the purge anchor; exit `0` verified,
|
||||
`3` not verified, `1` error, `2` authorization failure. Same gate as `verify-chain`.
|
||||
See Component-AuditLog.md, Retention & Purge.
|
||||
|
||||
### Health Commands
|
||||
```
|
||||
|
||||
@@ -200,6 +200,7 @@ The two-person authorization workflow for writes through the MxAccess Gateway. B
|
||||
|
||||
- **QueryAuditLog**: Legacy configuration-change audit query (filtered by entity type, user, date range, etc.) routed through `/management`. Gated to the `Admin` role; superseded for the centralized Audit Log component (#23) by the dedicated `/api/audit/*` REST endpoints described below.
|
||||
- **VerifyAuditChain** (ScadaBridge#41): verifies one UTC calendar month (`Month`, `YYYY-MM`) of the central AuditLog hash chain and returns an `AuditChainVerificationResult` (status, rows verified, tombstones applied, purge anchor, head sequence, unchained rows, first break with partition / `ChainSeq` / `EventId` / kind). Gated on the OperationalAudit roles (`Administrator`, `Viewer`); a site-scoped non-Administrator caller is refused (`ManagementUnauthorized`) because a month's chain spans every site. Long-running Ask timeout. Errors: `INVALID_ARGUMENT` (bad month), `CHAIN_BUSY` (a chaining/purge pass held the chain lock past 60 s). Backs `scadabridge audit verify-chain`.
|
||||
- **ListAuditArchives** / **VerifyAuditArchive** (ScadaBridge#42): list the AuditLog Parquet cold archives under `AuditLog:Archive:Directory` (`AuditArchiveListResult`: enabled flag + every manifest), and verify one by `Name` (`AuditArchiveVerificationResult`: file integrity, per-month chain recomputed from the archived rows, anchor `Matches`/`Superseded`/`NotPurged`/`Mismatch`, overall `Verified`). Same gate as VerifyAuditChain (OperationalAudit roles, site-scoped non-Administrator refused). VerifyAuditArchive uses the long-running Ask timeout. Errors: `ARCHIVE_NOT_CONFIGURED`, `INVALID_ARGUMENT` (not an archive name), `NOT_FOUND`. Back `scadabridge audit archive list|verify`.
|
||||
|
||||
### Transport (Bundle Import / Export)
|
||||
|
||||
|
||||
@@ -480,7 +480,7 @@ Sections 10.1–10.4 cover **configuration-database audit** (config-mutating use
|
||||
- **AL-9**: The site SQLite Audit Log is purged only when `ForwardState ∈ {Forwarded, Reconciled}` — i.e., a row must be either confirmed-forwarded *or* confirmed-reconciled before it can be removed. A central outage therefore **cannot cause audit loss at sites**.
|
||||
- **AL-10**: The Central UI exposes an **Audit Log page** with a cross-channel filter (by site, target, status, time range, correlation ID), plus **drill-ins from existing operational pages** (Site Calls, Notification Outbox, Inbound API).
|
||||
- **AL-11**: Append-only semantics are **enforced via DB roles** (no UPDATE/DELETE granted on the `AuditLog` table to application accounts); a **tamper-evidence hash chain** (per-month SHA-256 chain, default off) shipped later by ScadaBridge#41.
|
||||
- **AL-12**: The CLI provides a `scadabridge audit` command group for query, export, and hash-chain verification (verify-chain reports the first break of a month's hash chain) against the central Audit Log.
|
||||
- **AL-12**: The CLI provides a `scadabridge audit` command group for query, export (csv, jsonl, parquet), hash-chain verification (verify-chain reports the first break of a month's hash chain), and listing/verifying the Parquet cold archives (ScadaBridge#42) against the central Audit Log.
|
||||
|
||||
## 11. Health Monitoring
|
||||
|
||||
|
||||
@@ -1996,8 +1996,10 @@ scadabridge --url <url> audit export --since <time> --until <time> --format <fmt
|
||||
| `--target` | no | — | Filter by target |
|
||||
| `--actor` | no | — | Filter by actor |
|
||||
|
||||
> `--format parquet` is accepted by the CLI but the server returns `501 Not
|
||||
> Implemented` — Parquet archival is deferred to v1.x (see `Component-AuditLog.md`).
|
||||
> `--format parquet` (ScadaBridge#42) writes one Parquet file with the same 24 columns
|
||||
> as `csv`/`jsonl` (ids as strings, UTC microsecond timestamps, enums as names). A
|
||||
> central older than #42 answers `501 Not Implemented`; the CLI prints the message and
|
||||
> exits 1.
|
||||
> Use `csv` or `jsonl`.
|
||||
|
||||
#### `audit tree` (M5.3 T8)
|
||||
@@ -2069,6 +2071,40 @@ end, `DuplicateSequence`, `IncompleteLink`). `--format table` prints a summary;
|
||||
The chain only exists once `AuditLog:HashChain:Enabled` is on for the central nodes
|
||||
(default off); see `Component-AuditLog.md`, Security & Tamper-Evidence.
|
||||
|
||||
#### `audit archive list` / `audit archive verify`
|
||||
|
||||
List and verify the AuditLog Parquet cold archives (ScadaBridge#42) that the retention
|
||||
purge writes when `AuditLog:Archive:Enabled` is on (default off).
|
||||
|
||||
```sh
|
||||
scadabridge --url <url> audit archive list
|
||||
scadabridge --url <url> audit archive verify --name <archive>
|
||||
```
|
||||
|
||||
| Option | Required | Default | Description |
|
||||
|--------|----------|---------|-------------|
|
||||
| `--name` | yes (`verify`) | — | Archive name as `list` prints it: `auditlog-YYYY-MM-yyyyMMddTHHmmssfffZ` |
|
||||
|
||||
`list` sends `ListAuditArchives` and prints each archive's name, partition, status
|
||||
(`Archived` = written, purge not confirmed; `Purged` = the partition was switched out
|
||||
and the archive is the only copy), row counts and size. `verify` sends
|
||||
`VerifyAuditArchive`: the server checks both Parquet files' SHA-256 against the
|
||||
manifest, recomputes the archived month's hash chain from the archived rows and
|
||||
tombstones, and compares the chain end with the checkpoint anchor the purge left in
|
||||
`AuditLogChainHead` (`Matches`, `Superseded` by a later purge of late rows,
|
||||
`NotPurged`, or `Mismatch`).
|
||||
|
||||
| Exit code | Meaning |
|
||||
|-----------|---------|
|
||||
| `0` | Verified |
|
||||
| `3` | Not verified — a file was altered, the chain is broken, or the anchor mismatches |
|
||||
| `1` | Request error (`ARCHIVE_NOT_CONFIGURED`, `NOT_FOUND`, `INVALID_ARGUMENT`, server error) |
|
||||
| `2` | Authorization failure (needs `Administrator` or `Viewer`, not site-scoped) |
|
||||
|
||||
The archive files are plain Parquet; read them with any Parquet tool, e.g. DuckDB:
|
||||
`SELECT * FROM '<dir>/2025/auditlog-2025-10-….parquet'`. See `Component-AuditLog.md`,
|
||||
Retention & Purge.
|
||||
|
||||
---
|
||||
|
||||
### `audit-config` — Configuration-change audit log
|
||||
|
||||
Reference in New Issue
Block a user