docs(audit): Parquet cold archive shipped, default off (ScadaBridge#42)

Component-AuditLog Retention & Purge describes the archive: directory and
layout, the three files, consistency with the hash chain (SHARED read,
guarded switch, anchor = archived chain end), fail-closed purge, retention
interplay (per-channel overrides are not archived, their tombstones are),
verify/read, and the AuditLog:Archive configuration. alog.md §13.6/§15.2/§16,
CLAUDE.md, HighLevelReqs AL-12, Component-CLI, Component-ManagementService,
docs/components/{CLI,ManagementService}.md and the CLI README drop the
"deferred / 501" wording and document `audit archive list|verify` and the
parquet export.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

docs/components/CLI.md

@@ -23,7 +23,8 @@ Direct-to-node access (`http://localhost:9001` for `central-a`, `http://localhos
 Management commands use `POST /management`. Audit query, export, and tree use
 `GET /api/audit/query`, `/api/audit/export`, and `/api/audit/tree`; source-node
 backfill uses `POST /api/audit/backfill-source-node`. `audit verify-chain` sends the
-`VerifyAuditChain` management command (ScadaBridge#41). `debug stream` uses a persistent
+`VerifyAuditChain` management command (ScadaBridge#41); `audit archive list|verify` send
+`ListAuditArchives` / `VerifyAuditArchive` (ScadaBridge#42). `debug stream` uses a persistent
 SignalR connection. Audit export streams the response without buffering it all.
 
 ### Authentication and roles
@@ -120,7 +121,7 @@ scadabridge instance deploy --id 7
 | `db-connection` | `list`, `get`, `create`, `update`, `delete` | `Designer` |
 | `api-method` | `list`, `get`, `create`, `update`, `delete`, `validate`, `recompile` | `Designer` |
 | `bundle` | `export`, `preview`, `import` | Designer export; Administrator preview/import |
-| `audit` | `query`, `export`, `tree`, `backfill-source-node`, `verify-chain` | Audit roles; Administrator backfill; verify-chain needs an unscoped audit role |
+| `audit` | `query`, `export`, `tree`, `backfill-source-node`, `verify-chain`, `archive list`, `archive verify` | Audit roles; Administrator backfill; verify-chain and archive need an unscoped audit role |
 | `audit-config` | `query` (config-change audit trail; was `audit-log` pre-M8) | `Admin` |
 | `health` | `summary`, `pending-updates`, `site`, `event-log`, `parked-messages` | Authenticated reads with server-side scope checks |
 | `debug` | `snapshot`, `stream` | `Deployer` |
@@ -195,7 +196,7 @@ The management URL is resolved in this order: `--url` flag → `SCADABRIDGE_MANA
 
 - [Management Service (#18)](./ManagementService.md) — the server-side counterpart. Every CLI command (except the audit REST/placeholder and SignalR stream operations) translates to a named management command dispatched through `POST /management` to the `ManagementActor`. Role enforcement and LDAP authentication are applied there. The `ManagementCommandRegistry` in Commons maps command types to their names; both sides must stay in sync.
 - [Traefik Proxy (#20)](./TraefikProxy.md) — the recommended connection target. Pointing `--url` at the Traefik address ensures requests reach the active central node without per-command failover logic in the CLI. The `debug stream` command's WebSocket connection (SignalR `/hubs/debug-stream`) also traverses Traefik, which proxies the WebSocket upgrade natively.
-- [Audit Log (#23)](./AuditLog.md) — the `audit` command group targets the query, export, tree, and backfill REST endpoints exposed by the Audit Log component, bypassing the management command envelope; `verify-chain` is a local placeholder. The `audit-config` group (formerly `audit-log`) targets the configuration-change audit trail (`IAuditService`) via the standard management envelope.
+- [Audit Log (#23)](./AuditLog.md) — the `audit` command group targets the query, export, tree, and backfill REST endpoints exposed by the Audit Log component, bypassing the management command envelope; `verify-chain` and `archive list|verify` are management commands. The `audit-config` group (formerly `audit-log`) targets the configuration-change audit trail (`IAuditService`) via the standard management envelope.
 - [Security & Auth (#10)](./Security.md) — the server resolves LDAP credentials and maps group memberships to ScadaBridge roles (including `Administrator`, `Designer`, `Deployer`, `Operator`, `Verifier`, and audit roles). The CLI does not interact with LDAP directly.
 - [Commons (#16)](./Commons.md) — owns the management command record types and the `ManagementCommandRegistry` that maps each type to its wire name. The CLI project references Commons for these contracts.
 - [Transport (#24)](./Transport.md) — the `bundle` command group drives the Transport feature: `bundle export` requests a base64-encoded bundle from the server and streams it to a local `.scadabundle` file; `bundle preview` uploads a file and returns the diff manifest; `bundle import` uploads a file and applies it with a configurable conflict policy.

docs/components/ManagementService.md

@@ -128,7 +128,7 @@ private void HandleEnvelope(ManagementEnvelope envelope)
 `AuditEndpoints.MapAuditAPI` registers two GET endpoints that go directly to `IAuditLogRepository`, bypassing the actor:
 
 - `GET /api/audit/query` — keyset-paged JSON result. Requires `OperationalAudit` permission (Admin / Audit / AuditReadOnly roles). Accepts `channel`, `kind`, `status`, `sourceSiteId`, `correlationId`, `executionId`, `parentExecutionId`, `fromUtc`, `toUtc`, `pageSize`, and cursor params `afterOccurredAtUtc`/`afterEventId`. Returns `{ events, nextCursor }` where `nextCursor` is explicit `null` on the last page.
-- `GET /api/audit/export` — server-side streaming export (CSV or JSONL) of all matching rows, paging the repository internally at 1 000 rows per batch and flushing after each batch. Requires `AuditExport` permission (Admin / Audit roles). `format=parquet` returns HTTP 501 (deferred).
+- `GET /api/audit/export` — server-side streaming export (CSV, JSONL or Parquet) of all matching rows, paging the repository internally at 1 000 rows per batch and flushing after each batch (Parquet: after each 10 000-row row group; same 24 columns, content type `application/vnd.apache.parquet` — ScadaBridge#42). Requires `AuditExport` permission (Admin / Audit roles).
 
 Both endpoints apply the same HTTP Basic Auth / LDAP / role flow as `/management`. Site-scoped callers have their `sourceSiteId` filter intersected with their `PermittedSiteIds`; an explicit out-of-scope filter returns HTTP 403 rather than silently empty results.
 

docs/requirements/Component-AuditLog.md

@@ -759,6 +759,51 @@ MS SQL for direct-write events). Unredacted secrets never persist.
   partition switch-out loop. Values are validated to be in
   `[30, RetentionDays]`; keys that are not a recognized `AuditChannel` enum name
   are rejected at startup.
+- **Parquet cold archive (T2, ScadaBridge#42) — default OFF.** With
+  `AuditLog:Archive:Enabled` the purge actor archives every expired partition before
+  switching it out, so audit rows outlive `RetentionDays` as Parquet files:
+  - **Where.** `AuditLog:Archive:Directory` — an absolute path (validated at startup)
+    that both central nodes reach, because the purge runs on the active node and
+    `audit archive list|verify` on whichever node serves the call. Files land in
+    `{Directory}/{YYYY}/`. Object storage (S3 / Azure Blob) is not called directly —
+    mount it as a file system or copy the files on.
+  - **What.** Per partition, three files sharing the stem
+    `auditlog-{YYYY-MM}-{archivedAt:yyyyMMddTHHmmssfff}Z`: `.parquet` (every row of the
+    partition — the ten stored columns, `OccurredAtUtc` both as a UTC microsecond
+    timestamp and as exact `OccurredAtUtcTicks`, and `ChainSeq`/`RowHash`, null for
+    rows the chaining job had not reached; computed columns are not stored, they derive
+    from `DetailsJson`), `.tombstones.parquet` (the month's per-channel purge
+    tombstones) and `.manifest.json` (row / chained / unchained / tombstone counts,
+    SHA-256 and size of each file, and each chain month's head — `LastSeq`, `LastHash`,
+    and the anchor an earlier purge left). Each file is written to a `.tmp` sibling,
+    flushed and renamed; the manifest goes last and is the commit marker. Archives are
+    never overwritten. Row groups of `RowGroupSize` (default 50,000) bound memory.
+  - **Consistent with the hash chain.** The partition's rows, heads and tombstones are
+    read in one transaction holding the chain lock SHARED, so no chaining pass or purge
+    moves the chain mid-read. The switch then runs with an archive guard: under the
+    EXCLUSIVE chain lock it re-counts the partition and re-reads each month's
+    `LastSeq`, and THROWs (51002 row count changed / 51003 chain advanced — nothing
+    purged) unless both still equal the archive's. The switch's checkpoint anchor
+    (`PurgedThroughSeq/Hash`) therefore equals the archive's recorded chain end. A
+    stale archive keeps manifest status `Archived`; the next tick writes a fresh one.
+  - **Fail closed.** A partition that cannot be archived (volume full, lock busy past
+    `ChainLockTimeoutSeconds`, …) is not purged: the failure takes the usual
+    `AuditLogPurgeFailedEvent` / `PurgeFailures` path and the table keeps the rows.
+    A successful archive publishes `AuditLogArchivedEvent` before `AuditLogPurgedEvent`.
+  - **Retention interplay.** Archiving changes nothing about *when* a partition is
+    purged (`RetentionDays`). Per-channel overrides still delete their rows earlier,
+    so those rows are never archived — the archive carries their tombstones instead,
+    which is what keeps the archived chain verifiable. Archive files have no retention
+    of their own.
+  - **Verify / read.** `audit archive verify --name` (`VerifyAuditArchive`) checks both
+    files' SHA-256 and counts against the manifest, recomputes each archived month's
+    chain from the archived rows and tombstones (`AuditChainVerifier`, starting from
+    any earlier anchor), and compares the chain end with the database's current anchor:
+    `Matches`; `Superseded` (late rows for the month were chained, purged and archived
+    again afterwards — that later archive resumes from this one's end); `NotPurged`;
+    or `Mismatch` (archive or head altered). The files are plain Parquet — DuckDB,
+    pyarrow or Spark read them directly. There is no re-import into `dbo.AuditLog`:
+    a re-inserted expired row would be purged again by the next tick.
 - **Sites:** `SiteAuditRetentionService` (site-only `IHostedService`) runs the
   site SQLite retention purge on a timer — first tick after
   `AuditLog:SiteRetention:InitialDelay` (default 5 min, short so a daily-recycled
@@ -868,9 +913,12 @@ MS SQL for direct-write events). Unredacted secrets never persist.
     (SELECT on `AuditLog`, column-level `UPDATE (ChainSeq, RowHash)`, read/write on the
     head), grants the purger the head/tombstone rights its purge paths need, and DENYs
     INSERT/UPDATE/DELETE on both chain tables to `scadabridge_audit_writer`.
-- **Parquet archival (T2) — deferred to v1.x.** Long-term cold storage of purged
-  monthly partitions as Parquet files (suitable for offline analytics) will be
-  added in a future milestone. (T1 shipped later — see above; T2 is still deferred.)
+- **Parquet archival (T2) — shipped (ScadaBridge#42), default OFF.** See Retention &
+  Purge. The archive carries the hash-chain columns and the chain head, so tamper
+  evidence survives the purge: `audit archive verify` ties an archive to the checkpoint
+  anchor still in `AuditLogChainHead`. Like the chain itself, archives are unsigned;
+  a principal able to rewrite both the archive files and the head row can forge a
+  consistent pair.
 - **Site SQLite security.** File permissions: read/write by the ScadaBridge
   service account only. Not backed up off-machine — site SQLite is a buffer,
   not a record.
@@ -934,6 +982,21 @@ the global window; `PerChannelRetentionDays` specifies per-channel windows that
 are strictly shorter — any channel whose override equals or exceeds the global
 value is silently ignored (the global partition switch-out already governs it).
 
+The nested `AuditLog:Archive` section (ScadaBridge#42) turns on the Parquet cold
+archive (see Retention & Purge):
+
+```jsonc
+"AuditLog": {
+  "Archive": {
+    "Enabled": false,                 // default off
+    "Directory": "D:\\AuditArchive",   // absolute; required when Enabled; reachable by both centrals
+    "RowGroupSize": 50000,            // 1,000 – 1,000,000
+    "ReadCommandTimeoutMinutes": 60,  // reading one partition out of SQL Server
+    "ChainLockTimeoutSeconds": 300    // wait for a chaining pass before skipping the tick
+  }
+}
+```
+
 The nested `AuditLog:Purge` section controls the purge actor cadence and batch
 size:
 

docs/requirements/Component-CLI.md

@@ -304,7 +304,8 @@ The `scadabridge audit` group targets the centralized Audit Log component (#23)
 exposes the UI-equivalent operational audit surface. Permissions follow the same
 read-vs-export split the Central UI uses (see Component-AuditLog.md, Security &
 Tamper-Evidence, and Security & Auth #10): `audit query`, `audit tree`, and
-`audit verify-chain` require the `OperationalAudit` permission; `audit export`
+`audit verify-chain` and `audit archive list|verify` require the `OperationalAudit`
+permission; `audit export`
 additionally requires `AuditExport`; `audit backfill-source-node` requires the
 `Admin` role (maintenance path only). The server enforces permission checks and
 returns HTTP 403 (CLI exit code 2) on denial.
@@ -315,6 +316,8 @@ scadabridge audit export --since <t> --until <t> --format csv|jsonl|parquet --ou
 scadabridge audit tree --execution-id <guid> [--format table|json]
 scadabridge audit backfill-source-node --before <ISO-8601-UTC> [--sentinel <value>] [--batch <n>]
 scadabridge audit verify-chain --month <YYYY-MM>
+scadabridge audit archive list
+scadabridge audit archive verify --name <archive>
 ```
 
 - `audit query` — filtered query against the central `AuditLog` table, matching the
@@ -323,8 +326,9 @@ scadabridge audit verify-chain --month <YYYY-MM>
   JSON (default) or table.
 - `audit export` — server-side streaming export of the central `AuditLog` to the
   requested format (`csv`, `jsonl`, `parquet`) written to `--output`. The server
-  streams rows rather than materializing them in memory; the CLI writes bytes
-  through to disk. Supports the same scoping filters as `audit query`.
+  streams rows rather than materializing them in memory (Parquet: one row group of
+  10,000 rows at a time, the same 24 columns as csv/jsonl — ScadaBridge#42); the CLI
+  writes bytes through to disk. Supports the same scoping filters as `audit query`.
 - `audit tree --execution-id <guid>` (M5.3 T8) — renders the full execution-chain
   tree for the given `ExecutionId`. The server resolves the root from any node in
   the chain (walks `ParentExecutionId` to find the root, then traverses downward)
@@ -346,6 +350,13 @@ scadabridge audit verify-chain --month <YYYY-MM>
   `0` intact or not chained, `3` broken, `1` error, `2` authorization failure. Requires
   `OperationalAudit` and a caller not scoped to specific sites. See
   Component-AuditLog.md, Security & Tamper-Evidence.
+- `audit archive list` / `audit archive verify --name <archive>` (ScadaBridge#42) —
+  send the `ListAuditArchives` / `VerifyAuditArchive` management commands for the
+  AuditLog Parquet cold archives the retention purge writes (`AuditLog:Archive`,
+  default off). `verify` checks the files against their manifest, recomputes the
+  archived hash chain and compares its end with the purge anchor; exit `0` verified,
+  `3` not verified, `1` error, `2` authorization failure. Same gate as `verify-chain`.
+  See Component-AuditLog.md, Retention & Purge.
 
 ### Health Commands
 ```

docs/requirements/Component-ManagementService.md

@@ -200,6 +200,7 @@ The two-person authorization workflow for writes through the MxAccess Gateway. B
 
 - **QueryAuditLog**: Legacy configuration-change audit query (filtered by entity type, user, date range, etc.) routed through `/management`. Gated to the `Admin` role; superseded for the centralized Audit Log component (#23) by the dedicated `/api/audit/*` REST endpoints described below.
 - **VerifyAuditChain** (ScadaBridge#41): verifies one UTC calendar month (`Month`, `YYYY-MM`) of the central AuditLog hash chain and returns an `AuditChainVerificationResult` (status, rows verified, tombstones applied, purge anchor, head sequence, unchained rows, first break with partition / `ChainSeq` / `EventId` / kind). Gated on the OperationalAudit roles (`Administrator`, `Viewer`); a site-scoped non-Administrator caller is refused (`ManagementUnauthorized`) because a month's chain spans every site. Long-running Ask timeout. Errors: `INVALID_ARGUMENT` (bad month), `CHAIN_BUSY` (a chaining/purge pass held the chain lock past 60 s). Backs `scadabridge audit verify-chain`.
+- **ListAuditArchives** / **VerifyAuditArchive** (ScadaBridge#42): list the AuditLog Parquet cold archives under `AuditLog:Archive:Directory` (`AuditArchiveListResult`: enabled flag + every manifest), and verify one by `Name` (`AuditArchiveVerificationResult`: file integrity, per-month chain recomputed from the archived rows, anchor `Matches`/`Superseded`/`NotPurged`/`Mismatch`, overall `Verified`). Same gate as VerifyAuditChain (OperationalAudit roles, site-scoped non-Administrator refused). VerifyAuditArchive uses the long-running Ask timeout. Errors: `ARCHIVE_NOT_CONFIGURED`, `INVALID_ARGUMENT` (not an archive name), `NOT_FOUND`. Back `scadabridge audit archive list|verify`.
 
 ### Transport (Bundle Import / Export)
 

docs/requirements/HighLevelReqs.md

@@ -480,7 +480,7 @@ Sections 10.1–10.4 cover **configuration-database audit** (config-mutating use
 - **AL-9**: The site SQLite Audit Log is purged only when `ForwardState ∈ {Forwarded, Reconciled}` — i.e., a row must be either confirmed-forwarded *or* confirmed-reconciled before it can be removed. A central outage therefore **cannot cause audit loss at sites**.
 - **AL-10**: The Central UI exposes an **Audit Log page** with a cross-channel filter (by site, target, status, time range, correlation ID), plus **drill-ins from existing operational pages** (Site Calls, Notification Outbox, Inbound API).
 - **AL-11**: Append-only semantics are **enforced via DB roles** (no UPDATE/DELETE granted on the `AuditLog` table to application accounts); a **tamper-evidence hash chain** (per-month SHA-256 chain, default off) shipped later by ScadaBridge#41.
-- **AL-12**: The CLI provides a `scadabridge audit` command group for query, export, and hash-chain verification (verify-chain reports the first break of a month's hash chain) against the central Audit Log.
+- **AL-12**: The CLI provides a `scadabridge audit` command group for query, export (csv, jsonl, parquet), hash-chain verification (verify-chain reports the first break of a month's hash chain), and listing/verifying the Parquet cold archives (ScadaBridge#42) against the central Audit Log.
 
 ## 11. Health Monitoring