fix(auditlog): page the site audit purge, bound incremental_vacuum per page, fail fast instead of parking writers on a full queue (review 2026-10-21 06/MEM-1)

This commit is contained in:
Joseph Doherty
2026-10-22 06:23:35 -04:00
parent 268b55e350
commit 1da5d0bac4
6 changed files with 261 additions and 56 deletions
+21 -3
View File
@@ -574,6 +574,17 @@ MS SQL for direct-write events). Unredacted secrets never persist.
(disk full, schema lock, transient IO), events buffer in a small in-memory
ring (default 1024 rows); oldest are discarded with a Site Event Log warning
per drop.
- **A saturated writer fails fast; it never parks callers.** The site writer's
bounded queue (`AuditLog:SiteWriter:ChannelCapacity`, default 4096) used to
make `WriteAsync` *await* room once full — and because every hot-path caller is
fire-and-continue, a writer stuck behind a long purge accumulated an unbounded
number of parked waiters, each holding its event, while reporting healthy. A
full queue now faults the write immediately with
`SqliteAuditWriterSaturatedException`; `FallbackAuditWriter` treats it like
any other primary failure — the event goes to the drop-oldest ring above and
`SiteAuditWriteFailures` is bumped — so the degradation is the documented
best-effort drop with a counter, never unbounded memory (review 2026-10-21
06/MEM-1). Audit failure still never aborts the action.
- **Reconciliation as fallback.** If two consecutive reconciliation cycles
report a non-draining backlog, the supervisor restarts the telemetry actor
and a `SiteAuditTelemetryStalled` event fires.
@@ -655,9 +666,16 @@ MS SQL for direct-write events). Unredacted secrets never persist.
node still purges soon after start), then every
`AuditLog:SiteRetention:PurgeInterval` (default 24 h, clamped to ≥ 1 min).
Each tick calls `ISiteAuditQueue.PurgeExpiredAsync(UtcNow − RetentionDays)`,
which deletes eligible rows in one transaction (sidecar first, then the
canonical row) and then runs `PRAGMA incremental_vacuum` to return the freed
pages to the OS. `AuditLog:SiteRetention:RetentionDays` defaults to 7 and is
which deletes eligible rows **in pages** of at most
`AuditLog:SiteWriter:PurgeBatchSize` rows (default 5000) — one transaction per
page, sidecar first, then the canonical row — and after each page runs a
**bounded** `PRAGMA incremental_vacuum(N)` (`AuditLog:SiteWriter:PurgeVacuumPagesPerBatch`,
default 1000 pages) to return freed pages to the OS. The write lock is taken per
page and released between pages, so the batched hot-path writer interleaves with a
multi-million-row purge instead of stalling behind one giant DELETE plus an
unbounded vacuum (review 2026-10-21 06/MEM-1 — the same hazard register row 34
closed for `site_events`). One call still removes every eligible row; only the
lock grain changed. `AuditLog:SiteRetention:RetentionDays` defaults to 7 and is
clamped to `[1, 90]`. Respects the hard `ForwardState` invariant — a `Pending`
row is never purged on age alone; only `Forwarded`/`Reconciled` rows older than
the cutoff are removed. Per-tick failures are swallowed and logged so a transient