fix(audit): write-through on completed channel, poison-batch isolation, drain-fault fallback

This commit is contained in:
Joseph Doherty
2026-08-15 12:37:09 -04:00
parent 07b83561d1
commit 7b2d04605e
6 changed files with 492 additions and 274 deletions
+30 -5
View File
@@ -244,14 +244,39 @@ authenticated call reads. The schema bootstrap now runs once, from the drain's `
When the channel is full the newest event is dropped and counted rather than blocking the
producer: a stalled audit database must cost audit completeness, not gateway availability. Drops
are logged once and reported in aggregate on each sweep. Shutdown drains what is buffered under a
2-second cap. Where no hosted service runs — the `apikey` admin CLI — the writer falls back to the
synchronous path, so audit is never buffered into a channel nobody drains.
2-second cap.
Every other failure mode degrades to synchronous writes rather than to silent loss. The writer
falls back to the direct path whenever nothing is draining: before the drain attaches, after it
detaches, where no hosted service runs at all (the `apikey` admin CLI), and when the channel has
been completed — so no attach/detach sequence can leave producers filling a buffer with no reader.
If the drain loop itself dies it detaches the writer on the way out, which reverts every producer
to the direct path. A batch that will not commit is retried one event at a time, so an unwritable
row costs only itself instead of the up-to-63 good events sharing its transaction.
**All** audit is channelled, including admin and CRUD records — dashboard key create/revoke/rotate,
session Close/Kill, and the library-forwarded API-key lifecycle entries. The alternative considered
was keeping those on the synchronous writer and channelling only high-volume denial audit. It was
rejected because a single dashboard key-create emits two records through two different seams (the
library's `create-key` via `IApiKeyAuditStore`, and the enriching `dashboard-create-key` via
`IAuditWriter`); splitting them across two durability regimes gives an auditor a per-producer
matrix to reason about instead of one rule. The residual exposure is explicit: **if the gateway
process dies between the enqueue and the batch commit, buffered audit events are lost.** The window
is bounded by drain latency — the drain wakes on every write and commits immediately, so it is
sub-millisecond under normal load — and it does not apply to the `apikey` CLI, which writes
synchronously. Audit is a best-effort record of what the gateway did, not a write-ahead log of what
it is about to do; a deployment that needs crash-durable admin audit should ship the events off-box
rather than rely on this table.
`MxGateway:Security:AuditRetentionDays` (default 90, minimum 1) bounds the table: the drain sweeps
at startup and hourly, deleting older rows. Retention cannot be configured off. The sweep compares
through SQLite's `datetime()` rather than on the stored ISO-8601 text, because timestamps written
from a non-UTC offset do not sort lexicographically against a UTC cutoff; a row whose timestamp
cannot be parsed yields NULL and is kept rather than deleted.
through SQLite's `datetime()` rather than on the stored ISO-8601 text. Text comparison is correct
only while every row is UTC-normalized — which the canonical model guarantees for rows written
through the store, but not for rows that entered the table any other way — and on a mixed-format
column it silently deletes live audit, because `2026-05-17T09:00:00-05:00` is two hours after a
`2026-05-17T12:00:00+00:00` cutoff yet sorts before it. Comparing instants is correct however the
text got there, and a timestamp `datetime()` cannot parse yields NULL, so undateable audit is kept
rather than swept.
## Authorization