fix(audit): fail closed when a configured redactor is unavailable (#35)

Component-AuditLog.md has always required "we over-redact, never under-redact,
on configuration faults", but the body / SQL-parameter redactors violated it.

AuditRegexCache rejects a pattern that is malformed OR whose compile exceeds a
100 ms budget, caching the rejection for the process lifetime.
ScadaBridgeAuditRedactor then simply dropped the rejected pattern from its
redactor set and emitted the payload anyway — publishing precisely the values
the operator configured it to suppress, onto a row that looks entirely normal
downstream. Recovery required a process restart and the only signal was one
Warning line. The SQL path was worse: TryGetSqlParamRedactor returned a bare
false for both "no redactor configured for this connection" and "the configured
one will not compile", and CLAUDE.md records SQL parameter capture as on by
default.

Two changes:

1. Fail closed. A pattern that is CONFIGURED but unavailable now over-redacts
   the whole payload and increments AuditRedactionFailure, reusing the existing
   safety-net path. "Not configured at all" stays permissive — conflating those
   two states is the actual defect, so both are pinned by tests.

2. Precompile off the hot path. The audit-log roadmap specifies patterns are
   "precompiled at startup; rejected if compile takes >100ms"; the implementation
   had drifted to compiling lazily on first event, which put a wall-clock budget
   on a hot path under production load. RegexOptions.Compiled emits IL during
   construction, so a busy node could blow the budget on a perfectly valid
   pattern. Warm-up now runs at construction and on every options reload. The
   residual window between a reload and its warm-up is safe because that path
   now fails closed.

Warm-up deliberately does not fail the boot — an unusable pattern degrades the
node to over-redaction (safe, loud) rather than refusing to start. Reading
CurrentValue happens inside the warm-up try so an options provider that throws
still surfaces via Apply's over-redact path, not the constructor
(OuterCatch_OptionsThrows_NeverLeaks_AllSensitiveFieldsOverRedacted).

Also de-flakes GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer,
which is how this was found. It black-holed node A behind a 300 ms deadline, but
on a saturated machine the call could fail to even START — a genuinely-unsent
failure that IsConnectFailure correctly fails over on, so node B's ack arrived
instead of the expected Status.Failure. The test read as a flake while actually
reporting that its own premise had not held. Split in two: the hard rule now
injects an explicit DeadlineExceeded via a trailers-only response (deterministic,
load-independent), and a new BlackHoledNode_DoesNotHang covers the
deadline-is-actually-applied half with both nodes black-holed so no ack can
arrive down any path.

Verified: both fixes were confirmed to fail before they pass — reverting the
fail-closed guard fails exactly the 5 fail-closed tests while the 4 controls
still pass, and adding DeadlineExceeded to IsConnectFailure fails the rewritten
transport test. AuditLog 367/367, Host.Tests GrpcCentralTransport 8/8, solution
build clean. The previously-intermittent
Filter_PicksUp_NewBodyRedactor_OnConfigReload is green in a full sweep for the
first time.

Not addressed here, and noted on #35: the 100 ms wall-clock budget remains a
weak proxy for catastrophic backtracking (RegexOptions.Compiled defers JIT to
first match, so construction time measures the wrong thing), and a rejection is
still cached permanently. Both are now safe rather than dangerous, so they are
hardening rather than a leak.
This commit is contained in:
Joseph Doherty
2026-08-12 03:04:50 -04:00
parent 7e594054e4
commit 006202f3c7
5 changed files with 654 additions and 17 deletions
+25
View File
@@ -406,6 +406,31 @@ operational `SiteCalls` shape for the dispatcher and UI.
- **Safety net** — if a configured redactor throws, the affected payload becomes
`"<redacted: redactor error>"` and `AuditRedactionFailure` increments. We
over-redact, never under-redact, on configuration faults.
- **Unavailable redactors fail closed** — a configured pattern that will not
compile (malformed, or over the regex cache's 100 ms compile budget) makes the
whole payload over-redact for that row, exactly as a throwing redactor does.
It is NOT dropped from the redactor set. Dropping it published precisely the
values the operator configured it to suppress, on a row that looked entirely
normal downstream — a silent under-redaction, which this section has always
forbidden (Gitea #35). The distinction that matters is *configured but
unavailable* (suppress) versus *not configured at all* (capture as normal);
conflating the two is what caused the defect, so both cases are pinned by
`AuditRedactorFailClosedTests`.
Redactor patterns are precompiled when the options snapshot is bound and again on
every reload, so the compile budget is spent off the audit hot path. Compiling
lazily on first event instead put a wall-clock budget on a hot path under
production load, where `RegexOptions.Compiled` IL emission can exceed it for a
perfectly valid pattern — and a rejection is cached for the process lifetime, so
one unlucky moment disabled that redactor until restart. A narrow window remains
between a reload and its warm-up; an event landing there compiles on the hot path
and, if rejected, fails closed, so the worst case is over-redaction.
An unusable pattern does **not** fail the boot: the node degrades to
over-redaction (safe and loud via `AuditRedactionFailure` plus a startup warning
naming the count) rather than refusing to start. Operators should treat a
non-zero `AuditRedactionFailure` with over-redacted payloads as "fix the
pattern", not as a payload-capture problem.
Redaction happens at the write site, before the row touches SQLite (or central
MS SQL for direct-write events). Unredacted secrets never persist.