Component-AuditLog.md has always required "we over-redact, never under-redact,
on configuration faults", but the body / SQL-parameter redactors violated it.
AuditRegexCache rejects a pattern that is malformed OR whose compile exceeds a
100 ms budget, caching the rejection for the process lifetime.
ScadaBridgeAuditRedactor then simply dropped the rejected pattern from its
redactor set and emitted the payload anyway — publishing precisely the values
the operator configured it to suppress, onto a row that looks entirely normal
downstream. Recovery required a process restart and the only signal was one
Warning line. The SQL path was worse: TryGetSqlParamRedactor returned a bare
false for both "no redactor configured for this connection" and "the configured
one will not compile", and CLAUDE.md records SQL parameter capture as on by
default.
Two changes:
1. Fail closed. A pattern that is CONFIGURED but unavailable now over-redacts
the whole payload and increments AuditRedactionFailure, reusing the existing
safety-net path. "Not configured at all" stays permissive — conflating those
two states is the actual defect, so both are pinned by tests.
2. Precompile off the hot path. The audit-log roadmap specifies patterns are
"precompiled at startup; rejected if compile takes >100ms"; the implementation
had drifted to compiling lazily on first event, which put a wall-clock budget
on a hot path under production load. RegexOptions.Compiled emits IL during
construction, so a busy node could blow the budget on a perfectly valid
pattern. Warm-up now runs at construction and on every options reload. The
residual window between a reload and its warm-up is safe because that path
now fails closed.
Warm-up deliberately does not fail the boot — an unusable pattern degrades the
node to over-redaction (safe, loud) rather than refusing to start. Reading
CurrentValue happens inside the warm-up try so an options provider that throws
still surfaces via Apply's over-redact path, not the constructor
(OuterCatch_OptionsThrows_NeverLeaks_AllSensitiveFieldsOverRedacted).
Also de-flakes GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer,
which is how this was found. It black-holed node A behind a 300 ms deadline, but
on a saturated machine the call could fail to even START — a genuinely-unsent
failure that IsConnectFailure correctly fails over on, so node B's ack arrived
instead of the expected Status.Failure. The test read as a flake while actually
reporting that its own premise had not held. Split in two: the hard rule now
injects an explicit DeadlineExceeded via a trailers-only response (deterministic,
load-independent), and a new BlackHoledNode_DoesNotHang covers the
deadline-is-actually-applied half with both nodes black-holed so no ack can
arrive down any path.
Verified: both fixes were confirmed to fail before they pass — reverting the
fail-closed guard fails exactly the 5 fail-closed tests while the 4 controls
still pass, and adding DeadlineExceeded to IsConnectFailure fails the rewritten
transport test. AuditLog 367/367, Host.Tests GrpcCentralTransport 8/8, solution
build clean. The previously-intermittent
Filter_PicksUp_NewBodyRedactor_OnConfigReload is green in a full sweep for the
first time.
Not addressed here, and noted on #35: the 100 ms wall-clock budget remains a
weak proxy for catastrophic backtracking (RegexOptions.Compiled defers JIT to
first match, so construction time measures the wrong thing), and a rejection is
still cached permanently. Both are now safe rather than dangerous, so they are
hardening rather than a leak.
Perf re-baseline (HotPathLatencyTests): empirical p95 on Apple M-series Release
build: 4KB DetailsJson slow path ≈14 µs, small-DetailsJson no-redactors ≈2 µs,
true no-op fast path ≈0 µs. Thresholds updated: 200 µs / 30 µs / 5 µs (≈15×
headroom for contested CI runners). Old thresholds (50 µs / 10 µs) were set for
the pre-C3 typed-field path; canonical JSON parse+rewrite is empirically faster.
Adds a third test (Filter_Apply_NoDetailsJson_FastPath) that asserts same-instance
return on the DetailsJson-null + within-cap fast path. Env-var overrides retained.
CollapseAuditLogToCanonicalMigrationTests (new): three MSSQL-gated [SkippableFact]
tests verifying Action/Category/Outcome projection, NULL Actor, DetailsJson codec
round-trip, and all six persisted computed columns (Kind/Status/SourceSiteId/
ExecutionId/ParentExecutionId) for ApiOutbound, InboundAuthFailure, and Failed-
status rows.
AddAuditLogTableMigrationTests: rename CreatesFiveNamedIndexes →
CreatesNineNamedIndexes; expand coverage from 5 original indexes to all 9 named
non-clustered indexes present after CollapseAuditLogToCanonical (adds
IX_AuditLog_Execution, IX_AuditLog_ParentExecution, IX_AuditLog_Node_Occurred,
UX_AuditLog_EventId).
Dead-cref cleanup: zero references to the deleted IAuditPayloadFilter /
DefaultAuditPayloadFilter / SafeDefaultAuditPayloadFilter types remain in any
.cs file (source or test). 26 occurrences across 13 files replaced with correct
references to IAuditRedactor / ScadaBridgeAuditRedactor / SafeDefaultAuditRedactor
or reworded as plain prose.
Residual sweep: no unused transitional code found beyond the acknowledged
"C3 transitional shim" comments on IngestedAtUtc stamping (active code, not dead).
I1 (security): OverRedact() in ScadaBridgeAuditRedactor now suppresses ErrorDetail,
ErrorMessage, and Extra (in addition to RequestSummary/ResponseSummary) to the
over-redacted marker in BOTH code paths (Deserialize+with path and the fallback
new-AuditDetails path). SafeDefaultAuditRedactor catch block aligned to match.
M3 (test): OuterCatch_OptionsThrows_NeverLeaks_AllSensitiveFieldsOverRedacted forces
the outer try/catch → OverRedact path via a ThrowingMonitor that throws from
CurrentValue (the first statement in the try block). Asserts (a) Apply does not
throw, and (b) all five sensitive free-text fields are suppressed to the
over-redacted marker with PayloadTruncated=true.
M1 (consistency): SafeDefaultAuditRedactor now uses AuditRedactionPrimitives
constants (RedactedMarker for line-format header values, OverRedactedEventMarker
for the catch block), eliminating the divergent [REDACTED]/[redacted by ...]
strings. AuditRedactionPrimitives gains OverRedactedEventMarker = RedactorErrorMarker.
SafeDefaultAuditRedactorTests updated from [REDACTED] → <redacted>.
M2 (comment): Added one-line note in TruncateField explaining why the char-count
(result.Length != value.Length) truncation check is sufficient given TruncateUtf8
only ever shortens.
Joseph Doherty