Merge branch 'feature/audit-log-m5-payload-redaction': Audit Log #23 M5 Payload + Redaction
M5 ships the payload filter pipeline. IAuditPayloadFilter runs between
event construction and writer call:
- Stage 1: HTTP header redaction (Authorization/Cookie/Set-Cookie/X-Api-Key
default list from M1-T9; case-insensitive name match against JSON
{headers,body} shape).
- Stage 2: Body regex redaction (global + per-target). Patterns compiled
at startup with 100ms budget; runtime 50ms timeout guard against
catastrophic backtracking. Over-redact on exception + increment counter.
- Stage 3: SQL parameter redaction (Channel=DbOutbound, per-connection
opt-in via PerTargetOverrides[connection].RedactSqlParamsMatching).
- Stage 4: UTF-8 boundary-safe truncation. Default cap 8 KB; error cap
64 KB on Status NOT IN (Delivered/Submitted/Forwarded). PayloadTruncated
set to true when applied.
Filter wired into all three writer entry points:
- FallbackAuditWriter (site chain) — filter before SqliteAuditWriter.
- CentralAuditWriter (central direct-write) — filter before
IAuditLogRepository.InsertIfNotExistsAsync (NotificationOutbox dispatcher,
AuditWriteMiddleware).
- AuditLogIngestActor — filter before dual-write transaction.
Health metric SiteAuditRedactionFailureCounter wired through the existing
M2 Bundle G + M4 Bundle B health-bridge pattern; central-side counter
deferred to M6 (the milestone that ships the full central health surface).
Hot-reload via IOptionsMonitor + per-call CurrentValue read. Regex cache
keyed by pattern string so changing the config naturally invalidates old
patterns.
Shipped: 11 commits, ~49 net new tests across AuditLog.Tests,
HealthMonitoring.Tests, PerformanceTests. Full solution 24/24 test projects
green. infra/* untouched on any branch commit.
This commit is contained in:
20
docs/plans/2026-05-20-auditlog-m5-payload-redaction.md
Normal file
20
docs/plans/2026-05-20-auditlog-m5-payload-redaction.md
Normal file
@@ -0,0 +1,20 @@
|
|||||||
|
# Audit Log #23 — M5 Payload + Redaction Implementation Plan
|
||||||
|
|
||||||
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development (bundled cadence).
|
||||||
|
|
||||||
|
**Goal:** Filter pipeline (IAuditPayloadFilter) runs between event construction and writer call. Truncates to 8 KB / 64 KB on error; applies HTTP header redactors (default list from M1-T9 AuditLogOptions); applies body regex redactors (global + per-target); applies SQL parameter redactors (per-connection opt-in); over-redacts on regex error and increments AuditRedactionFailure metric. Hot-reloadable config via IOptionsMonitor.
|
||||||
|
|
||||||
|
**Vocabulary (M1 reality):** Error-row cap (64 KB) triggers when `Status NOT IN (AuditStatus.Delivered, AuditStatus.Submitted, AuditStatus.Forwarded)` — i.e., on `Failed/Parked/Discarded/Attempted/Skipped`. The roadmap's M5-T2 step references (Status=TransientFailure/PermanentFailure) are stale pre-M1 wording. Translation: `TransientFailure` = `Attempted` with HttpStatus 5xx OR `Failed`; `PermanentFailure` = `Failed`.
|
||||||
|
|
||||||
|
**M4 realities baked in:** AuditingDb decorators, NotificationOutboxActor, AuditWriteMiddleware, site emission paths all need filter pluggin. Filter is invoked in:
|
||||||
|
- FallbackAuditWriter.WriteAsync (site chain) — before SqliteAuditWriter.WriteAsync.
|
||||||
|
- CentralAuditWriter.WriteAsync (central direct-write) — before IAuditLogRepository.InsertIfNotExistsAsync.
|
||||||
|
- AuditLogIngestActor handlers — before InsertIfNotExistsAsync/UpsertAsync.
|
||||||
|
|
||||||
|
**Bundles:**
|
||||||
|
- Bundle A — Filter contract + truncation (T1, T2).
|
||||||
|
- Bundle B — Header + body + SQL param redaction (T3, T4, T5).
|
||||||
|
- Bundle C — Wire into emission paths + health metric (T6, T7).
|
||||||
|
- Bundle D — Configuration binding + perf + safety-net edge cases (T8, T9, T10).
|
||||||
|
|
||||||
|
Final cross-bundle review + merge.
|
||||||
@@ -1,6 +1,7 @@
|
|||||||
using Akka.Actor;
|
using Akka.Actor;
|
||||||
using Microsoft.Extensions.DependencyInjection;
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
|
using ScadaLink.AuditLog.Payload;
|
||||||
using ScadaLink.Commons.Entities.Audit;
|
using ScadaLink.Commons.Entities.Audit;
|
||||||
using ScadaLink.Commons.Interfaces.Repositories;
|
using ScadaLink.Commons.Interfaces.Repositories;
|
||||||
using ScadaLink.Commons.Messages.Audit;
|
using ScadaLink.Commons.Messages.Audit;
|
||||||
@@ -114,8 +115,15 @@ public class AuditLogIngestActor : ReceiveActor
|
|||||||
// Resolve the repository for the whole batch — one DbContext per
|
// Resolve the repository for the whole batch — one DbContext per
|
||||||
// message, mirroring NotificationOutboxActor. The injected-repository
|
// message, mirroring NotificationOutboxActor. The injected-repository
|
||||||
// mode (Bundle D tests) skips the scope entirely.
|
// mode (Bundle D tests) skips the scope entirely.
|
||||||
|
// Bundle C (M5-T6): the IAuditPayloadFilter is also resolved from the
|
||||||
|
// per-message scope when one is available so the row is truncated +
|
||||||
|
// redacted before InsertIfNotExistsAsync. The single-repository test
|
||||||
|
// ctor has no service provider — it falls through with no filter,
|
||||||
|
// which preserves the small-payload assumptions baked into the
|
||||||
|
// existing D2 fixtures.
|
||||||
IServiceScope? scope = null;
|
IServiceScope? scope = null;
|
||||||
IAuditLogRepository repository;
|
IAuditLogRepository repository;
|
||||||
|
IAuditPayloadFilter? filter = null;
|
||||||
if (_injectedRepository is not null)
|
if (_injectedRepository is not null)
|
||||||
{
|
{
|
||||||
repository = _injectedRepository;
|
repository = _injectedRepository;
|
||||||
@@ -124,6 +132,7 @@ public class AuditLogIngestActor : ReceiveActor
|
|||||||
{
|
{
|
||||||
scope = _serviceProvider!.CreateScope();
|
scope = _serviceProvider!.CreateScope();
|
||||||
repository = scope.ServiceProvider.GetRequiredService<IAuditLogRepository>();
|
repository = scope.ServiceProvider.GetRequiredService<IAuditLogRepository>();
|
||||||
|
filter = scope.ServiceProvider.GetService<IAuditPayloadFilter>();
|
||||||
}
|
}
|
||||||
|
|
||||||
try
|
try
|
||||||
@@ -136,7 +145,11 @@ public class AuditLogIngestActor : ReceiveActor
|
|||||||
// repository hardening already swallows duplicate-key races,
|
// repository hardening already swallows duplicate-key races,
|
||||||
// so the same id arriving twice (site retry, reconciliation)
|
// so the same id arriving twice (site retry, reconciliation)
|
||||||
// is a silent no-op.
|
// is a silent no-op.
|
||||||
var ingested = evt with { IngestedAtUtc = nowUtc };
|
// Filter BEFORE the IngestedAtUtc stamp so the redacted
|
||||||
|
// copy carries the central-side ingest timestamp. Filter
|
||||||
|
// is contract-bound to never throw; null = pass-through.
|
||||||
|
var filtered = filter?.Apply(evt) ?? evt;
|
||||||
|
var ingested = filtered with { IngestedAtUtc = nowUtc };
|
||||||
await repository.InsertIfNotExistsAsync(ingested).ConfigureAwait(false);
|
await repository.InsertIfNotExistsAsync(ingested).ConfigureAwait(false);
|
||||||
accepted.Add(evt.EventId);
|
accepted.Add(evt.EventId);
|
||||||
}
|
}
|
||||||
@@ -185,6 +198,12 @@ public class AuditLogIngestActor : ReceiveActor
|
|||||||
var auditRepo = scope.ServiceProvider.GetRequiredService<IAuditLogRepository>();
|
var auditRepo = scope.ServiceProvider.GetRequiredService<IAuditLogRepository>();
|
||||||
var siteCallRepo = scope.ServiceProvider.GetRequiredService<ISiteCallAuditRepository>();
|
var siteCallRepo = scope.ServiceProvider.GetRequiredService<ISiteCallAuditRepository>();
|
||||||
var dbContext = scope.ServiceProvider.GetRequiredService<ScadaLinkDbContext>();
|
var dbContext = scope.ServiceProvider.GetRequiredService<ScadaLinkDbContext>();
|
||||||
|
// Bundle C (M5-T6): resolve the filter for the whole batch from
|
||||||
|
// the scope; null = pass-through for test composition roots that
|
||||||
|
// skip the filter registration. The filter is contract-bound to
|
||||||
|
// never throw, so we can apply it inside the per-entry try
|
||||||
|
// without risking an unbounded blast radius.
|
||||||
|
var filter = scope.ServiceProvider.GetService<IAuditPayloadFilter>();
|
||||||
|
|
||||||
foreach (var entry in cmd.Entries)
|
foreach (var entry in cmd.Entries)
|
||||||
{
|
{
|
||||||
@@ -199,7 +218,12 @@ public class AuditLogIngestActor : ReceiveActor
|
|||||||
// matching timestamps (debugging convenience, not a
|
// matching timestamps (debugging convenience, not a
|
||||||
// correctness invariant).
|
// correctness invariant).
|
||||||
var ingestedAt = DateTime.UtcNow;
|
var ingestedAt = DateTime.UtcNow;
|
||||||
var auditStamped = entry.Audit with { IngestedAtUtc = ingestedAt };
|
// Filter the audit half BEFORE the dual-write — only the
|
||||||
|
// AuditLog row's payload columns are filterable; SiteCalls
|
||||||
|
// carries operational state only (status, retry count) and
|
||||||
|
// is left untouched.
|
||||||
|
var filteredAudit = filter?.Apply(entry.Audit) ?? entry.Audit;
|
||||||
|
var auditStamped = filteredAudit with { IngestedAtUtc = ingestedAt };
|
||||||
var siteCallStamped = entry.SiteCall with { IngestedAtUtc = ingestedAt };
|
var siteCallStamped = entry.SiteCall with { IngestedAtUtc = ingestedAt };
|
||||||
|
|
||||||
await auditRepo.InsertIfNotExistsAsync(auditStamped)
|
await auditRepo.InsertIfNotExistsAsync(auditStamped)
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
using Microsoft.Extensions.DependencyInjection;
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
|
using ScadaLink.AuditLog.Payload;
|
||||||
using ScadaLink.Commons.Entities.Audit;
|
using ScadaLink.Commons.Entities.Audit;
|
||||||
using ScadaLink.Commons.Interfaces.Repositories;
|
using ScadaLink.Commons.Interfaces.Repositories;
|
||||||
using ScadaLink.Commons.Interfaces.Services;
|
using ScadaLink.Commons.Interfaces.Services;
|
||||||
@@ -40,11 +41,24 @@ public sealed class CentralAuditWriter : ICentralAuditWriter
|
|||||||
{
|
{
|
||||||
private readonly IServiceProvider _services;
|
private readonly IServiceProvider _services;
|
||||||
private readonly ILogger<CentralAuditWriter> _logger;
|
private readonly ILogger<CentralAuditWriter> _logger;
|
||||||
|
private readonly IAuditPayloadFilter? _filter;
|
||||||
|
|
||||||
public CentralAuditWriter(IServiceProvider services, ILogger<CentralAuditWriter> logger)
|
/// <summary>
|
||||||
|
/// Bundle C (M5-T6) — the central direct-write path used by the
|
||||||
|
/// NotificationOutboxActor dispatch and the Inbound API middleware also
|
||||||
|
/// needs to truncate + redact before the row hits MS SQL. The filter is
|
||||||
|
/// optional so the M4 test composition roots that don't pass one keep
|
||||||
|
/// working (they only ever write small payloads); production DI registers
|
||||||
|
/// the real filter via <see cref="ServiceCollectionExtensions.AddAuditLog"/>.
|
||||||
|
/// </summary>
|
||||||
|
public CentralAuditWriter(
|
||||||
|
IServiceProvider services,
|
||||||
|
ILogger<CentralAuditWriter> logger,
|
||||||
|
IAuditPayloadFilter? filter = null)
|
||||||
{
|
{
|
||||||
_services = services ?? throw new ArgumentNullException(nameof(services));
|
_services = services ?? throw new ArgumentNullException(nameof(services));
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
|
_filter = filter;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
@@ -65,9 +79,14 @@ public sealed class CentralAuditWriter : ICentralAuditWriter
|
|||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
|
// Filter BEFORE stamping IngestedAtUtc + handing to the repo. The
|
||||||
|
// filter contract is "never throws"; the null-coalesce keeps the
|
||||||
|
// M4 test composition roots (no filter passed) working unchanged.
|
||||||
|
var filtered = _filter?.Apply(evt) ?? evt;
|
||||||
|
|
||||||
await using var scope = _services.CreateAsyncScope();
|
await using var scope = _services.CreateAsyncScope();
|
||||||
var repo = scope.ServiceProvider.GetRequiredService<IAuditLogRepository>();
|
var repo = scope.ServiceProvider.GetRequiredService<IAuditLogRepository>();
|
||||||
var stamped = evt with { IngestedAtUtc = DateTime.UtcNow };
|
var stamped = filtered with { IngestedAtUtc = DateTime.UtcNow };
|
||||||
await repo.InsertIfNotExistsAsync(stamped, ct).ConfigureAwait(false);
|
await repo.InsertIfNotExistsAsync(stamped, ct).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
|
|||||||
@@ -14,4 +14,15 @@ public sealed class PerTargetRedactionOverride
|
|||||||
|
|
||||||
/// <summary>Additional body redactor regex patterns (appended to the global list).</summary>
|
/// <summary>Additional body redactor regex patterns (appended to the global list).</summary>
|
||||||
public List<string>? AdditionalBodyRedactors { get; set; }
|
public List<string>? AdditionalBodyRedactors { get; set; }
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Opt-in SQL parameter redaction: case-insensitive regex matched against
|
||||||
|
/// each SQL parameter NAME in the M4 <c>AuditingDbCommand</c> RequestSummary
|
||||||
|
/// JSON (<c>{"sql":"...","parameters":{"@name":"value", ...}}</c>); values
|
||||||
|
/// whose name matches are replaced with <c><redacted></c>. Null (the
|
||||||
|
/// default) means parameter values are captured verbatim. Only applied to
|
||||||
|
/// <see cref="ScadaLink.Commons.Types.Enums.AuditChannel.DbOutbound"/>
|
||||||
|
/// rows.
|
||||||
|
/// </summary>
|
||||||
|
public string? RedactSqlParamsMatching { get; set; }
|
||||||
}
|
}
|
||||||
|
|||||||
573
src/ScadaLink.AuditLog/Payload/DefaultAuditPayloadFilter.cs
Normal file
573
src/ScadaLink.AuditLog/Payload/DefaultAuditPayloadFilter.cs
Normal file
@@ -0,0 +1,573 @@
|
|||||||
|
using System.Collections.Concurrent;
|
||||||
|
using System.Text;
|
||||||
|
using System.Text.Encodings.Web;
|
||||||
|
using System.Text.Json;
|
||||||
|
using System.Text.Json.Nodes;
|
||||||
|
using System.Text.RegularExpressions;
|
||||||
|
using Microsoft.Extensions.Logging;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using ScadaLink.AuditLog.Configuration;
|
||||||
|
using ScadaLink.Commons.Entities.Audit;
|
||||||
|
using ScadaLink.Commons.Types.Enums;
|
||||||
|
|
||||||
|
namespace ScadaLink.AuditLog.Payload;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Default <see cref="IAuditPayloadFilter"/>. Bundle A established the
|
||||||
|
/// truncation backbone; Bundle B chains HTTP header redaction (M5-T3) BEFORE
|
||||||
|
/// truncation so redactors operate on the full payload and the cap then trims
|
||||||
|
/// the redacted result.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <para>
|
||||||
|
/// Uses <see cref="IOptionsMonitor{TOptions}"/> (not <see cref="IOptions{TOptions}"/>)
|
||||||
|
/// so the M5-T8 hot-reload path sees fresh values without re-resolving the
|
||||||
|
/// singleton. <see cref="Apply"/> reads <see cref="IOptionsMonitor{T}.CurrentValue"/>
|
||||||
|
/// on every call, and the regex cache is keyed by pattern string — patterns
|
||||||
|
/// added via a live config change compile on first use of the next event;
|
||||||
|
/// patterns removed simply stop being looked up. No <c>OnChange</c> subscription
|
||||||
|
/// or explicit cache invalidation is required (the
|
||||||
|
/// <c>AuditLogOptionsBindingTests</c> fixture in <c>ScadaLink.AuditLog.Tests</c>
|
||||||
|
/// pins this behaviour).
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// "Error row" = <see cref="AuditEvent.Status"/> NOT IN (<c>Delivered</c>,
|
||||||
|
/// <c>Submitted</c>, <c>Forwarded</c>) — every other status, including the
|
||||||
|
/// non-terminal <c>Attempted</c>, the parked/discarded terminals, and the
|
||||||
|
/// short-circuit <c>Skipped</c>, receives the larger error cap so a verbose
|
||||||
|
/// error body survives.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Apply MUST NOT throw — on internal failure the filter over-redacts by
|
||||||
|
/// returning the input with <see cref="AuditEvent.PayloadTruncated"/> set and
|
||||||
|
/// increments the <c>AuditRedactionFailure</c> health metric via the injected
|
||||||
|
/// <see cref="IAuditRedactionFailureCounter"/>. Each redactor stage runs in
|
||||||
|
/// its own try/catch — a failure in (say) the header redactor still lets the
|
||||||
|
/// SQL parameter redactor and the truncator run on the remaining fields.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Stage order (each runs on every applicable field):
|
||||||
|
/// header redaction → body regex redaction → truncation. The SQL-parameter
|
||||||
|
/// stage piggybacks on the body-redactor path; both run BEFORE truncation so
|
||||||
|
/// the cap trims the redacted result, never bytes the redactor intended to
|
||||||
|
/// hide.
|
||||||
|
/// </para>
|
||||||
|
/// </remarks>
|
||||||
|
public sealed class DefaultAuditPayloadFilter : IAuditPayloadFilter
|
||||||
|
{
|
||||||
|
private const string RedactedMarker = "<redacted>";
|
||||||
|
private const string RedactorErrorMarker = "<redacted: redactor error>";
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Per-match regex timeout. Catastrophic-backtracking patterns trip a
|
||||||
|
/// <see cref="RegexMatchTimeoutException"/> when a single match takes
|
||||||
|
/// longer than this; the offending field is then over-redacted with
|
||||||
|
/// <see cref="RedactorErrorMarker"/> and the failure counter is bumped.
|
||||||
|
/// 50 ms is generous for normal patterns yet short enough that the
|
||||||
|
/// audit hot-path isn't held up by a misconfigured regex.
|
||||||
|
/// </summary>
|
||||||
|
private static readonly TimeSpan RegexMatchTimeout = TimeSpan.FromMilliseconds(50);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// JSON serializer options used to re-emit redacted summaries. The
|
||||||
|
/// UnsafeRelaxedJsonEscaping encoder is required so the redaction marker
|
||||||
|
/// (which contains <c><</c> / <c>></c>) survives unescaped — the
|
||||||
|
/// header-redaction tests grep for the literal marker, and the downstream
|
||||||
|
/// UI / log readers would rather see <c><redacted></c> than
|
||||||
|
/// <c><redacted></c>. The summaries are persisted to the audit
|
||||||
|
/// table and rendered in trusted-internal contexts only, so the relaxed
|
||||||
|
/// HTML-escaping rules do not introduce an XSS surface.
|
||||||
|
/// </summary>
|
||||||
|
private static readonly JsonSerializerOptions RedactedSummaryJsonOptions = new()
|
||||||
|
{
|
||||||
|
Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping,
|
||||||
|
};
|
||||||
|
|
||||||
|
private readonly IOptionsMonitor<AuditLogOptions> _options;
|
||||||
|
private readonly ILogger<DefaultAuditPayloadFilter> _logger;
|
||||||
|
private readonly IAuditRedactionFailureCounter _failureCounter;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Compiled-regex cache keyed by pattern string. Lazy population: each
|
||||||
|
/// pattern is compiled on first use and cached forever (the entry's
|
||||||
|
/// <see cref="CompiledRegex"/> carries either the working <see cref="Regex"/>
|
||||||
|
/// or a sentinel marking the pattern as invalid so we don't retry the
|
||||||
|
/// failing compile on every call). ConcurrentDictionary is the right
|
||||||
|
/// thread-safety primitive here because the filter is a DI singleton
|
||||||
|
/// shared across the audit hot-path.
|
||||||
|
/// </summary>
|
||||||
|
private readonly ConcurrentDictionary<string, CompiledRegex> _regexCache = new();
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Primary constructor used by DI — pulls the optional redaction-failure
|
||||||
|
/// counter from the container; a NoOp default is registered in
|
||||||
|
/// <see cref="ServiceCollectionExtensions.AddAuditLog"/>.
|
||||||
|
/// </summary>
|
||||||
|
public DefaultAuditPayloadFilter(
|
||||||
|
IOptionsMonitor<AuditLogOptions> options,
|
||||||
|
ILogger<DefaultAuditPayloadFilter> logger,
|
||||||
|
IAuditRedactionFailureCounter? failureCounter = null)
|
||||||
|
{
|
||||||
|
_options = options ?? throw new ArgumentNullException(nameof(options));
|
||||||
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
|
_failureCounter = failureCounter ?? new NoOpAuditRedactionFailureCounter();
|
||||||
|
}
|
||||||
|
|
||||||
|
public AuditEvent Apply(AuditEvent rawEvent)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var opts = _options.CurrentValue;
|
||||||
|
var cap = IsErrorStatus(rawEvent.Status) ? opts.ErrorCapBytes : opts.DefaultCapBytes;
|
||||||
|
|
||||||
|
// --- Header-redaction stage (runs BEFORE truncation) ----------
|
||||||
|
var request = RedactHeaders(rawEvent.RequestSummary, opts.HeaderRedactList);
|
||||||
|
var response = RedactHeaders(rawEvent.ResponseSummary, opts.HeaderRedactList);
|
||||||
|
var errorDetail = rawEvent.ErrorDetail;
|
||||||
|
var extra = rawEvent.Extra;
|
||||||
|
|
||||||
|
// --- Body-regex stage (also runs BEFORE truncation) -----------
|
||||||
|
// Resolves the active regex set per event so per-target overrides
|
||||||
|
// bound to AuditEvent.Target are picked up; effectively a no-op
|
||||||
|
// when neither GlobalBodyRedactors nor the per-target additions
|
||||||
|
// are configured.
|
||||||
|
var bodyRegexes = ResolveBodyRegexes(opts, rawEvent.Target);
|
||||||
|
if (bodyRegexes.Count > 0)
|
||||||
|
{
|
||||||
|
request = RedactBody(request, bodyRegexes);
|
||||||
|
response = RedactBody(response, bodyRegexes);
|
||||||
|
errorDetail = RedactBody(errorDetail, bodyRegexes);
|
||||||
|
extra = RedactBody(extra, bodyRegexes);
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- SQL parameter redaction stage (DbOutbound only) ----------
|
||||||
|
// Parses the M4 AuditingDbCommand RequestSummary shape
|
||||||
|
// {"sql":"...","parameters":{...}} and redacts parameter VALUES
|
||||||
|
// whose NAME matches the per-connection regex. Opt-in: no
|
||||||
|
// PerTargetOverrides[connectionName].RedactSqlParamsMatching =>
|
||||||
|
// no-op. Channel-guarded so the same regex can never accidentally
|
||||||
|
// touch an ApiOutbound row.
|
||||||
|
if (rawEvent.Channel == AuditChannel.DbOutbound
|
||||||
|
&& TryGetSqlParamRedactor(opts, rawEvent.Target, out var sqlParamRegex))
|
||||||
|
{
|
||||||
|
request = RedactSqlParameters(request, sqlParamRegex!);
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- Truncation stage -----------------------------------------
|
||||||
|
var truncated = false;
|
||||||
|
request = TruncateField(request, cap, ref truncated);
|
||||||
|
response = TruncateField(response, cap, ref truncated);
|
||||||
|
errorDetail = TruncateField(errorDetail, cap, ref truncated);
|
||||||
|
extra = TruncateField(extra, cap, ref truncated);
|
||||||
|
|
||||||
|
return rawEvent with
|
||||||
|
{
|
||||||
|
RequestSummary = request,
|
||||||
|
ResponseSummary = response,
|
||||||
|
ErrorDetail = errorDetail,
|
||||||
|
Extra = extra,
|
||||||
|
PayloadTruncated = rawEvent.PayloadTruncated || truncated,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
// Audit is best-effort: over-redact rather than fail the caller.
|
||||||
|
// The per-stage try/catches above already handle redactor faults
|
||||||
|
// and increment the counter; this catch covers any unexpected
|
||||||
|
// surprise in the surrounding orchestration code.
|
||||||
|
_logger.LogWarning(
|
||||||
|
ex,
|
||||||
|
"Payload filter failed; returning raw event with PayloadTruncated=true");
|
||||||
|
try { _failureCounter.Increment(); } catch { /* swallow per §7 */ }
|
||||||
|
return rawEvent with { PayloadTruncated = true };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Parse <paramref name="json"/> as the documented
|
||||||
|
/// <c>{"headers": {...}, "body": ...}</c> shape and replace values whose
|
||||||
|
/// header NAME (case-insensitive) is in <paramref name="redactList"/> with
|
||||||
|
/// <see cref="RedactedMarker"/>. Re-serialises and returns the result.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// No-op pass-through for inputs that aren't JSON-shaped — emitters that
|
||||||
|
/// have not yet adopted the convention (the M2 site emitters today, which
|
||||||
|
/// leave RequestSummary null on outbound API calls) get a transparent
|
||||||
|
/// pass. If the redactor itself throws, we over-redact the whole field
|
||||||
|
/// with <see cref="RedactorErrorMarker"/> and bump the failure counter.
|
||||||
|
/// </remarks>
|
||||||
|
private string? RedactHeaders(string? json, IList<string> redactList)
|
||||||
|
{
|
||||||
|
if (json is null)
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Cheap structural pre-check: only attempt JSON parsing when the input
|
||||||
|
// actually looks like a JSON object. Saves the JsonDocument allocation
|
||||||
|
// on the (very common) non-JSON ErrorDetail / Extra fields.
|
||||||
|
var trimmed = json.AsSpan().TrimStart();
|
||||||
|
if (trimmed.Length == 0 || trimmed[0] != '{')
|
||||||
|
{
|
||||||
|
return json;
|
||||||
|
}
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
JsonNode? root;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
root = JsonNode.Parse(json);
|
||||||
|
}
|
||||||
|
catch (JsonException)
|
||||||
|
{
|
||||||
|
// Not parseable JSON — leave the field alone (no error, no
|
||||||
|
// redaction). Emitters not yet using the documented shape get
|
||||||
|
// a transparent pass; Bundle C will update them.
|
||||||
|
return json;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (root is not JsonObject obj || obj["headers"] is not JsonObject headers)
|
||||||
|
{
|
||||||
|
// No "headers" object at the top level — nothing to redact.
|
||||||
|
return json;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build a case-insensitive lookup of the redact list so we can do
|
||||||
|
// one O(1) check per header name without an inner Any() loop.
|
||||||
|
var redactSet = new HashSet<string>(redactList, StringComparer.OrdinalIgnoreCase);
|
||||||
|
|
||||||
|
// Take a snapshot of names first — we cannot mutate while
|
||||||
|
// enumerating the JsonObject.
|
||||||
|
var names = new List<string>(headers.Count);
|
||||||
|
foreach (var kvp in headers)
|
||||||
|
{
|
||||||
|
names.Add(kvp.Key);
|
||||||
|
}
|
||||||
|
foreach (var name in names)
|
||||||
|
{
|
||||||
|
if (redactSet.Contains(name))
|
||||||
|
{
|
||||||
|
headers[name] = JsonValue.Create(RedactedMarker);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return obj.ToJsonString(RedactedSummaryJsonOptions);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(
|
||||||
|
ex,
|
||||||
|
"Header redactor faulted; over-redacting field with '{Marker}'",
|
||||||
|
RedactorErrorMarker);
|
||||||
|
try { _failureCounter.Increment(); } catch { /* swallow per §7 */ }
|
||||||
|
return RedactorErrorMarker;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Combine the global and per-target body-redactor lists for a single
|
||||||
|
/// event, returning the compiled-regex set to apply. Patterns that failed
|
||||||
|
/// compilation are silently skipped — the compile-time failure was logged
|
||||||
|
/// once on first encounter; we never let one bad pattern starve the rest.
|
||||||
|
/// </summary>
|
||||||
|
private IReadOnlyList<Regex> ResolveBodyRegexes(AuditLogOptions opts, string? target)
|
||||||
|
{
|
||||||
|
var hasGlobal = opts.GlobalBodyRedactors is { Count: > 0 };
|
||||||
|
var perTargetAdditions = (target != null
|
||||||
|
&& opts.PerTargetOverrides.TryGetValue(target, out var over)
|
||||||
|
&& over.AdditionalBodyRedactors is { Count: > 0 })
|
||||||
|
? over.AdditionalBodyRedactors
|
||||||
|
: null;
|
||||||
|
|
||||||
|
if (!hasGlobal && perTargetAdditions == null)
|
||||||
|
{
|
||||||
|
return Array.Empty<Regex>();
|
||||||
|
}
|
||||||
|
|
||||||
|
var result = new List<Regex>();
|
||||||
|
if (hasGlobal)
|
||||||
|
{
|
||||||
|
foreach (var pattern in opts.GlobalBodyRedactors)
|
||||||
|
{
|
||||||
|
if (TryGetCompiledRegex(pattern, out var rx))
|
||||||
|
{
|
||||||
|
result.Add(rx!);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (perTargetAdditions != null)
|
||||||
|
{
|
||||||
|
foreach (var pattern in perTargetAdditions)
|
||||||
|
{
|
||||||
|
if (TryGetCompiledRegex(pattern, out var rx))
|
||||||
|
{
|
||||||
|
result.Add(rx!);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Resolve a compiled regex from the cache, compiling it on first use.
|
||||||
|
/// Returns <c>false</c> for patterns that are invalid OR whose compile
|
||||||
|
/// took longer than 100 ms (the spec calls catastrophic-backtracking
|
||||||
|
/// guesses at compile time "invalid"); the failure is logged once and
|
||||||
|
/// the sentinel cache entry prevents repeat compile attempts.
|
||||||
|
/// </summary>
|
||||||
|
private bool TryGetCompiledRegex(string pattern, out Regex? regex)
|
||||||
|
{
|
||||||
|
var entry = _regexCache.GetOrAdd(pattern, CompileRegex);
|
||||||
|
regex = entry.Regex;
|
||||||
|
return entry.Regex != null;
|
||||||
|
}
|
||||||
|
|
||||||
|
private CompiledRegex CompileRegex(string pattern)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var swStart = System.Diagnostics.Stopwatch.GetTimestamp();
|
||||||
|
var rx = new Regex(pattern, RegexOptions.Compiled, RegexMatchTimeout);
|
||||||
|
var elapsedMs = (System.Diagnostics.Stopwatch.GetTimestamp() - swStart)
|
||||||
|
* 1000d / System.Diagnostics.Stopwatch.Frequency;
|
||||||
|
if (elapsedMs > 100)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(
|
||||||
|
"Body redactor pattern compiled in {Elapsed}ms (> 100ms cap); rejecting '{Pattern}'",
|
||||||
|
elapsedMs, pattern);
|
||||||
|
return CompiledRegex.Invalid;
|
||||||
|
}
|
||||||
|
return new CompiledRegex(rx);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(
|
||||||
|
ex,
|
||||||
|
"Body redactor pattern '{Pattern}' failed to compile; skipping",
|
||||||
|
pattern);
|
||||||
|
return CompiledRegex.Invalid;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Apply each compiled body-redactor regex to <paramref name="value"/> in
|
||||||
|
/// turn, replacing every match with <see cref="RedactedMarker"/>. If any
|
||||||
|
/// single regex match throws (most commonly
|
||||||
|
/// <see cref="RegexMatchTimeoutException"/>) the field is over-redacted
|
||||||
|
/// with <see cref="RedactorErrorMarker"/> and the failure counter is
|
||||||
|
/// incremented — the user-facing action is never aborted.
|
||||||
|
/// </summary>
|
||||||
|
private string? RedactBody(string? value, IReadOnlyList<Regex> regexes)
|
||||||
|
{
|
||||||
|
if (value is null)
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
var current = value;
|
||||||
|
foreach (var rx in regexes)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
current = rx.Replace(current, RedactedMarker);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(
|
||||||
|
ex,
|
||||||
|
"Body redactor '{Pattern}' faulted; over-redacting field with '{Marker}'",
|
||||||
|
rx.ToString(), RedactorErrorMarker);
|
||||||
|
try { _failureCounter.Increment(); } catch { /* swallow per §7 */ }
|
||||||
|
return RedactorErrorMarker;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return current;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Resolve the per-connection SQL parameter redaction regex for the given
|
||||||
|
/// DbOutbound event target. Target shape (M4 AuditingDbCommand): the
|
||||||
|
/// connection name optionally followed by <c>.<sql-snippet></c> for
|
||||||
|
/// disambiguation; the per-target dictionary is keyed by the connection
|
||||||
|
/// name alone, so we strip the snippet suffix before lookup. Patterns are
|
||||||
|
/// compiled with case-insensitive matching to match the documented
|
||||||
|
/// behaviour.
|
||||||
|
/// </summary>
|
||||||
|
private bool TryGetSqlParamRedactor(AuditLogOptions opts, string? target, out Regex? regex)
|
||||||
|
{
|
||||||
|
regex = null;
|
||||||
|
if (string.IsNullOrEmpty(target))
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
var dot = target.IndexOf('.');
|
||||||
|
var connectionKey = dot < 0 ? target : target[..dot];
|
||||||
|
|
||||||
|
if (!opts.PerTargetOverrides.TryGetValue(connectionKey, out var over)
|
||||||
|
|| string.IsNullOrEmpty(over.RedactSqlParamsMatching))
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Force case-insensitivity per the spec — even if the operator wrote
|
||||||
|
// the pattern without an IgnoreCase flag. The compile cache key folds
|
||||||
|
// the option to keep the entries unambiguous.
|
||||||
|
var cacheKey = "(?i)" + over.RedactSqlParamsMatching;
|
||||||
|
if (!TryGetCompiledRegex(cacheKey, out regex))
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Walk the M4 <c>{"sql":"...","parameters":{...}}</c> RequestSummary
|
||||||
|
/// shape; for each parameter whose NAME matches
|
||||||
|
/// <paramref name="paramNameRegex"/>, replace its value with
|
||||||
|
/// <see cref="RedactedMarker"/>. Re-serialise.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// No-op pass-through when the input isn't parseable JSON, isn't a JSON
|
||||||
|
/// object, or doesn't carry a top-level <c>"parameters"</c> object. On
|
||||||
|
/// any unexpected fault the field is over-redacted with
|
||||||
|
/// <see cref="RedactorErrorMarker"/> and the failure counter is bumped.
|
||||||
|
/// </remarks>
|
||||||
|
private string? RedactSqlParameters(string? json, Regex paramNameRegex)
|
||||||
|
{
|
||||||
|
if (json is null)
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
var trimmed = json.AsSpan().TrimStart();
|
||||||
|
if (trimmed.Length == 0 || trimmed[0] != '{')
|
||||||
|
{
|
||||||
|
return json;
|
||||||
|
}
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
JsonNode? root;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
root = JsonNode.Parse(json);
|
||||||
|
}
|
||||||
|
catch (JsonException)
|
||||||
|
{
|
||||||
|
return json;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (root is not JsonObject obj || obj["parameters"] is not JsonObject parameters)
|
||||||
|
{
|
||||||
|
return json;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Snapshot the names — mutating during enumeration is unsupported.
|
||||||
|
var names = new List<string>(parameters.Count);
|
||||||
|
foreach (var kvp in parameters)
|
||||||
|
{
|
||||||
|
names.Add(kvp.Key);
|
||||||
|
}
|
||||||
|
var anyChanged = false;
|
||||||
|
foreach (var name in names)
|
||||||
|
{
|
||||||
|
bool matched;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
matched = paramNameRegex.IsMatch(name);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(
|
||||||
|
ex,
|
||||||
|
"SQL parameter redactor faulted; over-redacting field with '{Marker}'",
|
||||||
|
RedactorErrorMarker);
|
||||||
|
try { _failureCounter.Increment(); } catch { /* swallow per §7 */ }
|
||||||
|
return RedactorErrorMarker;
|
||||||
|
}
|
||||||
|
if (matched)
|
||||||
|
{
|
||||||
|
parameters[name] = JsonValue.Create(RedactedMarker);
|
||||||
|
anyChanged = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Avoid re-serialising (which would normalise whitespace / order)
|
||||||
|
// when no parameter matched — keeps the on-disk row byte-identical
|
||||||
|
// to the emitter's output on the no-match path.
|
||||||
|
return anyChanged ? obj.ToJsonString(RedactedSummaryJsonOptions) : json;
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(
|
||||||
|
ex,
|
||||||
|
"SQL parameter redactor faulted; over-redacting field with '{Marker}'",
|
||||||
|
RedactorErrorMarker);
|
||||||
|
try { _failureCounter.Increment(); } catch { /* swallow per §7 */ }
|
||||||
|
return RedactorErrorMarker;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string? TruncateField(string? value, int cap, ref bool truncated)
|
||||||
|
{
|
||||||
|
if (value is null)
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
var result = TruncateUtf8(value, cap);
|
||||||
|
if (result.Length != value.Length)
|
||||||
|
{
|
||||||
|
truncated = true;
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// UTF-8 byte-safe truncation. Encodes the input to UTF-8, walks back from
|
||||||
|
/// the cap position until the byte is NOT a continuation byte
|
||||||
|
/// (<c>byte & 0xC0 == 0x80</c>), and decodes the resulting prefix —
|
||||||
|
/// guaranteeing the returned string never splits a multi-byte sequence.
|
||||||
|
/// </summary>
|
||||||
|
private static string TruncateUtf8(string value, int capBytes)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrEmpty(value))
|
||||||
|
{
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
var bytes = Encoding.UTF8.GetBytes(value);
|
||||||
|
if (bytes.Length <= capBytes)
|
||||||
|
{
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
var boundary = capBytes;
|
||||||
|
while (boundary > 0 && (bytes[boundary] & 0xC0) == 0x80)
|
||||||
|
{
|
||||||
|
boundary--;
|
||||||
|
}
|
||||||
|
return Encoding.UTF8.GetString(bytes, 0, boundary);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static bool IsErrorStatus(AuditStatus status) => status switch
|
||||||
|
{
|
||||||
|
AuditStatus.Delivered or AuditStatus.Submitted or AuditStatus.Forwarded => false,
|
||||||
|
_ => true,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Cache entry for a body-redactor pattern. Carries the working
|
||||||
|
/// <see cref="Regex"/> on the success path, or the
|
||||||
|
/// <see cref="Invalid"/> sentinel for patterns that failed to compile
|
||||||
|
/// (or exceeded the 100 ms compile budget). The sentinel lets us skip
|
||||||
|
/// repeat compile attempts on every event without re-throwing on the
|
||||||
|
/// hot-path.
|
||||||
|
/// </summary>
|
||||||
|
private readonly struct CompiledRegex
|
||||||
|
{
|
||||||
|
public static readonly CompiledRegex Invalid = new(null);
|
||||||
|
|
||||||
|
public Regex? Regex { get; }
|
||||||
|
|
||||||
|
public CompiledRegex(Regex? regex) => Regex = regex;
|
||||||
|
}
|
||||||
|
}
|
||||||
30
src/ScadaLink.AuditLog/Payload/IAuditPayloadFilter.cs
Normal file
30
src/ScadaLink.AuditLog/Payload/IAuditPayloadFilter.cs
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
using ScadaLink.Commons.Entities.Audit;
|
||||||
|
|
||||||
|
namespace ScadaLink.AuditLog.Payload;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Filters an <see cref="AuditEvent"/> between construction and persistence —
|
||||||
|
/// truncates oversized payload fields, applies header/body/SQL-parameter
|
||||||
|
/// redaction, sets <see cref="AuditEvent.PayloadTruncated"/>.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <para>
|
||||||
|
/// Pure function: returns a filtered COPY of the input via <c>with</c>
|
||||||
|
/// expressions; never throws (over-redacts on internal failure and increments
|
||||||
|
/// the <c>AuditRedactionFailure</c> health metric).
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Wired in M5 between event construction and the writer chain
|
||||||
|
/// (<c>FallbackAuditWriter.WriteAsync</c>, <c>CentralAuditWriter.WriteAsync</c>,
|
||||||
|
/// and the <c>AuditLogIngestActor</c> handlers).
|
||||||
|
/// </para>
|
||||||
|
/// </remarks>
|
||||||
|
public interface IAuditPayloadFilter
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// Apply the configured truncation + redaction policy to <paramref name="rawEvent"/>
|
||||||
|
/// and return a filtered copy. MUST NOT throw — on internal failure, over-redact
|
||||||
|
/// and surface the failure via the audit-redaction-failure health metric.
|
||||||
|
/// </summary>
|
||||||
|
AuditEvent Apply(AuditEvent rawEvent);
|
||||||
|
}
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
namespace ScadaLink.AuditLog.Payload;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Counter sink invoked by <see cref="DefaultAuditPayloadFilter"/> every time
|
||||||
|
/// a redactor (header / body regex / SQL parameter) throws and the filter has
|
||||||
|
/// to over-redact the offending field with the
|
||||||
|
/// <c><redacted: redactor error></c> marker. Bundle C bridges this into
|
||||||
|
/// the Site Health Monitoring report payload as <c>AuditRedactionFailure</c>.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Redaction failures must NEVER abort the user-facing action (alog.md §7) —
|
||||||
|
/// the filter over-redacts the field and surfaces the failure via this counter
|
||||||
|
/// instead. A NoOp default is the correct safe fallback while the health
|
||||||
|
/// metric is being wired in.
|
||||||
|
/// </remarks>
|
||||||
|
public interface IAuditRedactionFailureCounter
|
||||||
|
{
|
||||||
|
/// <summary>Increment the audit-redaction failure counter by one.</summary>
|
||||||
|
void Increment();
|
||||||
|
}
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
namespace ScadaLink.AuditLog.Payload;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Default <see cref="IAuditRedactionFailureCounter"/> binding used when the
|
||||||
|
/// Site Health Monitoring bridge has not been wired yet. Bundle C replaces
|
||||||
|
/// this registration with the real counter that surfaces in the site health
|
||||||
|
/// report payload as <c>AuditRedactionFailure</c>.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class NoOpAuditRedactionFailureCounter : IAuditRedactionFailureCounter
|
||||||
|
{
|
||||||
|
/// <inheritdoc/>
|
||||||
|
public void Increment()
|
||||||
|
{
|
||||||
|
// Intentionally empty — Bundle C overrides this binding with the real
|
||||||
|
// health-metric counter.
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -5,6 +5,7 @@ using Microsoft.Extensions.Logging;
|
|||||||
using Microsoft.Extensions.Options;
|
using Microsoft.Extensions.Options;
|
||||||
using ScadaLink.AuditLog.Central;
|
using ScadaLink.AuditLog.Central;
|
||||||
using ScadaLink.AuditLog.Configuration;
|
using ScadaLink.AuditLog.Configuration;
|
||||||
|
using ScadaLink.AuditLog.Payload;
|
||||||
using ScadaLink.AuditLog.Site;
|
using ScadaLink.AuditLog.Site;
|
||||||
using ScadaLink.AuditLog.Site.Telemetry;
|
using ScadaLink.AuditLog.Site.Telemetry;
|
||||||
using ScadaLink.Commons.Interfaces.Services;
|
using ScadaLink.Commons.Interfaces.Services;
|
||||||
@@ -59,6 +60,21 @@ public static class ServiceCollectionExtensions
|
|||||||
.ValidateOnStart();
|
.ValidateOnStart();
|
||||||
services.AddSingleton<IValidateOptions<AuditLogOptions>, AuditLogOptionsValidator>();
|
services.AddSingleton<IValidateOptions<AuditLogOptions>, AuditLogOptionsValidator>();
|
||||||
|
|
||||||
|
// M5 Bundle A: payload filter — truncates oversized RequestSummary /
|
||||||
|
// ResponseSummary / ErrorDetail / Extra fields between event
|
||||||
|
// construction and persistence. Bundle B layers header / body /
|
||||||
|
// SQL-parameter redaction onto the same singleton; Bundle C wires it
|
||||||
|
// into the FallbackAuditWriter / CentralAuditWriter / IngestActor
|
||||||
|
// paths. Singleton — the filter is stateless and the IOptionsMonitor
|
||||||
|
// dependency picks up M5-T8 hot reloads on its own.
|
||||||
|
services.AddSingleton<IAuditPayloadFilter, DefaultAuditPayloadFilter>();
|
||||||
|
|
||||||
|
// M5 Bundle B: per-stage redactor-failure counter. NoOp default;
|
||||||
|
// Bundle C replaces this binding with the Site Health Monitoring
|
||||||
|
// bridge that surfaces failures as AuditRedactionFailure on the site
|
||||||
|
// health report.
|
||||||
|
services.TryAddSingleton<IAuditRedactionFailureCounter, NoOpAuditRedactionFailureCounter>();
|
||||||
|
|
||||||
// M2 Bundle E: site writer + telemetry options bindings.
|
// M2 Bundle E: site writer + telemetry options bindings.
|
||||||
// BindConfiguration is not used because the configuration root supplied
|
// BindConfiguration is not used because the configuration root supplied
|
||||||
// by the caller may not be the application root — we go through the
|
// by the caller may not be the application root — we go through the
|
||||||
@@ -90,11 +106,16 @@ public static class ServiceCollectionExtensions
|
|||||||
// The script-thread surface is FallbackAuditWriter (primary + ring +
|
// The script-thread surface is FallbackAuditWriter (primary + ring +
|
||||||
// counter), not the raw SqliteAuditWriter — primary failures must NEVER
|
// counter), not the raw SqliteAuditWriter — primary failures must NEVER
|
||||||
// abort the user-facing action.
|
// abort the user-facing action.
|
||||||
|
// Bundle C (M5-T6): the IAuditPayloadFilter singleton above is wired
|
||||||
|
// through the factory so every event written through this surface is
|
||||||
|
// truncated + redacted before it hits SQLite (and the ring on
|
||||||
|
// failure).
|
||||||
services.AddSingleton<IAuditWriter>(sp => new FallbackAuditWriter(
|
services.AddSingleton<IAuditWriter>(sp => new FallbackAuditWriter(
|
||||||
primary: sp.GetRequiredService<SqliteAuditWriter>(),
|
primary: sp.GetRequiredService<SqliteAuditWriter>(),
|
||||||
ring: sp.GetRequiredService<RingBufferFallback>(),
|
ring: sp.GetRequiredService<RingBufferFallback>(),
|
||||||
failureCounter: sp.GetRequiredService<IAuditWriteFailureCounter>(),
|
failureCounter: sp.GetRequiredService<IAuditWriteFailureCounter>(),
|
||||||
logger: sp.GetRequiredService<ILogger<FallbackAuditWriter>>()));
|
logger: sp.GetRequiredService<ILogger<FallbackAuditWriter>>(),
|
||||||
|
filter: sp.GetRequiredService<IAuditPayloadFilter>()));
|
||||||
|
|
||||||
// ISiteStreamAuditClient: NoOp default. M6's reconciliation work brings
|
// ISiteStreamAuditClient: NoOp default. M6's reconciliation work brings
|
||||||
// the real gRPC-backed implementation (no site→central gRPC channel
|
// the real gRPC-backed implementation (no site→central gRPC channel
|
||||||
@@ -139,32 +160,50 @@ public static class ServiceCollectionExtensions
|
|||||||
// is intentionally distinct from IAuditWriter so site composition roots
|
// is intentionally distinct from IAuditWriter so site composition roots
|
||||||
// do not accidentally bind it; central composition roots that include
|
// do not accidentally bind it; central composition roots that include
|
||||||
// AddConfigurationDatabase get a working implementation transparently.
|
// AddConfigurationDatabase get a working implementation transparently.
|
||||||
services.AddSingleton<ICentralAuditWriter, CentralAuditWriter>();
|
// Bundle C (M5-T6): wire the IAuditPayloadFilter into the factory so
|
||||||
|
// NotificationOutboxActor + Inbound API rows are truncated + redacted
|
||||||
|
// before they hit MS SQL.
|
||||||
|
services.AddSingleton<ICentralAuditWriter>(sp => new CentralAuditWriter(
|
||||||
|
sp,
|
||||||
|
sp.GetRequiredService<ILogger<CentralAuditWriter>>(),
|
||||||
|
sp.GetRequiredService<IAuditPayloadFilter>()));
|
||||||
|
|
||||||
return services;
|
return services;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Audit Log (#23) M2 Bundle G — swap the default
|
/// Audit Log (#23) M2 Bundle G + M5 Bundle C — swap the default
|
||||||
/// <see cref="NoOpAuditWriteFailureCounter"/> registration for the real
|
/// <see cref="NoOpAuditWriteFailureCounter"/> and
|
||||||
/// <see cref="HealthMetricsAuditWriteFailureCounter"/> bridge so the
|
/// <see cref="NoOpAuditRedactionFailureCounter"/> registrations for the
|
||||||
/// FallbackAuditWriter primary-failure counter surfaces in the site health
|
/// real <see cref="HealthMetricsAuditWriteFailureCounter"/> /
|
||||||
/// report payload as <c>SiteHealthReport.SiteAuditWriteFailures</c>.
|
/// <see cref="HealthMetricsAuditRedactionFailureCounter"/> bridges so the
|
||||||
|
/// FallbackAuditWriter primary-failure counter AND the
|
||||||
|
/// DefaultAuditPayloadFilter redactor-failure counter both surface in the
|
||||||
|
/// site health report payload as
|
||||||
|
/// <c>SiteHealthReport.SiteAuditWriteFailures</c> +
|
||||||
|
/// <c>SiteHealthReport.AuditRedactionFailure</c>.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
/// <remarks>
|
/// <remarks>
|
||||||
/// <para>
|
/// <para>
|
||||||
/// Must be called AFTER both <see cref="AddAuditLog"/> (registers the
|
/// Must be called AFTER both <see cref="AddAuditLog"/> (registers the
|
||||||
/// NoOp default this method replaces) and
|
/// NoOp defaults this method replaces) and
|
||||||
/// <c>ScadaLink.HealthMonitoring.ServiceCollectionExtensions.AddHealthMonitoring</c>
|
/// <c>ScadaLink.HealthMonitoring.ServiceCollectionExtensions.AddHealthMonitoring</c>
|
||||||
/// or <c>AddSiteHealthMonitoring</c> (registers the
|
/// or <c>AddSiteHealthMonitoring</c> (registers the
|
||||||
/// <see cref="ISiteHealthCollector"/> the bridge depends on). Resolving
|
/// <see cref="ISiteHealthCollector"/> the bridges depend on). Resolving
|
||||||
/// <see cref="IAuditWriteFailureCounter"/> without the latter throws
|
/// <see cref="IAuditWriteFailureCounter"/> or
|
||||||
|
/// <see cref="IAuditRedactionFailureCounter"/> without the latter throws
|
||||||
/// <see cref="InvalidOperationException"/> at <c>GetRequiredService</c>
|
/// <see cref="InvalidOperationException"/> at <c>GetRequiredService</c>
|
||||||
/// time — by design, since a silent NoOp would mask a misconfiguration.
|
/// time — by design, since a silent NoOp would mask a misconfiguration.
|
||||||
/// </para>
|
/// </para>
|
||||||
/// <para>
|
/// <para>
|
||||||
/// Idempotent — calling twice replaces the descriptor each time without
|
/// Idempotent — calling twice replaces each descriptor without piling up
|
||||||
/// piling up registrations.
|
/// registrations.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Site-side only for M5: the central composition root keeps the NoOp
|
||||||
|
/// defaults; the central health-metric surface that would expose
|
||||||
|
/// <c>AuditRedactionFailure</c> next to the existing central counters
|
||||||
|
/// ships in M6.
|
||||||
/// </para>
|
/// </para>
|
||||||
/// </remarks>
|
/// </remarks>
|
||||||
public static IServiceCollection AddAuditLogHealthMetricsBridge(this IServiceCollection services)
|
public static IServiceCollection AddAuditLogHealthMetricsBridge(this IServiceCollection services)
|
||||||
@@ -173,6 +212,8 @@ public static class ServiceCollectionExtensions
|
|||||||
|
|
||||||
services.Replace(
|
services.Replace(
|
||||||
ServiceDescriptor.Singleton<IAuditWriteFailureCounter, HealthMetricsAuditWriteFailureCounter>());
|
ServiceDescriptor.Singleton<IAuditWriteFailureCounter, HealthMetricsAuditWriteFailureCounter>());
|
||||||
|
services.Replace(
|
||||||
|
ServiceDescriptor.Singleton<IAuditRedactionFailureCounter, HealthMetricsAuditRedactionFailureCounter>());
|
||||||
return services;
|
return services;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
using Microsoft.Extensions.Logging;
|
using Microsoft.Extensions.Logging;
|
||||||
|
using ScadaLink.AuditLog.Payload;
|
||||||
using ScadaLink.Commons.Entities.Audit;
|
using ScadaLink.Commons.Entities.Audit;
|
||||||
using ScadaLink.Commons.Interfaces.Services;
|
using ScadaLink.Commons.Interfaces.Services;
|
||||||
|
|
||||||
@@ -30,27 +31,48 @@ public sealed class FallbackAuditWriter : IAuditWriter
|
|||||||
private readonly RingBufferFallback _ring;
|
private readonly RingBufferFallback _ring;
|
||||||
private readonly IAuditWriteFailureCounter _failureCounter;
|
private readonly IAuditWriteFailureCounter _failureCounter;
|
||||||
private readonly ILogger<FallbackAuditWriter> _logger;
|
private readonly ILogger<FallbackAuditWriter> _logger;
|
||||||
|
private readonly IAuditPayloadFilter? _filter;
|
||||||
private readonly SemaphoreSlim _drainGate = new(1, 1);
|
private readonly SemaphoreSlim _drainGate = new(1, 1);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Bundle C (M5-T6) wires the singleton <see cref="IAuditPayloadFilter"/>
|
||||||
|
/// here so every event written via the site hot path is truncated +
|
||||||
|
/// header/body/SQL-param redacted before it hits both the primary SQLite
|
||||||
|
/// writer AND the ring fallback. The parameter is optional (defaults to
|
||||||
|
/// no filtering) so the long tail of test composition roots that don't
|
||||||
|
/// care about the filter need no change — the production
|
||||||
|
/// <see cref="ServiceCollectionExtensions.AddAuditLog"/> registration
|
||||||
|
/// always passes the real filter through.
|
||||||
|
/// </summary>
|
||||||
public FallbackAuditWriter(
|
public FallbackAuditWriter(
|
||||||
IAuditWriter primary,
|
IAuditWriter primary,
|
||||||
RingBufferFallback ring,
|
RingBufferFallback ring,
|
||||||
IAuditWriteFailureCounter failureCounter,
|
IAuditWriteFailureCounter failureCounter,
|
||||||
ILogger<FallbackAuditWriter> logger)
|
ILogger<FallbackAuditWriter> logger,
|
||||||
|
IAuditPayloadFilter? filter = null)
|
||||||
{
|
{
|
||||||
_primary = primary ?? throw new ArgumentNullException(nameof(primary));
|
_primary = primary ?? throw new ArgumentNullException(nameof(primary));
|
||||||
_ring = ring ?? throw new ArgumentNullException(nameof(ring));
|
_ring = ring ?? throw new ArgumentNullException(nameof(ring));
|
||||||
_failureCounter = failureCounter ?? throw new ArgumentNullException(nameof(failureCounter));
|
_failureCounter = failureCounter ?? throw new ArgumentNullException(nameof(failureCounter));
|
||||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
|
_filter = filter; // null = no-op pass-through; see WriteAsync.
|
||||||
}
|
}
|
||||||
|
|
||||||
public async Task WriteAsync(AuditEvent evt, CancellationToken ct = default)
|
public async Task WriteAsync(AuditEvent evt, CancellationToken ct = default)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(evt);
|
ArgumentNullException.ThrowIfNull(evt);
|
||||||
|
|
||||||
|
// Filter once, up-front. The filtered event flows BOTH to the primary
|
||||||
|
// and (on failure) to the ring buffer — so a primary outage that
|
||||||
|
// drains later still hands the SqliteAuditWriter a row that has
|
||||||
|
// already been truncated and redacted. The filter contract is
|
||||||
|
// "MUST NOT throw"; the null-coalesce keeps test composition roots
|
||||||
|
// that don't wire a filter working unchanged.
|
||||||
|
var filtered = _filter?.Apply(evt) ?? evt;
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
await _primary.WriteAsync(evt, ct).ConfigureAwait(false);
|
await _primary.WriteAsync(filtered, ct).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
@@ -62,8 +84,12 @@ public sealed class FallbackAuditWriter : IAuditWriter
|
|||||||
_failureCounter.Increment();
|
_failureCounter.Increment();
|
||||||
_logger.LogWarning(ex,
|
_logger.LogWarning(ex,
|
||||||
"Primary audit writer threw; routing EventId {EventId} to drop-oldest ring.",
|
"Primary audit writer threw; routing EventId {EventId} to drop-oldest ring.",
|
||||||
evt.EventId);
|
filtered.EventId);
|
||||||
_ring.TryEnqueue(evt);
|
// Ring stores the filtered copy so the eventual drain replays a
|
||||||
|
// payload that has already been capped/redacted — no second
|
||||||
|
// filter pass needed on recovery, and no risk of the ring
|
||||||
|
// holding the raw oversized blob in memory.
|
||||||
|
_ring.TryEnqueue(filtered);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,48 @@
|
|||||||
|
using ScadaLink.AuditLog.Payload;
|
||||||
|
using ScadaLink.HealthMonitoring;
|
||||||
|
|
||||||
|
namespace ScadaLink.AuditLog.Site;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Audit Log (#23) M5 Bundle C — bridges
|
||||||
|
/// <see cref="IAuditRedactionFailureCounter"/> (incremented by
|
||||||
|
/// <see cref="DefaultAuditPayloadFilter"/> every time a header / body / SQL
|
||||||
|
/// parameter redactor stage throws and the filter has to over-redact the
|
||||||
|
/// offending field) into <see cref="ISiteHealthCollector"/> so the count
|
||||||
|
/// surfaces in the site health report payload as
|
||||||
|
/// <c>SiteHealthReport.AuditRedactionFailure</c>.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <para>
|
||||||
|
/// Registered by <see cref="ServiceCollectionExtensions.AddAuditLogHealthMetricsBridge"/>;
|
||||||
|
/// callers must register <c>AddHealthMonitoring()</c> first so
|
||||||
|
/// <see cref="ISiteHealthCollector"/> resolves. The default <see cref="ServiceCollectionExtensions.AddAuditLog"/>
|
||||||
|
/// registration keeps <see cref="NoOpAuditRedactionFailureCounter"/> for nodes
|
||||||
|
/// where Site Health Monitoring is not wired (the silent-sink contract —
|
||||||
|
/// redaction failures must NEVER abort the user-facing action, alog.md §7).
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Mirrors the M2 Bundle G <see cref="HealthMetricsAuditWriteFailureCounter"/>
|
||||||
|
/// shape one-for-one so the two health-metric bridges age together.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Site-side only for M5: the redaction filter also runs on the central
|
||||||
|
/// writers (CentralAuditWriter + AuditLogIngestActor), but the central
|
||||||
|
/// health-metric surface that would expose <c>AuditRedactionFailure</c>
|
||||||
|
/// alongside the existing central counters ships in M6. Until then, the
|
||||||
|
/// central composition root keeps the NoOp default — the redactions still
|
||||||
|
/// happen, they just don't get counted into a health report.
|
||||||
|
/// </para>
|
||||||
|
/// </remarks>
|
||||||
|
public sealed class HealthMetricsAuditRedactionFailureCounter : IAuditRedactionFailureCounter
|
||||||
|
{
|
||||||
|
private readonly ISiteHealthCollector _collector;
|
||||||
|
|
||||||
|
public HealthMetricsAuditRedactionFailureCounter(ISiteHealthCollector collector)
|
||||||
|
{
|
||||||
|
_collector = collector ?? throw new ArgumentNullException(nameof(collector));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc/>
|
||||||
|
public void Increment() => _collector.IncrementAuditRedactionFailure();
|
||||||
|
}
|
||||||
@@ -25,7 +25,14 @@ public record SiteHealthReport(
|
|||||||
// primary failures (SQLite throws routed to the drop-oldest ring). Surfaces
|
// primary failures (SQLite throws routed to the drop-oldest ring). Surfaces
|
||||||
// a sustained audit-write outage on /monitoring/health. Defaults to 0 so
|
// a sustained audit-write outage on /monitoring/health. Defaults to 0 so
|
||||||
// existing producers / tests that don't construct the field stay valid.
|
// existing producers / tests that don't construct the field stay valid.
|
||||||
int SiteAuditWriteFailures = 0);
|
int SiteAuditWriteFailures = 0,
|
||||||
|
// Audit Log (#23) M5 Bundle C: per-interval count of payload-filter
|
||||||
|
// redactor over-redactions (header / body / SQL parameter stages all
|
||||||
|
// throwing → field replaced with the "<redacted: redactor error>"
|
||||||
|
// marker). Surfaces a misconfigured / catastrophic regex on
|
||||||
|
// /monitoring/health. Defaults to 0 for back-compat with existing
|
||||||
|
// producers and tests that don't construct the field.
|
||||||
|
int AuditRedactionFailure = 0);
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Broadcast wrapper used between central nodes to keep per-node
|
/// Broadcast wrapper used between central nodes to keep per-node
|
||||||
|
|||||||
@@ -19,6 +19,15 @@ public interface ISiteHealthCollector
|
|||||||
/// <c>AddAuditLogHealthMetricsBridge()</c>.
|
/// <c>AddAuditLogHealthMetricsBridge()</c>.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
void IncrementSiteAuditWriteFailures();
|
void IncrementSiteAuditWriteFailures();
|
||||||
|
/// <summary>
|
||||||
|
/// Audit Log (#23) M5 Bundle C — increment the per-interval count of
|
||||||
|
/// payload-filter redactor over-redactions (header / body / SQL
|
||||||
|
/// parameter stage throws routed to the
|
||||||
|
/// <c><redacted: redactor error></c> marker). Bridged from the
|
||||||
|
/// <c>IAuditRedactionFailureCounter</c> binding registered via
|
||||||
|
/// <c>AddAuditLogHealthMetricsBridge()</c>.
|
||||||
|
/// </summary>
|
||||||
|
void IncrementAuditRedactionFailure();
|
||||||
void UpdateConnectionHealth(string connectionName, ConnectionHealth health);
|
void UpdateConnectionHealth(string connectionName, ConnectionHealth health);
|
||||||
void RemoveConnection(string connectionName);
|
void RemoveConnection(string connectionName);
|
||||||
void UpdateTagResolution(string connectionName, int totalSubscribed, int successfullyResolved);
|
void UpdateTagResolution(string connectionName, int totalSubscribed, int successfullyResolved);
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ public class SiteHealthCollector : ISiteHealthCollector
|
|||||||
private int _alarmErrorCount;
|
private int _alarmErrorCount;
|
||||||
private int _deadLetterCount;
|
private int _deadLetterCount;
|
||||||
private int _siteAuditWriteFailures;
|
private int _siteAuditWriteFailures;
|
||||||
|
private int _auditRedactionFailures;
|
||||||
private readonly ConcurrentDictionary<string, ConnectionHealth> _connectionStatuses = new();
|
private readonly ConcurrentDictionary<string, ConnectionHealth> _connectionStatuses = new();
|
||||||
private readonly ConcurrentDictionary<string, TagResolutionStatus> _tagResolutionCounts = new();
|
private readonly ConcurrentDictionary<string, TagResolutionStatus> _tagResolutionCounts = new();
|
||||||
private readonly ConcurrentDictionary<string, string> _connectionEndpoints = new();
|
private readonly ConcurrentDictionary<string, string> _connectionEndpoints = new();
|
||||||
@@ -74,6 +75,20 @@ public class SiteHealthCollector : ISiteHealthCollector
|
|||||||
Interlocked.Increment(ref _siteAuditWriteFailures);
|
Interlocked.Increment(ref _siteAuditWriteFailures);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Audit Log (#23) M5 Bundle C — increment the per-interval count of
|
||||||
|
/// payload-filter redactor over-redactions (header / body / SQL
|
||||||
|
/// parameter stages routed to the
|
||||||
|
/// <c><redacted: redactor error></c> marker). Bridged from the
|
||||||
|
/// <c>IAuditRedactionFailureCounter</c> binding registered via
|
||||||
|
/// <c>AddAuditLogHealthMetricsBridge()</c>; reset every interval together
|
||||||
|
/// with the other per-interval counters.
|
||||||
|
/// </summary>
|
||||||
|
public void IncrementAuditRedactionFailure()
|
||||||
|
{
|
||||||
|
Interlocked.Increment(ref _auditRedactionFailures);
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Update the health status for a named data connection.
|
/// Update the health status for a named data connection.
|
||||||
/// Called by DCL when connection state changes.
|
/// Called by DCL when connection state changes.
|
||||||
@@ -158,6 +173,7 @@ public class SiteHealthCollector : ISiteHealthCollector
|
|||||||
var alarmErrors = Interlocked.Exchange(ref _alarmErrorCount, 0);
|
var alarmErrors = Interlocked.Exchange(ref _alarmErrorCount, 0);
|
||||||
var deadLetters = Interlocked.Exchange(ref _deadLetterCount, 0);
|
var deadLetters = Interlocked.Exchange(ref _deadLetterCount, 0);
|
||||||
var siteAuditWriteFailures = Interlocked.Exchange(ref _siteAuditWriteFailures, 0);
|
var siteAuditWriteFailures = Interlocked.Exchange(ref _siteAuditWriteFailures, 0);
|
||||||
|
var auditRedactionFailures = Interlocked.Exchange(ref _auditRedactionFailures, 0);
|
||||||
|
|
||||||
// Snapshot current connection and tag resolution state
|
// Snapshot current connection and tag resolution state
|
||||||
var connectionStatuses = new Dictionary<string, ConnectionHealth>(_connectionStatuses);
|
var connectionStatuses = new Dictionary<string, ConnectionHealth>(_connectionStatuses);
|
||||||
@@ -190,6 +206,7 @@ public class SiteHealthCollector : ISiteHealthCollector
|
|||||||
DataConnectionTagQuality: tagQuality,
|
DataConnectionTagQuality: tagQuality,
|
||||||
ParkedMessageCount: Interlocked.CompareExchange(ref _parkedMessageCount, 0, 0),
|
ParkedMessageCount: Interlocked.CompareExchange(ref _parkedMessageCount, 0, 0),
|
||||||
ClusterNodes: _clusterNodes?.ToList(),
|
ClusterNodes: _clusterNodes?.ToList(),
|
||||||
SiteAuditWriteFailures: siteAuditWriteFailures);
|
SiteAuditWriteFailures: siteAuditWriteFailures,
|
||||||
|
AuditRedactionFailure: auditRedactionFailures);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,220 @@
|
|||||||
|
using System.Text;
|
||||||
|
using Microsoft.Extensions.Configuration;
|
||||||
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using ScadaLink.AuditLog.Configuration;
|
||||||
|
using ScadaLink.AuditLog.Payload;
|
||||||
|
using ScadaLink.Commons.Entities.Audit;
|
||||||
|
using ScadaLink.Commons.Types.Enums;
|
||||||
|
|
||||||
|
namespace ScadaLink.AuditLog.Tests.Configuration;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Bundle D (M5-T8) tests for hot-reloadable <see cref="AuditLogOptions"/>
|
||||||
|
/// binding. The first test pins the JSON-realistic binding shape end-to-end
|
||||||
|
/// (scalars, lists, per-target overrides) so accidental drift in the section
|
||||||
|
/// layout breaks the build. The second test exercises the live hot-reload
|
||||||
|
/// path: a <see cref="DefaultAuditPayloadFilter"/> backed by a mutable
|
||||||
|
/// <see cref="IOptionsMonitor{TOptions}"/> must respond to config changes on
|
||||||
|
/// the very next event, with both cap-bytes and the regex-cache invalidation
|
||||||
|
/// flowing through without a restart.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Distinct from <see cref="AuditLogOptionsTests"/> (M1-T9) which covered
|
||||||
|
/// section binding + validator failures via single-key in-memory config — those
|
||||||
|
/// tests exist; these add (a) end-to-end binding from a realistic JSON literal
|
||||||
|
/// and (b) the hot-reload behavioural contract the M5-T8 spec calls out.
|
||||||
|
/// </remarks>
|
||||||
|
public class AuditLogOptionsBindingTests
|
||||||
|
{
|
||||||
|
[Fact]
|
||||||
|
public void AuditLog_Section_Binds_AllFields()
|
||||||
|
{
|
||||||
|
const string json = """
|
||||||
|
{
|
||||||
|
"AuditLog": {
|
||||||
|
"DefaultCapBytes": 4096,
|
||||||
|
"ErrorCapBytes": 32768,
|
||||||
|
"HeaderRedactList": ["Authorization", "Custom-Token"],
|
||||||
|
"GlobalBodyRedactors": ["\"password\":\\s*\"[^\"]*\""],
|
||||||
|
"PerTargetOverrides": {
|
||||||
|
"myconnection": {
|
||||||
|
"CapBytes": 16384,
|
||||||
|
"AdditionalBodyRedactors": [],
|
||||||
|
"RedactSqlParamsMatching": "@token|@secret"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"RetentionDays": 180
|
||||||
|
}
|
||||||
|
}
|
||||||
|
""";
|
||||||
|
|
||||||
|
using var stream = new MemoryStream(Encoding.UTF8.GetBytes(json));
|
||||||
|
var configuration = new ConfigurationBuilder()
|
||||||
|
.AddJsonStream(stream)
|
||||||
|
.Build();
|
||||||
|
var services = new ServiceCollection();
|
||||||
|
services.AddAuditLog(configuration);
|
||||||
|
using var provider = services.BuildServiceProvider();
|
||||||
|
|
||||||
|
var opts = provider.GetRequiredService<IOptions<AuditLogOptions>>().Value;
|
||||||
|
|
||||||
|
// Scalars.
|
||||||
|
Assert.Equal(4096, opts.DefaultCapBytes);
|
||||||
|
Assert.Equal(32768, opts.ErrorCapBytes);
|
||||||
|
Assert.Equal(180, opts.RetentionDays);
|
||||||
|
|
||||||
|
// HeaderRedactList: the Microsoft.Extensions.Configuration list binder
|
||||||
|
// APPENDS to the default list, so we assert containment rather than
|
||||||
|
// exact equality (see M1-T9 AuditLogOptionsTests for the rationale).
|
||||||
|
Assert.Contains("Authorization", opts.HeaderRedactList);
|
||||||
|
Assert.Contains("Custom-Token", opts.HeaderRedactList);
|
||||||
|
|
||||||
|
// GlobalBodyRedactors: pattern arrived intact, regex-escape sequences
|
||||||
|
// and all.
|
||||||
|
Assert.Contains("\"password\":\\s*\"[^\"]*\"", opts.GlobalBodyRedactors);
|
||||||
|
|
||||||
|
// PerTargetOverrides: keyed by connection name, each field bound.
|
||||||
|
Assert.True(opts.PerTargetOverrides.ContainsKey("myconnection"));
|
||||||
|
var ov = opts.PerTargetOverrides["myconnection"];
|
||||||
|
Assert.Equal(16384, ov.CapBytes);
|
||||||
|
// Microsoft.Extensions.Configuration JSON binder leaves an empty array
|
||||||
|
// null on a nullable List<T>; either null or empty is acceptable as
|
||||||
|
// "no additional redactors" — both result in zero patterns at use.
|
||||||
|
Assert.True(ov.AdditionalBodyRedactors is null || ov.AdditionalBodyRedactors.Count == 0);
|
||||||
|
Assert.Equal("@token|@secret", ov.RedactSqlParamsMatching);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Filter_Behavior_Updates_OnConfigReload()
|
||||||
|
{
|
||||||
|
// Start at the default cap (4096). A 5 KB body should be truncated;
|
||||||
|
// PayloadTruncated flips to true.
|
||||||
|
var initial = new AuditLogOptions { DefaultCapBytes = 4096 };
|
||||||
|
var monitor = new TestOptionsMonitor<AuditLogOptions>(initial);
|
||||||
|
var filter = new DefaultAuditPayloadFilter(
|
||||||
|
monitor,
|
||||||
|
NullLogger<DefaultAuditPayloadFilter>.Instance);
|
||||||
|
|
||||||
|
var body = new string('x', 5 * 1024);
|
||||||
|
var evt = new AuditEvent
|
||||||
|
{
|
||||||
|
EventId = Guid.NewGuid(),
|
||||||
|
OccurredAtUtc = DateTime.UtcNow,
|
||||||
|
Channel = AuditChannel.ApiOutbound,
|
||||||
|
Kind = AuditKind.ApiCall,
|
||||||
|
Status = AuditStatus.Delivered,
|
||||||
|
RequestSummary = body,
|
||||||
|
};
|
||||||
|
|
||||||
|
var resultBefore = filter.Apply(evt);
|
||||||
|
Assert.True(resultBefore.PayloadTruncated, "5KB body at 4096 cap must be truncated");
|
||||||
|
Assert.NotNull(resultBefore.RequestSummary);
|
||||||
|
Assert.True(Encoding.UTF8.GetByteCount(resultBefore.RequestSummary!) <= 4096);
|
||||||
|
|
||||||
|
// Reload: cap raised to 16384 — next event must NOT truncate. This is
|
||||||
|
// the M5-T8 contract: the filter sees the new value on the very next
|
||||||
|
// Apply, without process restart.
|
||||||
|
monitor.Set(new AuditLogOptions { DefaultCapBytes = 16384 });
|
||||||
|
|
||||||
|
var resultAfter = filter.Apply(evt);
|
||||||
|
Assert.False(resultAfter.PayloadTruncated, "5KB body at 16384 cap must NOT be truncated");
|
||||||
|
Assert.Equal(body, resultAfter.RequestSummary);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Filter_PicksUp_NewBodyRedactor_OnConfigReload()
|
||||||
|
{
|
||||||
|
// The regex cache is keyed by pattern string — a redactor added via
|
||||||
|
// config reload must compile + apply on the very next event without a
|
||||||
|
// process restart. Pre-reload: no redactor, hunter2 survives. After
|
||||||
|
// reload: hunter2 redacted.
|
||||||
|
var monitor = new TestOptionsMonitor<AuditLogOptions>(new AuditLogOptions());
|
||||||
|
var filter = new DefaultAuditPayloadFilter(
|
||||||
|
monitor,
|
||||||
|
NullLogger<DefaultAuditPayloadFilter>.Instance);
|
||||||
|
|
||||||
|
const string body = "{\"user\":\"alice\",\"password\":\"hunter2\"}";
|
||||||
|
var evt = new AuditEvent
|
||||||
|
{
|
||||||
|
EventId = Guid.NewGuid(),
|
||||||
|
OccurredAtUtc = DateTime.UtcNow,
|
||||||
|
Channel = AuditChannel.ApiOutbound,
|
||||||
|
Kind = AuditKind.ApiCall,
|
||||||
|
Status = AuditStatus.Delivered,
|
||||||
|
RequestSummary = body,
|
||||||
|
};
|
||||||
|
|
||||||
|
var before = filter.Apply(evt);
|
||||||
|
Assert.Contains("hunter2", before.RequestSummary!);
|
||||||
|
|
||||||
|
monitor.Set(new AuditLogOptions
|
||||||
|
{
|
||||||
|
GlobalBodyRedactors = new List<string> { "\"password\":\\s*\"[^\"]*\"" },
|
||||||
|
});
|
||||||
|
|
||||||
|
var after = filter.Apply(evt);
|
||||||
|
Assert.DoesNotContain("hunter2", after.RequestSummary!);
|
||||||
|
Assert.Contains("<redacted>", after.RequestSummary!);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// IOptionsMonitor test double — exposes a <see cref="Set"/> method that
|
||||||
|
/// updates the current value and fires registered OnChange callbacks.
|
||||||
|
/// Avoids depending on Microsoft.Extensions.Configuration's reload-token
|
||||||
|
/// plumbing, which is awkward to drive deterministically from xUnit.
|
||||||
|
/// </summary>
|
||||||
|
private sealed class TestOptionsMonitor<T> : IOptionsMonitor<T>
|
||||||
|
{
|
||||||
|
private T _current;
|
||||||
|
private readonly List<Action<T, string?>> _listeners = new();
|
||||||
|
|
||||||
|
public TestOptionsMonitor(T initial) => _current = initial;
|
||||||
|
|
||||||
|
public T CurrentValue => _current;
|
||||||
|
|
||||||
|
public T Get(string? name) => _current;
|
||||||
|
|
||||||
|
public IDisposable? OnChange(Action<T, string?> listener)
|
||||||
|
{
|
||||||
|
lock (_listeners)
|
||||||
|
{
|
||||||
|
_listeners.Add(listener);
|
||||||
|
}
|
||||||
|
return new Unsubscribe(_listeners, listener);
|
||||||
|
}
|
||||||
|
|
||||||
|
public void Set(T value)
|
||||||
|
{
|
||||||
|
_current = value;
|
||||||
|
Action<T, string?>[] snapshot;
|
||||||
|
lock (_listeners)
|
||||||
|
{
|
||||||
|
snapshot = _listeners.ToArray();
|
||||||
|
}
|
||||||
|
foreach (var l in snapshot)
|
||||||
|
{
|
||||||
|
l(_current, Options.DefaultName);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class Unsubscribe : IDisposable
|
||||||
|
{
|
||||||
|
private readonly List<Action<T, string?>> _listeners;
|
||||||
|
private readonly Action<T, string?> _listener;
|
||||||
|
public Unsubscribe(List<Action<T, string?>> listeners, Action<T, string?> listener)
|
||||||
|
{
|
||||||
|
_listeners = listeners;
|
||||||
|
_listener = listener;
|
||||||
|
}
|
||||||
|
public void Dispose()
|
||||||
|
{
|
||||||
|
lock (_listeners)
|
||||||
|
{
|
||||||
|
_listeners.Remove(_listener);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,207 @@
|
|||||||
|
using System.Text;
|
||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using ScadaLink.AuditLog.Configuration;
|
||||||
|
using ScadaLink.AuditLog.Payload;
|
||||||
|
using ScadaLink.Commons.Entities.Audit;
|
||||||
|
using ScadaLink.Commons.Types.Enums;
|
||||||
|
|
||||||
|
namespace ScadaLink.AuditLog.Tests.Payload;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Bundle B (M5-T4) tests for body regex redaction in
|
||||||
|
/// <see cref="DefaultAuditPayloadFilter"/>. The body-redactor stage runs
|
||||||
|
/// regex replace against RequestSummary / ResponseSummary / ErrorDetail /
|
||||||
|
/// Extra, replacing every match with <c><redacted></c>. Regexes come
|
||||||
|
/// from <see cref="AuditLogOptions.GlobalBodyRedactors"/> plus the per-target
|
||||||
|
/// <see cref="PerTargetRedactionOverride.AdditionalBodyRedactors"/>. Each
|
||||||
|
/// regex is compiled with a 50 ms timeout so catastrophic-backtracking
|
||||||
|
/// patterns trip a <see cref="System.Text.RegularExpressions.RegexMatchTimeoutException"/>;
|
||||||
|
/// when that happens the offending field is over-redacted with
|
||||||
|
/// <c><redacted: redactor error></c> and the
|
||||||
|
/// <see cref="IAuditRedactionFailureCounter"/> is incremented. The stage runs
|
||||||
|
/// BEFORE truncation.
|
||||||
|
/// </summary>
|
||||||
|
public class BodyRegexRedactionTests
|
||||||
|
{
|
||||||
|
private static IOptionsMonitor<AuditLogOptions> Monitor(AuditLogOptions? opts = null) =>
|
||||||
|
new StaticMonitor(opts ?? new AuditLogOptions());
|
||||||
|
|
||||||
|
private static DefaultAuditPayloadFilter Filter(
|
||||||
|
AuditLogOptions? opts = null,
|
||||||
|
IAuditRedactionFailureCounter? counter = null) =>
|
||||||
|
new(Monitor(opts), NullLogger<DefaultAuditPayloadFilter>.Instance, counter);
|
||||||
|
|
||||||
|
private static AuditEvent NewEvent(
|
||||||
|
AuditStatus status = AuditStatus.Delivered,
|
||||||
|
string? request = null,
|
||||||
|
string? response = null,
|
||||||
|
string? errorDetail = null,
|
||||||
|
string? extra = null,
|
||||||
|
string? target = null) => new()
|
||||||
|
{
|
||||||
|
EventId = Guid.NewGuid(),
|
||||||
|
OccurredAtUtc = DateTime.UtcNow,
|
||||||
|
Channel = AuditChannel.ApiOutbound,
|
||||||
|
Kind = AuditKind.ApiCall,
|
||||||
|
Status = status,
|
||||||
|
Target = target,
|
||||||
|
RequestSummary = request,
|
||||||
|
ResponseSummary = response,
|
||||||
|
ErrorDetail = errorDetail,
|
||||||
|
Extra = extra,
|
||||||
|
};
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void GlobalRegex_HunterPassword_Redacted()
|
||||||
|
{
|
||||||
|
var opts = new AuditLogOptions
|
||||||
|
{
|
||||||
|
GlobalBodyRedactors = new List<string> { "\"password\":\\s*\"[^\"]*\"" },
|
||||||
|
};
|
||||||
|
const string input = "{\"user\":\"alice\",\"password\":\"hunter2\"}";
|
||||||
|
var evt = NewEvent(request: input);
|
||||||
|
|
||||||
|
var result = Filter(opts).Apply(evt);
|
||||||
|
|
||||||
|
Assert.NotNull(result.RequestSummary);
|
||||||
|
Assert.Contains("<redacted>", result.RequestSummary);
|
||||||
|
Assert.DoesNotContain("hunter2", result.RequestSummary);
|
||||||
|
Assert.Contains("alice", result.RequestSummary);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void PerTargetRegex_OnlyAppliedToMatchingTarget()
|
||||||
|
{
|
||||||
|
var opts = new AuditLogOptions
|
||||||
|
{
|
||||||
|
PerTargetOverrides = new Dictionary<string, PerTargetRedactionOverride>
|
||||||
|
{
|
||||||
|
["esg.A"] = new PerTargetRedactionOverride
|
||||||
|
{
|
||||||
|
AdditionalBodyRedactors = new List<string> { "SECRET-[A-Z0-9]+" },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
const string input = "token=SECRET-XYZ123 normal-text";
|
||||||
|
|
||||||
|
var matchedEvt = NewEvent(request: input, target: "esg.A");
|
||||||
|
var matchedResult = Filter(opts).Apply(matchedEvt);
|
||||||
|
Assert.Contains("<redacted>", matchedResult.RequestSummary!);
|
||||||
|
Assert.DoesNotContain("SECRET-XYZ123", matchedResult.RequestSummary!);
|
||||||
|
|
||||||
|
var unmatchedEvt = NewEvent(request: input, target: "esg.B");
|
||||||
|
var unmatchedResult = Filter(opts).Apply(unmatchedEvt);
|
||||||
|
Assert.Equal(input, unmatchedResult.RequestSummary);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void RegexThrowsTimeout_FieldBecomesRedactedMarker_CounterIncrements()
|
||||||
|
{
|
||||||
|
// Catastrophic backtracking pattern: alternation with overlapping
|
||||||
|
// groups + non-matching suffix forces the engine into exponential
|
||||||
|
// work that blows past the 50 ms timeout. Append a non-'a' character
|
||||||
|
// so the suffix anchor fails and the engine has to exhaust every
|
||||||
|
// permutation.
|
||||||
|
var opts = new AuditLogOptions
|
||||||
|
{
|
||||||
|
GlobalBodyRedactors = new List<string> { "^(a+)+$" },
|
||||||
|
};
|
||||||
|
// 30 'a's followed by '!' — small enough to keep the test fast, big
|
||||||
|
// enough to overflow the 50 ms regex timeout on every machine the CI
|
||||||
|
// grid runs on.
|
||||||
|
var input = new string('a', 30) + "!";
|
||||||
|
var counter = new CountingRedactionFailureCounter();
|
||||||
|
var evt = NewEvent(request: input);
|
||||||
|
|
||||||
|
var result = Filter(opts, counter).Apply(evt);
|
||||||
|
|
||||||
|
Assert.Equal("<redacted: redactor error>", result.RequestSummary);
|
||||||
|
Assert.True(counter.Count >= 1, $"expected counter >= 1, got {counter.Count}");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void NoRegexConfigured_FieldUnchanged()
|
||||||
|
{
|
||||||
|
var opts = new AuditLogOptions(); // no GlobalBodyRedactors, no per-target
|
||||||
|
const string input = "{\"password\":\"hunter2\"}";
|
||||||
|
var evt = NewEvent(request: input);
|
||||||
|
|
||||||
|
var result = Filter(opts).Apply(evt);
|
||||||
|
|
||||||
|
Assert.Equal(input, result.RequestSummary);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void RedactionAppliedBeforeTruncation()
|
||||||
|
{
|
||||||
|
// A pattern that matches a long secret in the body. The full input is
|
||||||
|
// > 8 KB so truncation must run. After redaction:
|
||||||
|
// * the marker survives the cap (redaction ran first),
|
||||||
|
// * the original secret bytes do NOT survive,
|
||||||
|
// * PayloadTruncated is set.
|
||||||
|
var opts = new AuditLogOptions
|
||||||
|
{
|
||||||
|
GlobalBodyRedactors = new List<string> { "SECRET-[A-Z0-9]+" },
|
||||||
|
};
|
||||||
|
var secret = "SECRET-ABCDEF123";
|
||||||
|
var padding = new string('x', 9 * 1024);
|
||||||
|
var input = secret + padding;
|
||||||
|
Assert.True(Encoding.UTF8.GetByteCount(input) > 8192);
|
||||||
|
|
||||||
|
var evt = NewEvent(AuditStatus.Delivered, request: input);
|
||||||
|
|
||||||
|
var result = Filter(opts).Apply(evt);
|
||||||
|
|
||||||
|
Assert.NotNull(result.RequestSummary);
|
||||||
|
Assert.True(Encoding.UTF8.GetByteCount(result.RequestSummary!) <= 8192);
|
||||||
|
Assert.Contains("<redacted>", result.RequestSummary);
|
||||||
|
Assert.DoesNotContain(secret, result.RequestSummary);
|
||||||
|
Assert.True(result.PayloadTruncated);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void CatastrophicBacktrackingRegex_AtCompileTime_RejectedAtStartup()
|
||||||
|
{
|
||||||
|
// .NET's regex engine has no compile-time detection for catastrophic
|
||||||
|
// backtracking (only structural validation), so the filter's
|
||||||
|
// protection is RUNTIME — the 50 ms per-match timeout. We assert the
|
||||||
|
// safety net behaviour: a known evil pattern compiles cleanly but
|
||||||
|
// matches time out at runtime, the field is over-redacted, and the
|
||||||
|
// failure counter is incremented. Future engines that DO support
|
||||||
|
// compile-time analysis can tighten this further; the contract here
|
||||||
|
// is that the user-facing action is never aborted.
|
||||||
|
var evilPattern = "^(a+)+$";
|
||||||
|
var opts = new AuditLogOptions
|
||||||
|
{
|
||||||
|
GlobalBodyRedactors = new List<string> { evilPattern },
|
||||||
|
};
|
||||||
|
var input = new string('a', 30) + "!";
|
||||||
|
var counter = new CountingRedactionFailureCounter();
|
||||||
|
var evt = NewEvent(request: input);
|
||||||
|
|
||||||
|
var result = Filter(opts, counter).Apply(evt);
|
||||||
|
|
||||||
|
Assert.Equal("<redacted: redactor error>", result.RequestSummary);
|
||||||
|
Assert.True(counter.Count >= 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Test double that counts increments.</summary>
|
||||||
|
private sealed class CountingRedactionFailureCounter : IAuditRedactionFailureCounter
|
||||||
|
{
|
||||||
|
private int _count;
|
||||||
|
public int Count => _count;
|
||||||
|
public void Increment() => System.Threading.Interlocked.Increment(ref _count);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>IOptionsMonitor test double — returns the same snapshot on every read.</summary>
|
||||||
|
private sealed class StaticMonitor : IOptionsMonitor<AuditLogOptions>
|
||||||
|
{
|
||||||
|
private readonly AuditLogOptions _value;
|
||||||
|
public StaticMonitor(AuditLogOptions value) => _value = value;
|
||||||
|
public AuditLogOptions CurrentValue => _value;
|
||||||
|
public AuditLogOptions Get(string? name) => _value;
|
||||||
|
public IDisposable? OnChange(Action<AuditLogOptions, string?> listener) => null;
|
||||||
|
}
|
||||||
|
}
|
||||||
301
tests/ScadaLink.AuditLog.Tests/Payload/FilterIntegrationTests.cs
Normal file
301
tests/ScadaLink.AuditLog.Tests/Payload/FilterIntegrationTests.cs
Normal file
@@ -0,0 +1,301 @@
|
|||||||
|
using System.Text;
|
||||||
|
using Akka.Actor;
|
||||||
|
using Akka.TestKit.Xunit2;
|
||||||
|
using Microsoft.Data.Sqlite;
|
||||||
|
using Microsoft.EntityFrameworkCore;
|
||||||
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using NSubstitute;
|
||||||
|
using ScadaLink.AuditLog.Central;
|
||||||
|
using ScadaLink.AuditLog.Configuration;
|
||||||
|
using ScadaLink.AuditLog.Payload;
|
||||||
|
using ScadaLink.AuditLog.Site;
|
||||||
|
using ScadaLink.Commons.Entities.Audit;
|
||||||
|
using ScadaLink.Commons.Interfaces.Repositories;
|
||||||
|
using ScadaLink.Commons.Interfaces.Services;
|
||||||
|
using ScadaLink.Commons.Messages.Audit;
|
||||||
|
using ScadaLink.Commons.Types;
|
||||||
|
using ScadaLink.Commons.Types.Enums;
|
||||||
|
using ScadaLink.ConfigurationDatabase;
|
||||||
|
using ScadaLink.ConfigurationDatabase.Repositories;
|
||||||
|
using ScadaLink.ConfigurationDatabase.Tests.Migrations;
|
||||||
|
|
||||||
|
namespace ScadaLink.AuditLog.Tests.Payload;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Bundle C (M5-T6) integration tests verifying that the
|
||||||
|
/// <see cref="IAuditPayloadFilter"/> wires correctly into each of the three
|
||||||
|
/// writer entry points — <see cref="FallbackAuditWriter"/> on the site hot
|
||||||
|
/// path, <see cref="CentralAuditWriter"/> on the central direct-write path,
|
||||||
|
/// and <see cref="AuditLogIngestActor"/> on the site→central telemetry ingest
|
||||||
|
/// path (both the per-row <c>IngestAuditEventsCommand</c> handler and the
|
||||||
|
/// combined <c>IngestCachedTelemetryCommand</c> dual-write handler).
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Bundle B established the filter's behaviour in isolation (truncation,
|
||||||
|
/// header redaction, body-regex redaction, SQL-parameter redaction). Bundle C
|
||||||
|
/// proves that filtering actually happens before persistence — a 10 KB
|
||||||
|
/// RequestSummary on a Delivered row must land on disk capped to 8192 bytes
|
||||||
|
/// with <c>PayloadTruncated=true</c>, regardless of whether the row was
|
||||||
|
/// written via the site's SQLite hot path, the central direct-write path, or
|
||||||
|
/// the site→central ingest pipeline. We use the production
|
||||||
|
/// <see cref="DefaultAuditPayloadFilter"/> through every test so the
|
||||||
|
/// integration is real end-to-end, not a fake-filter assertion.
|
||||||
|
/// </remarks>
|
||||||
|
public class FilterIntegrationTests
|
||||||
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// Default-options filter — 8 KiB cap on success rows, 64 KiB on error
|
||||||
|
/// rows. Cached and reused; the filter is stateless w.r.t. the per-event
|
||||||
|
/// inputs and the regex cache is happy under sharing.
|
||||||
|
/// </summary>
|
||||||
|
private static IAuditPayloadFilter NewDefaultFilter()
|
||||||
|
{
|
||||||
|
var monitor = Microsoft.Extensions.Options.Options.Create(new AuditLogOptions());
|
||||||
|
return new DefaultAuditPayloadFilter(
|
||||||
|
new StaticMonitor(monitor.Value),
|
||||||
|
NullLogger<DefaultAuditPayloadFilter>.Instance);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static AuditEvent NewEvent(string? request = null, Guid? eventId = null) => new()
|
||||||
|
{
|
||||||
|
EventId = eventId ?? Guid.NewGuid(),
|
||||||
|
OccurredAtUtc = new DateTime(2026, 5, 20, 10, 0, 0, DateTimeKind.Utc),
|
||||||
|
Channel = AuditChannel.ApiOutbound,
|
||||||
|
Kind = AuditKind.ApiCall,
|
||||||
|
// Delivered = success cap (8 KiB). Picking a success status so the
|
||||||
|
// 10 KB payload reliably trips the filter.
|
||||||
|
Status = AuditStatus.Delivered,
|
||||||
|
RequestSummary = request,
|
||||||
|
PayloadTruncated = false,
|
||||||
|
ForwardState = AuditForwardState.Pending,
|
||||||
|
};
|
||||||
|
|
||||||
|
// -- C1.1: FallbackAuditWriter applies the filter before SQLite write ----
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task FallbackAuditWriter_AppliesFilter_BeforeSqliteWrite()
|
||||||
|
{
|
||||||
|
var dataSource =
|
||||||
|
$"file:filter-fbw-{Guid.NewGuid():N}?mode=memory&cache=shared";
|
||||||
|
// Hold the in-memory database alive for the verifier connection —
|
||||||
|
// SQLite frees a Cache=Shared in-memory DB when the last connection
|
||||||
|
// closes, so without this keep-alive the FallbackAuditWriter's
|
||||||
|
// dispose would wipe the data before we could query it.
|
||||||
|
using var keepAlive = new SqliteConnection($"Data Source={dataSource};Cache=Shared");
|
||||||
|
keepAlive.Open();
|
||||||
|
|
||||||
|
var sqliteWriter = new SqliteAuditWriter(
|
||||||
|
Microsoft.Extensions.Options.Options.Create(new SqliteAuditWriterOptions { DatabasePath = dataSource }),
|
||||||
|
NullLogger<SqliteAuditWriter>.Instance,
|
||||||
|
connectionStringOverride: $"Data Source={dataSource};Cache=Shared");
|
||||||
|
await using var _disposeSqlite = sqliteWriter;
|
||||||
|
|
||||||
|
var fallback = new FallbackAuditWriter(
|
||||||
|
sqliteWriter,
|
||||||
|
new RingBufferFallback(),
|
||||||
|
new NoOpAuditWriteFailureCounter(),
|
||||||
|
NullLogger<FallbackAuditWriter>.Instance,
|
||||||
|
NewDefaultFilter());
|
||||||
|
|
||||||
|
var bigRequest = new string('a', 10 * 1024);
|
||||||
|
var evt = NewEvent(request: bigRequest);
|
||||||
|
await fallback.WriteAsync(evt);
|
||||||
|
|
||||||
|
// Read back via a fresh connection so we observe what actually
|
||||||
|
// landed in SQLite — not what the writer was handed.
|
||||||
|
using var verifier = new SqliteConnection($"Data Source={dataSource};Cache=Shared");
|
||||||
|
verifier.Open();
|
||||||
|
using var cmd = verifier.CreateCommand();
|
||||||
|
cmd.CommandText = "SELECT RequestSummary, PayloadTruncated FROM AuditLog WHERE EventId = $id;";
|
||||||
|
cmd.Parameters.AddWithValue("$id", evt.EventId.ToString());
|
||||||
|
using var reader = cmd.ExecuteReader();
|
||||||
|
Assert.True(reader.Read());
|
||||||
|
var persistedRequest = reader.GetString(0);
|
||||||
|
var truncatedFlag = reader.GetInt32(1);
|
||||||
|
|
||||||
|
Assert.Equal(8192, Encoding.UTF8.GetByteCount(persistedRequest));
|
||||||
|
Assert.Equal(1, truncatedFlag);
|
||||||
|
}
|
||||||
|
|
||||||
|
// -- C1.2: CentralAuditWriter applies the filter before repo insert ------
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task CentralAuditWriter_AppliesFilter_BeforeRepoInsert()
|
||||||
|
{
|
||||||
|
var repo = Substitute.For<IAuditLogRepository>();
|
||||||
|
var services = new ServiceCollection();
|
||||||
|
services.AddScoped(_ => repo);
|
||||||
|
services.AddSingleton(NewDefaultFilter());
|
||||||
|
var provider = services.BuildServiceProvider();
|
||||||
|
|
||||||
|
var writer = new CentralAuditWriter(
|
||||||
|
provider, NullLogger<CentralAuditWriter>.Instance, NewDefaultFilter());
|
||||||
|
|
||||||
|
var bigRequest = new string('b', 10 * 1024);
|
||||||
|
var evt = NewEvent(request: bigRequest);
|
||||||
|
await writer.WriteAsync(evt);
|
||||||
|
|
||||||
|
// Verify the repository saw the FILTERED event, not the raw one.
|
||||||
|
// The filter caps RequestSummary to 8192 bytes on a Delivered row
|
||||||
|
// and flags PayloadTruncated.
|
||||||
|
await repo.Received(1).InsertIfNotExistsAsync(
|
||||||
|
Arg.Is<AuditEvent>(e =>
|
||||||
|
e.EventId == evt.EventId
|
||||||
|
&& e.RequestSummary != null
|
||||||
|
&& Encoding.UTF8.GetByteCount(e.RequestSummary) == 8192
|
||||||
|
&& e.PayloadTruncated == true),
|
||||||
|
Arg.Any<CancellationToken>());
|
||||||
|
}
|
||||||
|
|
||||||
|
// -- C1.3 + C1.4: AuditLogIngestActor applies the filter on both paths ---
|
||||||
|
|
||||||
|
public class IngestActorTests : TestKit, IClassFixture<MsSqlMigrationFixture>
|
||||||
|
{
|
||||||
|
private readonly MsSqlMigrationFixture _fixture;
|
||||||
|
|
||||||
|
public IngestActorTests(MsSqlMigrationFixture fixture)
|
||||||
|
{
|
||||||
|
_fixture = fixture;
|
||||||
|
}
|
||||||
|
|
||||||
|
private ScadaLinkDbContext CreateReadContext()
|
||||||
|
{
|
||||||
|
var options = new DbContextOptionsBuilder<ScadaLinkDbContext>()
|
||||||
|
.UseSqlServer(_fixture.ConnectionString)
|
||||||
|
.Options;
|
||||||
|
return new ScadaLinkDbContext(options);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string NewSiteId() =>
|
||||||
|
"test-bundle-c1-filter-" + Guid.NewGuid().ToString("N").Substring(0, 8);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Build the IServiceProvider in the production-flavoured shape —
|
||||||
|
/// scoped repositories + a singleton <see cref="IAuditPayloadFilter"/>
|
||||||
|
/// resolved per-message from the actor's scope. Matches the
|
||||||
|
/// AddAuditLog registrations Bundle B established.
|
||||||
|
/// </summary>
|
||||||
|
private IServiceProvider BuildServiceProvider()
|
||||||
|
{
|
||||||
|
var services = new ServiceCollection();
|
||||||
|
services.AddDbContext<ScadaLinkDbContext>(opts =>
|
||||||
|
opts.UseSqlServer(_fixture.ConnectionString)
|
||||||
|
.ConfigureWarnings(w => w.Ignore(
|
||||||
|
Microsoft.EntityFrameworkCore.Diagnostics.RelationalEventId.PendingModelChangesWarning)));
|
||||||
|
services.AddScoped<IAuditLogRepository>(sp =>
|
||||||
|
new AuditLogRepository(sp.GetRequiredService<ScadaLinkDbContext>()));
|
||||||
|
services.AddScoped<ISiteCallAuditRepository>(sp =>
|
||||||
|
new SiteCallAuditRepository(sp.GetRequiredService<ScadaLinkDbContext>()));
|
||||||
|
services.AddSingleton(NewDefaultFilter());
|
||||||
|
return services.BuildServiceProvider();
|
||||||
|
}
|
||||||
|
|
||||||
|
[SkippableFact]
|
||||||
|
public async Task AuditLogIngestActor_AppliesFilter_BeforeBatchInsert()
|
||||||
|
{
|
||||||
|
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
|
||||||
|
|
||||||
|
var siteId = NewSiteId();
|
||||||
|
var bigRequest = new string('c', 10 * 1024);
|
||||||
|
var evt = new AuditEvent
|
||||||
|
{
|
||||||
|
EventId = Guid.NewGuid(),
|
||||||
|
OccurredAtUtc = new DateTime(2026, 5, 20, 10, 0, 0, DateTimeKind.Utc),
|
||||||
|
Channel = AuditChannel.ApiOutbound,
|
||||||
|
Kind = AuditKind.ApiCall,
|
||||||
|
Status = AuditStatus.Delivered,
|
||||||
|
SourceSiteId = siteId,
|
||||||
|
RequestSummary = bigRequest,
|
||||||
|
PayloadTruncated = false,
|
||||||
|
};
|
||||||
|
|
||||||
|
var sp = BuildServiceProvider();
|
||||||
|
var actor = Sys.ActorOf(Props.Create(() => new AuditLogIngestActor(
|
||||||
|
sp, NullLogger<AuditLogIngestActor>.Instance)));
|
||||||
|
|
||||||
|
actor.Tell(new IngestAuditEventsCommand(new[] { evt }), TestActor);
|
||||||
|
ExpectMsg<IngestAuditEventsReply>(TimeSpan.FromSeconds(15));
|
||||||
|
|
||||||
|
// Verify the persisted row was filtered before INSERT.
|
||||||
|
await using var read = CreateReadContext();
|
||||||
|
var row = await read.Set<AuditEvent>()
|
||||||
|
.SingleAsync(e => e.EventId == evt.EventId);
|
||||||
|
Assert.NotNull(row.RequestSummary);
|
||||||
|
Assert.Equal(8192, Encoding.UTF8.GetByteCount(row.RequestSummary!));
|
||||||
|
Assert.True(row.PayloadTruncated);
|
||||||
|
}
|
||||||
|
|
||||||
|
[SkippableFact]
|
||||||
|
public async Task AuditLogIngestActor_CachedTelemetry_AppliesFilter()
|
||||||
|
{
|
||||||
|
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
|
||||||
|
|
||||||
|
var siteId = NewSiteId();
|
||||||
|
var trackedId = TrackedOperationId.New();
|
||||||
|
var bigRequest = new string('d', 10 * 1024);
|
||||||
|
var audit = new AuditEvent
|
||||||
|
{
|
||||||
|
EventId = Guid.NewGuid(),
|
||||||
|
OccurredAtUtc = new DateTime(2026, 5, 20, 10, 0, 0, DateTimeKind.Utc),
|
||||||
|
Channel = AuditChannel.ApiOutbound,
|
||||||
|
Kind = AuditKind.CachedSubmit,
|
||||||
|
Status = AuditStatus.Submitted,
|
||||||
|
SourceSiteId = siteId,
|
||||||
|
CorrelationId = trackedId.Value,
|
||||||
|
RequestSummary = bigRequest,
|
||||||
|
PayloadTruncated = false,
|
||||||
|
};
|
||||||
|
var siteCall = new SiteCall
|
||||||
|
{
|
||||||
|
TrackedOperationId = trackedId,
|
||||||
|
Channel = "ApiOutbound",
|
||||||
|
Target = "ERP.GetOrder",
|
||||||
|
SourceSite = siteId,
|
||||||
|
Status = "Submitted",
|
||||||
|
RetryCount = 0,
|
||||||
|
CreatedAtUtc = new DateTime(2026, 5, 20, 10, 0, 0, DateTimeKind.Utc),
|
||||||
|
UpdatedAtUtc = new DateTime(2026, 5, 20, 10, 0, 0, DateTimeKind.Utc),
|
||||||
|
IngestedAtUtc = new DateTime(2026, 5, 20, 10, 0, 0, DateTimeKind.Utc),
|
||||||
|
};
|
||||||
|
|
||||||
|
var sp = BuildServiceProvider();
|
||||||
|
var actor = Sys.ActorOf(Props.Create(() => new AuditLogIngestActor(
|
||||||
|
sp, NullLogger<AuditLogIngestActor>.Instance)));
|
||||||
|
|
||||||
|
actor.Tell(
|
||||||
|
new IngestCachedTelemetryCommand(new[] { new CachedTelemetryEntry(audit, siteCall) }),
|
||||||
|
TestActor);
|
||||||
|
ExpectMsg<IngestCachedTelemetryReply>(TimeSpan.FromSeconds(15));
|
||||||
|
|
||||||
|
await using var read = CreateReadContext();
|
||||||
|
var auditRow = await read.Set<AuditEvent>()
|
||||||
|
.SingleAsync(e => e.EventId == audit.EventId);
|
||||||
|
Assert.NotNull(auditRow.RequestSummary);
|
||||||
|
// Bundle C filter must run before the dual-write transaction
|
||||||
|
// commits, so the persisted AuditLog row carries the truncated
|
||||||
|
// payload.
|
||||||
|
Assert.Equal(8192, Encoding.UTF8.GetByteCount(auditRow.RequestSummary!));
|
||||||
|
Assert.True(auditRow.PayloadTruncated);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// IOptionsMonitor test double — returns the same snapshot on every read,
|
||||||
|
/// no change-token plumbing required for these tests. Mirrors the helper
|
||||||
|
/// used in <c>TruncationTests</c>.
|
||||||
|
/// </summary>
|
||||||
|
private sealed class StaticMonitor : IOptionsMonitor<AuditLogOptions>
|
||||||
|
{
|
||||||
|
private readonly AuditLogOptions _value;
|
||||||
|
|
||||||
|
public StaticMonitor(AuditLogOptions value) => _value = value;
|
||||||
|
|
||||||
|
public AuditLogOptions CurrentValue => _value;
|
||||||
|
|
||||||
|
public AuditLogOptions Get(string? name) => _value;
|
||||||
|
|
||||||
|
public IDisposable? OnChange(Action<AuditLogOptions, string?> listener) => null;
|
||||||
|
}
|
||||||
|
}
|
||||||
217
tests/ScadaLink.AuditLog.Tests/Payload/HeaderRedactionTests.cs
Normal file
217
tests/ScadaLink.AuditLog.Tests/Payload/HeaderRedactionTests.cs
Normal file
@@ -0,0 +1,217 @@
|
|||||||
|
using System.Text;
|
||||||
|
using System.Text.Json;
|
||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using ScadaLink.AuditLog.Configuration;
|
||||||
|
using ScadaLink.AuditLog.Payload;
|
||||||
|
using ScadaLink.Commons.Entities.Audit;
|
||||||
|
using ScadaLink.Commons.Types.Enums;
|
||||||
|
|
||||||
|
namespace ScadaLink.AuditLog.Tests.Payload;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Bundle B (M5-T3) tests for <see cref="DefaultAuditPayloadFilter"/> HTTP header
|
||||||
|
/// redaction. Redaction parses <see cref="AuditEvent.RequestSummary"/> /
|
||||||
|
/// <see cref="AuditEvent.ResponseSummary"/> as JSON of shape
|
||||||
|
/// <c>{"headers": {"name": "value", ...}, "body": "..."}</c>, replaces values
|
||||||
|
/// whose header NAME (case-insensitive) is in
|
||||||
|
/// <see cref="AuditLogOptions.HeaderRedactList"/> with <c>"<redacted>"</c>,
|
||||||
|
/// and re-serialises. Non-JSON inputs pass through unchanged (no-op for
|
||||||
|
/// emitters that have not yet adopted the convention). The stage runs BEFORE
|
||||||
|
/// truncation so the redaction marker survives the cap.
|
||||||
|
/// </summary>
|
||||||
|
public class HeaderRedactionTests
|
||||||
|
{
|
||||||
|
private static IOptionsMonitor<AuditLogOptions> Monitor(AuditLogOptions? opts = null) =>
|
||||||
|
new StaticMonitor(opts ?? new AuditLogOptions());
|
||||||
|
|
||||||
|
private static DefaultAuditPayloadFilter Filter(AuditLogOptions? opts = null) =>
|
||||||
|
new(Monitor(opts), NullLogger<DefaultAuditPayloadFilter>.Instance);
|
||||||
|
|
||||||
|
private static AuditEvent NewEvent(
|
||||||
|
AuditStatus status = AuditStatus.Delivered,
|
||||||
|
string? request = null,
|
||||||
|
string? response = null) => new()
|
||||||
|
{
|
||||||
|
EventId = Guid.NewGuid(),
|
||||||
|
OccurredAtUtc = DateTime.UtcNow,
|
||||||
|
Channel = AuditChannel.ApiOutbound,
|
||||||
|
Kind = AuditKind.ApiCall,
|
||||||
|
Status = status,
|
||||||
|
RequestSummary = request,
|
||||||
|
ResponseSummary = response,
|
||||||
|
};
|
||||||
|
|
||||||
|
private static string BuildSummary(IDictionary<string, string> headers, string body)
|
||||||
|
{
|
||||||
|
// Serialize via System.Text.Json so we get a representative shape.
|
||||||
|
return JsonSerializer.Serialize(new
|
||||||
|
{
|
||||||
|
headers = headers,
|
||||||
|
body = body,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IDictionary<string, JsonElement> ParseSummary(string? summary)
|
||||||
|
{
|
||||||
|
Assert.NotNull(summary);
|
||||||
|
using var doc = JsonDocument.Parse(summary!);
|
||||||
|
var dict = new Dictionary<string, JsonElement>();
|
||||||
|
foreach (var property in doc.RootElement.EnumerateObject())
|
||||||
|
{
|
||||||
|
dict[property.Name] = property.Value.Clone();
|
||||||
|
}
|
||||||
|
return dict;
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void HeaderRedaction_AuthorizationBearer_Redacted()
|
||||||
|
{
|
||||||
|
var headers = new Dictionary<string, string>
|
||||||
|
{
|
||||||
|
["Authorization"] = "Bearer secret-token-xyz",
|
||||||
|
["Content-Type"] = "application/json",
|
||||||
|
};
|
||||||
|
var input = BuildSummary(headers, "hello");
|
||||||
|
var evt = NewEvent(request: input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
var parsed = ParseSummary(result.RequestSummary);
|
||||||
|
var resultHeaders = parsed["headers"];
|
||||||
|
Assert.Equal("<redacted>", resultHeaders.GetProperty("Authorization").GetString());
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void HeaderRedaction_CaseInsensitive_LowercaseAuthorization_Redacted()
|
||||||
|
{
|
||||||
|
var headers = new Dictionary<string, string>
|
||||||
|
{
|
||||||
|
["authorization"] = "Bearer secret-token-xyz",
|
||||||
|
};
|
||||||
|
var input = BuildSummary(headers, "hello");
|
||||||
|
var evt = NewEvent(request: input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
var parsed = ParseSummary(result.RequestSummary);
|
||||||
|
var resultHeaders = parsed["headers"];
|
||||||
|
Assert.Equal("<redacted>", resultHeaders.GetProperty("authorization").GetString());
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void HeaderRedaction_CustomRedactList_RedactsCustomHeaderName()
|
||||||
|
{
|
||||||
|
var opts = new AuditLogOptions
|
||||||
|
{
|
||||||
|
HeaderRedactList = new List<string> { "X-Custom-Secret" },
|
||||||
|
};
|
||||||
|
var headers = new Dictionary<string, string>
|
||||||
|
{
|
||||||
|
["X-Custom-Secret"] = "topsecret",
|
||||||
|
["Authorization"] = "Bearer keep-me", // not in list anymore
|
||||||
|
};
|
||||||
|
var input = BuildSummary(headers, "hi");
|
||||||
|
var evt = NewEvent(request: input);
|
||||||
|
|
||||||
|
var result = Filter(opts).Apply(evt);
|
||||||
|
|
||||||
|
var parsed = ParseSummary(result.RequestSummary);
|
||||||
|
var resultHeaders = parsed["headers"];
|
||||||
|
Assert.Equal("<redacted>", resultHeaders.GetProperty("X-Custom-Secret").GetString());
|
||||||
|
// Authorization no longer listed -> preserved verbatim.
|
||||||
|
Assert.Equal("Bearer keep-me", resultHeaders.GetProperty("Authorization").GetString());
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void HeaderRedaction_NonJson_RequestSummary_Unchanged()
|
||||||
|
{
|
||||||
|
const string input = "this is not JSON at all";
|
||||||
|
var evt = NewEvent(request: input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
Assert.Equal(input, result.RequestSummary);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void HeaderRedaction_NoHeadersField_Unchanged()
|
||||||
|
{
|
||||||
|
var input = JsonSerializer.Serialize(new { body = "only a body, no headers" });
|
||||||
|
var evt = NewEvent(request: input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
// The stage may re-serialise but the content must be semantically identical.
|
||||||
|
var parsed = ParseSummary(result.RequestSummary);
|
||||||
|
Assert.Equal("only a body, no headers", parsed["body"].GetString());
|
||||||
|
Assert.False(parsed.ContainsKey("headers"));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void HeaderRedaction_Other_Headers_Preserved()
|
||||||
|
{
|
||||||
|
var headers = new Dictionary<string, string>
|
||||||
|
{
|
||||||
|
["Authorization"] = "Bearer secret",
|
||||||
|
["Content-Type"] = "application/json",
|
||||||
|
["X-Request-Id"] = "abc-123",
|
||||||
|
["Accept"] = "application/json",
|
||||||
|
};
|
||||||
|
var input = BuildSummary(headers, "payload");
|
||||||
|
var evt = NewEvent(request: input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
var parsed = ParseSummary(result.RequestSummary);
|
||||||
|
var resultHeaders = parsed["headers"];
|
||||||
|
Assert.Equal("<redacted>", resultHeaders.GetProperty("Authorization").GetString());
|
||||||
|
Assert.Equal("application/json", resultHeaders.GetProperty("Content-Type").GetString());
|
||||||
|
Assert.Equal("abc-123", resultHeaders.GetProperty("X-Request-Id").GetString());
|
||||||
|
Assert.Equal("application/json", resultHeaders.GetProperty("Accept").GetString());
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void HeaderRedaction_AppliedBeforeTruncation()
|
||||||
|
{
|
||||||
|
// Build a summary whose Authorization header value is enormous AND whose
|
||||||
|
// body padding pushes the total beyond the 8 KB cap. After redaction the
|
||||||
|
// Authorization value becomes "<redacted>" — then truncation caps the
|
||||||
|
// re-serialised string. Result must:
|
||||||
|
// * carry "<redacted>" (header redaction ran first),
|
||||||
|
// * NOT carry the original secret bytes (proves redaction won, not order swap),
|
||||||
|
// * be capped at the configured DefaultCapBytes,
|
||||||
|
// * have PayloadTruncated == true.
|
||||||
|
const string secret = "SUPER-SECRET-TOKEN-DO-NOT-LEAK";
|
||||||
|
var headers = new Dictionary<string, string>
|
||||||
|
{
|
||||||
|
["Authorization"] = "Bearer " + secret,
|
||||||
|
};
|
||||||
|
var body = new string('x', 9 * 1024);
|
||||||
|
var input = BuildSummary(headers, body);
|
||||||
|
Assert.True(Encoding.UTF8.GetByteCount(input) > 8192);
|
||||||
|
|
||||||
|
var evt = NewEvent(AuditStatus.Delivered, request: input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
Assert.NotNull(result.RequestSummary);
|
||||||
|
Assert.True(Encoding.UTF8.GetByteCount(result.RequestSummary!) <= 8192);
|
||||||
|
Assert.Contains("<redacted>", result.RequestSummary);
|
||||||
|
Assert.DoesNotContain(secret, result.RequestSummary);
|
||||||
|
Assert.True(result.PayloadTruncated);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// IOptionsMonitor test double — returns the same snapshot on every read,
|
||||||
|
/// no change-token plumbing required for these tests.
|
||||||
|
/// </summary>
|
||||||
|
private sealed class StaticMonitor : IOptionsMonitor<AuditLogOptions>
|
||||||
|
{
|
||||||
|
private readonly AuditLogOptions _value;
|
||||||
|
public StaticMonitor(AuditLogOptions value) => _value = value;
|
||||||
|
public AuditLogOptions CurrentValue => _value;
|
||||||
|
public AuditLogOptions Get(string? name) => _value;
|
||||||
|
public IDisposable? OnChange(Action<AuditLogOptions, string?> listener) => null;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
using System.Linq;
|
||||||
|
using System.Reflection;
|
||||||
|
using ScadaLink.AuditLog.Payload;
|
||||||
|
using ScadaLink.Commons.Entities.Audit;
|
||||||
|
|
||||||
|
namespace ScadaLink.AuditLog.Tests.Payload;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Bundle A (M5-T1) contract test for <see cref="IAuditPayloadFilter"/>. The
|
||||||
|
/// interface is the seam between event construction and writer persistence;
|
||||||
|
/// later bundles register the production implementation as a singleton and
|
||||||
|
/// invoke it from the site/central writer paths. We pin the surface area here
|
||||||
|
/// via reflection so accidental signature drift breaks the build before the
|
||||||
|
/// downstream wiring goes red.
|
||||||
|
/// </summary>
|
||||||
|
public class PayloadFilterContractTests
|
||||||
|
{
|
||||||
|
[Fact]
|
||||||
|
public void Interface_Exists_InPayloadNamespace()
|
||||||
|
{
|
||||||
|
var type = typeof(IAuditPayloadFilter);
|
||||||
|
|
||||||
|
Assert.True(type.IsInterface, "IAuditPayloadFilter must be an interface");
|
||||||
|
Assert.Equal("ScadaLink.AuditLog.Payload", type.Namespace);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Apply_Method_HasDocumentedSignature()
|
||||||
|
{
|
||||||
|
var type = typeof(IAuditPayloadFilter);
|
||||||
|
|
||||||
|
var method = type.GetMethod(
|
||||||
|
"Apply",
|
||||||
|
BindingFlags.Instance | BindingFlags.Public,
|
||||||
|
binder: null,
|
||||||
|
types: new[] { typeof(AuditEvent) },
|
||||||
|
modifiers: null);
|
||||||
|
|
||||||
|
Assert.NotNull(method);
|
||||||
|
Assert.Equal(typeof(AuditEvent), method!.ReturnType);
|
||||||
|
|
||||||
|
var parameters = method.GetParameters();
|
||||||
|
Assert.Single(parameters);
|
||||||
|
Assert.Equal("rawEvent", parameters[0].Name);
|
||||||
|
Assert.Equal(typeof(AuditEvent), parameters[0].ParameterType);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Interface_DeclaresExactlyOneMethod()
|
||||||
|
{
|
||||||
|
var type = typeof(IAuditPayloadFilter);
|
||||||
|
var methods = type.GetMethods(BindingFlags.Instance | BindingFlags.Public)
|
||||||
|
.Where(m => !m.IsSpecialName)
|
||||||
|
.ToArray();
|
||||||
|
|
||||||
|
Assert.Single(methods);
|
||||||
|
Assert.Equal("Apply", methods[0].Name);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,270 @@
|
|||||||
|
using Microsoft.Extensions.Logging;
|
||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using ScadaLink.AuditLog.Configuration;
|
||||||
|
using ScadaLink.AuditLog.Payload;
|
||||||
|
using ScadaLink.Commons.Entities.Audit;
|
||||||
|
using ScadaLink.Commons.Types.Enums;
|
||||||
|
|
||||||
|
namespace ScadaLink.AuditLog.Tests.Payload;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Bundle D (M5-T10) safety-net edge cases for
|
||||||
|
/// <see cref="DefaultAuditPayloadFilter"/>. Bundle B already pinned the
|
||||||
|
/// happy-path safety net (catastrophic-backtracking timeout →
|
||||||
|
/// <c><redacted: redactor error></c> + counter bump); this fixture covers
|
||||||
|
/// the pathological / config-mistake corners that production operators will
|
||||||
|
/// hit when typoing a regex or shipping a half-baked redactor list.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <para>
|
||||||
|
/// The invariants under test:
|
||||||
|
/// </para>
|
||||||
|
/// <list type="bullet">
|
||||||
|
/// <item>An UNCOMPILABLE pattern (e.g. <c>[unclosed</c>) is logged at warning
|
||||||
|
/// on first encounter and cached as invalid so it never throws again,
|
||||||
|
/// but the redactor-failure COUNTER is not bumped at bind time —
|
||||||
|
/// per the contract on <see cref="IAuditRedactionFailureCounter"/>
|
||||||
|
/// the counter tracks RUNTIME redaction failures only.</item>
|
||||||
|
/// <item>One throwing regex in the middle of a list does NOT poison the
|
||||||
|
/// other patterns — the filter stops at the failing pattern,
|
||||||
|
/// over-redacts the offending field, but lets every other field keep
|
||||||
|
/// the prior cleanly-redacted state and lets the rest of the writer
|
||||||
|
/// pipeline run.</item>
|
||||||
|
/// <item>A live config change that introduces a broken pattern does not
|
||||||
|
/// crash the filter — the bad pattern is silently dropped (logged once)
|
||||||
|
/// and the still-valid patterns continue to redact normally.</item>
|
||||||
|
/// </list>
|
||||||
|
/// </remarks>
|
||||||
|
public class RedactionSafetyNetTests
|
||||||
|
{
|
||||||
|
private static IOptionsMonitor<AuditLogOptions> Monitor(AuditLogOptions? opts = null) =>
|
||||||
|
new StaticMonitor(opts ?? new AuditLogOptions());
|
||||||
|
|
||||||
|
private static AuditEvent NewEvent(
|
||||||
|
AuditStatus status = AuditStatus.Delivered,
|
||||||
|
string? request = null,
|
||||||
|
string? response = null,
|
||||||
|
string? errorDetail = null,
|
||||||
|
string? extra = null,
|
||||||
|
string? target = null) => new()
|
||||||
|
{
|
||||||
|
EventId = Guid.NewGuid(),
|
||||||
|
OccurredAtUtc = DateTime.UtcNow,
|
||||||
|
Channel = AuditChannel.ApiOutbound,
|
||||||
|
Kind = AuditKind.ApiCall,
|
||||||
|
Status = status,
|
||||||
|
Target = target,
|
||||||
|
RequestSummary = request,
|
||||||
|
ResponseSummary = response,
|
||||||
|
ErrorDetail = errorDetail,
|
||||||
|
Extra = extra,
|
||||||
|
};
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void RegexNotCompilable_AtBindTime_LoggedAndSkipped()
|
||||||
|
{
|
||||||
|
// `[unclosed` is a structurally invalid character class — the .NET
|
||||||
|
// regex engine throws ArgumentException at compile time. We assert:
|
||||||
|
// * the filter does NOT throw,
|
||||||
|
// * the OTHER (valid) pattern still redacts hunter2,
|
||||||
|
// * the failure counter is NOT incremented at compile time
|
||||||
|
// (it tracks runtime redaction failures only),
|
||||||
|
// * a warning is logged exactly once.
|
||||||
|
const string badPattern = "[unclosed";
|
||||||
|
const string goodPattern = "\"password\":\\s*\"[^\"]*\"";
|
||||||
|
var opts = new AuditLogOptions
|
||||||
|
{
|
||||||
|
GlobalBodyRedactors = new List<string> { badPattern, goodPattern },
|
||||||
|
};
|
||||||
|
var counter = new CountingRedactionFailureCounter();
|
||||||
|
var spy = new SpyLogger<DefaultAuditPayloadFilter>();
|
||||||
|
var filter = new DefaultAuditPayloadFilter(Monitor(opts), spy, counter);
|
||||||
|
|
||||||
|
var evt = NewEvent(request: "{\"user\":\"alice\",\"password\":\"hunter2\"}");
|
||||||
|
|
||||||
|
var result = filter.Apply(evt);
|
||||||
|
|
||||||
|
Assert.NotNull(result.RequestSummary);
|
||||||
|
Assert.DoesNotContain("hunter2", result.RequestSummary);
|
||||||
|
Assert.Contains("<redacted>", result.RequestSummary);
|
||||||
|
Assert.Equal(0, counter.Count);
|
||||||
|
// Apply twice — the invalid-pattern compile must run AT MOST once;
|
||||||
|
// the sentinel-cache entry stops repeat compile attempts.
|
||||||
|
_ = filter.Apply(evt);
|
||||||
|
var badPatternWarnings = spy.Entries
|
||||||
|
.Where(e => e.Level == LogLevel.Warning && e.Message.Contains(badPattern))
|
||||||
|
.Count();
|
||||||
|
Assert.Equal(1, badPatternWarnings);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void MultipleRedactors_OneThrows_OthersStillApply_ToOtherFields()
|
||||||
|
{
|
||||||
|
// Pattern set: [valid-A, evil, valid-B]. The evil pattern is
|
||||||
|
// catastrophic-backtracking on the RequestSummary input (all-'a's +
|
||||||
|
// mismatching suffix) — that field is over-redacted with the error
|
||||||
|
// marker as soon as evil throws. ResponseSummary is processed
|
||||||
|
// INDEPENDENTLY; its input does not trigger evil's backtracking, so
|
||||||
|
// valid-A and valid-B both still apply on that field. This proves a
|
||||||
|
// per-field redactor failure does not poison the rest of the writer
|
||||||
|
// call (the SQL-param stage, the truncation stage, and the other
|
||||||
|
// fields all continue normally).
|
||||||
|
const string validA = "SECRET-[A-Z0-9]+";
|
||||||
|
const string evil = "^(a+)+$"; // catastrophic on long all-'a' string
|
||||||
|
const string validB = "PIN-\\d{4}";
|
||||||
|
var opts = new AuditLogOptions
|
||||||
|
{
|
||||||
|
GlobalBodyRedactors = new List<string> { validA, evil, validB },
|
||||||
|
};
|
||||||
|
var counter = new CountingRedactionFailureCounter();
|
||||||
|
var filter = new DefaultAuditPayloadFilter(
|
||||||
|
Monitor(opts),
|
||||||
|
NullLogger<DefaultAuditPayloadFilter>.Instance,
|
||||||
|
counter);
|
||||||
|
|
||||||
|
// Request: ALL 'a's + a non-'a' suffix character. valid-A does not
|
||||||
|
// match (no SECRET-X prefix), so the buffer reaches `evil` untouched
|
||||||
|
// and triggers the backtracking explosion.
|
||||||
|
var request = new string('a', 30) + "!";
|
||||||
|
// Response: short, mismatches the evil pattern cleanly (no
|
||||||
|
// backtracking), so both valid-A and valid-B run and redact.
|
||||||
|
const string response = "SECRET-ABC456 PIN-9999 other-text";
|
||||||
|
|
||||||
|
var result = filter.Apply(NewEvent(request: request, response: response));
|
||||||
|
|
||||||
|
// RequestSummary: over-redacted (evil pattern threw).
|
||||||
|
Assert.Equal("<redacted: redactor error>", result.RequestSummary);
|
||||||
|
Assert.True(counter.Count >= 1, $"expected counter >= 1, got {counter.Count}");
|
||||||
|
|
||||||
|
// ResponseSummary: clean — both valid regexes still applied; the evil
|
||||||
|
// one ran without throwing on this short input.
|
||||||
|
Assert.NotNull(result.ResponseSummary);
|
||||||
|
Assert.DoesNotContain("SECRET-ABC456", result.ResponseSummary);
|
||||||
|
Assert.DoesNotContain("PIN-9999", result.ResponseSummary);
|
||||||
|
Assert.Contains("<redacted>", result.ResponseSummary);
|
||||||
|
Assert.Contains("other-text", result.ResponseSummary);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Edge case 3 (RedactorReturnsNonStringExceptionType) intentionally
|
||||||
|
// skipped — the brief permits dropping it: there is no portable way to
|
||||||
|
// artificially trigger an OutOfMemoryException inside System.Text.RegularExpressions
|
||||||
|
// from a unit test without writing native interop, and the existing
|
||||||
|
// per-stage try/catch already covers Exception (which OOM and similar
|
||||||
|
// would derive from). Bundle B's RegexThrowsTimeout coverage exercises
|
||||||
|
// the same catch path with a deterministic trigger.
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void ConfigChange_WithBadRegex_LiveTrafficKeepsApplyingValidRegexes()
|
||||||
|
{
|
||||||
|
// Initial config: one valid global redactor — hunter2 is redacted.
|
||||||
|
// Reload: ADD a malformed pattern alongside the original. Per the
|
||||||
|
// safety contract, the bad pattern is logged + skipped, the original
|
||||||
|
// valid pattern keeps redacting, and the filter NEVER throws on the
|
||||||
|
// hot path. The counter must not be bumped at reload time (the
|
||||||
|
// CompiledRegex sentinel covers the bind error before runtime even
|
||||||
|
// sees it).
|
||||||
|
var monitor = new MutableMonitor(new AuditLogOptions
|
||||||
|
{
|
||||||
|
GlobalBodyRedactors = new List<string> { "\"password\":\\s*\"[^\"]*\"" },
|
||||||
|
});
|
||||||
|
var counter = new CountingRedactionFailureCounter();
|
||||||
|
var spy = new SpyLogger<DefaultAuditPayloadFilter>();
|
||||||
|
var filter = new DefaultAuditPayloadFilter(monitor, spy, counter);
|
||||||
|
|
||||||
|
var evt = NewEvent(request: "{\"user\":\"alice\",\"password\":\"hunter2\"}");
|
||||||
|
|
||||||
|
var before = filter.Apply(evt);
|
||||||
|
Assert.DoesNotContain("hunter2", before.RequestSummary!);
|
||||||
|
|
||||||
|
// Reload: malformed pattern added to the list.
|
||||||
|
monitor.Set(new AuditLogOptions
|
||||||
|
{
|
||||||
|
GlobalBodyRedactors = new List<string>
|
||||||
|
{
|
||||||
|
"\"password\":\\s*\"[^\"]*\"",
|
||||||
|
"[unclosed",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
var after = filter.Apply(evt);
|
||||||
|
Assert.NotNull(after.RequestSummary);
|
||||||
|
Assert.DoesNotContain("hunter2", after.RequestSummary);
|
||||||
|
Assert.Contains("<redacted>", after.RequestSummary);
|
||||||
|
Assert.Equal(0, counter.Count);
|
||||||
|
// Compile-time warning logged for the broken pattern.
|
||||||
|
Assert.Contains(
|
||||||
|
spy.Entries,
|
||||||
|
e => e.Level == LogLevel.Warning && e.Message.Contains("[unclosed"));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Counts <see cref="IAuditRedactionFailureCounter.Increment"/> calls.</summary>
|
||||||
|
private sealed class CountingRedactionFailureCounter : IAuditRedactionFailureCounter
|
||||||
|
{
|
||||||
|
private int _count;
|
||||||
|
public int Count => _count;
|
||||||
|
public void Increment() => System.Threading.Interlocked.Increment(ref _count);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>IOptionsMonitor test double — returns the same snapshot on every read.</summary>
|
||||||
|
private sealed class StaticMonitor : IOptionsMonitor<AuditLogOptions>
|
||||||
|
{
|
||||||
|
private readonly AuditLogOptions _value;
|
||||||
|
public StaticMonitor(AuditLogOptions value) => _value = value;
|
||||||
|
public AuditLogOptions CurrentValue => _value;
|
||||||
|
public AuditLogOptions Get(string? name) => _value;
|
||||||
|
public IDisposable? OnChange(Action<AuditLogOptions, string?> listener) => null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// IOptionsMonitor test double that supports a live <see cref="Set"/> —
|
||||||
|
/// mirrors the helper used in
|
||||||
|
/// <see cref="ScadaLink.AuditLog.Tests.Configuration.AuditLogOptionsBindingTests"/>;
|
||||||
|
/// kept private here so the safety-net test file remains self-contained.
|
||||||
|
/// </summary>
|
||||||
|
private sealed class MutableMonitor : IOptionsMonitor<AuditLogOptions>
|
||||||
|
{
|
||||||
|
private AuditLogOptions _current;
|
||||||
|
public MutableMonitor(AuditLogOptions initial) => _current = initial;
|
||||||
|
public AuditLogOptions CurrentValue => _current;
|
||||||
|
public AuditLogOptions Get(string? name) => _current;
|
||||||
|
public IDisposable? OnChange(Action<AuditLogOptions, string?> listener) => null;
|
||||||
|
public void Set(AuditLogOptions value) => _current = value;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Minimal ILogger that records each formatted log line so tests can
|
||||||
|
/// assert on the compile-time warning emission contract — counting
|
||||||
|
/// warnings and grepping the message text.
|
||||||
|
/// </summary>
|
||||||
|
private sealed class SpyLogger<T> : ILogger<T>
|
||||||
|
{
|
||||||
|
private readonly List<LogEntry> _entries = new();
|
||||||
|
|
||||||
|
public IReadOnlyList<LogEntry> Entries
|
||||||
|
{
|
||||||
|
get { lock (_entries) return _entries.ToArray(); }
|
||||||
|
}
|
||||||
|
|
||||||
|
public IDisposable BeginScope<TState>(TState state) where TState : notnull => NullScope.Instance;
|
||||||
|
public bool IsEnabled(LogLevel logLevel) => true;
|
||||||
|
public void Log<TState>(
|
||||||
|
LogLevel logLevel,
|
||||||
|
EventId eventId,
|
||||||
|
TState state,
|
||||||
|
Exception? exception,
|
||||||
|
Func<TState, Exception?, string> formatter)
|
||||||
|
{
|
||||||
|
var msg = formatter(state, exception);
|
||||||
|
lock (_entries) _entries.Add(new LogEntry(logLevel, msg));
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class NullScope : IDisposable
|
||||||
|
{
|
||||||
|
public static readonly NullScope Instance = new();
|
||||||
|
public void Dispose() { }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public sealed record LogEntry(LogLevel Level, string Message);
|
||||||
|
}
|
||||||
212
tests/ScadaLink.AuditLog.Tests/Payload/SqlParamRedactionTests.cs
Normal file
212
tests/ScadaLink.AuditLog.Tests/Payload/SqlParamRedactionTests.cs
Normal file
@@ -0,0 +1,212 @@
|
|||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using ScadaLink.AuditLog.Configuration;
|
||||||
|
using ScadaLink.AuditLog.Payload;
|
||||||
|
using ScadaLink.Commons.Entities.Audit;
|
||||||
|
using ScadaLink.Commons.Types.Enums;
|
||||||
|
|
||||||
|
namespace ScadaLink.AuditLog.Tests.Payload;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Bundle B (M5-T5) tests for SQL parameter redaction in
|
||||||
|
/// <see cref="DefaultAuditPayloadFilter"/>. M4 Bundle A's
|
||||||
|
/// <c>AuditingDbCommand</c> emits <c>RequestSummary</c> as
|
||||||
|
/// <c>{"sql":"...","parameters":{"@name":"value", ...}}</c>; the SQL-parameter
|
||||||
|
/// redactor parses this shape on
|
||||||
|
/// <see cref="AuditChannel.DbOutbound"/> rows, replaces values whose key
|
||||||
|
/// matches the configured case-insensitive regex with <c><redacted></c>,
|
||||||
|
/// and re-serialises. Default behaviour with no opt-in: parameter values are
|
||||||
|
/// captured verbatim. Connection lookup uses the connection-name prefix of
|
||||||
|
/// <see cref="AuditEvent.Target"/> (everything before the first <c>.</c>) so
|
||||||
|
/// the same per-connection regex applies regardless of the SQL-snippet suffix
|
||||||
|
/// that <c>AuditingDbCommand</c> appends to disambiguate rows.
|
||||||
|
/// </summary>
|
||||||
|
public class SqlParamRedactionTests
|
||||||
|
{
|
||||||
|
private static IOptionsMonitor<AuditLogOptions> Monitor(AuditLogOptions? opts = null) =>
|
||||||
|
new StaticMonitor(opts ?? new AuditLogOptions());
|
||||||
|
|
||||||
|
private static DefaultAuditPayloadFilter Filter(AuditLogOptions? opts = null) =>
|
||||||
|
new(Monitor(opts), NullLogger<DefaultAuditPayloadFilter>.Instance);
|
||||||
|
|
||||||
|
private static AuditEvent NewDbEvent(string target, string requestSummary) => new()
|
||||||
|
{
|
||||||
|
EventId = Guid.NewGuid(),
|
||||||
|
OccurredAtUtc = DateTime.UtcNow,
|
||||||
|
Channel = AuditChannel.DbOutbound,
|
||||||
|
Kind = AuditKind.DbWrite,
|
||||||
|
Status = AuditStatus.Delivered,
|
||||||
|
Target = target,
|
||||||
|
RequestSummary = requestSummary,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Build a RequestSummary in the exact shape M4's <c>AuditingDbCommand</c>
|
||||||
|
/// emits — hand-rolled JSON with <c>"sql"</c> + <c>"parameters"</c> keys.
|
||||||
|
/// Tests depend on this format; if AuditingDbCommand ever changes, this
|
||||||
|
/// helper updates in lockstep.
|
||||||
|
/// </summary>
|
||||||
|
private static string DbRequestSummary(string sql, params (string name, string value)[] parameters)
|
||||||
|
{
|
||||||
|
var sb = new System.Text.StringBuilder();
|
||||||
|
sb.Append("{\"sql\":\"").Append(sql).Append('"');
|
||||||
|
if (parameters.Length > 0)
|
||||||
|
{
|
||||||
|
sb.Append(",\"parameters\":{");
|
||||||
|
for (var i = 0; i < parameters.Length; i++)
|
||||||
|
{
|
||||||
|
if (i > 0) sb.Append(',');
|
||||||
|
sb.Append('"').Append(parameters[i].name).Append("\":\"")
|
||||||
|
.Append(parameters[i].value).Append('"');
|
||||||
|
}
|
||||||
|
sb.Append('}');
|
||||||
|
}
|
||||||
|
sb.Append('}');
|
||||||
|
return sb.ToString();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void NoOptIn_ParamsVerbatim_Unchanged()
|
||||||
|
{
|
||||||
|
var input = DbRequestSummary(
|
||||||
|
"INSERT INTO Users (Name, Token) VALUES (@name, @token)",
|
||||||
|
("@name", "Alice"),
|
||||||
|
("@token", "secret-xyz"));
|
||||||
|
var evt = NewDbEvent("PrimaryDb.INSERT INTO Users", input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
Assert.Equal(input, result.RequestSummary);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void OptInRegex_AtToken_OrAtApikey_RedactsThoseValues_KeepsOthers()
|
||||||
|
{
|
||||||
|
var opts = new AuditLogOptions
|
||||||
|
{
|
||||||
|
PerTargetOverrides = new Dictionary<string, PerTargetRedactionOverride>
|
||||||
|
{
|
||||||
|
["PrimaryDb"] = new PerTargetRedactionOverride
|
||||||
|
{
|
||||||
|
RedactSqlParamsMatching = "^@(token|apikey)$",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
var input = DbRequestSummary(
|
||||||
|
"INSERT INTO Users (Name, Token, ApiKey) VALUES (@name, @token, @apikey)",
|
||||||
|
("@name", "Alice"),
|
||||||
|
("@token", "secret-xyz"),
|
||||||
|
("@apikey", "k-987"));
|
||||||
|
var evt = NewDbEvent("PrimaryDb.INSERT INTO Users", input);
|
||||||
|
|
||||||
|
var result = Filter(opts).Apply(evt);
|
||||||
|
|
||||||
|
Assert.NotNull(result.RequestSummary);
|
||||||
|
Assert.Contains("\"@name\":\"Alice\"", result.RequestSummary);
|
||||||
|
Assert.Contains("\"@token\":\"<redacted>\"", result.RequestSummary);
|
||||||
|
Assert.Contains("\"@apikey\":\"<redacted>\"", result.RequestSummary);
|
||||||
|
Assert.DoesNotContain("secret-xyz", result.RequestSummary);
|
||||||
|
Assert.DoesNotContain("k-987", result.RequestSummary);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void RegexCaseInsensitive_MatchesParamNames()
|
||||||
|
{
|
||||||
|
var opts = new AuditLogOptions
|
||||||
|
{
|
||||||
|
PerTargetOverrides = new Dictionary<string, PerTargetRedactionOverride>
|
||||||
|
{
|
||||||
|
["PrimaryDb"] = new PerTargetRedactionOverride
|
||||||
|
{
|
||||||
|
RedactSqlParamsMatching = "token",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
var input = DbRequestSummary(
|
||||||
|
"UPDATE x SET Token = @TOKEN",
|
||||||
|
("@TOKEN", "uppercased-secret"));
|
||||||
|
var evt = NewDbEvent("PrimaryDb.UPDATE x SET Token", input);
|
||||||
|
|
||||||
|
var result = Filter(opts).Apply(evt);
|
||||||
|
|
||||||
|
Assert.NotNull(result.RequestSummary);
|
||||||
|
Assert.Contains("\"@TOKEN\":\"<redacted>\"", result.RequestSummary);
|
||||||
|
Assert.DoesNotContain("uppercased-secret", result.RequestSummary);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void NonDbOutboundChannel_NotAffected()
|
||||||
|
{
|
||||||
|
// ApiOutbound row whose RequestSummary happens to look like the
|
||||||
|
// DbOutbound JSON shape (worst-case false positive). The SQL
|
||||||
|
// redactor must NOT touch it — channel guards the stage.
|
||||||
|
var opts = new AuditLogOptions
|
||||||
|
{
|
||||||
|
PerTargetOverrides = new Dictionary<string, PerTargetRedactionOverride>
|
||||||
|
{
|
||||||
|
["PrimaryDb"] = new PerTargetRedactionOverride
|
||||||
|
{
|
||||||
|
RedactSqlParamsMatching = "^@token$",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
var input = DbRequestSummary(
|
||||||
|
"SELECT @token",
|
||||||
|
("@token", "should-survive"));
|
||||||
|
var evt = new AuditEvent
|
||||||
|
{
|
||||||
|
EventId = Guid.NewGuid(),
|
||||||
|
OccurredAtUtc = DateTime.UtcNow,
|
||||||
|
Channel = AuditChannel.ApiOutbound,
|
||||||
|
Kind = AuditKind.ApiCall,
|
||||||
|
Status = AuditStatus.Delivered,
|
||||||
|
Target = "PrimaryDb.SELECT", // doesn't matter — channel guards
|
||||||
|
RequestSummary = input,
|
||||||
|
};
|
||||||
|
|
||||||
|
var result = Filter(opts).Apply(evt);
|
||||||
|
|
||||||
|
Assert.Equal(input, result.RequestSummary);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void PerTargetSetting_MatchesByTarget()
|
||||||
|
{
|
||||||
|
// Two connections — A is configured to redact tokens, B is not. Same
|
||||||
|
// payload through each must yield different results.
|
||||||
|
var opts = new AuditLogOptions
|
||||||
|
{
|
||||||
|
PerTargetOverrides = new Dictionary<string, PerTargetRedactionOverride>
|
||||||
|
{
|
||||||
|
["ConnA"] = new PerTargetRedactionOverride
|
||||||
|
{
|
||||||
|
RedactSqlParamsMatching = "^@token$",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
var input = DbRequestSummary(
|
||||||
|
"SELECT @token",
|
||||||
|
("@token", "the-secret"));
|
||||||
|
|
||||||
|
var aEvt = NewDbEvent("ConnA.SELECT @token", input);
|
||||||
|
var bEvt = NewDbEvent("ConnB.SELECT @token", input);
|
||||||
|
|
||||||
|
var aResult = Filter(opts).Apply(aEvt);
|
||||||
|
var bResult = Filter(opts).Apply(bEvt);
|
||||||
|
|
||||||
|
Assert.Contains("<redacted>", aResult.RequestSummary!);
|
||||||
|
Assert.DoesNotContain("the-secret", aResult.RequestSummary!);
|
||||||
|
|
||||||
|
Assert.Equal(input, bResult.RequestSummary);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>IOptionsMonitor test double.</summary>
|
||||||
|
private sealed class StaticMonitor : IOptionsMonitor<AuditLogOptions>
|
||||||
|
{
|
||||||
|
private readonly AuditLogOptions _value;
|
||||||
|
public StaticMonitor(AuditLogOptions value) => _value = value;
|
||||||
|
public AuditLogOptions CurrentValue => _value;
|
||||||
|
public AuditLogOptions Get(string? name) => _value;
|
||||||
|
public IDisposable? OnChange(Action<AuditLogOptions, string?> listener) => null;
|
||||||
|
}
|
||||||
|
}
|
||||||
226
tests/ScadaLink.AuditLog.Tests/Payload/TruncationTests.cs
Normal file
226
tests/ScadaLink.AuditLog.Tests/Payload/TruncationTests.cs
Normal file
@@ -0,0 +1,226 @@
|
|||||||
|
using System.Linq;
|
||||||
|
using System.Text;
|
||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using ScadaLink.AuditLog.Configuration;
|
||||||
|
using ScadaLink.AuditLog.Payload;
|
||||||
|
using ScadaLink.Commons.Entities.Audit;
|
||||||
|
using ScadaLink.Commons.Types.Enums;
|
||||||
|
|
||||||
|
namespace ScadaLink.AuditLog.Tests.Payload;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Bundle A (M5-T2) tests for <see cref="DefaultAuditPayloadFilter"/> truncation.
|
||||||
|
/// The filter caps RequestSummary / ResponseSummary / ErrorDetail / Extra at
|
||||||
|
/// <see cref="AuditLogOptions.DefaultCapBytes"/> (8 KiB) on success rows and
|
||||||
|
/// <see cref="AuditLogOptions.ErrorCapBytes"/> (64 KiB) on error rows. "Error
|
||||||
|
/// row" = <see cref="AuditEvent.Status"/> NOT IN (<c>Delivered</c>,
|
||||||
|
/// <c>Submitted</c>, <c>Forwarded</c>). Truncation must respect UTF-8 character
|
||||||
|
/// boundaries (never split a multi-byte sequence mid-character) and must set
|
||||||
|
/// <see cref="AuditEvent.PayloadTruncated"/> true when any field is shortened.
|
||||||
|
/// </summary>
|
||||||
|
public class TruncationTests
|
||||||
|
{
|
||||||
|
private static IOptionsMonitor<AuditLogOptions> Monitor(AuditLogOptions? opts = null)
|
||||||
|
{
|
||||||
|
var snapshot = opts ?? new AuditLogOptions();
|
||||||
|
return new StaticMonitor(snapshot);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static DefaultAuditPayloadFilter Filter(AuditLogOptions? opts = null) =>
|
||||||
|
new(Monitor(opts), NullLogger<DefaultAuditPayloadFilter>.Instance);
|
||||||
|
|
||||||
|
private static AuditEvent NewEvent(
|
||||||
|
AuditStatus status = AuditStatus.Delivered,
|
||||||
|
string? request = null,
|
||||||
|
string? response = null,
|
||||||
|
string? errorDetail = null,
|
||||||
|
string? extra = null,
|
||||||
|
bool payloadTruncated = false) => new()
|
||||||
|
{
|
||||||
|
EventId = Guid.NewGuid(),
|
||||||
|
OccurredAtUtc = DateTime.UtcNow,
|
||||||
|
Channel = AuditChannel.ApiOutbound,
|
||||||
|
Kind = AuditKind.ApiCall,
|
||||||
|
Status = status,
|
||||||
|
RequestSummary = request,
|
||||||
|
ResponseSummary = response,
|
||||||
|
ErrorDetail = errorDetail,
|
||||||
|
Extra = extra,
|
||||||
|
PayloadTruncated = payloadTruncated,
|
||||||
|
};
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void SuccessRow_10KB_RequestSummary_TruncatedTo8KB_PayloadTruncatedTrue()
|
||||||
|
{
|
||||||
|
var input = new string('a', 10 * 1024);
|
||||||
|
var evt = NewEvent(AuditStatus.Delivered, request: input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
Assert.NotNull(result.RequestSummary);
|
||||||
|
Assert.Equal(8192, Encoding.UTF8.GetByteCount(result.RequestSummary!));
|
||||||
|
Assert.True(result.PayloadTruncated);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void ErrorRow_10KB_RequestSummary_NotTruncated_UnderErrorCap()
|
||||||
|
{
|
||||||
|
var input = new string('b', 10 * 1024);
|
||||||
|
var evt = NewEvent(AuditStatus.Failed, request: input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
Assert.Equal(input, result.RequestSummary);
|
||||||
|
Assert.False(result.PayloadTruncated);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void ErrorRow_70KB_RequestSummary_TruncatedTo64KB_PayloadTruncatedTrue()
|
||||||
|
{
|
||||||
|
var input = new string('c', 70 * 1024);
|
||||||
|
var evt = NewEvent(AuditStatus.Failed, request: input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
Assert.NotNull(result.RequestSummary);
|
||||||
|
Assert.Equal(65536, Encoding.UTF8.GetByteCount(result.RequestSummary!));
|
||||||
|
Assert.True(result.PayloadTruncated);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Multibyte_UTF8_TruncatedAtCharacterBoundary_NotMidByte()
|
||||||
|
{
|
||||||
|
// U+1F600 (grinning face) encodes to 4 UTF-8 bytes; 2000 of them = 8000 bytes,
|
||||||
|
// safely under the 8192 default cap so the boundary scan kicks in mid-character
|
||||||
|
// when we push past it. Pad with a few extra emoji so the *input* is > 8192 bytes
|
||||||
|
// and forces truncation.
|
||||||
|
var emoji = "😀"; // surrogate pair => one code point => 4 UTF-8 bytes
|
||||||
|
var sb = new StringBuilder();
|
||||||
|
for (int i = 0; i < 2100; i++)
|
||||||
|
{
|
||||||
|
sb.Append(emoji);
|
||||||
|
}
|
||||||
|
var input = sb.ToString();
|
||||||
|
Assert.True(Encoding.UTF8.GetByteCount(input) > 8192);
|
||||||
|
|
||||||
|
var evt = NewEvent(AuditStatus.Delivered, request: input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
Assert.NotNull(result.RequestSummary);
|
||||||
|
var resultBytes = Encoding.UTF8.GetByteCount(result.RequestSummary!);
|
||||||
|
Assert.True(resultBytes <= 8192, $"expected <= 8192 bytes, got {resultBytes}");
|
||||||
|
// 4-byte emoji boundary: the kept byte length must be a multiple of 4.
|
||||||
|
Assert.Equal(0, resultBytes % 4);
|
||||||
|
// And round-tripping the result must not introduce a U+FFFD replacement char.
|
||||||
|
Assert.DoesNotContain('<27>', result.RequestSummary);
|
||||||
|
Assert.True(result.PayloadTruncated);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void NullSummary_PassesThrough_AsNull()
|
||||||
|
{
|
||||||
|
var evt = NewEvent(AuditStatus.Delivered, request: null, response: null, errorDetail: null, extra: null);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
Assert.Null(result.RequestSummary);
|
||||||
|
Assert.Null(result.ResponseSummary);
|
||||||
|
Assert.Null(result.ErrorDetail);
|
||||||
|
Assert.Null(result.Extra);
|
||||||
|
Assert.False(result.PayloadTruncated);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void RawEventAlreadyTruncated_PayloadTruncatedRemainsTrue()
|
||||||
|
{
|
||||||
|
// Small payload that requires no truncation, but the caller already
|
||||||
|
// flagged PayloadTruncated upstream — the filter must not clear it.
|
||||||
|
var evt = NewEvent(AuditStatus.Delivered, request: "small", payloadTruncated: true);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
Assert.Equal("small", result.RequestSummary);
|
||||||
|
Assert.True(result.PayloadTruncated);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void StatusAttempted_TreatedAsError_UsesErrorCap()
|
||||||
|
{
|
||||||
|
// 10 KB is under the 64 KB error cap; if Attempted were a success status
|
||||||
|
// the value would be truncated to 8 KB. We assert it is NOT truncated.
|
||||||
|
var input = new string('d', 10 * 1024);
|
||||||
|
var evt = NewEvent(AuditStatus.Attempted, request: input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
Assert.Equal(input, result.RequestSummary);
|
||||||
|
Assert.False(result.PayloadTruncated);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void StatusParked_TreatedAsError_UsesErrorCap()
|
||||||
|
{
|
||||||
|
var input = new string('e', 10 * 1024);
|
||||||
|
var evt = NewEvent(AuditStatus.Parked, request: input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
Assert.Equal(input, result.RequestSummary);
|
||||||
|
Assert.False(result.PayloadTruncated);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void StatusSkipped_TreatedAsError_UsesErrorCap()
|
||||||
|
{
|
||||||
|
var input = new string('f', 10 * 1024);
|
||||||
|
var evt = NewEvent(AuditStatus.Skipped, request: input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
Assert.Equal(input, result.RequestSummary);
|
||||||
|
Assert.False(result.PayloadTruncated);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void ErrorDetail_AndExtra_Truncated_Independently()
|
||||||
|
{
|
||||||
|
// Each field is capped on its own — a 10 KB RequestSummary and a 10 KB
|
||||||
|
// ErrorDetail on the same Delivered row should both be cut to 8 KB and
|
||||||
|
// the row flagged truncated.
|
||||||
|
var input = new string('g', 10 * 1024);
|
||||||
|
var evt = NewEvent(
|
||||||
|
AuditStatus.Delivered,
|
||||||
|
request: input,
|
||||||
|
response: input,
|
||||||
|
errorDetail: input,
|
||||||
|
extra: input);
|
||||||
|
|
||||||
|
var result = Filter().Apply(evt);
|
||||||
|
|
||||||
|
Assert.Equal(8192, Encoding.UTF8.GetByteCount(result.RequestSummary!));
|
||||||
|
Assert.Equal(8192, Encoding.UTF8.GetByteCount(result.ResponseSummary!));
|
||||||
|
Assert.Equal(8192, Encoding.UTF8.GetByteCount(result.ErrorDetail!));
|
||||||
|
Assert.Equal(8192, Encoding.UTF8.GetByteCount(result.Extra!));
|
||||||
|
Assert.True(result.PayloadTruncated);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// IOptionsMonitor test double — returns the same snapshot on every read,
|
||||||
|
/// no change-token plumbing required for these tests (Bundle D wires the
|
||||||
|
/// real hot-reload path).
|
||||||
|
/// </summary>
|
||||||
|
private sealed class StaticMonitor : IOptionsMonitor<AuditLogOptions>
|
||||||
|
{
|
||||||
|
private readonly AuditLogOptions _value;
|
||||||
|
|
||||||
|
public StaticMonitor(AuditLogOptions value) => _value = value;
|
||||||
|
|
||||||
|
public AuditLogOptions CurrentValue => _value;
|
||||||
|
|
||||||
|
public AuditLogOptions Get(string? name) => _value;
|
||||||
|
|
||||||
|
public IDisposable? OnChange(Action<AuditLogOptions, string?> listener) => null;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
using NSubstitute;
|
||||||
|
using ScadaLink.AuditLog.Site;
|
||||||
|
using ScadaLink.HealthMonitoring;
|
||||||
|
|
||||||
|
namespace ScadaLink.AuditLog.Tests.Site;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Bundle C (M5-T7) — the <see cref="HealthMetricsAuditRedactionFailureCounter"/>
|
||||||
|
/// adapter is the production binding for
|
||||||
|
/// <see cref="ScadaLink.AuditLog.Payload.IAuditRedactionFailureCounter"/> on
|
||||||
|
/// site nodes; it forwards every <see cref="DefaultAuditPayloadFilter"/>
|
||||||
|
/// redactor over-redaction event into the shared
|
||||||
|
/// <see cref="ISiteHealthCollector"/> so the site health report surfaces the
|
||||||
|
/// count as <c>AuditRedactionFailure</c>. Mirrors the M2 Bundle G
|
||||||
|
/// HealthMetricsAuditWriteFailureCounter shape one-for-one.
|
||||||
|
/// </summary>
|
||||||
|
public class HealthMetricsAuditRedactionFailureCounterTests
|
||||||
|
{
|
||||||
|
[Fact]
|
||||||
|
public void Increment_Routes_To_Collector_IncrementAuditRedactionFailure()
|
||||||
|
{
|
||||||
|
var collector = Substitute.For<ISiteHealthCollector>();
|
||||||
|
var counter = new HealthMetricsAuditRedactionFailureCounter(collector);
|
||||||
|
|
||||||
|
counter.Increment();
|
||||||
|
|
||||||
|
collector.Received(1).IncrementAuditRedactionFailure();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Increment_Multiple_Calls_Route_To_Collector_Each_Time()
|
||||||
|
{
|
||||||
|
var collector = Substitute.For<ISiteHealthCollector>();
|
||||||
|
var counter = new HealthMetricsAuditRedactionFailureCounter(collector);
|
||||||
|
|
||||||
|
counter.Increment();
|
||||||
|
counter.Increment();
|
||||||
|
counter.Increment();
|
||||||
|
|
||||||
|
collector.Received(3).IncrementAuditRedactionFailure();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Construction_With_Null_Collector_Throws_ArgumentNullException()
|
||||||
|
{
|
||||||
|
Assert.Throws<ArgumentNullException>(
|
||||||
|
() => new HealthMetricsAuditRedactionFailureCounter(null!));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
namespace ScadaLink.HealthMonitoring.Tests;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Bundle C (M5-T7) regression coverage. The Audit Log payload filter
|
||||||
|
/// (<c>DefaultAuditPayloadFilter</c>) increments
|
||||||
|
/// <c>IAuditRedactionFailureCounter</c> every time a header/body/SQL-param
|
||||||
|
/// redactor stage throws and the filter has to over-redact the field with
|
||||||
|
/// the <c><redacted: redactor error></c> marker. Bundle C bridges that
|
||||||
|
/// counter into the Site Health Monitoring report payload as
|
||||||
|
/// <c>AuditRedactionFailure</c> so a misconfigured / catastrophic regex
|
||||||
|
/// surfaces on /monitoring/health rather than disappearing into a NoOp sink.
|
||||||
|
/// Mirrors the Bundle G <c>SiteAuditWriteFailures</c> metric shape — same
|
||||||
|
/// per-interval increment-and-reset semantics, same defaults-to-zero
|
||||||
|
/// contract.
|
||||||
|
/// </summary>
|
||||||
|
public class AuditRedactionFailureMetricTests
|
||||||
|
{
|
||||||
|
private readonly SiteHealthCollector _collector = new();
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Increment_Three_Times_Counter_Reports_3()
|
||||||
|
{
|
||||||
|
_collector.IncrementAuditRedactionFailure();
|
||||||
|
_collector.IncrementAuditRedactionFailure();
|
||||||
|
_collector.IncrementAuditRedactionFailure();
|
||||||
|
|
||||||
|
var report = _collector.CollectReport("site-1");
|
||||||
|
|
||||||
|
Assert.Equal(3, report.AuditRedactionFailure);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Report_Payload_Includes_AuditRedactionFailure_AsZeroByDefault()
|
||||||
|
{
|
||||||
|
var report = _collector.CollectReport("site-1");
|
||||||
|
|
||||||
|
Assert.Equal(0, report.AuditRedactionFailure);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Mirrors the existing per-interval reset semantics for ScriptErrorCount /
|
||||||
|
/// AlarmEvaluationErrorCount / DeadLetterCount / SiteAuditWriteFailures —
|
||||||
|
/// AuditRedactionFailure is an interval count, not a running total.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void CollectReport_Resets_AuditRedactionFailure()
|
||||||
|
{
|
||||||
|
_collector.IncrementAuditRedactionFailure();
|
||||||
|
_collector.IncrementAuditRedactionFailure();
|
||||||
|
|
||||||
|
var first = _collector.CollectReport("site-1");
|
||||||
|
Assert.Equal(2, first.AuditRedactionFailure);
|
||||||
|
|
||||||
|
var second = _collector.CollectReport("site-1");
|
||||||
|
Assert.Equal(0, second.AuditRedactionFailure);
|
||||||
|
}
|
||||||
|
}
|
||||||
163
tests/ScadaLink.PerformanceTests/AuditLog/HotPathLatencyTests.cs
Normal file
163
tests/ScadaLink.PerformanceTests/AuditLog/HotPathLatencyTests.cs
Normal file
@@ -0,0 +1,163 @@
|
|||||||
|
using System.Diagnostics;
|
||||||
|
using System.Text;
|
||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using ScadaLink.AuditLog.Configuration;
|
||||||
|
using ScadaLink.AuditLog.Payload;
|
||||||
|
using ScadaLink.Commons.Entities.Audit;
|
||||||
|
using ScadaLink.Commons.Types.Enums;
|
||||||
|
|
||||||
|
namespace ScadaLink.PerformanceTests.AuditLog;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Bundle D (M5-T9) hot-path latency budget for <see cref="IAuditPayloadFilter"/>.
|
||||||
|
/// The filter sits between event construction and persistence on every audit
|
||||||
|
/// row — site SQLite hot-path and central direct-write both — so it MUST stay
|
||||||
|
/// out of the way of script-thread latency.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <para>
|
||||||
|
/// Methodology: warm-up + N iterations, time each <see cref="IAuditPayloadFilter.Apply"/>
|
||||||
|
/// with <see cref="Stopwatch"/>, sort, take p95, assert under threshold. Matches
|
||||||
|
/// the simple-loop style of the existing <c>StaggeredStartupTests</c> /
|
||||||
|
/// <c>HealthAggregationTests</c> in this project (no BenchmarkDotNet).
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Threshold note: the spec says "set during M5 brainstorm" — pick targets that
|
||||||
|
/// are an order of magnitude faster than the SQLite write they precede (the
|
||||||
|
/// site writer's bottleneck is the disk fsync, not the in-memory filter).
|
||||||
|
/// Reality may diverge on slow CI; the assertions include the empirical
|
||||||
|
/// fall-back the task brief calls for (p95 + 30% regression guard) wired
|
||||||
|
/// through environment-variable override so a slow shared runner doesn't
|
||||||
|
/// flake the build but a 10x regression still does.
|
||||||
|
/// </para>
|
||||||
|
/// </remarks>
|
||||||
|
public class HotPathLatencyTests
|
||||||
|
{
|
||||||
|
private const int WarmupIterations = 200;
|
||||||
|
private const int MeasureIterations = 2_000;
|
||||||
|
|
||||||
|
private static DefaultAuditPayloadFilter Filter(AuditLogOptions opts) => new(
|
||||||
|
new StaticMonitor(opts),
|
||||||
|
NullLogger<DefaultAuditPayloadFilter>.Instance);
|
||||||
|
|
||||||
|
private static AuditEvent NewEvent(string request)
|
||||||
|
{
|
||||||
|
return new AuditEvent
|
||||||
|
{
|
||||||
|
EventId = Guid.NewGuid(),
|
||||||
|
OccurredAtUtc = DateTime.UtcNow,
|
||||||
|
Channel = AuditChannel.ApiOutbound,
|
||||||
|
Kind = AuditKind.ApiCall,
|
||||||
|
Status = AuditStatus.Delivered,
|
||||||
|
Target = "esg.target",
|
||||||
|
RequestSummary = request,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Run <paramref name="fn"/> N times, returning the p95 in microseconds.
|
||||||
|
/// Single-threaded; <see cref="Stopwatch.GetTimestamp"/> for high-res
|
||||||
|
/// timing.
|
||||||
|
/// </summary>
|
||||||
|
private static double MeasureP95Microseconds(int iterations, Action fn)
|
||||||
|
{
|
||||||
|
var samples = new double[iterations];
|
||||||
|
var ticksToMicroseconds = 1_000_000d / Stopwatch.Frequency;
|
||||||
|
for (var i = 0; i < iterations; i++)
|
||||||
|
{
|
||||||
|
var start = Stopwatch.GetTimestamp();
|
||||||
|
fn();
|
||||||
|
var end = Stopwatch.GetTimestamp();
|
||||||
|
samples[i] = (end - start) * ticksToMicroseconds;
|
||||||
|
}
|
||||||
|
Array.Sort(samples);
|
||||||
|
var p95Index = (int)Math.Ceiling(iterations * 0.95) - 1;
|
||||||
|
return samples[p95Index];
|
||||||
|
}
|
||||||
|
|
||||||
|
[Trait("Category", "Performance")]
|
||||||
|
[Fact]
|
||||||
|
public void Filter_Apply_4KB_Body_DefaultRedactors_P95_LessThan_50_Microseconds()
|
||||||
|
{
|
||||||
|
// 4 KiB body laced with a 16-digit token + a `password` field so the
|
||||||
|
// header-redact stage is a no-op (input isn't a JSON object with a
|
||||||
|
// headers field), the body regex stage matches twice, and the
|
||||||
|
// truncation stage runs after redaction. Mirrors a typical
|
||||||
|
// medium-sized HTTP POST body that an outbound API audit row would
|
||||||
|
// carry.
|
||||||
|
var opts = new AuditLogOptions
|
||||||
|
{
|
||||||
|
// Keep the cap modest so the truncation path actually fires.
|
||||||
|
DefaultCapBytes = 4096,
|
||||||
|
GlobalBodyRedactors = new List<string>
|
||||||
|
{
|
||||||
|
"\"password\":\\s*\"[^\"]*\"",
|
||||||
|
"\\d{16}",
|
||||||
|
},
|
||||||
|
};
|
||||||
|
var pad = new string('x', 4 * 1024);
|
||||||
|
var body = "{\"user\":\"alice\",\"password\":\"hunter2\",\"card\":\"4111111111111111\",\"pad\":\"" + pad + "\"}";
|
||||||
|
// Sanity: we actually want > 4 KiB so the truncate stage runs.
|
||||||
|
Assert.True(Encoding.UTF8.GetByteCount(body) > 4096);
|
||||||
|
|
||||||
|
var filter = Filter(opts);
|
||||||
|
var evt = NewEvent(body);
|
||||||
|
|
||||||
|
// Warm-up — JIT, regex compile, dictionary populate.
|
||||||
|
for (var i = 0; i < WarmupIterations; i++) _ = filter.Apply(evt);
|
||||||
|
|
||||||
|
var p95Us = MeasureP95Microseconds(MeasureIterations, () => _ = filter.Apply(evt));
|
||||||
|
|
||||||
|
// Default budget 50 µs (spec target). Override via env for slow CI:
|
||||||
|
// SCADALINK_AUDIT_FILTER_4KB_P95_US — interpret as the regression
|
||||||
|
// guard threshold. Print the observed value so a missed budget gives
|
||||||
|
// useful telemetry on the test output.
|
||||||
|
var threshold = GetThresholdMicroseconds("SCADALINK_AUDIT_FILTER_4KB_P95_US", 50d);
|
||||||
|
Assert.True(p95Us < threshold,
|
||||||
|
$"4KB body filter p95 = {p95Us:F1} µs; threshold = {threshold:F1} µs");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Trait("Category", "Performance")]
|
||||||
|
[Fact]
|
||||||
|
public void Filter_Apply_RawEvent_NoRedactors_P95_LessThan_10_Microseconds()
|
||||||
|
{
|
||||||
|
// No redactors configured — header redactor short-circuits on the
|
||||||
|
// non-JSON-object pre-check, body redactor list is empty, SQL param
|
||||||
|
// redactor is gated on AuditChannel.DbOutbound (we're ApiOutbound).
|
||||||
|
// Just the per-field truncation walk. Should be effectively free.
|
||||||
|
var opts = new AuditLogOptions();
|
||||||
|
var filter = Filter(opts);
|
||||||
|
|
||||||
|
// Small payload that fits under the 8 KiB default cap — no truncation,
|
||||||
|
// just the byte-count check per field.
|
||||||
|
var evt = NewEvent("hello world");
|
||||||
|
|
||||||
|
for (var i = 0; i < WarmupIterations; i++) _ = filter.Apply(evt);
|
||||||
|
|
||||||
|
var p95Us = MeasureP95Microseconds(MeasureIterations, () => _ = filter.Apply(evt));
|
||||||
|
|
||||||
|
var threshold = GetThresholdMicroseconds("SCADALINK_AUDIT_FILTER_RAW_P95_US", 10d);
|
||||||
|
Assert.True(p95Us < threshold,
|
||||||
|
$"Raw-event filter p95 = {p95Us:F1} µs; threshold = {threshold:F1} µs");
|
||||||
|
}
|
||||||
|
|
||||||
|
private static double GetThresholdMicroseconds(string envVar, double defaultUs)
|
||||||
|
{
|
||||||
|
var raw = Environment.GetEnvironmentVariable(envVar);
|
||||||
|
if (raw != null && double.TryParse(raw, out var parsed) && parsed > 0)
|
||||||
|
{
|
||||||
|
return parsed;
|
||||||
|
}
|
||||||
|
return defaultUs;
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class StaticMonitor : IOptionsMonitor<AuditLogOptions>
|
||||||
|
{
|
||||||
|
private readonly AuditLogOptions _value;
|
||||||
|
public StaticMonitor(AuditLogOptions value) => _value = value;
|
||||||
|
public AuditLogOptions CurrentValue => _value;
|
||||||
|
public AuditLogOptions Get(string? name) => _value;
|
||||||
|
public IDisposable? OnChange(Action<AuditLogOptions, string?> listener) => null;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -21,6 +21,7 @@
|
|||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
|
<ProjectReference Include="../../src/ScadaLink.AuditLog/ScadaLink.AuditLog.csproj" />
|
||||||
<ProjectReference Include="../../src/ScadaLink.Commons/ScadaLink.Commons.csproj" />
|
<ProjectReference Include="../../src/ScadaLink.Commons/ScadaLink.Commons.csproj" />
|
||||||
<ProjectReference Include="../../src/ScadaLink.HealthMonitoring/ScadaLink.HealthMonitoring.csproj" />
|
<ProjectReference Include="../../src/ScadaLink.HealthMonitoring/ScadaLink.HealthMonitoring.csproj" />
|
||||||
<ProjectReference Include="../../src/ScadaLink.StoreAndForward/ScadaLink.StoreAndForward.csproj" />
|
<ProjectReference Include="../../src/ScadaLink.StoreAndForward/ScadaLink.StoreAndForward.csproj" />
|
||||||
|
|||||||
@@ -70,6 +70,7 @@ public class DeploymentManagerRedeployTests : TestKit, IDisposable
|
|||||||
public void IncrementAlarmError() { }
|
public void IncrementAlarmError() { }
|
||||||
public void IncrementDeadLetter() { }
|
public void IncrementDeadLetter() { }
|
||||||
public void IncrementSiteAuditWriteFailures() { }
|
public void IncrementSiteAuditWriteFailures() { }
|
||||||
|
public void IncrementAuditRedactionFailure() { }
|
||||||
public void UpdateConnectionHealth(string connectionName, ConnectionHealth health) { }
|
public void UpdateConnectionHealth(string connectionName, ConnectionHealth health) { }
|
||||||
public void RemoveConnection(string connectionName) { }
|
public void RemoveConnection(string connectionName) { }
|
||||||
public void UpdateTagResolution(string connectionName, int totalSubscribed, int successfullyResolved) { }
|
public void UpdateTagResolution(string connectionName, int totalSubscribed, int successfullyResolved) { }
|
||||||
|
|||||||
Reference in New Issue
Block a user