fix(central): review findings — no client-side audit truncation, insert-first upsert, QI-safe scripts, honest operator replies

Six adversarial-review findings in the central SQL/ingest layer. F1 (AuditLogRepository.InsertChunkAsync) — the set-based ingest declared each string parameter at its COLUMN width (Actor/Target 256, Action 64, Outcome 16, Category 32, SourceNode 64), so SqlClient truncated an over-long value at bind time and committed the mutilated row — silent, in an append-only store, with no PayloadTruncated flag — while the per-row and reconciliation paths sent the same value in full and let the server reject it with 2628. Bind at the value's own length instead; explicit SqlDbType is kept (it fixes the VALUES constructor's derived column types and datetime2 precision). Design: reject everywhere, truncate nowhere — matching today's per-row behaviour. F2 (SiteCallAuditRepository.UpsertAsync) — the single-statement upsert ran the monotonic UPDATE first and INSERTed only if nothing matched. Two writers racing the first packet of one TrackedOperationId (the cached dual-write and the reconciliation pull carry DIFFERENT lifecycle states) both matched nothing, and the loser then skipped its INSERT or swallowed a 2627 — dropping its Status/RetryCount/HttpStatus/TerminalAtUtc. Legs swapped to `IF NOT EXISTS … INSERT; UPDATE <monotonic>` — still one round trip, and the loser's UPDATE now lands on the winner's row. The duplicate-key catch re-runs the monotonic UPDATE for the same reason. Moved to raw SQL with explicitly-typed parameters so the intricate rank predicate exists in exactly one place (an untyped DateTime would bind as `datetime` and round the freshness tiebreaker). F3 (docs/plans/sql/*.sql) — filtered-index DDL failed with error 1934 under the documented `docker exec … sqlcmd` path, which defaults QUOTED_IDENTIFIER OFF; once IX_Notifications_Delivered exists, QI-OFF DML on Notifications fails too. All four scripts now open with `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO` (own batch, so it is in force when the next batch parses), and the migration convention in Component-ConfigurationDatabase.md documents `sqlcmd -I`. Verified live: the pre-fix script fails 1934 without -I, the fixed one applies. F4 (SiteCallAuditActor) — the off-mailbox reconciliation/purge passes reuse the injected repository, so tests drove one DbContext from the pass and a mailbox handler concurrently. Serialized at the CALL via a private SerializedRepository wrapper applied only by the test constructors, rather than running the pass on-mailbox: production keeps its PipeTo shape untouched, and the existing "a blocked drain does not stall ingest/query/KPI" regression tests stay meaningful (they would have been invalidated by suspending the mailbox). F5 (AuditLogIngestActor) — when the batch failed because the 20 s IngestBudget expired, the per-row fallback reused the same expired token: N instant failures, N counter bumps, zero accepted. The fallback now gets a fresh 5 s budget (inside the 30 s outer Ask), and a blown budget bumps the failure counter ONCE for the batch instead of once per row. F6 (NotificationOutboxRepository.UpdateAsync) — ExecuteUpdate's row count was discarded, so an operator Retry/Discard of a notification the retention purge had already deleted reported success (the pre-ExecuteUpdate code threw DbUpdateConcurrencyException). UpdateAsync now returns whether a row matched; the operator one-shots answer "notification not found" and emit no audit row for the action that did not happen, while the dispatcher logs a warning (its delivery already happened; nothing to retry). GetByIdAsync switched to AsNoTracking since the write is out-of-band. Tests: 5 new SQL-backed regressions (over-long Target rejected on both paths + boundary round-trip; concurrent first-write and already-created-by-another-writer upserts; vanished-row UpdateAsync), a token-identity pin on the ingest fallback, a repository-concurrency detector for the SiteCallAudit passes, and vanished-row operator-path tests. The F1/F2/F4 regressions were each confirmed failing against the pre-fix code. Suites: ConfigurationDatabase 369, AuditLog 378, SiteCallAudit 66, NotificationOutbox 152 — all green, solution builds with 0 warnings.
2026-08-14 23:46:28 -04:00
parent b1de9dfdd4
commit 5d075f1374
30 changed files with 1042 additions and 126 deletions
@@ -84,6 +84,22 @@ public class AuditLogIngestActor : ReceiveActor
    /// </summary>
    internal static readonly TimeSpan IngestSqlCommandTimeout = TimeSpan.FromSeconds(15);

+    /// <summary>
+    /// Budget for the per-row fallback loop, granted as a FRESH
+    /// <see cref="CancellationTokenSource"/> rather than reusing the batch's.
+    /// </summary>
+    /// <remarks>
+    /// The fallback used to run on the batch's own token. When the batch failed
+    /// BECAUSE that token expired, every fallback insert was handed an
+    /// already-cancelled token and failed instantly: N logged errors, N counter
+    /// bumps, zero rows accepted, and the site retried the whole packet anyway.
+    /// A fresh, deliberately SHORT budget (shorter than <see cref="IngestBudget"/>,
+    /// so the reply still beats the caller's Ask even when the batch consumed its
+    /// whole allowance) gives the poison-row isolation the fallback exists for a
+    /// real chance to land the good rows.
+    /// </remarks>
+    internal static readonly TimeSpan IngestFallbackBudget = TimeSpan.FromSeconds(5);
+
    private readonly IServiceProvider? _serviceProvider;
    private readonly IAuditLogRepository? _injectedRepository;
    private readonly ILogger<AuditLogIngestActor> _logger;
@@ -265,15 +281,36 @@ public class AuditLogIngestActor : ReceiveActor
            // back to the per-row path here, so a poison row still costs only
            // itself. Re-running rows the failed batch may already have committed
            // is safe: every insert is idempotent on EventId.
+            //
+            // The fallback gets its OWN token. Reusing the batch's meant that a
+            // batch which failed because the budget EXPIRED handed the loop an
+            // already-cancelled token, so every row failed instantly and the
+            // fallback did nothing but multiply the log and counter noise by the
+            // row count.
+            var budgetExpired = budget.IsCancellationRequested;
+            using var fallbackBudget = new CancellationTokenSource(IngestFallbackBudget);
+
            _logger.LogWarning(ex,
-                "Set-based ingest of {Count} audit event(s) failed; falling back to per-row inserts so one bad row does not sink the batch.",
-                cmd.Events.Count);
+                "Set-based ingest of {Count} audit event(s) failed{BudgetNote}; falling back to per-row inserts (fresh {FallbackSeconds}s budget) so one bad row does not sink the batch.",
+                cmd.Events.Count,
+                budgetExpired ? " after exhausting the ingest budget" : string.Empty,
+                IngestFallbackBudget.TotalSeconds);
+
+            if (budgetExpired)
+            {
+                // A blown budget is ONE failure event for the batch, not one per
+                // row — bump the health counter once here and suppress the
+                // per-row bumps below so the dashboard sees a timeout as a
+                // timeout rather than as N independent write failures.
+                try { failureCounter?.Increment(); }
+                catch { /* counter must never throw — defence in depth */ }
+            }

            for (var i = 0; i < projected.Count; i++)
            {
                try
                {
-                    await repository.InsertIfNotExistsAsync(projected[i], budget.Token).ConfigureAwait(false);
+                    await repository.InsertIfNotExistsAsync(projected[i], fallbackBudget.Token).ConfigureAwait(false);
                    accepted.Add(cmd.Events[i].EventId);
                }
                catch (Exception rowEx)
@@ -281,9 +318,14 @@ public class AuditLogIngestActor : ReceiveActor
                    // Per-row catch — one bad row never sinks the whole batch.
                    // The row stays Pending at the site; the next drain retries.
                    // Bump the central health counter so a sustained insert-throw
-                    // failure surfaces on the dashboard.
-                    try { failureCounter?.Increment(); }
-                    catch { /* counter must never throw — defence in depth */ }
+                    // failure surfaces on the dashboard — unless the batch already
+                    // counted itself as a single budget-exhaustion failure.
+                    if (!budgetExpired)
+                    {
+                        try { failureCounter?.Increment(); }
+                        catch { /* counter must never throw — defence in depth */ }
+                    }
+
                    _logger.LogError(rowEx,
                        "Failed to persist audit event {EventId} during batch ingest; row will be retried by the site.",
                        cmd.Events[i].EventId);