fix(auditlog)+fix(configdb): budget the ingest fallback loop; index + varchar-type the SiteCalls list query (arch-review 2026-08-19 P5-1 + P5-2)

P5-1: AuditLogIngestActor.OnCachedTelemetryAsync's per-entry fallback loop
(used when the cached-telemetry batch fast path fails) ran with no
budget/cancellation, unlike OnIngestAsync's fallback. A slow/wedged DB could
freeze the central ingest singleton's mailbox for minutes. The loop now runs
under its own fresh IngestFallbackBudget CancellationTokenSource (mirroring
OnIngestAsync), passes the token into BeginTransactionAsync/
InsertIfNotExistsAsync/UpsertAsync, and stops early on budget exhaustion,
logging how many entries were left for the periodic reconciliation pull to
pick up.

P5-2: SiteCallAuditRepository.QueryAsync ordered by
(CreatedAtUtc DESC, TrackedOperationId DESC) with no index leading
unconditionally on CreatedAtUtc, forcing a full scan+sort on an unfiltered
page. Added IX_SiteCalls_Created_TrackedOperationId (migration
AddSiteCallsListSortIndex + matching hand-written production-parity script
under docs/plans/sql/). Also switched the query from FromSqlInterpolated
(which binds string filters as nvarchar(4000)) to FromSqlRaw with explicitly
SqlDbType.VarChar-typed parameters, matching the SiteCalls columns'
IsUnicode(false) declaration and the write path's existing Param(...)
pattern, so seeks — including the keyset predicate — are no longer blocked
by an implicit conversion.

Tests: new AuditLogIngestActorCombinedTelemetryTests case proves the
fallback budget bounds the wait (hanging repo stub, ~5s not indefinite),
that remaining entries are skipped, and that a non-default/cancellable
token reaches the repository call; verified fail-unfixed by reverting the
actor change (test times out at its 20s bound). Existing
SiteCallAuditRepositoryTests QueryAsync/keyset/stuck-cutoff cases (real
MSSQL) confirm the raw-SQL rewrite still returns correct pages.
This commit is contained in:
Joseph Doherty
2026-08-19 14:44:44 -04:00
parent 6c11362d9f
commit eca0ab768d
8 changed files with 2488 additions and 32 deletions
@@ -325,7 +325,12 @@ public class AuditLogIngestActor : ReceiveActor
/// <see cref="Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.BeginTransactionAsync"/>
/// scope and a try/catch around it). Audit-write failure NEVER aborts the
/// user-facing action — the site keeps the row Pending and retries on the
/// next drain.
/// next drain. The per-entry loop (used when <see cref="TryIngestCachedBatchAsync"/>
/// falls back) runs under its own fresh <see cref="IngestFallbackBudget"/>
/// token, mirroring <see cref="OnIngestAsync"/>'s fallback — a slow or wedged
/// database can only cost this handler that one short budget, not an
/// unbounded per-entry loop, so the singleton's mailbox is never frozen for
/// the life of a large packet.
/// </remarks>
private async Task OnCachedTelemetryAsync(IngestCachedTelemetryCommand cmd)
{
@@ -369,14 +374,38 @@ public class AuditLogIngestActor : ReceiveActor
return;
}
// The per-entry fallback gets its OWN fresh, short budget — mirroring
// OnIngestAsync's fallback (see IngestFallbackBudget above). Without
// this, an unbounded loop of per-entry transactions (each wrapped in
// SqlServerRetryingExecutionStrategy's 5x30s retry ladder) can run for
// minutes inside this ReceiveAsync handler, freezing the singleton's
// mailbox: every queued IngestAuditEventsCommand/IngestCachedTelemetryCommand
// stalls behind it, sites time out and re-send, and the pile-up
// compounds. Budget exhaustion stops the loop early; entries left
// over are simply never added to `accepted`, so the site keeps them
// Pending and the existing lost-telemetry path (periodic
// reconciliation pull) picks up the remainder.
using var fallbackBudget = new CancellationTokenSource(IngestFallbackBudget);
var entryIndex = 0;
foreach (var entry in cmd.Entries)
{
entryIndex++;
if (fallbackBudget.IsCancellationRequested)
{
var remaining = cmd.Entries.Count - entryIndex + 1;
_logger.LogWarning(
"Combined telemetry per-entry fallback exhausted its {FallbackSeconds}s budget; leaving {Remaining} entr(ies) unaccepted for this drain — the site keeps them Pending and periodic reconciliation will pick up the remainder.",
IngestFallbackBudget.TotalSeconds,
remaining);
break;
}
try
{
await strategy.ExecuteAsync(async () =>
{
await using var tx = await dbContext.Database
.BeginTransactionAsync()
.BeginTransactionAsync(fallbackBudget.Token)
.ConfigureAwait(false);
// Stamp IngestedAtUtc on both rows from a single
@@ -396,12 +425,12 @@ public class AuditLogIngestActor : ReceiveActor
var auditStamped = AuditRowProjection.WithIngestedAtUtc(filteredAudit, ingestedAt);
var siteCallStamped = entry.SiteCall with { IngestedAtUtc = ingestedAt };
await auditRepo.InsertIfNotExistsAsync(auditStamped)
await auditRepo.InsertIfNotExistsAsync(auditStamped, fallbackBudget.Token)
.ConfigureAwait(false);
await siteCallRepo.UpsertAsync(siteCallStamped)
await siteCallRepo.UpsertAsync(siteCallStamped, fallbackBudget.Token)
.ConfigureAwait(false);
await tx.CommitAsync().ConfigureAwait(false);
await tx.CommitAsync(fallbackBudget.Token).ConfigureAwait(false);
}).ConfigureAwait(false);
accepted.Add(entry.Audit.EventId);