fix(auditlog)+fix(configdb): budget the ingest fallback loop; index + varchar-type the SiteCalls list query (arch-review 2026-08-19 P5-1 + P5-2)
P5-1: AuditLogIngestActor.OnCachedTelemetryAsync's per-entry fallback loop (used when the cached-telemetry batch fast path fails) ran with no budget/cancellation, unlike OnIngestAsync's fallback. A slow/wedged DB could freeze the central ingest singleton's mailbox for minutes. The loop now runs under its own fresh IngestFallbackBudget CancellationTokenSource (mirroring OnIngestAsync), passes the token into BeginTransactionAsync/ InsertIfNotExistsAsync/UpsertAsync, and stops early on budget exhaustion, logging how many entries were left for the periodic reconciliation pull to pick up. P5-2: SiteCallAuditRepository.QueryAsync ordered by (CreatedAtUtc DESC, TrackedOperationId DESC) with no index leading unconditionally on CreatedAtUtc, forcing a full scan+sort on an unfiltered page. Added IX_SiteCalls_Created_TrackedOperationId (migration AddSiteCallsListSortIndex + matching hand-written production-parity script under docs/plans/sql/). Also switched the query from FromSqlInterpolated (which binds string filters as nvarchar(4000)) to FromSqlRaw with explicitly SqlDbType.VarChar-typed parameters, matching the SiteCalls columns' IsUnicode(false) declaration and the write path's existing Param(...) pattern, so seeks — including the keyset predicate — are no longer blocked by an implicit conversion. Tests: new AuditLogIngestActorCombinedTelemetryTests case proves the fallback budget bounds the wait (hanging repo stub, ~5s not indefinite), that remaining entries are skipped, and that a non-default/cancellable token reaches the repository call; verified fail-unfixed by reverting the actor change (test times out at its 20s bound). Existing SiteCallAuditRepositoryTests QueryAsync/keyset/stuck-cutoff cases (real MSSQL) confirm the raw-SQL rewrite still returns correct pages.
This commit is contained in:
@@ -325,7 +325,12 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
/// <see cref="Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.BeginTransactionAsync"/>
|
||||
/// scope and a try/catch around it). Audit-write failure NEVER aborts the
|
||||
/// user-facing action — the site keeps the row Pending and retries on the
|
||||
/// next drain.
|
||||
/// next drain. The per-entry loop (used when <see cref="TryIngestCachedBatchAsync"/>
|
||||
/// falls back) runs under its own fresh <see cref="IngestFallbackBudget"/>
|
||||
/// token, mirroring <see cref="OnIngestAsync"/>'s fallback — a slow or wedged
|
||||
/// database can only cost this handler that one short budget, not an
|
||||
/// unbounded per-entry loop, so the singleton's mailbox is never frozen for
|
||||
/// the life of a large packet.
|
||||
/// </remarks>
|
||||
private async Task OnCachedTelemetryAsync(IngestCachedTelemetryCommand cmd)
|
||||
{
|
||||
@@ -369,14 +374,38 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
return;
|
||||
}
|
||||
|
||||
// The per-entry fallback gets its OWN fresh, short budget — mirroring
|
||||
// OnIngestAsync's fallback (see IngestFallbackBudget above). Without
|
||||
// this, an unbounded loop of per-entry transactions (each wrapped in
|
||||
// SqlServerRetryingExecutionStrategy's 5x30s retry ladder) can run for
|
||||
// minutes inside this ReceiveAsync handler, freezing the singleton's
|
||||
// mailbox: every queued IngestAuditEventsCommand/IngestCachedTelemetryCommand
|
||||
// stalls behind it, sites time out and re-send, and the pile-up
|
||||
// compounds. Budget exhaustion stops the loop early; entries left
|
||||
// over are simply never added to `accepted`, so the site keeps them
|
||||
// Pending and the existing lost-telemetry path (periodic
|
||||
// reconciliation pull) picks up the remainder.
|
||||
using var fallbackBudget = new CancellationTokenSource(IngestFallbackBudget);
|
||||
var entryIndex = 0;
|
||||
foreach (var entry in cmd.Entries)
|
||||
{
|
||||
entryIndex++;
|
||||
if (fallbackBudget.IsCancellationRequested)
|
||||
{
|
||||
var remaining = cmd.Entries.Count - entryIndex + 1;
|
||||
_logger.LogWarning(
|
||||
"Combined telemetry per-entry fallback exhausted its {FallbackSeconds}s budget; leaving {Remaining} entr(ies) unaccepted for this drain — the site keeps them Pending and periodic reconciliation will pick up the remainder.",
|
||||
IngestFallbackBudget.TotalSeconds,
|
||||
remaining);
|
||||
break;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await strategy.ExecuteAsync(async () =>
|
||||
{
|
||||
await using var tx = await dbContext.Database
|
||||
.BeginTransactionAsync()
|
||||
.BeginTransactionAsync(fallbackBudget.Token)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
// Stamp IngestedAtUtc on both rows from a single
|
||||
@@ -396,12 +425,12 @@ public class AuditLogIngestActor : ReceiveActor
|
||||
var auditStamped = AuditRowProjection.WithIngestedAtUtc(filteredAudit, ingestedAt);
|
||||
var siteCallStamped = entry.SiteCall with { IngestedAtUtc = ingestedAt };
|
||||
|
||||
await auditRepo.InsertIfNotExistsAsync(auditStamped)
|
||||
await auditRepo.InsertIfNotExistsAsync(auditStamped, fallbackBudget.Token)
|
||||
.ConfigureAwait(false);
|
||||
await siteCallRepo.UpsertAsync(siteCallStamped)
|
||||
await siteCallRepo.UpsertAsync(siteCallStamped, fallbackBudget.Token)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
await tx.CommitAsync().ConfigureAwait(false);
|
||||
await tx.CommitAsync(fallbackBudget.Token).ConfigureAwait(false);
|
||||
}).ConfigureAwait(false);
|
||||
|
||||
accepted.Add(entry.Audit.EventId);
|
||||
|
||||
Reference in New Issue
Block a user