perf(central): set-based ingest, aligned partition purge, KPI query shapes, EF hygiene

This commit is contained in:
Joseph Doherty
2026-08-14 21:07:12 -04:00
parent ee193cd2bb
commit 5db2a810c0
29 changed files with 3790 additions and 266 deletions
+32 -4
View File
@@ -291,6 +291,27 @@ stay `Pending` for the next sweep. Cadence is short (default 5 s) when
non-empty, longer (default 30 s) when idle; telemetry runs on a dedicated
dispatcher.
**Central-side ingest is set-based.** `AuditLogIngestActor` writes a whole
packet with ONE `InsertManyIfNotExistsAsync` statement rather than one
`IF NOT EXISTS … INSERT` round trip per event; the cached-telemetry dual-write
similarly runs the whole packet in ONE transaction (set-based audit insert plus
one single-statement `SiteCalls` upsert per entry) instead of a transaction per
entry. Idempotency is unchanged: duplicates that repeat *within* a packet
collapse first-write-wins before the statement is built, duplicates *across*
packets are eliminated by the anti-semi-join, and any failure falls back to the
per-row / per-entry path — so the documented invariant that one bad row cannot
sink the rest of the batch still holds, it is simply no longer paid for on the
healthy path.
**Timeout ladder.** The ingest budget is deliberately the smallest on the path:
the site's Ask and the central gRPC handler's Ask are both 30 s, the actor's
own database budget is 20 s and the per-statement SQL timeout is 15 s. Before
this the three were identical, so they expired at the same instant and the
caller learned nothing but "it took 30 s" — no partial ack, no way to tell a
slow database from a wedged singleton. With the inner budgets strictly smaller,
a slow batch is abandoned by the actor first and the accepted-so-far ids are
still replied while the outer Asks are still waiting.
### Reconciliation pull (self-healing for missed telemetry)
A central `SiteAuditReconciliationActor` periodically (default 5 min per site)
@@ -490,12 +511,19 @@ MS SQL for direct-write events). Unredacted secrets never persist.
silently failing retention job would otherwise be invisible until the table grew
unbounded. Per-boundary/per-channel error isolation is unchanged — a single failure
still never abandons the rest of the tick.
- **Maintenance command timeout:** the switch-out drop-and-rebuild dance and each
- **Maintenance command timeout:** the switch-out staging batch and each
per-channel `DELETE TOP` batch run with an explicit command timeout
(`AuditLog:Purge:MaintenanceCommandTimeoutMinutes`, default 30, floor 1 min) rather
than the ~30 s ADO.NET default, which could abort the metadata-only SWITCH mid-dance
on a large or contended partition and leave the live table without
`UX_AuditLog_EventId` until a later tick's CATCH branch rebuilt it.
than the ~30 s ADO.NET default, which could abort the metadata-only SWITCH mid-batch
on a large or contended partition and leave an orphaned staging table for the next
tick's CATCH branch to clean up.
- **Partition-aligned uniqueness:** the switch no longer drops and rebuilds an index.
`EventId` uniqueness rides the clustered `PK_AuditLog (EventId, OccurredAtUtc)`,
which is aligned on `ps_AuditLog_Month(OccurredAtUtc)`, so `SWITCH PARTITION` has no
non-aligned unique index to object to. The predecessor `UX_AuditLog_EventId` forced
an offline whole-table index build inside the switch transaction — blocking every
audit writer for its duration — and left a window in which the idempotency-supporting
index did not exist at all. See migration `AlignAuditLogEventIdUniqueness`.
- **Per-channel retention overrides (M5.5 T3):** `AuditLog:PerChannelRetentionDays`
is a dictionary keyed by canonical channel name (`ApiOutbound`, `DbOutbound`,
`Notification`, `ApiInbound`, `SecuredWrite` — all five `AuditChannel` values are