fix: outbox KPI per-status legs, composite event-log indexes, AuditLog ingest FORCESEEK (#122, #123, #124)

#122: the Notification Outbox KPI snapshots ORed every status in one
aggregation, which scanned the clustered index once the table held ~1 M
Parked rows (0.7-2.6 s; Health dashboard up to 5 s). Each snapshot now runs
three status-equality legs that seek their own index (~0.12 s); a cold Health
load on the rig is 0.12-0.52 s.

#123: site_events event_type/severity indexes are now composites with
timestamp, so a filtered page is read in order and stops at the limit
(severity=Error over 1 GB: 2,287 -> 15 ms cold, 115 -> 1.6 ms warm). Costs
454 -> 530 B/row, ~14 % fewer rows under the same cap. The schema apply drops
the old indexes in place; the cap size model samples timestamp three times
and re-scores inside its band with unchanged constants.

#124: the set-based AuditLog insert's NOT EXISTS probe scanned the EventId
index and concurrent writers deadlocked on S/IX page locks. WITH (FORCESEEK):
4 writers 775 rows/s + 76 deadlocks in 30 s -> 5,214 rows/s + 0. New
concurrent-writer regression test fails 4/4 without the hint.
This commit is contained in:
Joseph Doherty
2026-11-10 16:46:42 -05:00
parent 2b80a6defa
commit 424db1555d
12 changed files with 386 additions and 158 deletions
@@ -250,7 +250,18 @@ VALUES
}
sql.Append("\n) AS v (").Append(CanonicalColumnList).Append(")\n");
sql.Append("WHERE NOT EXISTS (SELECT 1 FROM dbo.AuditLog x WHERE x.EventId = v.EventId);");
// FORCESEEK (#124): left to itself the optimizer answered the anti-semi-join for
// a 256-row packet by SCANNING the EventId index. Under READ COMMITTED a scan
// holds its current page's S lock while it requests the next page, so two
// concurrent inserts — each holding IX on the pages it had just written —
// deadlocked on each other's pages (error 1205; captured graph: two INSERTs,
// S/IX page locks on AuditLog). Measured with 4 concurrent writers on a fresh
// database (ResidualBench audit-contention): 76 victims in 30 s, 775 rows/s.
// One index seek per VALUES row locks only the key it reads, and only while
// reading it: 0 victims, 5,214 rows/s. A seek can still wait on another
// writer's uncommitted row with the SAME EventId — a genuine duplicate, which
// the duplicate-key fallback below already resolves.
sql.Append("WHERE NOT EXISTS (SELECT 1 FROM dbo.AuditLog x WITH (FORCESEEK) WHERE x.EventId = v.EventId);");
var conn = _context.Database.GetDbConnection();
var openedHere = false;