diff --git a/docs/plans/2026-11-09-auditlog-clustered-key-benchmark.md b/docs/plans/2026-11-09-auditlog-clustered-key-benchmark.md index 4fc7c034..8f6da78b 100644 --- a/docs/plans/2026-11-09-auditlog-clustered-key-benchmark.md +++ b/docs/plans/2026-11-09-auditlog-clustered-key-benchmark.md @@ -22,9 +22,9 @@ Queries measured cold (DROPCLEANBUFFERS) and warm with `SET STATISTICS IO, TIME` |---|---|---|---|---|---| | A (current) | 4,022 | 1,122,830 | 735,066 / 5,743 | 99.2% | 64.0% | | B (no EventId index) | 7,631 | 816,961 | 500,001 / 3,906 | 0.6% | 94.2% | -| B2 (B + unique EventId NC) | 4,065 | 1,141,726 | 735,769 / 5,748 (this is the EventId NC index; clustered was compact) | 99.1% (EventId index) | 63.9% | +| B2 (B + unique EventId NC) | 4,065 | 1,141,726 | 735,769 / 5,748 | 99.1% | 63.9% | -(For B2 the first row of `sys.dm_db_index_physical_stats` sorted by size was the new unique EventId index; the clustered index itself behaved like B.) The NC indexes keyed on site/target/channel etc. fragment similarly in every variant because their leading columns are not time-ordered; B's total page-split count is lower but still large for that reason. +B2's clustered index ended up as fragmented as A's even though its key is time-ordered. The most likely cause (plan not inspected) is that the `NOT EXISTS (EventId)` probe against the new EventId index makes the optimizer feed the insert in EventId order, so the clustered inserts become random again; the unique EventId index itself is 16,099 pages / 99.2% fragmented. The NC indexes keyed on site/target/channel etc. fragment in every variant because their leading columns are not time-ordered. ### Query shapes (logical reads, warm elapsed ms; cold in brackets) | Shape | A | B | B2 | @@ -48,12 +48,12 @@ Important side finding: in variant A the unfiltered/time-range Audit Log queries ### Findings 1. The flip is a large, real win for read paths, orders of magnitude on the Audit Log page (the default view) and on every time-range query, plus ~13x fewer reads for the hash-chain job's selection, and sequential month reads. -2. The flip alone also roughly halves insert cost and removes clustered fragmentation (0.6% vs 99.2%, fill 94% vs 64%, 32% smaller table) **because the clustered insert becomes append-only**. -3. But without an EventId index, idempotent ingest (`WHERE NOT EXISTS (EventId)`) degrades to scanning a narrow NC index (B) - the cost grows with table size, and the table loses its uniqueness guarantee. Restoring it with a unique aligned `(EventId, OccurredAtUtc)` index (B2) re-introduces a random-GUID B-tree: insert throughput and fragmentation then match A exactly. So with B2 the flip does **not** help ingest, and the page-split bill moves from the clustered index to the EventId index. +2. The flip alone (B) roughly halves insert cost and removes clustered fragmentation (0.6% vs 99.2%, fill 94% vs 64%, 32% smaller table) because the clustered insert becomes append-only - but B is not viable without an EventId index (finding 3). +3. But without an EventId index, idempotent ingest (`WHERE NOT EXISTS (EventId)`) degrades to scanning a narrow NC index (B) - the cost grows with table size, and the table loses its uniqueness guarantee. Restoring it with a unique aligned `(EventId, OccurredAtUtc)` index (B2) re-introduces a random-GUID B-tree, and in this test also made the clustered inserts random: insert throughput, size and fragmentation matched A. So with B2 the flip does **not** help ingest (no regression either). Whether an insert path that does not sort by EventId (e.g. skipping the NOT EXISTS probe and catching 2627) would keep the clustered side append-only was not tested. 4. Chain stamp (Q12) and purge (Q15) are unchanged within noise; the chain verify read is better in B, equal in B2. ### Recommendation -**Flip: yes, as B2 (clustered `(OccurredAtUtc, EventId)` + aligned unique NC `(EventId, OccurredAtUtc)`), but schedule it deliberately - it is a priority-appropriate (P3) performance fix, not an urgent one, and the read win (default Audit Log page doing a full-table scan) justifies it as table size grows.** B alone is not recommended (no cheap idempotency probe, no uniqueness guard). Ingest stays at today's cost under B2 (no regression), reads improve by 2-4 orders of magnitude. An optional later step: probe idempotency by the NC index only for rows that might be duplicates (e.g. reconciliation), and drop the probe on the hot telemetry path in favour of catching 2627, which would recover most of B's insert win - not benchmarked here. +**Flip: yes, as B2 (clustered `(OccurredAtUtc, EventId)` + aligned unique NC `(EventId, OccurredAtUtc)`), but schedule it deliberately - it is a priority-appropriate (P3) performance fix, not an urgent one, and the read win (default Audit Log page doing a full-table scan) justifies it as table size grows.** B alone is not recommended (no cheap idempotency probe, no uniqueness guard). Ingest stays at today's cost under B2 (no regression), reads improve by 2-4 orders of magnitude. An optional later experiment: drop the pre-insert probe on the hot telemetry path in favour of catching 2627, which might recover B's insert win - not benchmarked. ### Migration cost, risk and compatibility (not implemented) - Rebuild is an offline-ish data move of the whole table (`DROP/CREATE` clustered index, or the `CollapseAuditLogToCanonical`-style new-table-and-copy pattern on the preserved partition scheme). Every one of the 12 NC indexes is rebuilt too (locator changes). On a large table that is hours of I/O and log, and a write outage for the audit path unless `ONLINE = ON` (Enterprise/Developer only; the deployment does not guarantee it - see `AlignAuditLogEventIdUniqueness`). Prefer a maintenance window or per-partition copy while draining ingest (site buffers retain unforwarded rows, so a window is tolerable).