fix: outbox KPI per-status legs, composite event-log indexes, AuditLog ingest FORCESEEK (#122, #123, #124)

#122: the Notification Outbox KPI snapshots ORed every status in one
aggregation, which scanned the clustered index once the table held ~1 M
Parked rows (0.7-2.6 s; Health dashboard up to 5 s). Each snapshot now runs
three status-equality legs that seek their own index (~0.12 s); a cold Health
load on the rig is 0.12-0.52 s.

#123: site_events event_type/severity indexes are now composites with
timestamp, so a filtered page is read in order and stops at the limit
(severity=Error over 1 GB: 2,287 -> 15 ms cold, 115 -> 1.6 ms warm). Costs
454 -> 530 B/row, ~14 % fewer rows under the same cap. The schema apply drops
the old indexes in place; the cap size model samples timestamp three times
and re-scores inside its band with unchanged constants.

#124: the set-based AuditLog insert's NOT EXISTS probe scanned the EventId
index and concurrent writers deadlocked on S/IX page locks. WITH (FORCESEEK):
4 writers 775 rows/s + 76 deadlocks in 30 s -> 5,214 rows/s + 0. New
concurrent-writer regression test fails 4/4 without the hint.
This commit is contained in:
Joseph Doherty
2026-11-10 16:46:42 -05:00
parent 2b80a6defa
commit 424db1555d
12 changed files with 386 additions and 158 deletions
+6 -2
View File
@@ -57,12 +57,16 @@ cmd.CommandText = """
details TEXT
);
CREATE INDEX IF NOT EXISTS idx_events_timestamp ON site_events(timestamp);
CREATE INDEX IF NOT EXISTS idx_events_type ON site_events(event_type);
CREATE INDEX IF NOT EXISTS idx_events_instance ON site_events(instance_id);
CREATE INDEX IF NOT EXISTS idx_events_severity ON site_events(severity);
DROP INDEX IF EXISTS idx_events_type;
DROP INDEX IF EXISTS idx_events_severity;
CREATE INDEX IF NOT EXISTS idx_events_type_timestamp ON site_events(event_type, timestamp);
CREATE INDEX IF NOT EXISTS idx_events_severity_timestamp ON site_events(severity, timestamp);
""";
```
The `event_type` and `severity` indexes are composites with `timestamp` (#123). Every query orders by `(timestamp, id)` and returns one page, so with a single-column index an equality filter read and sorted every matching row: `severity = Error` over a full 1 GB log took 2.3 s cold and 115 ms warm. The composite index returns rows already in timestamp order and stops at the page limit, in about 1 ms. The cost is about 69 bytes more index per row, which the storage-cap model charges, so a 1 GB cap now holds roughly 15 % fewer events. The two `DROP INDEX` statements migrate a pre-#123 file in place on its first boot; rebuilding the two indexes on a full 1 GB table takes seconds and happens once. Indexes are not part of LocalDb's replicated-table digest, so this change does not stop a mixed-version pair from syncing.
Keyword search (`KeywordFilter`) runs as `LIKE '%…%' ESCAPE '\'` on `message` and `source`. A leading-wildcard `LIKE` cannot use a B-tree index, so keyword-only queries full-scan the table. All other filters (`event_type`, `severity`, `instance_id`, `timestamp`) are covered by the indexes above.
### Connection lock
@@ -171,6 +171,24 @@ stream whose row width changes sharply *in the middle* of the history — neithe
tail — is mis-sized until the trim reaches it. Self-correcting in practice, since the cap is
re-evaluated every tick and the head sample advances as the oldest rows go.
### 3.2 Recalibration for the composite indexes (#123, 2026-11-10)
#123 replaced the single-column `event_type` and `severity` indexes with `(event_type, timestamp)`
and `(severity, timestamp)`. There are still four secondary indexes, so `IndexedColumnCount = 4`
stands. The sampler's `indexed_bytes` now counts `timestamp` three times: once for its own index
and once for each composite. The other constants are unchanged.
The model was re-scored with `EstimateSiteEventsSize_TracksTheRealFootprint`:
| Workload | estimated | actual | error | before #123 |
| --- | ---: | ---: | ---: | ---: |
| narrow rows, no details | 3 859 626 | 3 551 232 | **+8.7 %** | +8.4 % |
| typical rows, 200 B details | 4 520 096 | 4 005 888 | **+12.8 %** | +12.8 % |
| wide rows, 2 KB details | 13 012 048 | 13 078 528 | **−0.5 %** | −0.8 % |
The band and the high bias are unchanged. `SiteEventsSchemaCalibrationTests` pins the new index
set and the columns each index covers.
## 4. The wipe-prevention guard, restated
The register asked whether the "size did not decrease" guard should survive. It should, and it