feat(audit): flip AuditLog clustered key to (OccurredAtUtc, EventId) (ScadaBridge#111)
Variant B2 of the #78 benchmark: migration FlipAuditLogClusteredKey drops the twelve non-clustered indexes, re-adds PK_AuditLog as (OccurredAtUtc, EventId) on ps_AuditLog_Month, adds the aligned unique UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc) for the idempotency probe, and rebuilds the twelve indexes unchanged. Guarded (re-runnable) Up, inverse-rebuild Down; production scripts docs/plans/sql/FlipAuditLogClusteredKey{,.rollback}.sql. The partition-switch purge staging table now reads the live clustered key order from sys.index_columns, so it switches either shape (old binaries hard-code the old order: ship with the binaries, both centrals stopped). Evidence: 300k rows, page 1 90,532 -> 34 logical reads (ordered backward clustered scan under Top), keyset page 90,532 -> 46, probe 14 -> 14; FlipAuditLogClusteredKeyMigrationTests pins indexes, grants, Down/Up, the staging key (mutation: hard-coded old key fails SWITCH) and the read ratio. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -69,7 +69,7 @@ The configuration database stores all central system data, organized by domain a
|
||||
|
||||
- **New tables use UTC `DateTime` + the UTC value converter**, not `DateTimeOffset`. The `Notification` entity's `DateTimeOffset` timestamps are the **documented legacy exception** — on the SQLite test harness the `DateTimeOffset`→ISO-8601-string converter forced in-memory reductions in a few KPI paths (now pushed server-side where translatable). Prefer `datetime2` UTC columns for any new table.
|
||||
- **New repositories write MSSQL-only SQL with MSSQL-backed test fixtures** (`SkippableFact` gated on the live SQL Server container). The `NotificationOutboxRepository` dual SQLite/T-SQL dialect is the **documented legacy exception** (finding S8) — it predates the MSSQL-fixture convention; do not copy that pattern.
|
||||
- **AuditLog clustered key.** The `dbo.AuditLog` primary/clustered key is `(EventId, OccurredAtUtc)` — `OccurredAtUtc` is required by the monthly partition scheme (`ps_AuditLog_Month`), and `EventId` carries a separate unique index for `InsertIfNotExists` idempotency. **Tracked follow-up (P3):** benchmark whether flipping the clustered key to `(OccurredAtUtc, EventId)` improves range-scan locality once real volume exists. → #78
|
||||
- **AuditLog clustered key.** The `dbo.AuditLog` primary/clustered key is `(OccurredAtUtc, EventId)` since migration `FlipAuditLogClusteredKey` (#111; variant B2 of the #78 benchmark, `docs/plans/2026-11-09-auditlog-clustered-key-benchmark.md`) — time-leading so the Audit Log page's newest-first TOP-N and keyset cursor read an ordered range instead of the whole partitioned table (300k-row local run: 90,532 → 34 logical reads for page 1, 90,532 → 46 for a keyset page). `OccurredAtUtc` is also the partition column of `ps_AuditLog_Month`. The aligned unique `UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc)` keeps the `InsertIfNotExists` idempotency probe a seek (14 reads, unchanged). Before the flip the key was `(EventId, OccurredAtUtc)`. Production script `docs/plans/sql/FlipAuditLogClusteredKey.sql` (+ `.rollback.sql`): an offline whole-table rebuild, run with both centrals stopped (old binaries' partition-switch staging table hard-codes the old key); DEV/QA sizing and the write window are tracked in #111.
|
||||
- **AuditLog reconciliation-pull paging (keyset, implemented).** `SiteAuditReconciliationActor` keeps a per-site `(Since, AfterId)` cursor and `GrpcPullAuditEventsClient` sends both `since_utc` and `after_id` on every `PullAuditEvents` call; the site (`ReadPendingSinceAsync`) serves rows strictly after the `(OccurredAtUtc, EventId)` pair when `after_id` is set and falls back to inclusive `>=` on the timestamp only for a first pull / empty `after_id`. Ingest stays idempotent on `EventId`, so the cursor is an exactness optimisation, not a correctness requirement. (The earlier note that central still sent a bare `sinceUtc` was stale.)
|
||||
- **KPI sampling cadence.** There is a single recorder cadence (`KpiHistory:SampleInterval`, default 60 s); per-site and per-node series are sampled in the same pass as the global ones, with no separate per-node cadence. See Component-KpiHistory.md. Benchmark record: `docs/plans/2026-11-09-auditlog-clustered-key-benchmark.md` (ScadaBridge#78).
|
||||
|
||||
@@ -98,11 +98,12 @@ The configuration database stores all central system data, organized by domain a
|
||||
- `ParentExecutionId` — `CAST(JSON_VALUE(DetailsJson,'$.parentExecutionId') AS uniqueidentifier)` PERSISTED — spawner's `ExecutionId`; null for top-level runs.
|
||||
- `IngestedAtUtc` — `CAST(SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson,'$.ingestedAtUtc') AS datetimeoffset), 0) AS datetime2(7))` — central ingest timestamp; **not** persisted (SQL Server rejects PERSISTED on the non-deterministic `SWITCHOFFSET` expression).
|
||||
|
||||
*Clustered primary key:* `(EventId, OccurredAtUtc)` — composite so the key is partition-aligned, and the **sole** `EventId` uniqueness enforcement backing `InsertIfNotExistsAsync` idempotency. `EventId` is a GUID minted once at the emitting site in the same operation that stamps `OccurredAtUtc`, and neither field is ever re-stamped downstream, so a given `EventId` always arrives with the same `OccurredAtUtc` and can only map to one partition — pair-uniqueness is `EventId`-uniqueness for every row the system produces. The idempotency probe (`WHERE EventId = @id`) still seeks the clustered key's leading column, at the cost of one seek per partition rather than one seek overall.
|
||||
*Clustered primary key:* `(OccurredAtUtc, EventId)` (since `FlipAuditLogClusteredKey`, #111; previously `(EventId, OccurredAtUtc)`) — composite so the key is partition-aligned, time-leading so newest-first range reads seek. *EventId uniqueness:* `UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc)`, unique and partition-aligned, backing `InsertIfNotExistsAsync` idempotency (the PK enforces the same pair). `EventId` is a GUID minted once at the emitting site in the same operation that stamps `OccurredAtUtc`, and neither field is ever re-stamped downstream, so a given `EventId` always arrives with the same `OccurredAtUtc` and can only map to one partition — pair-uniqueness is `EventId`-uniqueness for every row the system produces. The idempotency probe (`WHERE EventId = @id`) seeks the unique index's leading column, at the cost of one seek per partition rather than one seek overall. The partition-switch purge reads the live clustered key order and builds its staging table to match, so it works on either shape.
|
||||
|
||||
A non-aligned single-column `UX_AuditLog_EventId` on `[PRIMARY]` used to carry that uniqueness; it was dropped by the `AlignAuditLogEventIdUniqueness` migration because a non-aligned index blocks `ALTER TABLE … SWITCH PARTITION`, forcing the retention purge to drop it, switch, and rebuild it **offline inside the switch transaction** on every run.
|
||||
|
||||
*Indexes* (all non-clustered, partition-aligned on `ps_AuditLog_Month(OccurredAtUtc)`):
|
||||
*Indexes* (all non-clustered, partition-aligned on `ps_AuditLog_Month(OccurredAtUtc)`; `FlipAuditLogClusteredKey` rebuilt all of them unchanged):
|
||||
- `UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc)` UNIQUE (EventId uniqueness + idempotency probe, #111)
|
||||
- `IX_AuditLog_OccurredAtUtc` (primary time-range index for global scans)
|
||||
- `IX_AuditLog_Site_Occurred (SourceSiteId, OccurredAtUtc)` (per-site filters)
|
||||
- `IX_AuditLog_CorrelationId (CorrelationId) WHERE CorrelationId IS NOT NULL` (drilldown from a single operation)
|
||||
|
||||
Reference in New Issue
Block a user