feat(audit): flip AuditLog clustered key to (OccurredAtUtc, EventId) (ScadaBridge#111)

Variant B2 of the #78 benchmark: migration FlipAuditLogClusteredKey drops the twelve non-clustered indexes, re-adds PK_AuditLog as (OccurredAtUtc, EventId) on ps_AuditLog_Month, adds the aligned unique UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc) for the idempotency probe, and rebuilds the twelve indexes unchanged. Guarded (re-runnable) Up, inverse-rebuild Down; production scripts docs/plans/sql/FlipAuditLogClusteredKey{,.rollback}.sql. The partition-switch purge staging table now reads the live clustered key order from sys.index_columns, so it switches either shape (old binaries hard-code the old order: ship with the binaries, both centrals stopped). Evidence: 300k rows, page 1 90,532 -> 34 logical reads (ordered backward clustered scan under Top), keyset page 90,532 -> 46, probe 14 -> 14; FlipAuditLogClusteredKeyMigrationTests pins indexes, grants, Down/Up, the staging key (mutation: hard-coded old key fails SWITCH) and the read ratio. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 22:50:35 -05:00
parent 559e02f5c8
commit 5e7d5389d4
14 changed files with 3032 additions and 43 deletions
@@ -1,6 +1,6 @@
 # AuditLog clustered key benchmark and P4 / P6 follow-ups (ScadaBridge#78)

-Date: 2026-11-09. Status: benchmark record + recommendation only; no migration written.
+Date: 2026-11-09. Status: benchmark record + recommendation. **Implemented 2026-11-10 as migration `FlipAuditLogClusteredKey` (variant B2, #111)** — script `docs/plans/sql/FlipAuditLogClusteredKey.sql` + `.rollback.sql`; the partition-switch staging table now follows the live key order (the "old binaries work on either shape" claim below is wrong for that one statement — binaries before the flip cannot switch a flipped table). Reduced re-run, SQL Server 2022 dev container, 300k rows / 631 MB: page 1 90,532 → 34 logical reads (plan: `Clustered Index Scan … ORDERED BACKWARD` under `Top`, no sort), keyset page 90,532 → 46, idempotency probe 14 → 14; `FlipAuditLogClusteredKeyMigrationTests` pins the same ratio on 20k rows (76 vs 7,420 / 274 vs 7,393).

 ## P3 - flip `dbo.AuditLog` clustered key `(EventId, OccurredAtUtc)` -> `(OccurredAtUtc, EventId)`?

@@ -0,0 +1,50 @@
+-- FlipAuditLogClusteredKey ROLLBACK (ScadaBridge#111): the inverse rebuild back to clustered
+-- (EventId, OccurredAtUtc), UX_AuditLog_EventId_Occurred dropped, the twelve non-clustered
+-- indexes rebuilt unchanged. Same cost and window as the forward script; a no-op on a table that
+-- already has the old key. Binaries from FlipAuditLogClusteredKey on purge either shape.
+-- Run with QUOTED_IDENTIFIER ON (sqlcmd -I, or keep this header).
+SET QUOTED_IDENTIFIER ON;
+SET ANSI_NULLS ON;
+GO
+BEGIN TRANSACTION;
+
+IF 
+(SELECT c.name FROM sys.index_columns ic
+   JOIN sys.columns c ON c.object_id = ic.object_id AND c.column_id = ic.column_id
+  WHERE ic.object_id = OBJECT_ID('dbo.AuditLog') AND ic.index_id = 1 AND ic.key_ordinal = 1) = N'OccurredAtUtc'
+BEGIN
+
+DECLARE @dropSql nvarchar(max) = N'';
+SELECT @dropSql = @dropSql + N'DROP INDEX ' + QUOTENAME(i.name) + N' ON dbo.AuditLog;' + CHAR(10)
+FROM sys.indexes i
+WHERE i.object_id = OBJECT_ID('dbo.AuditLog') AND i.type = 2 AND i.name IN (
+    N'IX_AuditLog_OccurredAtUtc', N'IX_AuditLog_Site_Occurred', N'IX_AuditLog_CorrelationId',
+    N'IX_AuditLog_Channel_Status_Occurred', N'IX_AuditLog_Target_Occurred', N'IX_AuditLog_Execution',
+    N'IX_AuditLog_ParentExecution', N'IX_AuditLog_Node_Occurred', N'IX_AuditLog_Kind_Occurred',
+    N'IX_AuditLog_Actor_Occurred', N'IX_AuditLog_Unchained', N'IX_AuditLog_ChainSeq',
+    N'UX_AuditLog_EventId_Occurred');
+EXEC sp_executesql @dropSql;
+    ALTER TABLE dbo.AuditLog DROP CONSTRAINT PK_AuditLog;
+    ALTER TABLE dbo.AuditLog ADD CONSTRAINT PK_AuditLog PRIMARY KEY CLUSTERED (EventId, OccurredAtUtc)
+        ON ps_AuditLog_Month(OccurredAtUtc);
+
+CREATE NONCLUSTERED INDEX IX_AuditLog_OccurredAtUtc ON dbo.AuditLog (OccurredAtUtc DESC) INCLUDE (Status) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Site_Occurred ON dbo.AuditLog (SourceSiteId ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_CorrelationId ON dbo.AuditLog (CorrelationId) WHERE CorrelationId IS NOT NULL ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Channel_Status_Occurred ON dbo.AuditLog (Category ASC, Status ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Target_Occurred ON dbo.AuditLog (Target ASC, OccurredAtUtc DESC) WHERE Target IS NOT NULL ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Execution ON dbo.AuditLog (ExecutionId) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_ParentExecution ON dbo.AuditLog (ParentExecutionId) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Node_Occurred ON dbo.AuditLog (SourceNode, OccurredAtUtc) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Kind_Occurred ON dbo.AuditLog (Kind ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Actor_Occurred ON dbo.AuditLog (Actor ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Unchained ON dbo.AuditLog (OccurredAtUtc, EventId) WHERE ChainSeq IS NULL ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_ChainSeq ON dbo.AuditLog (ChainSeq) WHERE ChainSeq IS NOT NULL ON ps_AuditLog_Month(OccurredAtUtc);
+END
+
+DELETE FROM [__EFMigrationsHistory]
+WHERE [MigrationId] = N'20261110033732_FlipAuditLogClusteredKey';
+
+COMMIT;
+GO
+
@@ -0,0 +1,63 @@
+-- FlipAuditLogClusteredKey (2026-11-10, ScadaBridge#111, variant B2 of
+-- docs/plans/2026-11-09-auditlog-clustered-key-benchmark.md). dbo.AuditLog clustered key
+-- (EventId, OccurredAtUtc) -> (OccurredAtUtc, EventId) + aligned unique UX_AuditLog_EventId_Occurred
+-- (EventId, OccurredAtUtc); the twelve non-clustered indexes are dropped first and rebuilt last,
+-- unchanged and partition-aligned. Re-runnable (no-op on an already-flipped table).
+--
+-- OFFLINE whole-table rebuild: size the window from the real table first
+--   SELECT SUM(used_page_count)*8/1024 AS MB FROM sys.dm_db_partition_stats WHERE object_id = OBJECT_ID('dbo.AuditLog');
+-- (local reference: 631 MB / 300k rows rebuilt in ~9 s on an idle dev container; production I/O and
+-- log growth differ — the whole table is rewritten in one transaction, so the log needs ~2x the
+-- table size free). Take a backup first. Stop BOTH central nodes (old binaries' partition-switch
+-- purge cannot switch a flipped table: their staging table hard-codes the old key — msg 4947,
+-- rolled back, nothing purged). Rollback: FlipAuditLogClusteredKey.rollback.sql (same cost).
+--
+-- Run with QUOTED_IDENTIFIER ON (sqlcmd -I, or keep this header): AuditLog has persisted computed
+-- columns and filtered indexes and SQL Server raises error 1934 otherwise.
+SET QUOTED_IDENTIFIER ON;
+SET ANSI_NULLS ON;
+GO
+BEGIN TRANSACTION;
+
+IF 
+(SELECT c.name FROM sys.index_columns ic
+   JOIN sys.columns c ON c.object_id = ic.object_id AND c.column_id = ic.column_id
+  WHERE ic.object_id = OBJECT_ID('dbo.AuditLog') AND ic.index_id = 1 AND ic.key_ordinal = 1) = N'EventId'
+BEGIN
+
+DECLARE @dropSql nvarchar(max) = N'';
+SELECT @dropSql = @dropSql + N'DROP INDEX ' + QUOTENAME(i.name) + N' ON dbo.AuditLog;' + CHAR(10)
+FROM sys.indexes i
+WHERE i.object_id = OBJECT_ID('dbo.AuditLog') AND i.type = 2 AND i.name IN (
+    N'IX_AuditLog_OccurredAtUtc', N'IX_AuditLog_Site_Occurred', N'IX_AuditLog_CorrelationId',
+    N'IX_AuditLog_Channel_Status_Occurred', N'IX_AuditLog_Target_Occurred', N'IX_AuditLog_Execution',
+    N'IX_AuditLog_ParentExecution', N'IX_AuditLog_Node_Occurred', N'IX_AuditLog_Kind_Occurred',
+    N'IX_AuditLog_Actor_Occurred', N'IX_AuditLog_Unchained', N'IX_AuditLog_ChainSeq',
+    N'UX_AuditLog_EventId_Occurred');
+EXEC sp_executesql @dropSql;
+    ALTER TABLE dbo.AuditLog DROP CONSTRAINT PK_AuditLog;
+    ALTER TABLE dbo.AuditLog ADD CONSTRAINT PK_AuditLog PRIMARY KEY CLUSTERED (OccurredAtUtc, EventId)
+        ON ps_AuditLog_Month(OccurredAtUtc);
+    CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId_Occurred ON dbo.AuditLog (EventId, OccurredAtUtc)
+        ON ps_AuditLog_Month(OccurredAtUtc);
+
+CREATE NONCLUSTERED INDEX IX_AuditLog_OccurredAtUtc ON dbo.AuditLog (OccurredAtUtc DESC) INCLUDE (Status) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Site_Occurred ON dbo.AuditLog (SourceSiteId ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_CorrelationId ON dbo.AuditLog (CorrelationId) WHERE CorrelationId IS NOT NULL ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Channel_Status_Occurred ON dbo.AuditLog (Category ASC, Status ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Target_Occurred ON dbo.AuditLog (Target ASC, OccurredAtUtc DESC) WHERE Target IS NOT NULL ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Execution ON dbo.AuditLog (ExecutionId) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_ParentExecution ON dbo.AuditLog (ParentExecutionId) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Node_Occurred ON dbo.AuditLog (SourceNode, OccurredAtUtc) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Kind_Occurred ON dbo.AuditLog (Kind ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Actor_Occurred ON dbo.AuditLog (Actor ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_Unchained ON dbo.AuditLog (OccurredAtUtc, EventId) WHERE ChainSeq IS NULL ON ps_AuditLog_Month(OccurredAtUtc);
+CREATE NONCLUSTERED INDEX IX_AuditLog_ChainSeq ON dbo.AuditLog (ChainSeq) WHERE ChainSeq IS NOT NULL ON ps_AuditLog_Month(OccurredAtUtc);
+END
+
+INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
+VALUES (N'20261110033732_FlipAuditLogClusteredKey', N'10.0.13');
+
+COMMIT;
+GO
+
@@ -116,7 +116,8 @@ row per lifecycle event across all channels.

 **Indexes (first cut):**

- `IX_AuditLog_OccurredAtUtc` — primary time-range index for global scans.
+- Clustered `PK_AuditLog (OccurredAtUtc, EventId)` + unique `UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc)` (#111) — the time-leading clustered key serves the Audit Log page's newest-first TOP-N and keyset paging as an ordered range; the EventId-leading unique index serves the idempotency probe.
+- `IX_AuditLog_OccurredAtUtc` — covering (`INCLUDE (Status)`) time-range index for the KPI window.
 - `IX_AuditLog_Site_Occurred (SourceSiteId, OccurredAtUtc)` — per-site filters.
 - `IX_AuditLog_Node_Occurred (SourceNode, OccurredAtUtc)` — per-node filters ("everything `central-a` did in window X", or pinning a misbehaving site node).
 - `IX_AuditLog_CorrelationId (CorrelationId)` — drilldown from a single operation.
@@ -731,9 +732,13 @@ MS SQL for direct-write events). Unredacted secrets never persist.
  on a large or contended partition and leave an orphaned staging table for the next
  tick's CATCH branch to clean up.
 - **Partition-aligned uniqueness:** the switch no longer drops and rebuilds an index.
-  `EventId` uniqueness rides the clustered `PK_AuditLog (EventId, OccurredAtUtc)`,
-  which is aligned on `ps_AuditLog_Month(OccurredAtUtc)`, so `SWITCH PARTITION` has no
-  non-aligned unique index to object to. The predecessor `UX_AuditLog_EventId` forced
+  `EventId` uniqueness rides the clustered `PK_AuditLog (OccurredAtUtc, EventId)` and the
+  unique `UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc)` (migration
+  `FlipAuditLogClusteredKey`, #111 — before it, the PK alone, keyed `(EventId, OccurredAtUtc)`),
+  all aligned on `ps_AuditLog_Month(OccurredAtUtc)`, so `SWITCH PARTITION` has no
+  non-aligned unique index to object to. The staging table takes the live table's clustered
+  key order (read from `sys.index_columns` at purge time), so the purge works on either key
+  shape; binaries older than the flip hard-code the old order and cannot switch a flipped table. The predecessor `UX_AuditLog_EventId` forced
  an offline whole-table index build inside the switch transaction — blocking every
  audit writer for its duration — and left a window in which the idempotency-supporting
  index did not exist at all. See migration `AlignAuditLogEventIdUniqueness`.
@@ -69,7 +69,7 @@ The configuration database stores all central system data, organized by domain a

 - **New tables use UTC `DateTime` + the UTC value converter**, not `DateTimeOffset`. The `Notification` entity's `DateTimeOffset` timestamps are the **documented legacy exception** — on the SQLite test harness the `DateTimeOffset`→ISO-8601-string converter forced in-memory reductions in a few KPI paths (now pushed server-side where translatable). Prefer `datetime2` UTC columns for any new table.
 - **New repositories write MSSQL-only SQL with MSSQL-backed test fixtures** (`SkippableFact` gated on the live SQL Server container). The `NotificationOutboxRepository` dual SQLite/T-SQL dialect is the **documented legacy exception** (finding S8) — it predates the MSSQL-fixture convention; do not copy that pattern.
- **AuditLog clustered key.** The `dbo.AuditLog` primary/clustered key is `(EventId, OccurredAtUtc)` — `OccurredAtUtc` is required by the monthly partition scheme (`ps_AuditLog_Month`), and `EventId` carries a separate unique index for `InsertIfNotExists` idempotency. **Tracked follow-up (P3):** benchmark whether flipping the clustered key to `(OccurredAtUtc, EventId)` improves range-scan locality once real volume exists. → #78
+- **AuditLog clustered key.** The `dbo.AuditLog` primary/clustered key is `(OccurredAtUtc, EventId)` since migration `FlipAuditLogClusteredKey` (#111; variant B2 of the #78 benchmark, `docs/plans/2026-11-09-auditlog-clustered-key-benchmark.md`) — time-leading so the Audit Log page's newest-first TOP-N and keyset cursor read an ordered range instead of the whole partitioned table (300k-row local run: 90,532 → 34 logical reads for page 1, 90,532 → 46 for a keyset page). `OccurredAtUtc` is also the partition column of `ps_AuditLog_Month`. The aligned unique `UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc)` keeps the `InsertIfNotExists` idempotency probe a seek (14 reads, unchanged). Before the flip the key was `(EventId, OccurredAtUtc)`. Production script `docs/plans/sql/FlipAuditLogClusteredKey.sql` (+ `.rollback.sql`): an offline whole-table rebuild, run with both centrals stopped (old binaries' partition-switch staging table hard-codes the old key); DEV/QA sizing and the write window are tracked in #111.
 - **AuditLog reconciliation-pull paging (keyset, implemented).** `SiteAuditReconciliationActor` keeps a per-site `(Since, AfterId)` cursor and `GrpcPullAuditEventsClient` sends both `since_utc` and `after_id` on every `PullAuditEvents` call; the site (`ReadPendingSinceAsync`) serves rows strictly after the `(OccurredAtUtc, EventId)` pair when `after_id` is set and falls back to inclusive `>=` on the timestamp only for a first pull / empty `after_id`. Ingest stays idempotent on `EventId`, so the cursor is an exactness optimisation, not a correctness requirement. (The earlier note that central still sent a bare `sinceUtc` was stale.)
 - **KPI sampling cadence.** There is a single recorder cadence (`KpiHistory:SampleInterval`, default 60 s); per-site and per-node series are sampled in the same pass as the global ones, with no separate per-node cadence. See Component-KpiHistory.md. Benchmark record: `docs/plans/2026-11-09-auditlog-clustered-key-benchmark.md` (ScadaBridge#78).

@@ -98,11 +98,12 @@ The configuration database stores all central system data, organized by domain a
  - `ParentExecutionId` — `CAST(JSON_VALUE(DetailsJson,'$.parentExecutionId') AS uniqueidentifier)` PERSISTED — spawner's `ExecutionId`; null for top-level runs.
  - `IngestedAtUtc` — `CAST(SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson,'$.ingestedAtUtc') AS datetimeoffset), 0) AS datetime2(7))` — central ingest timestamp; **not** persisted (SQL Server rejects PERSISTED on the non-deterministic `SWITCHOFFSET` expression).

-  *Clustered primary key:* `(EventId, OccurredAtUtc)` — composite so the key is partition-aligned, and the **sole** `EventId` uniqueness enforcement backing `InsertIfNotExistsAsync` idempotency. `EventId` is a GUID minted once at the emitting site in the same operation that stamps `OccurredAtUtc`, and neither field is ever re-stamped downstream, so a given `EventId` always arrives with the same `OccurredAtUtc` and can only map to one partition — pair-uniqueness is `EventId`-uniqueness for every row the system produces. The idempotency probe (`WHERE EventId = @id`) still seeks the clustered key's leading column, at the cost of one seek per partition rather than one seek overall.
+  *Clustered primary key:* `(OccurredAtUtc, EventId)` (since `FlipAuditLogClusteredKey`, #111; previously `(EventId, OccurredAtUtc)`) — composite so the key is partition-aligned, time-leading so newest-first range reads seek. *EventId uniqueness:* `UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc)`, unique and partition-aligned, backing `InsertIfNotExistsAsync` idempotency (the PK enforces the same pair). `EventId` is a GUID minted once at the emitting site in the same operation that stamps `OccurredAtUtc`, and neither field is ever re-stamped downstream, so a given `EventId` always arrives with the same `OccurredAtUtc` and can only map to one partition — pair-uniqueness is `EventId`-uniqueness for every row the system produces. The idempotency probe (`WHERE EventId = @id`) seeks the unique index's leading column, at the cost of one seek per partition rather than one seek overall. The partition-switch purge reads the live clustered key order and builds its staging table to match, so it works on either shape.

  A non-aligned single-column `UX_AuditLog_EventId` on `[PRIMARY]` used to carry that uniqueness; it was dropped by the `AlignAuditLogEventIdUniqueness` migration because a non-aligned index blocks `ALTER TABLE … SWITCH PARTITION`, forcing the retention purge to drop it, switch, and rebuild it **offline inside the switch transaction** on every run.

-  *Indexes* (all non-clustered, partition-aligned on `ps_AuditLog_Month(OccurredAtUtc)`):
+  *Indexes* (all non-clustered, partition-aligned on `ps_AuditLog_Month(OccurredAtUtc)`; `FlipAuditLogClusteredKey` rebuilt all of them unchanged):
+  - `UX_AuditLog_EventId_Occurred (EventId, OccurredAtUtc)` UNIQUE (EventId uniqueness + idempotency probe, #111)
  - `IX_AuditLog_OccurredAtUtc` (primary time-range index for global scans)
  - `IX_AuditLog_Site_Occurred (SourceSiteId, OccurredAtUtc)` (per-site filters)
  - `IX_AuditLog_CorrelationId (CorrelationId) WHERE CorrelationId IS NOT NULL` (drilldown from a single operation)