Merge branch 'worktree-agent-a3b474b485c0288de' into arch-review-remediation

docs/components/ConfigurationDatabase.md

@@ -78,13 +78,15 @@ builder.Property(e => e.ExecutionId)
         "CAST(JSON_VALUE(DetailsJson,'$.executionId') AS uniqueidentifier)", stored: true)
     .ValueGeneratedOnAddOrUpdate();
 
-// Composite PK includes OccurredAtUtc for partition alignment
+// Composite PK includes OccurredAtUtc for partition alignment — and is the ONLY
+// EventId uniqueness enforcement (WP2.2 / AlignAuditLogEventIdUniqueness). EventId
+// is a GUID minted once at the site alongside OccurredAtUtc, so a given EventId
+// always lands in one partition and pair-uniqueness is EventId-uniqueness.
 builder.HasKey(e => new { e.EventId, e.OccurredAtUtc });
-
-builder.HasIndex(e => e.EventId).IsUnique()
-    .HasDatabaseName("UX_AuditLog_EventId");
 ```
 
+The predecessor `UX_AuditLog_EventId` — a single-column unique index on `[PRIMARY]`, deliberately NOT partition-aligned — was dropped: a non-aligned index blocks `ALTER TABLE … SWITCH PARTITION`, so the retention purge had to drop it, switch, and rebuild it offline inside the switch transaction on every run.
+
 **`TemplateConfiguration`** (representative of the domain-area configs) sets up the self-referencing parent FK, folder FK, cascade-delete relationships to attributes/alarms/scripts/compositions/native alarm sources, and the filtered unique index that enforces name uniqueness only on non-derived (base) templates.
 
 **`SiteCallEntityTypeConfiguration`** maps `SiteCall` to `dbo.SiteCalls` with a `TrackedOperationId` PK stored as `varchar(36)` (GUID in `"D"` format) so the column shape matches the wire format and the site SQLite store — one consistent format for operational debugging.
@@ -108,11 +110,13 @@ VALUES
 
 `FormattableString` interpolation parameterises every value so there is no injection surface. SQL error numbers `2601` and `2627` (unique-index violation) are swallowed as no-ops because the IF NOT EXISTS check has a race window; both the check-loser and the retrying telemetry path are semantically correct duplicates.
 
+`InsertManyIfNotExistsAsync` is the set-based form the ingest hot path uses: one `INSERT … SELECT … FROM (VALUES …) WHERE NOT EXISTS` per chunk of 100 rows (ten bound parameters per row against SQL Server's 2,100-parameter ceiling), so a telemetry packet costs one round trip instead of one per event. Because the anti-semi-join only sees committed rows, the packet is de-duplicated by `EventId` in C# first (first-write-wins, matching the single-row contract); a duplicate-key fault from a concurrent writer falls back to the per-row path so the batch is never a correctness dependency. It uses raw ADO.NET with explicitly typed `SqlParameter`s — the `VALUES` constructor derives its column types from the first row's parameters, so an untyped null would give the derived column the wrong type — and enlists in the DbContext's ambient transaction when one is open (the cached-telemetry dual-write).
+
 `QueryAsync` builds LINQ predicates over `AuditLogRow` using `AsNoTracking()`, translating filter dimensions (`Channels`, `Kinds`, `Statuses`, `SourceSiteIds`, `SourceNodes`, `ExecutionId`, `ParentExecutionId`, time range) to server-side SQL IN/equality predicates and using keyset pagination on `(OccurredAtUtc DESC, EventId DESC)`.
 
 `GetExecutionTreeAsync` walks the `ParentExecutionId` graph in two phases: a loop climbs to the root (bounded at 32 levels), then a recursive CTE descends the full tree and LEFT JOINs back to `AuditLog` so stub nodes (purged or row-less executions) still appear with `RowCount = 0`.
 
-`SwitchOutPartitionAsync` executes a drop-and-rebuild dance — dropping `UX_AuditLog_EventId`, creating a byte-identical staging table (including the computed-column definitions), switching the target partition to staging, dropping staging, and rebuilding the unique index — all inside a single `BEGIN TRY / BEGIN CATCH` block that guarantees the index is present whether the switch succeeds or rolls back.
+`SwitchOutPartitionAsync` creates a byte-identical staging table (including the computed-column definitions), switches the target partition to staging, and drops staging, inside a single `BEGIN TRY / BEGIN CATCH` block whose CATCH cleans up the staging table on any failure. No index is dropped or rebuilt: uniqueness rides the partition-aligned clustered PK, so the switch is metadata-only. A guarded defensive `DROP INDEX UX_AuditLog_EventId` remains at the head of the batch purely so a database restored from a pre-alignment backup still purges.
 
 ### IAuditService — config-change audit
 
@@ -241,7 +245,7 @@ The host is running in production mode and `GetPendingMigrationsAsync` found una
 
 ### AuditLog partition switch fails mid-operation
 
-`SwitchOutPartitionAsync` wraps the drop-and-rebuild dance in `BEGIN TRY / BEGIN CATCH`. On failure the CATCH block drops the staging table if it exists and rebuilds `UX_AuditLog_EventId` if it was dropped before the failure. The original exception is re-thrown so the Audit Log purge actor logs it and retries on the next daily tick. Verify that the `scadabridge_audit_purger` role still holds `ALTER ON SCHEMA::dbo` if the operation fails with a permissions error.
+`SwitchOutPartitionAsync` wraps the staging/switch batch in `BEGIN TRY / BEGIN CATCH`. On failure the CATCH block drops the staging table if it exists, so no orphaned `AuditLog_Staging_*` object is left behind. There is no index to repair — uniqueness lives on the clustered PK, which the switch never touches. The original exception is re-thrown so the Audit Log purge actor logs it and retries on the next daily tick. Verify that the `scadabridge_audit_purger` role still holds `ALTER ON SCHEMA::dbo` if the operation fails with a permissions error.
 
 ### Design-time `dotnet ef` tooling cannot find a connection string
 

docs/plans/sql/AlignAuditLogEventIdUniqueness.sql

@@ -0,0 +1,24 @@
+BEGIN TRANSACTION;
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20260815004957_AlignAuditLogEventIdUniqueness'
+)
+BEGIN
+
+    IF EXISTS (SELECT 1 FROM sys.indexes
+               WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog'))
+        DROP INDEX UX_AuditLog_EventId ON dbo.AuditLog;
+END;
+
+IF NOT EXISTS (
+    SELECT * FROM [__EFMigrationsHistory]
+    WHERE [MigrationId] = N'20260815004957_AlignAuditLogEventIdUniqueness'
+)
+BEGIN
+    INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
+    VALUES (N'20260815004957_AlignAuditLogEventIdUniqueness', N'10.0.7');
+END;
+
+COMMIT;
+GO
+

docs/requirements/Component-AuditLog.md

@@ -291,6 +291,27 @@ stay `Pending` for the next sweep. Cadence is short (default 5 s) when
 non-empty, longer (default 30 s) when idle; telemetry runs on a dedicated
 dispatcher.
 
+**Central-side ingest is set-based.** `AuditLogIngestActor` writes a whole
+packet with ONE `InsertManyIfNotExistsAsync` statement rather than one
+`IF NOT EXISTS … INSERT` round trip per event; the cached-telemetry dual-write
+similarly runs the whole packet in ONE transaction (set-based audit insert plus
+one single-statement `SiteCalls` upsert per entry) instead of a transaction per
+entry. Idempotency is unchanged: duplicates that repeat *within* a packet
+collapse first-write-wins before the statement is built, duplicates *across*
+packets are eliminated by the anti-semi-join, and any failure falls back to the
+per-row / per-entry path — so the documented invariant that one bad row cannot
+sink the rest of the batch still holds, it is simply no longer paid for on the
+healthy path.
+
+**Timeout ladder.** The ingest budget is deliberately the smallest on the path:
+the site's Ask and the central gRPC handler's Ask are both 30 s, the actor's
+own database budget is 20 s and the per-statement SQL timeout is 15 s. Before
+this the three were identical, so they expired at the same instant and the
+caller learned nothing but "it took 30 s" — no partial ack, no way to tell a
+slow database from a wedged singleton. With the inner budgets strictly smaller,
+a slow batch is abandoned by the actor first and the accepted-so-far ids are
+still replied while the outer Asks are still waiting.
+
 ### Reconciliation pull (self-healing for missed telemetry)
 
 A central `SiteAuditReconciliationActor` periodically (default 5 min per site)
@@ -490,12 +511,19 @@ MS SQL for direct-write events). Unredacted secrets never persist.
   silently failing retention job would otherwise be invisible until the table grew
   unbounded. Per-boundary/per-channel error isolation is unchanged — a single failure
   still never abandons the rest of the tick.
-- **Maintenance command timeout:** the switch-out drop-and-rebuild dance and each
+- **Maintenance command timeout:** the switch-out staging batch and each
   per-channel `DELETE TOP` batch run with an explicit command timeout
   (`AuditLog:Purge:MaintenanceCommandTimeoutMinutes`, default 30, floor 1 min) rather
-  than the ~30 s ADO.NET default, which could abort the metadata-only SWITCH mid-dance
-  on a large or contended partition and leave the live table without
-  `UX_AuditLog_EventId` until a later tick's CATCH branch rebuilt it.
+  than the ~30 s ADO.NET default, which could abort the metadata-only SWITCH mid-batch
+  on a large or contended partition and leave an orphaned staging table for the next
+  tick's CATCH branch to clean up.
+- **Partition-aligned uniqueness:** the switch no longer drops and rebuilds an index.
+  `EventId` uniqueness rides the clustered `PK_AuditLog (EventId, OccurredAtUtc)`,
+  which is aligned on `ps_AuditLog_Month(OccurredAtUtc)`, so `SWITCH PARTITION` has no
+  non-aligned unique index to object to. The predecessor `UX_AuditLog_EventId` forced
+  an offline whole-table index build inside the switch transaction — blocking every
+  audit writer for its duration — and left a window in which the idempotency-supporting
+  index did not exist at all. See migration `AlignAuditLogEventIdUniqueness`.
 - **Per-channel retention overrides (M5.5 T3):** `AuditLog:PerChannelRetentionDays`
   is a dictionary keyed by canonical channel name (`ApiOutbound`, `DbOutbound`,
   `Notification`, `ApiInbound`, `SecuredWrite` — all five `AuditChannel` values are

docs/requirements/Component-ConfigurationDatabase.md

@@ -95,9 +95,11 @@ The configuration database stores all central system data, organized by domain a
   - `ParentExecutionId` — `CAST(JSON_VALUE(DetailsJson,'$.parentExecutionId') AS uniqueidentifier)` PERSISTED — spawner's `ExecutionId`; null for top-level runs.
   - `IngestedAtUtc` — `CAST(SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson,'$.ingestedAtUtc') AS datetimeoffset), 0) AS datetime2(7))` — central ingest timestamp; **not** persisted (SQL Server rejects PERSISTED on the non-deterministic `SWITCHOFFSET` expression).
 
-  *Clustered primary key:* `(EventId, OccurredAtUtc)` — composite so the key is partition-aligned. `UX_AuditLog_EventId` (unique, non-aligned on `[PRIMARY]`) enforces global `EventId` uniqueness for `InsertIfNotExistsAsync` idempotency.
+  *Clustered primary key:* `(EventId, OccurredAtUtc)` — composite so the key is partition-aligned, and the **sole** `EventId` uniqueness enforcement backing `InsertIfNotExistsAsync` idempotency. `EventId` is a GUID minted once at the emitting site in the same operation that stamps `OccurredAtUtc`, and neither field is ever re-stamped downstream, so a given `EventId` always arrives with the same `OccurredAtUtc` and can only map to one partition — pair-uniqueness is `EventId`-uniqueness for every row the system produces. The idempotency probe (`WHERE EventId = @id`) still seeks the clustered key's leading column, at the cost of one seek per partition rather than one seek overall.
 
-  *Indexes* (all non-clustered, partition-aligned on `ps_AuditLog_Month(OccurredAtUtc)` except `UX_AuditLog_EventId`):
+  A non-aligned single-column `UX_AuditLog_EventId` on `[PRIMARY]` used to carry that uniqueness; it was dropped by the `AlignAuditLogEventIdUniqueness` migration because a non-aligned index blocks `ALTER TABLE … SWITCH PARTITION`, forcing the retention purge to drop it, switch, and rebuild it **offline inside the switch transaction** on every run.
+
+  *Indexes* (all non-clustered, partition-aligned on `ps_AuditLog_Month(OccurredAtUtc)`):
   - `IX_AuditLog_OccurredAtUtc` (primary time-range index for global scans)
   - `IX_AuditLog_Site_Occurred (SourceSiteId, OccurredAtUtc)` (per-site filters)
   - `IX_AuditLog_CorrelationId (CorrelationId) WHERE CorrelationId IS NOT NULL` (drilldown from a single operation)

docs/requirements/Component-SiteCallAudit.md

@@ -135,6 +135,28 @@ Pinned)` on the EventStream (transition-only, mirroring
 health-observable condition rather than a silent log line, and the latch clears
 with `Pinned=false` once a later tick makes progress.
 
+**The drain runs off the mailbox.** A reconciliation pass is unbounded work —
+every site, up to a page ceiling of network pulls each, one upsert per row — so
+it runs as a background task with a `PipeTo`-delivered completion message and a
+single-flight guard, not as an actor message handler. A handler occupies the
+actor for its whole duration, so a post-outage catch-up used to park telemetry
+ingest, UI queries and KPI Asks behind it; those callers time out rather than
+queue, which made a slow site look like a dead central. The single-flight guard
+is raised and lowered ON the actor thread, so the per-site cursor and pinned-latch
+dictionaries the pass mutates are still only ever touched by one task at a time,
+and the mailbox supplies the memory barrier between consecutive passes. The
+daily terminal-row purge uses the same shape. (`NotificationOutboxActor`'s
+dispatch sweep is the in-repo reference.)
+
+**The central upsert is one statement.** `SiteCallAuditRepository.UpsertAsync`
+issues a single batch that runs the monotonic UPDATE first and INSERTs only when
+nothing matched *and* the row genuinely does not exist — instead of an
+unconditional insert-if-absent followed by the update, which cost two round trips
+on every packet and wasted the insert half for every packet after the first. The
+existence re-check is load-bearing: a zero row count also means "the monotonic
+guard rejected this packet", and inserting there would fork the mirror with a
+second row for an id that already exists.
+
 ## Retry / Discard Relay
 
 Parked cached calls live in the owning site's S&F buffer. Operator Retry/Discard