Merge branch 'worktree-agent-a3b474b485c0288de' into arch-review-remediation

This commit is contained in:
Joseph Doherty
2026-08-14 21:15:31 -04:00
29 changed files with 3790 additions and 266 deletions
+32 -4
View File
@@ -291,6 +291,27 @@ stay `Pending` for the next sweep. Cadence is short (default 5 s) when
non-empty, longer (default 30 s) when idle; telemetry runs on a dedicated
dispatcher.
**Central-side ingest is set-based.** `AuditLogIngestActor` writes a whole
packet with ONE `InsertManyIfNotExistsAsync` statement rather than one
`IF NOT EXISTS … INSERT` round trip per event; the cached-telemetry dual-write
similarly runs the whole packet in ONE transaction (set-based audit insert plus
one single-statement `SiteCalls` upsert per entry) instead of a transaction per
entry. Idempotency is unchanged: duplicates that repeat *within* a packet
collapse first-write-wins before the statement is built, duplicates *across*
packets are eliminated by the anti-semi-join, and any failure falls back to the
per-row / per-entry path — so the documented invariant that one bad row cannot
sink the rest of the batch still holds, it is simply no longer paid for on the
healthy path.
**Timeout ladder.** The ingest budget is deliberately the smallest on the path:
the site's Ask and the central gRPC handler's Ask are both 30 s, the actor's
own database budget is 20 s and the per-statement SQL timeout is 15 s. Before
this the three were identical, so they expired at the same instant and the
caller learned nothing but "it took 30 s" — no partial ack, no way to tell a
slow database from a wedged singleton. With the inner budgets strictly smaller,
a slow batch is abandoned by the actor first and the accepted-so-far ids are
still replied while the outer Asks are still waiting.
### Reconciliation pull (self-healing for missed telemetry)
A central `SiteAuditReconciliationActor` periodically (default 5 min per site)
@@ -490,12 +511,19 @@ MS SQL for direct-write events). Unredacted secrets never persist.
silently failing retention job would otherwise be invisible until the table grew
unbounded. Per-boundary/per-channel error isolation is unchanged — a single failure
still never abandons the rest of the tick.
- **Maintenance command timeout:** the switch-out drop-and-rebuild dance and each
- **Maintenance command timeout:** the switch-out staging batch and each
per-channel `DELETE TOP` batch run with an explicit command timeout
(`AuditLog:Purge:MaintenanceCommandTimeoutMinutes`, default 30, floor 1 min) rather
than the ~30 s ADO.NET default, which could abort the metadata-only SWITCH mid-dance
on a large or contended partition and leave the live table without
`UX_AuditLog_EventId` until a later tick's CATCH branch rebuilt it.
than the ~30 s ADO.NET default, which could abort the metadata-only SWITCH mid-batch
on a large or contended partition and leave an orphaned staging table for the next
tick's CATCH branch to clean up.
- **Partition-aligned uniqueness:** the switch no longer drops and rebuilds an index.
`EventId` uniqueness rides the clustered `PK_AuditLog (EventId, OccurredAtUtc)`,
which is aligned on `ps_AuditLog_Month(OccurredAtUtc)`, so `SWITCH PARTITION` has no
non-aligned unique index to object to. The predecessor `UX_AuditLog_EventId` forced
an offline whole-table index build inside the switch transaction — blocking every
audit writer for its duration — and left a window in which the idempotency-supporting
index did not exist at all. See migration `AlignAuditLogEventIdUniqueness`.
- **Per-channel retention overrides (M5.5 T3):** `AuditLog:PerChannelRetentionDays`
is a dictionary keyed by canonical channel name (`ApiOutbound`, `DbOutbound`,
`Notification`, `ApiInbound`, `SecuredWrite` — all five `AuditChannel` values are
@@ -95,9 +95,11 @@ The configuration database stores all central system data, organized by domain a
- `ParentExecutionId``CAST(JSON_VALUE(DetailsJson,'$.parentExecutionId') AS uniqueidentifier)` PERSISTED — spawner's `ExecutionId`; null for top-level runs.
- `IngestedAtUtc``CAST(SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson,'$.ingestedAtUtc') AS datetimeoffset), 0) AS datetime2(7))` — central ingest timestamp; **not** persisted (SQL Server rejects PERSISTED on the non-deterministic `SWITCHOFFSET` expression).
*Clustered primary key:* `(EventId, OccurredAtUtc)` — composite so the key is partition-aligned. `UX_AuditLog_EventId` (unique, non-aligned on `[PRIMARY]`) enforces global `EventId` uniqueness for `InsertIfNotExistsAsync` idempotency.
*Clustered primary key:* `(EventId, OccurredAtUtc)` — composite so the key is partition-aligned, and the **sole** `EventId` uniqueness enforcement backing `InsertIfNotExistsAsync` idempotency. `EventId` is a GUID minted once at the emitting site in the same operation that stamps `OccurredAtUtc`, and neither field is ever re-stamped downstream, so a given `EventId` always arrives with the same `OccurredAtUtc` and can only map to one partition — pair-uniqueness is `EventId`-uniqueness for every row the system produces. The idempotency probe (`WHERE EventId = @id`) still seeks the clustered key's leading column, at the cost of one seek per partition rather than one seek overall.
*Indexes* (all non-clustered, partition-aligned on `ps_AuditLog_Month(OccurredAtUtc)` except `UX_AuditLog_EventId`):
A non-aligned single-column `UX_AuditLog_EventId` on `[PRIMARY]` used to carry that uniqueness; it was dropped by the `AlignAuditLogEventIdUniqueness` migration because a non-aligned index blocks `ALTER TABLE … SWITCH PARTITION`, forcing the retention purge to drop it, switch, and rebuild it **offline inside the switch transaction** on every run.
*Indexes* (all non-clustered, partition-aligned on `ps_AuditLog_Month(OccurredAtUtc)`):
- `IX_AuditLog_OccurredAtUtc` (primary time-range index for global scans)
- `IX_AuditLog_Site_Occurred (SourceSiteId, OccurredAtUtc)` (per-site filters)
- `IX_AuditLog_CorrelationId (CorrelationId) WHERE CorrelationId IS NOT NULL` (drilldown from a single operation)
@@ -135,6 +135,28 @@ Pinned)` on the EventStream (transition-only, mirroring
health-observable condition rather than a silent log line, and the latch clears
with `Pinned=false` once a later tick makes progress.
**The drain runs off the mailbox.** A reconciliation pass is unbounded work —
every site, up to a page ceiling of network pulls each, one upsert per row — so
it runs as a background task with a `PipeTo`-delivered completion message and a
single-flight guard, not as an actor message handler. A handler occupies the
actor for its whole duration, so a post-outage catch-up used to park telemetry
ingest, UI queries and KPI Asks behind it; those callers time out rather than
queue, which made a slow site look like a dead central. The single-flight guard
is raised and lowered ON the actor thread, so the per-site cursor and pinned-latch
dictionaries the pass mutates are still only ever touched by one task at a time,
and the mailbox supplies the memory barrier between consecutive passes. The
daily terminal-row purge uses the same shape. (`NotificationOutboxActor`'s
dispatch sweep is the in-repo reference.)
**The central upsert is one statement.** `SiteCallAuditRepository.UpsertAsync`
issues a single batch that runs the monotonic UPDATE first and INSERTs only when
nothing matched *and* the row genuinely does not exist — instead of an
unconditional insert-if-absent followed by the update, which cost two round trips
on every packet and wasted the insert half for every packet after the first. The
existence re-check is load-bearing: a zero row count also means "the monotonic
guard rejected this packet", and inserting there would fork the mirror with a
second row for an id that already exists.
## Retry / Discard Relay
Parked cached calls live in the owning site's S&F buffer. Operator Retry/Discard