Merge branch 'worktree-agent-a3b474b485c0288de' into arch-review-remediation
This commit is contained in:
@@ -291,6 +291,27 @@ stay `Pending` for the next sweep. Cadence is short (default 5 s) when
|
||||
non-empty, longer (default 30 s) when idle; telemetry runs on a dedicated
|
||||
dispatcher.
|
||||
|
||||
**Central-side ingest is set-based.** `AuditLogIngestActor` writes a whole
|
||||
packet with ONE `InsertManyIfNotExistsAsync` statement rather than one
|
||||
`IF NOT EXISTS … INSERT` round trip per event; the cached-telemetry dual-write
|
||||
similarly runs the whole packet in ONE transaction (set-based audit insert plus
|
||||
one single-statement `SiteCalls` upsert per entry) instead of a transaction per
|
||||
entry. Idempotency is unchanged: duplicates that repeat *within* a packet
|
||||
collapse first-write-wins before the statement is built, duplicates *across*
|
||||
packets are eliminated by the anti-semi-join, and any failure falls back to the
|
||||
per-row / per-entry path — so the documented invariant that one bad row cannot
|
||||
sink the rest of the batch still holds, it is simply no longer paid for on the
|
||||
healthy path.
|
||||
|
||||
**Timeout ladder.** The ingest budget is deliberately the smallest on the path:
|
||||
the site's Ask and the central gRPC handler's Ask are both 30 s, the actor's
|
||||
own database budget is 20 s and the per-statement SQL timeout is 15 s. Before
|
||||
this the three were identical, so they expired at the same instant and the
|
||||
caller learned nothing but "it took 30 s" — no partial ack, no way to tell a
|
||||
slow database from a wedged singleton. With the inner budgets strictly smaller,
|
||||
a slow batch is abandoned by the actor first and the accepted-so-far ids are
|
||||
still replied while the outer Asks are still waiting.
|
||||
|
||||
### Reconciliation pull (self-healing for missed telemetry)
|
||||
|
||||
A central `SiteAuditReconciliationActor` periodically (default 5 min per site)
|
||||
@@ -490,12 +511,19 @@ MS SQL for direct-write events). Unredacted secrets never persist.
|
||||
silently failing retention job would otherwise be invisible until the table grew
|
||||
unbounded. Per-boundary/per-channel error isolation is unchanged — a single failure
|
||||
still never abandons the rest of the tick.
|
||||
- **Maintenance command timeout:** the switch-out drop-and-rebuild dance and each
|
||||
- **Maintenance command timeout:** the switch-out staging batch and each
|
||||
per-channel `DELETE TOP` batch run with an explicit command timeout
|
||||
(`AuditLog:Purge:MaintenanceCommandTimeoutMinutes`, default 30, floor 1 min) rather
|
||||
than the ~30 s ADO.NET default, which could abort the metadata-only SWITCH mid-dance
|
||||
on a large or contended partition and leave the live table without
|
||||
`UX_AuditLog_EventId` until a later tick's CATCH branch rebuilt it.
|
||||
than the ~30 s ADO.NET default, which could abort the metadata-only SWITCH mid-batch
|
||||
on a large or contended partition and leave an orphaned staging table for the next
|
||||
tick's CATCH branch to clean up.
|
||||
- **Partition-aligned uniqueness:** the switch no longer drops and rebuilds an index.
|
||||
`EventId` uniqueness rides the clustered `PK_AuditLog (EventId, OccurredAtUtc)`,
|
||||
which is aligned on `ps_AuditLog_Month(OccurredAtUtc)`, so `SWITCH PARTITION` has no
|
||||
non-aligned unique index to object to. The predecessor `UX_AuditLog_EventId` forced
|
||||
an offline whole-table index build inside the switch transaction — blocking every
|
||||
audit writer for its duration — and left a window in which the idempotency-supporting
|
||||
index did not exist at all. See migration `AlignAuditLogEventIdUniqueness`.
|
||||
- **Per-channel retention overrides (M5.5 T3):** `AuditLog:PerChannelRetentionDays`
|
||||
is a dictionary keyed by canonical channel name (`ApiOutbound`, `DbOutbound`,
|
||||
`Notification`, `ApiInbound`, `SecuredWrite` — all five `AuditChannel` values are
|
||||
|
||||
@@ -95,9 +95,11 @@ The configuration database stores all central system data, organized by domain a
|
||||
- `ParentExecutionId` — `CAST(JSON_VALUE(DetailsJson,'$.parentExecutionId') AS uniqueidentifier)` PERSISTED — spawner's `ExecutionId`; null for top-level runs.
|
||||
- `IngestedAtUtc` — `CAST(SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson,'$.ingestedAtUtc') AS datetimeoffset), 0) AS datetime2(7))` — central ingest timestamp; **not** persisted (SQL Server rejects PERSISTED on the non-deterministic `SWITCHOFFSET` expression).
|
||||
|
||||
*Clustered primary key:* `(EventId, OccurredAtUtc)` — composite so the key is partition-aligned. `UX_AuditLog_EventId` (unique, non-aligned on `[PRIMARY]`) enforces global `EventId` uniqueness for `InsertIfNotExistsAsync` idempotency.
|
||||
*Clustered primary key:* `(EventId, OccurredAtUtc)` — composite so the key is partition-aligned, and the **sole** `EventId` uniqueness enforcement backing `InsertIfNotExistsAsync` idempotency. `EventId` is a GUID minted once at the emitting site in the same operation that stamps `OccurredAtUtc`, and neither field is ever re-stamped downstream, so a given `EventId` always arrives with the same `OccurredAtUtc` and can only map to one partition — pair-uniqueness is `EventId`-uniqueness for every row the system produces. The idempotency probe (`WHERE EventId = @id`) still seeks the clustered key's leading column, at the cost of one seek per partition rather than one seek overall.
|
||||
|
||||
*Indexes* (all non-clustered, partition-aligned on `ps_AuditLog_Month(OccurredAtUtc)` except `UX_AuditLog_EventId`):
|
||||
A non-aligned single-column `UX_AuditLog_EventId` on `[PRIMARY]` used to carry that uniqueness; it was dropped by the `AlignAuditLogEventIdUniqueness` migration because a non-aligned index blocks `ALTER TABLE … SWITCH PARTITION`, forcing the retention purge to drop it, switch, and rebuild it **offline inside the switch transaction** on every run.
|
||||
|
||||
*Indexes* (all non-clustered, partition-aligned on `ps_AuditLog_Month(OccurredAtUtc)`):
|
||||
- `IX_AuditLog_OccurredAtUtc` (primary time-range index for global scans)
|
||||
- `IX_AuditLog_Site_Occurred (SourceSiteId, OccurredAtUtc)` (per-site filters)
|
||||
- `IX_AuditLog_CorrelationId (CorrelationId) WHERE CorrelationId IS NOT NULL` (drilldown from a single operation)
|
||||
|
||||
@@ -135,6 +135,28 @@ Pinned)` on the EventStream (transition-only, mirroring
|
||||
health-observable condition rather than a silent log line, and the latch clears
|
||||
with `Pinned=false` once a later tick makes progress.
|
||||
|
||||
**The drain runs off the mailbox.** A reconciliation pass is unbounded work —
|
||||
every site, up to a page ceiling of network pulls each, one upsert per row — so
|
||||
it runs as a background task with a `PipeTo`-delivered completion message and a
|
||||
single-flight guard, not as an actor message handler. A handler occupies the
|
||||
actor for its whole duration, so a post-outage catch-up used to park telemetry
|
||||
ingest, UI queries and KPI Asks behind it; those callers time out rather than
|
||||
queue, which made a slow site look like a dead central. The single-flight guard
|
||||
is raised and lowered ON the actor thread, so the per-site cursor and pinned-latch
|
||||
dictionaries the pass mutates are still only ever touched by one task at a time,
|
||||
and the mailbox supplies the memory barrier between consecutive passes. The
|
||||
daily terminal-row purge uses the same shape. (`NotificationOutboxActor`'s
|
||||
dispatch sweep is the in-repo reference.)
|
||||
|
||||
**The central upsert is one statement.** `SiteCallAuditRepository.UpsertAsync`
|
||||
issues a single batch that runs the monotonic UPDATE first and INSERTs only when
|
||||
nothing matched *and* the row genuinely does not exist — instead of an
|
||||
unconditional insert-if-absent followed by the update, which cost two round trips
|
||||
on every packet and wasted the insert half for every packet after the first. The
|
||||
existence re-check is load-bearing: a zero row count also means "the monotonic
|
||||
guard rejected this packet", and inserting there would fork the mirror with a
|
||||
second row for an id that already exists.
|
||||
|
||||
## Retry / Discard Relay
|
||||
|
||||
Parked cached calls live in the owning site's S&F buffer. Operator Retry/Discard
|
||||
|
||||
Reference in New Issue
Block a user