Joseph Doherty
5d075f1374
Six adversarial-review findings in the central SQL/ingest layer. F1 (AuditLogRepository.InsertChunkAsync) — the set-based ingest declared each string parameter at its COLUMN width (Actor/Target 256, Action 64, Outcome 16, Category 32, SourceNode 64), so SqlClient truncated an over-long value at bind time and committed the mutilated row — silent, in an append-only store, with no PayloadTruncated flag — while the per-row and reconciliation paths sent the same value in full and let the server reject it with 2628. Bind at the value's own length instead; explicit SqlDbType is kept (it fixes the VALUES constructor's derived column types and datetime2 precision). Design: reject everywhere, truncate nowhere — matching today's per-row behaviour. F2 (SiteCallAuditRepository.UpsertAsync) — the single-statement upsert ran the monotonic UPDATE first and INSERTed only if nothing matched. Two writers racing the first packet of one TrackedOperationId (the cached dual-write and the reconciliation pull carry DIFFERENT lifecycle states) both matched nothing, and the loser then skipped its INSERT or swallowed a 2627 — dropping its Status/RetryCount/HttpStatus/TerminalAtUtc. Legs swapped to `IF NOT EXISTS … INSERT; UPDATE <monotonic>` — still one round trip, and the loser's UPDATE now lands on the winner's row. The duplicate-key catch re-runs the monotonic UPDATE for the same reason. Moved to raw SQL with explicitly-typed parameters so the intricate rank predicate exists in exactly one place (an untyped DateTime would bind as `datetime` and round the freshness tiebreaker). F3 (docs/plans/sql/*.sql) — filtered-index DDL failed with error 1934 under the documented `docker exec … sqlcmd` path, which defaults QUOTED_IDENTIFIER OFF; once IX_Notifications_Delivered exists, QI-OFF DML on Notifications fails too. All four scripts now open with `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO` (own batch, so it is in force when the next batch parses), and the migration convention in Component-ConfigurationDatabase.md documents `sqlcmd -I`. Verified live: the pre-fix script fails 1934 without -I, the fixed one applies. F4 (SiteCallAuditActor) — the off-mailbox reconciliation/purge passes reuse the injected repository, so tests drove one DbContext from the pass and a mailbox handler concurrently. Serialized at the CALL via a private SerializedRepository wrapper applied only by the test constructors, rather than running the pass on-mailbox: production keeps its PipeTo shape untouched, and the existing "a blocked drain does not stall ingest/query/KPI" regression tests stay meaningful (they would have been invalidated by suspending the mailbox). F5 (AuditLogIngestActor) — when the batch failed because the 20 s IngestBudget expired, the per-row fallback reused the same expired token: N instant failures, N counter bumps, zero accepted. The fallback now gets a fresh 5 s budget (inside the 30 s outer Ask), and a blown budget bumps the failure counter ONCE for the batch instead of once per row. F6 (NotificationOutboxRepository.UpdateAsync) — ExecuteUpdate's row count was discarded, so an operator Retry/Discard of a notification the retention purge had already deleted reported success (the pre-ExecuteUpdate code threw DbUpdateConcurrencyException). UpdateAsync now returns whether a row matched; the operator one-shots answer "notification not found" and emit no audit row for the action that did not happen, while the dispatcher logs a warning (its delivery already happened; nothing to retry). GetByIdAsync switched to AsNoTracking since the write is out-of-band. Tests: 5 new SQL-backed regressions (over-long Target rejected on both paths + boundary round-trip; concurrent first-write and already-created-by-another-writer upserts; vanished-row UpdateAsync), a token-identity pin on the ingest fallback, a repository-concurrency detector for the SiteCallAudit passes, and vanished-row operator-path tests. The F1/F2/F4 regressions were each confirmed failing against the pre-fix code. Suites: ConfigurationDatabase 369, AuditLog 378, SiteCallAudit 66, NotificationOutbox 152 — all green, solution builds with 0 warnings.
247 lines
14 KiB
Markdown
247 lines
14 KiB
Markdown
# Component: Site Call Audit
|
||
|
||
## Purpose
|
||
|
||
Provides central, queryable audit and operational visibility for cached calls
|
||
made by site scripts — `ExternalSystem.CachedCall()` and `Database.CachedWrite()`.
|
||
Each such call carries a `TrackedOperationId`; sites report lifecycle telemetry
|
||
to this component, which maintains a central audit record, computes KPIs, and
|
||
relays Retry/Discard actions back to the owning site.
|
||
|
||
This is the second centrally-hosted observability component for site
|
||
store-and-forward activity (the Notification Outbox is the first). Unlike the
|
||
Notification Outbox, Site Call Audit is **not a dispatcher** — it never delivers
|
||
anything. Cached calls are delivered by the site's Store-and-Forward Engine
|
||
against site-local external systems and databases, which central cannot reach.
|
||
|
||
## Location
|
||
|
||
Central cluster only. A singleton actor (`SiteCallAuditActor`) on the active
|
||
central node. Registered as component #22 in the Host role configuration.
|
||
|
||
## Responsibilities
|
||
|
||
- Ingest cached-call lifecycle telemetry from sites into the central `SiteCalls`
|
||
table.
|
||
- Run periodic per-site reconciliation pulls so missed telemetry self-heals.
|
||
- Compute point-in-time KPIs (global and per-site) from the `SiteCalls` table.
|
||
- Relay operator Retry/Discard actions for parked cached calls to the owning
|
||
site over the command/control channel.
|
||
- Purge terminal audit rows after a configurable retention window.
|
||
|
||
## The `SiteCalls` Table
|
||
|
||
Lives in the central MS SQL configuration database — a sibling of the
|
||
`Notifications` table. One row per `TrackedOperationId` (the shipped columns, as
|
||
mapped by `SiteCallEntityTypeConfiguration` — the source of truth is
|
||
`Component-ConfigurationDatabase.md` § Site Calls):
|
||
|
||
- **TrackedOperationId** — GUID (`varchar(36)`, "D"-format), primary key. Generated
|
||
site-side at call time.
|
||
- **Channel** — `varchar(32)`, the trust-boundary channel that produced the call:
|
||
`ApiOutbound` (`ExternalSystem.CachedCall()`) or `DbOutbound` (`Database.CachedWrite()`).
|
||
- **Target** — `varchar(256)`, human-readable target (e.g. `ERP.GetOrder` for an
|
||
external call, or the database connection name for a cached write — intentionally
|
||
not the SQL statement or table, a deliberate scoping choice).
|
||
- **SourceSite** — `varchar(64)`, site that issued the call.
|
||
- **SourceNode** — `varchar(64)` NULL, the cluster node on which the call was issued
|
||
(`node-a` / `node-b`, qualified by `SourceSite`). Stamped site-side at submit time
|
||
and carried verbatim through the combined `CachedCallTelemetry` packet,
|
||
reconciliation pulls, and the central upsert; NULL for reconciled rows from a
|
||
retired node.
|
||
- **Status** — `varchar(32)`, the `AuditStatus` enum name (**not** the tracking-lifecycle
|
||
names): `Submitted`, `Forwarded`, `Attempted`, `Delivered`, `Failed`, `Parked`,
|
||
`Discarded`. The lifecycle is monotonic, so out-of-order/at-least-once telemetry is
|
||
harmless.
|
||
- **RetryCount** — `int`, attempts so far.
|
||
- **LastError** — `nvarchar(1024)` NULL, most recent error detail, if any.
|
||
- **HttpStatus** — `int` NULL, last HTTP status code for API calls.
|
||
- **CreatedAtUtc**, **UpdatedAtUtc** (`datetime2`), **TerminalAtUtc** (`datetime2` NULL),
|
||
**IngestedAtUtc** (`datetime2`, central ingest timestamp) — key timestamps.
|
||
|
||
There are **no `Kind`, `TargetSummary`, or provenance (instance/script) columns** —
|
||
those were an earlier design that did not ship; provenance detail for a cached call
|
||
lives on the site's own tracking store and in the AuditLog rows.
|
||
|
||
## Status Lifecycle
|
||
|
||
`Pending → Retrying → Delivered / Parked / Failed / Discarded`
|
||
|
||
> **Stored vs. tracking view.** This lifecycle is the operator-facing *tracking*
|
||
> view (what the site's tracking store and `Tracking.Status()` express). The
|
||
> persisted `SiteCalls.Status` **column** stores the monotonic `AuditStatus`-derived
|
||
> string (`Submitted` / `Forwarded` / `Attempted` / `Delivered` / `Failed` / `Parked`
|
||
> / `Discarded`) carried by the combined telemetry packet — the mirror records the
|
||
> audit-event status, not the tracking enum name. The two agree on the terminal
|
||
> outcomes; the non-terminal `Submitted`/`Forwarded`/`Attempted` strings are the
|
||
> ingest-phase equivalents of `Pending`/`Retrying`.
|
||
|
||
- **Pending** — non-terminal: buffered after a transient failure, awaiting its
|
||
first retry.
|
||
- **Retrying** — non-terminal: undergoing retry attempts.
|
||
- **Delivered** — terminal, success. A cached call that succeeds on its first
|
||
immediate attempt is recorded directly as `Delivered`.
|
||
- **Parked** — non-terminal: transient retries exhausted; awaiting manual action.
|
||
- **Failed** — terminal: permanent failure (e.g. HTTP 4xx). The error was also
|
||
returned synchronously to the calling script; the record captures it. `Failed`
|
||
rows are **not operator-actionable** — see Retry / Discard Relay.
|
||
- **Discarded** — terminal, reached **only by operator action** on a `Parked`
|
||
row. The row is kept (not deleted) so the table remains a complete audit
|
||
record.
|
||
|
||
The site is the source of truth. The `SiteCalls` row is an eventually-consistent
|
||
mirror — never queried by scripts (`Tracking.Status()` is answered site-locally).
|
||
|
||
## Ingest & Idempotency
|
||
|
||
Telemetry ingestion is **insert-if-not-exists** keyed on `TrackedOperationId`,
|
||
then **upsert-on-newer-status, with a newest-`UpdatedAtUtc` tiebreaker within
|
||
equal non-terminal rank**. The lifecycle is monotonic on status rank, so status
|
||
never regresses. Within an equal *non-terminal* rank (the `Attempted`/`Skipped`
|
||
retry phase), the packet with the newest `UpdatedAtUtc` wins — so a retrying
|
||
call's `RetryCount`/`LastError`/`HttpStatus` stay live instead of freezing at the
|
||
first `Attempted` write. Equal *terminal* rank stays immutable (a later
|
||
`Delivered` never overwrites an earlier `Parked`), equal stamps are an idempotent
|
||
no-op, and a lower rank is always a no-op — so at-least-once and out-of-order
|
||
telemetry remain harmless.
|
||
|
||
From v1.x onward, the `CachedCallTelemetry` message additively carries the
|
||
`AuditEvent` content alongside the existing operational fields. Central's
|
||
`AuditLogIngestActor` (Audit Log #23) performs both the immutable `AuditLog`
|
||
insert and the `SiteCalls` upsert in a single transaction. Idempotency keys
|
||
remain `EventId` (for `AuditLog`) and `TrackedOperationId` (for `SiteCalls`).
|
||
See [Component-AuditLog.md](Component-AuditLog.md), Cached Operations —
|
||
Combined Telemetry, for the dual-write contract.
|
||
|
||
## Reconciliation
|
||
|
||
Because telemetry is best-effort, `SiteCallAuditActor` periodically — and on site
|
||
reconnect — pulls "all tracking rows changed since cursor X" from each site.
|
||
Gaps left by lost telemetry self-heal. Central converges to the site; the site
|
||
never depends on central.
|
||
|
||
The per-site cursor is a **composite `(UpdatedAtUtc, TrackedOperationId)`
|
||
keyset**, not a single timestamp. Each pull asks for rows strictly greater than
|
||
the cursor pair and advances it to the maximum row seen; a burst of more rows
|
||
than one batch all sharing one exact `UpdatedAtUtc` therefore drains via the
|
||
`TrackedOperationId` tiebreak instead of pinning the timestamp forever. The
|
||
`after_id` keyset field is additive on the pull contract — a first pull (or a
|
||
**legacy** site that predates it) sends no `after_id` and keeps the inclusive
|
||
`>=` timestamp behaviour. When such a legacy site keeps reporting
|
||
`MoreAvailable` yet the composite cursor cannot advance, the actor latches the
|
||
site as *pinned* and publishes `SiteCallReconciliationPinnedChanged(siteId,
|
||
Pinned)` on the EventStream (transition-only, mirroring
|
||
`SiteAuditTelemetryStalledChanged`) — the un-drainable tail is a
|
||
health-observable condition rather than a silent log line, and the latch clears
|
||
with `Pinned=false` once a later tick makes progress.
|
||
|
||
**The drain runs off the mailbox.** A reconciliation pass is unbounded work —
|
||
every site, up to a page ceiling of network pulls each, one upsert per row — so
|
||
it runs as a background task with a `PipeTo`-delivered completion message and a
|
||
single-flight guard, not as an actor message handler. A handler occupies the
|
||
actor for its whole duration, so a post-outage catch-up used to park telemetry
|
||
ingest, UI queries and KPI Asks behind it; those callers time out rather than
|
||
queue, which made a slow site look like a dead central. The single-flight guard
|
||
is raised and lowered ON the actor thread, so the per-site cursor and pinned-latch
|
||
dictionaries the pass mutates are still only ever touched by one task at a time,
|
||
and the mailbox supplies the memory barrier between consecutive passes. The
|
||
daily terminal-row purge uses the same shape. (`NotificationOutboxActor`'s
|
||
dispatch sweep is the in-repo reference.)
|
||
|
||
**The central upsert is one round trip, INSERT-first.**
|
||
`SiteCallAuditRepository.UpsertAsync` ships both statements in a single command
|
||
text — `IF NOT EXISTS … INSERT;` then the monotonic `UPDATE` — so a packet costs
|
||
one round trip, not two. The insert leg is gated on the row's existence *alone*,
|
||
never on "the update matched nothing": a zero row count also means "the monotonic
|
||
guard rejected this packet", and inserting there would fork the mirror with a
|
||
second row for an id that already exists.
|
||
|
||
The ORDER is load-bearing, and running the UPDATE first is a data-loss bug. The
|
||
two writers — the cached dual-write and the reconciliation pull — routinely carry
|
||
*different* lifecycle states for the same `TrackedOperationId`, and both can race
|
||
its very first packet. Under UPDATE-first both find no row, so both updates match
|
||
nothing; the loser then either fails its existence re-check and skips its insert,
|
||
or attempts it and takes a duplicate-key fault — either way its
|
||
`Status`/`RetryCount`/`HttpStatus`/`TerminalAtUtc` are dropped, because it never
|
||
ran an update against the winner's row. Under INSERT-first the loser's insert is
|
||
skipped or faults and its monotonic update still applies to whichever row won, so
|
||
the newer state survives every interleaving while a stale one is still rejected by
|
||
the rank guard. The duplicate-key catch re-runs the monotonic update for the same
|
||
reason (it is idempotent and rank-guarded, so a redundant re-run is inert).
|
||
|
||
## Retry / Discard Relay
|
||
|
||
Parked cached calls live in the owning site's S&F buffer. Operator Retry/Discard
|
||
from the Central UI is relayed to that site as a `RetryParkedOperation` /
|
||
`DiscardParkedOperation` command over the command/control channel. The site
|
||
applies the change and emits telemetry reflecting the new state; central never
|
||
mutates the `SiteCalls` row directly. If the site is offline the command fails
|
||
fast and the UI surfaces a "site unreachable" message.
|
||
|
||
On a **successful** relay (the site acks `Applied`), `SiteCallAuditActor` emits
|
||
one best-effort central **direct-write** audit row (`CachedResolve`, status
|
||
`Submitted` for a Retry / `Discarded` for a Discard) carrying the authenticated
|
||
operator as `Actor` and the `TrackedOperationId` as `CorrelationId` — recording
|
||
*who asked*. The operator identity flows in on `RetrySiteCallRequest` /
|
||
`DiscardSiteCallRequest` (`RequestedBy`, captured at the Central UI). This row
|
||
only adds provenance: the **site remains the source of truth** for the state
|
||
change itself, and central reads the stored `SiteCalls` row solely to enrich the
|
||
audit row's channel/target (a benign read, never a mutation). Audit is
|
||
best-effort — a writer fault never changes the relay outcome.
|
||
|
||
Only `Parked` rows are operator-actionable. `Failed` rows offer no Retry or
|
||
Discard: a permanent failure (e.g. HTTP 4xx) would simply fail again, and the
|
||
error was already returned synchronously to the calling script — there is
|
||
nothing for an operator to recover.
|
||
|
||
## KPIs
|
||
|
||
Point-in-time, computed from the `SiteCalls` table, global and per-source-site,
|
||
mirroring the Notification Outbox KPI shape:
|
||
|
||
- Buffered count (`Pending` + `Retrying`)
|
||
- Parked count
|
||
- Failed-last-interval
|
||
- Delivered-last-interval
|
||
- Oldest-pending age
|
||
- Stuck count — `Pending`/`Retrying` older than a configurable threshold
|
||
(default 10 minutes); display-only, no escalation.
|
||
|
||
## Retention
|
||
|
||
Daily purge of terminal rows (`Delivered`, `Failed`, `Discarded`) after a
|
||
configurable window (default 365 days), matching the `Notifications` purge.
|
||
|
||
## Dependencies
|
||
|
||
- **Configuration Database**: hosts the `SiteCalls` table and its repository.
|
||
- **Central–Site Communication**: receives cached-call telemetry and reconciliation
|
||
responses; sends Retry/Discard commands.
|
||
- **Store-and-Forward Engine**: the site-side origin of cached-call telemetry and
|
||
the executor of relayed Retry/Discard commands.
|
||
- **Audit Log (#23)**: shares the `CachedCallTelemetry` packet — each lifecycle
|
||
transition (`CachedEnqueued`, `CachedAttempt`, `CachedTerminal`) carries an
|
||
`AuditEvent` alongside the operational fields, and central's
|
||
`AuditLogIngestActor` performs the `AuditLog` insert and the `SiteCalls`
|
||
upsert in a single transaction (see [Component-AuditLog.md](Component-AuditLog.md),
|
||
Cached Operations — Combined Telemetry).
|
||
- **Commons**: `TrackedOperationId`, status enum, telemetry message contracts.
|
||
|
||
## Interactions
|
||
|
||
- **Central UI**: the Site Calls page queries this component and issues
|
||
Retry/Discard actions.
|
||
- **Health Monitoring**: surfaces Site Call Audit KPI tiles on the dashboard.
|
||
- **Cluster Infrastructure**: hosts the `SiteCallAuditActor` singleton with
|
||
active/standby failover.
|
||
- **KPI History (#26)**: emits `IKpiSampleSource`
|
||
(`SiteCallAuditKpiSampleSource`, Global + per-Site + per-Node) consumed by the
|
||
KpiHistory recorder (#26), reusing the existing KPI reads. All six metrics —
|
||
`buffered` / `parked` / `failedLastInterval` / `deliveredLastInterval` /
|
||
`stuck` / `oldestPendingAgeSeconds` — are sampled into the `KpiSample` history
|
||
store, but only the three charted via the public `KpiMetrics.SiteCallAudit`
|
||
catalog (`buffered` / `parked` / `failedLastInterval`) render as trends on the
|
||
Site Calls page via `KpiTrendChart`; `deliveredLastInterval` / `stuck` /
|
||
`oldestPendingAgeSeconds` are sampled-but-not-yet-charted (available for future
|
||
trend panels / ad-hoc query). See [Component-KpiHistory.md](Component-KpiHistory.md).
|