# Component: Site Call Audit ## Purpose Provides central, queryable audit and operational visibility for cached calls made by site scripts — `ExternalSystem.CachedCall()` and `Database.CachedWrite()`. Each such call carries a `TrackedOperationId`; sites report lifecycle telemetry to this component, which maintains a central audit record, computes KPIs, and relays Retry/Discard actions back to the owning site. This is the second centrally-hosted observability component for site store-and-forward activity (the Notification Outbox is the first). Unlike the Notification Outbox, Site Call Audit is **not a dispatcher** — it never delivers anything. Cached calls are delivered by the site's Store-and-Forward Engine against site-local external systems and databases, which central cannot reach. ## Location Central cluster only. A singleton actor (`SiteCallAuditActor`) on the active central node. Registered as component #22 in the Host role configuration. ## Responsibilities - Ingest cached-call lifecycle telemetry from sites into the central `SiteCalls` table. - Run periodic per-site reconciliation pulls so missed telemetry self-heals. - Compute point-in-time KPIs (global and per-site) from the `SiteCalls` table. - Relay operator Retry/Discard actions for parked cached calls to the owning site over the command/control channel. - Purge terminal audit rows after a configurable retention window. ## The `SiteCalls` Table Lives in the central MS SQL configuration database — a sibling of the `Notifications` table. One row per `TrackedOperationId` (the shipped columns, as mapped by `SiteCallEntityTypeConfiguration` — the source of truth is `Component-ConfigurationDatabase.md` § Site Calls): - **TrackedOperationId** — GUID (`varchar(36)`, "D"-format), primary key. Generated site-side at call time. - **Channel** — `varchar(32)`, the trust-boundary channel that produced the call: `ApiOutbound` (`ExternalSystem.CachedCall()`) or `DbOutbound` (`Database.CachedWrite()`). - **Target** — `varchar(256)`, human-readable target (e.g. `ERP.GetOrder` for an external call, or the database connection name for a cached write — intentionally not the SQL statement or table, a deliberate scoping choice). - **SourceSite** — `varchar(64)`, site that issued the call. - **SourceNode** — `varchar(64)` NULL, the cluster node on which the call was issued (`node-a` / `node-b`, qualified by `SourceSite`). Stamped site-side at submit time and carried verbatim through the combined `CachedCallTelemetry` packet, reconciliation pulls, and the central upsert; NULL for reconciled rows from a retired node. - **Status** — `varchar(32)`, the `AuditStatus` enum name (**not** the tracking-lifecycle names): `Submitted`, `Forwarded`, `Attempted`, `Delivered`, `Failed`, `Parked`, `Discarded`. The lifecycle is monotonic, so out-of-order/at-least-once telemetry is harmless. - **RetryCount** — `int`, attempts so far. - **LastError** — `nvarchar(1024)` NULL, most recent error detail, if any. - **HttpStatus** — `int` NULL, last HTTP status code for API calls. - **CreatedAtUtc**, **UpdatedAtUtc** (`datetime2`), **TerminalAtUtc** (`datetime2` NULL), **IngestedAtUtc** (`datetime2`, central ingest timestamp) — key timestamps. There are **no `Kind`, `TargetSummary`, or provenance (instance/script) columns** — those were an earlier design that did not ship; provenance detail for a cached call lives on the site's own tracking store and in the AuditLog rows. ## Status Lifecycle `Pending → Retrying → Delivered / Parked / Failed / Discarded` > **Stored vs. tracking view.** This lifecycle is the operator-facing *tracking* > view (what the site's tracking store and `Tracking.Status()` express). The > persisted `SiteCalls.Status` **column** stores the monotonic `AuditStatus`-derived > string (`Submitted` / `Forwarded` / `Attempted` / `Delivered` / `Failed` / `Parked` > / `Discarded`) carried by the combined telemetry packet — the mirror records the > audit-event status, not the tracking enum name. The two agree on the terminal > outcomes; the non-terminal `Submitted`/`Forwarded`/`Attempted` strings are the > ingest-phase equivalents of `Pending`/`Retrying`. - **Pending** — non-terminal: buffered after a transient failure, awaiting its first retry. - **Retrying** — non-terminal: undergoing retry attempts. - **Delivered** — terminal, success. A cached call that succeeds on its first immediate attempt is recorded directly as `Delivered`. - **Parked** — non-terminal: transient retries exhausted; awaiting manual action. - **Failed** — terminal: permanent failure (e.g. HTTP 4xx). The error was also returned synchronously to the calling script; the record captures it. `Failed` rows are **not operator-actionable** — see Retry / Discard Relay. - **Discarded** — terminal, reached **only by operator action** on a `Parked` row. The row is kept (not deleted) so the table remains a complete audit record. The site is the source of truth. The `SiteCalls` row is an eventually-consistent mirror — never queried by scripts (`Tracking.Status()` is answered site-locally). ## Ingest & Idempotency Telemetry ingestion is **insert-if-not-exists** keyed on `TrackedOperationId`, then **upsert-on-newer-status, with a newest-`UpdatedAtUtc` tiebreaker within equal non-terminal rank**. The lifecycle is monotonic on status rank, so status never regresses. Within an equal *non-terminal* rank (the `Attempted`/`Skipped` retry phase), the packet with the newest `UpdatedAtUtc` wins — so a retrying call's `RetryCount`/`LastError`/`HttpStatus` stay live instead of freezing at the first `Attempted` write. Equal *terminal* rank stays immutable (a later `Delivered` never overwrites an earlier `Parked`), equal stamps are an idempotent no-op, and a lower rank is always a no-op — so at-least-once and out-of-order telemetry remain harmless. From v1.x onward, the `CachedCallTelemetry` message additively carries the `AuditEvent` content alongside the existing operational fields. Central's `AuditLogIngestActor` (Audit Log #23) performs both the immutable `AuditLog` insert and the `SiteCalls` upsert in a single transaction. Idempotency keys remain `EventId` (for `AuditLog`) and `TrackedOperationId` (for `SiteCalls`). See [Component-AuditLog.md](Component-AuditLog.md), Cached Operations — Combined Telemetry, for the dual-write contract. ## Reconciliation Because telemetry is best-effort, `SiteCallAuditActor` periodically — and on site reconnect — pulls "all tracking rows changed since cursor X" from each site. Gaps left by lost telemetry self-heal. Central converges to the site; the site never depends on central. The per-site cursor is a **composite `(UpdatedAtUtc, TrackedOperationId)` keyset**, not a single timestamp. Each pull asks for rows strictly greater than the cursor pair and advances it to the maximum row seen; a burst of more rows than one batch all sharing one exact `UpdatedAtUtc` therefore drains via the `TrackedOperationId` tiebreak instead of pinning the timestamp forever. The `after_id` keyset field is additive on the pull contract — a first pull (or a **legacy** site that predates it) sends no `after_id` and keeps the inclusive `>=` timestamp behaviour. When such a legacy site keeps reporting `MoreAvailable` yet the composite cursor cannot advance, the actor latches the site as *pinned* and publishes `SiteCallReconciliationPinnedChanged(siteId, Pinned)` on the EventStream (transition-only, mirroring `SiteAuditTelemetryStalledChanged`) — the un-drainable tail is a health-observable condition rather than a silent log line, and the latch clears with `Pinned=false` once a later tick makes progress. ## Retry / Discard Relay Parked cached calls live in the owning site's S&F buffer. Operator Retry/Discard from the Central UI is relayed to that site as a `RetryParkedOperation` / `DiscardParkedOperation` command over the command/control channel. The site applies the change and emits telemetry reflecting the new state; central never mutates the `SiteCalls` row directly. If the site is offline the command fails fast and the UI surfaces a "site unreachable" message. On a **successful** relay (the site acks `Applied`), `SiteCallAuditActor` emits one best-effort central **direct-write** audit row (`CachedResolve`, status `Submitted` for a Retry / `Discarded` for a Discard) carrying the authenticated operator as `Actor` and the `TrackedOperationId` as `CorrelationId` — recording *who asked*. The operator identity flows in on `RetrySiteCallRequest` / `DiscardSiteCallRequest` (`RequestedBy`, captured at the Central UI). This row only adds provenance: the **site remains the source of truth** for the state change itself, and central reads the stored `SiteCalls` row solely to enrich the audit row's channel/target (a benign read, never a mutation). Audit is best-effort — a writer fault never changes the relay outcome. Only `Parked` rows are operator-actionable. `Failed` rows offer no Retry or Discard: a permanent failure (e.g. HTTP 4xx) would simply fail again, and the error was already returned synchronously to the calling script — there is nothing for an operator to recover. ## KPIs Point-in-time, computed from the `SiteCalls` table, global and per-source-site, mirroring the Notification Outbox KPI shape: - Buffered count (`Pending` + `Retrying`) - Parked count - Failed-last-interval - Delivered-last-interval - Oldest-pending age - Stuck count — `Pending`/`Retrying` older than a configurable threshold (default 10 minutes); display-only, no escalation. ## Retention Daily purge of terminal rows (`Delivered`, `Failed`, `Discarded`) after a configurable window (default 365 days), matching the `Notifications` purge. ## Dependencies - **Configuration Database**: hosts the `SiteCalls` table and its repository. - **Central–Site Communication**: receives cached-call telemetry and reconciliation responses; sends Retry/Discard commands. - **Store-and-Forward Engine**: the site-side origin of cached-call telemetry and the executor of relayed Retry/Discard commands. - **Audit Log (#23)**: shares the `CachedCallTelemetry` packet — each lifecycle transition (`CachedEnqueued`, `CachedAttempt`, `CachedTerminal`) carries an `AuditEvent` alongside the operational fields, and central's `AuditLogIngestActor` performs the `AuditLog` insert and the `SiteCalls` upsert in a single transaction (see [Component-AuditLog.md](Component-AuditLog.md), Cached Operations — Combined Telemetry). - **Commons**: `TrackedOperationId`, status enum, telemetry message contracts. ## Interactions - **Central UI**: the Site Calls page queries this component and issues Retry/Discard actions. - **Health Monitoring**: surfaces Site Call Audit KPI tiles on the dashboard. - **Cluster Infrastructure**: hosts the `SiteCallAuditActor` singleton with active/standby failover. - **KPI History (#26)**: emits `IKpiSampleSource` (`SiteCallAuditKpiSampleSource`, Global + per-Site + per-Node) consumed by the KpiHistory recorder (#26), reusing the existing KPI reads. All six metrics — `buffered` / `parked` / `failedLastInterval` / `deliveredLastInterval` / `stuck` / `oldestPendingAgeSeconds` — are sampled into the `KpiSample` history store, but only the three charted via the public `KpiMetrics.SiteCallAudit` catalog (`buffered` / `parked` / `failedLastInterval`) render as trends on the Site Calls page via `KpiTrendChart`; `deliveredLastInterval` / `stuck` / `oldestPendingAgeSeconds` are sampled-but-not-yet-charted (available for future trend panels / ad-hoc query). See [Component-KpiHistory.md](Component-KpiHistory.md).