ScadaBridge/docs/requirements/Component-SiteCallAudit.md

Component: Site Call Audit

Purpose

Provides central, queryable audit and operational visibility for cached calls made by site scripts — ExternalSystem.CachedCall() and Database.CachedWrite(). Each such call carries a TrackedOperationId; sites report lifecycle telemetry to this component, which maintains a central audit record, computes KPIs, and relays Retry/Discard actions back to the owning site.

This is the second centrally-hosted observability component for site store-and-forward activity (the Notification Outbox is the first). Unlike the Notification Outbox, Site Call Audit is not a dispatcher — it never delivers anything. Cached calls are delivered by the site's Store-and-Forward Engine against site-local external systems and databases, which central cannot reach.

Location

Central cluster only. A singleton actor (SiteCallAuditActor) on the active central node. Registered as component #22 in the Host role configuration.

Responsibilities

• Ingest cached-call lifecycle telemetry from sites into the central SiteCalls table.
• Run periodic per-site reconciliation pulls so missed telemetry self-heals.
• Compute point-in-time KPIs (global and per-site) from the SiteCalls table.
• Relay operator Retry/Discard actions for parked cached calls to the owning site over the command/control channel.
• Purge terminal audit rows after a configurable retention window.

The SiteCalls Table

Lives in the central MS SQL configuration database — a sibling of the Notifications table. One row per TrackedOperationId (the shipped columns, as mapped by SiteCallEntityTypeConfiguration — the source of truth is Component-ConfigurationDatabase.md § Site Calls):

• TrackedOperationId — GUID (varchar(36), "D"-format), primary key. Generated site-side at call time.
• Channel — varchar(32), the trust-boundary channel that produced the call: ApiOutbound (ExternalSystem.CachedCall()) or DbOutbound (Database.CachedWrite()).
• Target — varchar(256), human-readable target (e.g. ERP.GetOrder for an external call, or the database connection name for a cached write — intentionally not the SQL statement or table, a deliberate scoping choice).
• SourceSite — varchar(64), site that issued the call.
• SourceNode — varchar(64) NULL, the cluster node on which the call was issued (node-a / node-b, qualified by SourceSite). Stamped site-side at submit time and carried verbatim through the combined CachedCallTelemetry packet, reconciliation pulls, and the central upsert; NULL for reconciled rows from a retired node.
• Status — varchar(32), the AuditStatus enum name (not the tracking-lifecycle names): Submitted, Forwarded, Attempted, Delivered, Failed, Parked, Discarded. The lifecycle is monotonic, so out-of-order/at-least-once telemetry is harmless.
• RetryCount — int, attempts so far.
• LastError — nvarchar(1024) NULL, most recent error detail, if any.
• HttpStatus — int NULL, last HTTP status code for API calls.
• CreatedAtUtc, UpdatedAtUtc (datetime2), TerminalAtUtc (datetime2 NULL), IngestedAtUtc (datetime2, central ingest timestamp) — key timestamps.

There are no Kind, TargetSummary, or provenance (instance/script) columns — those were an earlier design that did not ship; provenance detail for a cached call lives on the site's own tracking store and in the AuditLog rows.

Status Lifecycle

Pending → Retrying → Delivered / Parked / Failed / Discarded

Stored vs. tracking view. This lifecycle is the operator-facing tracking view (what the site's tracking store and Tracking.Status() express). The persisted SiteCalls.Status column stores the monotonic AuditStatus-derived string (Submitted / Forwarded / Attempted / Delivered / Failed / Parked / Discarded) carried by the combined telemetry packet — the mirror records the audit-event status, not the tracking enum name. The two agree on the terminal outcomes; the non-terminal Submitted/Forwarded/Attempted strings are the ingest-phase equivalents of Pending/Retrying.

• Pending — non-terminal: buffered after a transient failure, awaiting its first retry.
• Retrying — non-terminal: undergoing retry attempts.
• Delivered — terminal, success. A cached call that succeeds on its first immediate attempt is recorded directly as Delivered.
• Parked — non-terminal: transient retries exhausted; awaiting manual action.
• Failed — terminal: permanent failure (e.g. HTTP 4xx). The error was also returned synchronously to the calling script; the record captures it. Failed rows are not operator-actionable — see Retry / Discard Relay.
• Discarded — terminal, reached only by operator action on a Parked row. The row is kept (not deleted) so the table remains a complete audit record.

The site is the source of truth. The SiteCalls row is an eventually-consistent mirror — never queried by scripts (Tracking.Status() is answered site-locally).

Ingest & Idempotency

Telemetry ingestion is insert-if-not-exists keyed on TrackedOperationId, then upsert-on-newer-status, with a newest-UpdatedAtUtc tiebreaker within equal non-terminal rank. The lifecycle is monotonic on status rank, so status never regresses. Within an equal non-terminal rank (the Attempted/Skipped retry phase), the packet with the newest UpdatedAtUtc wins — so a retrying call's RetryCount/LastError/HttpStatus stay live instead of freezing at the first Attempted write. Equal terminal rank stays immutable (a later Delivered never overwrites an earlier Parked), equal stamps are an idempotent no-op, and a lower rank is always a no-op — so at-least-once and out-of-order telemetry remain harmless.

From v1.x onward, the CachedCallTelemetry message additively carries the AuditEvent content alongside the existing operational fields. Central's AuditLogIngestActor (Audit Log #23) performs both the immutable AuditLog insert and the SiteCalls upsert in a single transaction. Idempotency keys remain EventId (for AuditLog) and TrackedOperationId (for SiteCalls). See Component-AuditLog.md, Cached Operations — Combined Telemetry, for the dual-write contract.

Reconciliation

Because telemetry is best-effort, SiteCallAuditActor periodically — and on site reconnect — pulls "all tracking rows changed since cursor X" from each site. Gaps left by lost telemetry self-heal. Central converges to the site; the site never depends on central.

The per-site cursor is a composite (UpdatedAtUtc, TrackedOperationId) keyset, not a single timestamp. Each pull asks for rows strictly greater than the cursor pair and advances it to the maximum row seen; a burst of more rows than one batch all sharing one exact UpdatedAtUtc therefore drains via the TrackedOperationId tiebreak instead of pinning the timestamp forever. The after_id keyset field is additive on the pull contract — a first pull (or a legacy site that predates it) sends no after_id and keeps the inclusive >= timestamp behaviour. When such a legacy site keeps reporting MoreAvailable yet the composite cursor cannot advance, the actor latches the site as pinned and publishes SiteCallReconciliationPinnedChanged(siteId, Pinned) on the EventStream (transition-only, mirroring SiteAuditTelemetryStalledChanged) — the un-drainable tail is a health-observable condition rather than a silent log line, and the latch clears with Pinned=false once a later tick makes progress.

The drain runs off the mailbox. A reconciliation pass is unbounded work — every site, up to a page ceiling of network pulls each, one upsert per row — so it runs as a background task with a PipeTo-delivered completion message and a single-flight guard, not as an actor message handler. A handler occupies the actor for its whole duration, so a post-outage catch-up used to park telemetry ingest, UI queries and KPI Asks behind it; those callers time out rather than queue, which made a slow site look like a dead central. The single-flight guard is raised and lowered ON the actor thread, so the per-site cursor and pinned-latch dictionaries the pass mutates are still only ever touched by one task at a time, and the mailbox supplies the memory barrier between consecutive passes. The daily terminal-row purge uses the same shape. (NotificationOutboxActor's dispatch sweep is the in-repo reference.)

The central upsert is one round trip, INSERT-first. SiteCallAuditRepository.UpsertAsync ships both statements in a single command text — IF NOT EXISTS … INSERT; then the monotonic UPDATE — so a packet costs one round trip, not two. The insert leg is gated on the row's existence alone, never on "the update matched nothing": a zero row count also means "the monotonic guard rejected this packet", and inserting there would fork the mirror with a second row for an id that already exists.

The ORDER is load-bearing, and running the UPDATE first is a data-loss bug. The two writers — the cached dual-write and the reconciliation pull — routinely carry different lifecycle states for the same TrackedOperationId, and both can race its very first packet. Under UPDATE-first both find no row, so both updates match nothing; the loser then either fails its existence re-check and skips its insert, or attempts it and takes a duplicate-key fault — either way its Status/RetryCount/HttpStatus/TerminalAtUtc are dropped, because it never ran an update against the winner's row. Under INSERT-first the loser's insert is skipped or faults and its monotonic update still applies to whichever row won, so the newer state survives every interleaving while a stale one is still rejected by the rank guard. The duplicate-key catch re-runs the monotonic update for the same reason (it is idempotent and rank-guarded, so a redundant re-run is inert).

Retry / Discard Relay

Parked cached calls live in the owning site's S&F buffer. Operator Retry/Discard from the Central UI is relayed to that site as a RetryParkedOperation / DiscardParkedOperation command over the command/control channel. The site applies the change and emits telemetry reflecting the new state; central never mutates the SiteCalls row directly. If the site is offline the command fails fast and the UI surfaces a "site unreachable" message.

On a successful relay (the site acks Applied), SiteCallAuditActor emits one best-effort central direct-write audit row (CachedResolve, status Submitted for a Retry / Discarded for a Discard) carrying the authenticated operator as Actor and the TrackedOperationId as CorrelationId — recording who asked. The operator identity flows in on RetrySiteCallRequest / DiscardSiteCallRequest (RequestedBy, captured at the Central UI). This row only adds provenance: the site remains the source of truth for the state change itself, and central reads the stored SiteCalls row solely to enrich the audit row's channel/target (a benign read, never a mutation). Audit is best-effort — a writer fault never changes the relay outcome.

Only Parked rows are operator-actionable. Failed rows offer no Retry or Discard: a permanent failure (e.g. HTTP 4xx) would simply fail again, and the error was already returned synchronously to the calling script — there is nothing for an operator to recover.

KPIs

Point-in-time, computed from the SiteCalls table, global and per-source-site, mirroring the Notification Outbox KPI shape:

• Buffered count (Pending + Retrying)
• Parked count
• Failed-last-interval
• Delivered-last-interval
• Oldest-pending age
• Stuck count — Pending/Retrying older than a configurable threshold (default 10 minutes); display-only, no escalation.

Retention

Daily purge of terminal rows (Delivered, Failed, Discarded) after a configurable window (default 365 days), matching the Notifications purge.

Dependencies

• Configuration Database: hosts the SiteCalls table and its repository.
• Central–Site Communication: receives cached-call telemetry and reconciliation responses; sends Retry/Discard commands.
• Store-and-Forward Engine: the site-side origin of cached-call telemetry and the executor of relayed Retry/Discard commands.
• Audit Log (#23): shares the CachedCallTelemetry packet — each lifecycle transition (CachedEnqueued, CachedAttempt, CachedTerminal) carries an AuditEvent alongside the operational fields, and central's AuditLogIngestActor performs the AuditLog insert and the SiteCalls upsert in a single transaction (see Component-AuditLog.md, Cached Operations — Combined Telemetry).
• Commons: TrackedOperationId, status enum, telemetry message contracts.

Interactions

• Central UI: the Site Calls page queries this component and issues Retry/Discard actions.
• Health Monitoring: surfaces Site Call Audit KPI tiles on the dashboard.
• Cluster Infrastructure: hosts the SiteCallAuditActor singleton with active/standby failover.
• KPI History (#26): emits IKpiSampleSource (SiteCallAuditKpiSampleSource, Global + per-Site + per-Node) consumed by the KpiHistory recorder (#26), reusing the existing KPI reads. All six metrics — buffered / parked / failedLastInterval / deliveredLastInterval / stuck / oldestPendingAgeSeconds — are sampled into the KpiSample history store, but only the three charted via the public KpiMetrics.SiteCallAudit catalog (buffered / parked / failedLastInterval) render as trends on the Site Calls page via KpiTrendChart; deliveredLastInterval / stuck / oldestPendingAgeSeconds are sampled-but-not-yet-charted (available for future trend panels / ad-hoc query). See Component-KpiHistory.md.