ScadaBridge/docs/requirements/Component-SiteCallAudit.md

Component: Site Call Audit

Purpose

Provides central, queryable audit and operational visibility for cached calls made by site scripts — ExternalSystem.CachedCall() and Database.CachedWrite(). Each such call carries a TrackedOperationId; sites report lifecycle telemetry to this component, which maintains a central audit record, computes KPIs, and relays Retry/Discard actions back to the owning site.

This is the second centrally-hosted observability component for site store-and-forward activity (the Notification Outbox is the first). Unlike the Notification Outbox, Site Call Audit is not a dispatcher — it never delivers anything. Cached calls are delivered by the site's Store-and-Forward Engine against site-local external systems and databases, which central cannot reach.

Location

Central cluster only. A singleton actor (SiteCallAuditActor) on the active central node. Registered as component #22 in the Host role configuration.

Responsibilities

• Ingest cached-call lifecycle telemetry from sites into the central SiteCalls table.
• Run periodic per-site reconciliation pulls so missed telemetry self-heals.
• Compute point-in-time KPIs (global and per-site) from the SiteCalls table.
• Relay operator Retry/Discard actions for parked cached calls to the owning site over the command/control channel.
• Purge terminal audit rows after a configurable retention window.

The SiteCalls Table

Lives in the central MS SQL configuration database — a sibling of the Notifications table. One row per TrackedOperationId (the shipped columns, as mapped by SiteCallEntityTypeConfiguration — the source of truth is Component-ConfigurationDatabase.md § Site Calls):

• TrackedOperationId — GUID (varchar(36), "D"-format), primary key. Generated site-side at call time.
• Channel — varchar(32), the trust-boundary channel that produced the call: ApiOutbound (ExternalSystem.CachedCall()) or DbOutbound (Database.CachedWrite()).
• Target — varchar(256), human-readable target (e.g. ERP.GetOrder for an external call, or the database connection name for a cached write — intentionally not the SQL statement or table, a deliberate scoping choice).
• SourceSite — varchar(64), site that issued the call.
• SourceNode — varchar(64) NULL, the cluster node on which the call was issued (node-a / node-b, qualified by SourceSite). Stamped site-side at submit time and carried verbatim through the combined CachedCallTelemetry packet, reconciliation pulls, and the central upsert; NULL for reconciled rows from a retired node.
• Status — varchar(32), the AuditStatus enum name (not the tracking-lifecycle names): Submitted, Forwarded, Attempted, Delivered, Failed, Parked, Discarded. The lifecycle is monotonic, so out-of-order/at-least-once telemetry is harmless.
• RetryCount — int, attempts so far.
• LastError — nvarchar(1024) NULL, most recent error detail, if any.
• HttpStatus — int NULL, last HTTP status code for API calls.
• CreatedAtUtc, UpdatedAtUtc (datetime2), TerminalAtUtc (datetime2 NULL), IngestedAtUtc (datetime2, central ingest timestamp) — key timestamps.

There are no Kind, TargetSummary, or provenance (instance/script) columns — those were an earlier design that did not ship; provenance detail for a cached call lives on the site's own tracking store and in the AuditLog rows.

Status Lifecycle

Pending → Retrying → Delivered / Parked / Failed / Discarded

Stored vs. tracking view. This lifecycle is the operator-facing tracking view (what the site's tracking store and Tracking.Status() express). The persisted SiteCalls.Status column stores the monotonic AuditStatus-derived string (Submitted / Forwarded / Attempted / Delivered / Failed / Parked / Discarded) carried by the combined telemetry packet — the mirror records the audit-event status, not the tracking enum name. The two agree on the terminal outcomes; the non-terminal Submitted/Forwarded/Attempted strings are the ingest-phase equivalents of Pending/Retrying.

• Pending — non-terminal: buffered after a transient failure, awaiting its first retry.
• Retrying — non-terminal: undergoing retry attempts.
• Delivered — terminal, success. A cached call that succeeds on its first immediate attempt is recorded directly as Delivered.
• Parked — non-terminal: transient retries exhausted; awaiting manual action.
• Failed — terminal: permanent failure (e.g. HTTP 4xx). The error was also returned synchronously to the calling script; the record captures it. Failed rows are not operator-actionable — see Retry / Discard Relay.
• Discarded — terminal, reached only by operator action on a Parked row. The row is kept (not deleted) so the table remains a complete audit record.

The site is the source of truth. The SiteCalls row is an eventually-consistent mirror — never queried by scripts (Tracking.Status() is answered site-locally).

Ingest & Idempotency

Telemetry ingestion is insert-if-not-exists keyed on TrackedOperationId, then upsert-on-newer-status, with a newest-UpdatedAtUtc tiebreaker within equal non-terminal rank. The lifecycle is monotonic on status rank, so status never regresses. Within an equal non-terminal rank (the Attempted/Skipped retry phase), the packet with the newest UpdatedAtUtc wins — so a retrying call's RetryCount/LastError/HttpStatus stay live instead of freezing at the first Attempted write. Equal terminal rank stays immutable (a later Delivered never overwrites an earlier Parked), equal stamps are an idempotent no-op, and a lower rank is always a no-op — so at-least-once and out-of-order telemetry remain harmless.

From v1.x onward, the CachedCallTelemetry message additively carries the AuditEvent content alongside the existing operational fields. Central's AuditLogIngestActor (Audit Log #23) performs both the immutable AuditLog insert and the SiteCalls upsert in a single transaction. Idempotency keys remain EventId (for AuditLog) and TrackedOperationId (for SiteCalls). See Component-AuditLog.md, Cached Operations — Combined Telemetry, for the dual-write contract.

Reconciliation

Because telemetry is best-effort, SiteCallAuditActor periodically — and on site reconnect — pulls "all tracking rows changed since cursor X" from each site. Gaps left by lost telemetry self-heal. Central converges to the site; the site never depends on central.

The per-site cursor is a composite (UpdatedAtUtc, TrackedOperationId) keyset, not a single timestamp. Each pull asks for rows strictly greater than the cursor pair and advances it to the maximum row seen; a burst of more rows than one batch all sharing one exact UpdatedAtUtc therefore drains via the TrackedOperationId tiebreak instead of pinning the timestamp forever. The after_id keyset field is additive on the pull contract — a first pull (or a legacy site that predates it) sends no after_id and keeps the inclusive >= timestamp behaviour. When such a legacy site keeps reporting MoreAvailable yet the composite cursor cannot advance, the actor latches the site as pinned and publishes SiteCallReconciliationPinnedChanged(siteId, Pinned) on the EventStream (transition-only, mirroring SiteAuditTelemetryStalledChanged) — the un-drainable tail is a health-observable condition rather than a silent log line, and the latch clears with Pinned=false once a later tick makes progress.

The drain runs off the mailbox. A reconciliation pass is unbounded work — every site, up to a page ceiling of network pulls each, one upsert per row — so it runs as a background task with a PipeTo-delivered completion message and a single-flight guard, not as an actor message handler. A handler occupies the actor for its whole duration, so a post-outage catch-up used to park telemetry ingest, UI queries and KPI Asks behind it; those callers time out rather than queue, which made a slow site look like a dead central. The single-flight guard is raised and lowered ON the actor thread, so the per-site cursor and pinned-latch dictionaries the pass mutates are still only ever touched by one task at a time, and the mailbox supplies the memory barrier between consecutive passes. The daily terminal-row purge uses the same shape. (NotificationOutboxActor's dispatch sweep is the in-repo reference.)

Stored site ownership is immutable. Authenticated control-plane requests must claim their verified site in both audit and operational halves. An existing TrackedOperationId cannot be advanced by another site even when the incoming packet claims its own site consistently: the SQL upsert rejects the owner mismatch before its monotonic update, including the duplicate-insert retry path. The update predicate independently requires an exact owner match. A rejection rolls back the paired audit/operational ingest transaction and produces no acknowledgment.

The central upsert is one round trip, INSERT-first. SiteCallAuditRepository.UpsertAsync ships both statements in a single command text — IF NOT EXISTS … INSERT; then the monotonic UPDATE — so a packet costs one round trip, not two. The insert leg is gated on the row's existence alone, never on "the update matched nothing": a zero row count also means "the monotonic guard rejected this packet", and inserting there would fork the mirror with a second row for an id that already exists.

The ORDER is load-bearing, and running the UPDATE first is a data-loss bug. The two writers — the cached dual-write and the reconciliation pull — routinely carry different lifecycle states for the same TrackedOperationId, and both can race its very first packet. Under UPDATE-first both find no row, so both updates match nothing; the loser then either fails its existence re-check and skips its insert, or attempts it and takes a duplicate-key fault — either way its Status/RetryCount/HttpStatus/TerminalAtUtc are dropped, because it never ran an update against the winner's row. Under INSERT-first the loser's insert is skipped or faults and its monotonic update still applies to whichever row won, so the newer state survives every interleaving while a stale one is still rejected by the rank guard. The duplicate-key catch re-runs the monotonic update for the same reason (it is idempotent and rank-guarded, so a redundant re-run is inert).

Retry / Discard Relay

Parked cached calls live in the owning site's S&F buffer. Operator Retry/Discard from the Central UI is relayed to that site as a RetryParkedOperation / DiscardParkedOperation command over the command/control channel. The site applies the change and emits telemetry reflecting the new state; central never mutates the SiteCalls row directly. If the site is offline the command fails fast and the UI surfaces a "site unreachable" message.

On a successful relay (the site acks Applied), SiteCallAuditActor emits one best-effort central direct-write audit row (CachedResolve, status Submitted for a Retry / Discarded for a Discard) carrying the authenticated operator as Actor and the TrackedOperationId as CorrelationId — recording who asked. The operator identity flows in on RetrySiteCallRequest / DiscardSiteCallRequest (RequestedBy, captured at the Central UI). This row only adds provenance: the site remains the source of truth for the state change itself, and central reads the stored SiteCalls row solely to enrich the audit row's channel/target (a benign read, never a mutation). Audit is best-effort — a writer fault never changes the relay outcome.

Only Parked rows are operator-actionable. Failed rows offer no Retry or Discard: a permanent failure (e.g. HTTP 4xx) would simply fail again, and the error was already returned synchronously to the calling script — there is nothing for an operator to recover.

KPIs

Point-in-time, computed from the SiteCalls table, global and per-source-site, mirroring the Notification Outbox KPI shape:

• Buffered count (Pending + Retrying)
• Parked count
• Failed-last-interval
• Delivered-last-interval
• Oldest-pending age
• Stuck count — Pending/Retrying older than a configurable threshold (default 10 minutes); display-only, no escalation.

Retention

Daily purge of terminal rows (Delivered, Failed, Discarded) after a configurable window (default 365 days), matching the Notifications purge.

Dependencies

• Configuration Database: hosts the SiteCalls table and its repository.
• Central–Site Communication: receives cached-call telemetry and reconciliation responses; sends Retry/Discard commands.
• Store-and-Forward Engine: the site-side origin of cached-call telemetry and the executor of relayed Retry/Discard commands.
• Audit Log (#23): shares the CachedCallTelemetry packet — each lifecycle transition (CachedEnqueued, CachedAttempt, CachedTerminal) carries an AuditEvent alongside the operational fields, and central's AuditLogIngestActor performs the AuditLog insert and the SiteCalls upsert in a single transaction (see Component-AuditLog.md, Cached Operations — Combined Telemetry).
• Commons: TrackedOperationId, status enum, telemetry message contracts.

Interactions

• Central UI: the Site Calls page queries this component and issues Retry/Discard actions.
• Health Monitoring: surfaces Site Call Audit KPI tiles on the dashboard.
• Cluster Infrastructure: hosts the SiteCallAuditActor singleton with active/standby failover.
• KPI History (#26): emits IKpiSampleSource (SiteCallAuditKpiSampleSource, Global + per-Site + per-Node) consumed by the KpiHistory recorder (#26), reusing the existing KPI reads. All six metrics — buffered / parked / failedLastInterval / deliveredLastInterval / stuck / oldestPendingAgeSeconds — are sampled into the KpiSample history store, but only the three charted via the public KpiMetrics.SiteCallAudit catalog (buffered / parked / failedLastInterval) render as trends on the Site Calls page via KpiTrendChart; deliveredLastInterval / stuck / oldestPendingAgeSeconds are sampled-but-not-yet-charted (available for future trend panels / ad-hoc query). See Component-KpiHistory.md.