scadalink-design/docs/plans/2026-05-19-cached-call-tracking-design.md

# Cached Call Tracking — Design

**Date**: 2026-05-19
**Status**: Approved
**Topic**: Trackable IDs for cached external system calls and cached database writes

## Problem

`ExternalSystem.CachedCall()` and `Database.CachedWrite()` are fire-and-forget: a
script gets no handle back, cannot confirm delivery, and an operator cannot tie a
parked S&F message to a known business operation. `Notify.Send()` already returns a
trackable `NotificationId`. The goal is to give cached external/database calls the
same first-class traceability, under a tracking model unified across all three
store-and-forward producers.

## Decision

Add a trackable ID to cached calls via **Approach B — a sibling central component
(`Site Call Audit`) plus shared tracking contracts in Commons**. The Notification
Outbox is left unchanged; unification lives in shared types and a consistent script
API, not in a merged table or component.

### Why a sibling, not a merged component

Delivery locality is the decisive constraint:

- **Notifications** are *central-delivered*: sites store-and-forward them to the
  central cluster, which delivers via SMTP. The `NotificationOutboxActor` runs a
  dispatcher loop. Central becomes the source of truth after handoff.
- **Cached calls / DB writes** are *site-delivered*: the external system or database
  often sits on the site's own network and is unreachable from central. The site's
  S&F Engine must always own delivery, and the **site remains the source of truth**
  for status. Central audit is an eventually-consistent mirror.

Merging both into one component (Approach A) would put a dispatcher loop that is live
for some rows and dormant for others into a single component, hiding a real
architectural difference. Approach B expresses the difference honestly while still
giving scripts a unified ID model and `Status()` API.

## Unified tracking model

### `TrackedOperationId`

A GUID, defined in Commons, generated **caller-side at the site at call time**. It is
both the tracking handle returned to the script and the idempotency key for telemetry
sent to central. `Notify.Send()`'s existing `NotificationId` is the notification-domain
name for this same type — no behavior change for notifications.

### Script API

| Call | Returns |
|---|---|
| `ExternalSystem.CachedCall(system, method, params)` | `TrackedOperationId` |
| `Database.CachedWrite(name, sql, params)` | `TrackedOperationId` |
| `Notify.Send(...)` | `TrackedOperationId` (unchanged) |
| `Tracking.Status(id)` | unified status record (status, retry count, last error, key timestamps) |

`Tracking.Status(id)` is the unified accessor. `Notify.Status(id)` is retained as a
thin alias for backward compatibility.

### Status lifecycle

`Pending → Retrying → Delivered / Parked / Failed / Discarded`

- **Delivered** — succeeded. A cached call that succeeds on its first immediate
  attempt goes straight here and never enters the S&F buffer.
- **Parked** — transient retries exhausted; awaiting manual action.
- **Failed** — permanent failure (e.g. HTTP 4xx). The error is *also* returned
  synchronously to the calling script, exactly as today; the record captures it.
  This is the one state beyond the notification lifecycle.
- **Discarded** — operator discarded a parked operation.

There is no `Forwarding` state for cached calls — that exists only because
notifications hand off to central. For cached calls, `Tracking.Status(id)` is always
answered site-locally and authoritatively.

## Site-side architecture

### Site-local operation tracking table

A new SQLite table alongside the existing S&F buffer DB. One row per
`TrackedOperationId`, created the moment the script issues the cached call,
regardless of outcome:

- Fields: kind, target summary (system+method, or DB name), status, retry count,
  last error, created/updated/terminal timestamps, source provenance
  (instance/script).
- This table is the **status record**. The S&F buffer remains purely the **retry
  mechanism**; a buffered message references its `TrackedOperationId`.
- Immediate success writes a terminal `Delivered` row directly here, with nothing
  placed in the S&F buffer.
- `Tracking.Status(id)` reads this table — local, authoritative, available even when
  central is unreachable.
- Retention: terminal rows purged after a configurable window (default 7 days; the
  site holds live operational state, central holds long-term audit).

### Telemetry to central

On every lifecycle transition (`Created → Retrying → Delivered/Parked/Failed/
Discarded`) the site emits a telemetry event over the existing site→central channel:
`TrackedOperationId`, kind, summary, status, retry count, last error, timestamps,
source site. Best-effort, at-least-once, idempotent on the ID.

### Reconciliation

Because telemetry is best-effort, the central side periodically (and on reconnect)
pulls "all tracking rows changed since cursor X" per site. Missed telemetry
self-heals. The site never depends on central; central converges to the site.

### Carried-over rules (unchanged)

- Tracking rows, like buffered messages, are not cleared on instance deletion.
- Cached-call idempotency remains the caller's responsibility — a retry can still
  double-deliver.

## Central — Site Call Audit component (new component #22)

### `SiteCalls` table (central MS SQL)

Sibling of the `Notifications` table. One row per `TrackedOperationId`: source site,
kind, target summary, status, retry count, last error, created/updated/terminal
timestamps. Fed only by site telemetry and reconciliation pulls.

Ingestion is **insert-if-not-exists**, then **upsert-on-newer-status**. The lifecycle
is monotonic, so status only advances, never regresses — making at-least-once and
out-of-order telemetry harmless. Daily purge of terminal rows after a configurable
window (default 365 days, mirroring `Notifications`).

### `SiteCallAuditActor`

Singleton on the active central node. Ingests telemetry, runs the periodic
reconciliation pulls, computes KPIs, and relays Retry/Discard commands to sites.

It is **not a dispatcher** — the crucial difference from `NotificationOutboxActor`.
Central has no path to a site's external systems or databases; this component is an
audit sink, a query surface, and a command relay only.

### KPIs

Point-in-time from the `SiteCalls` table, global and per-site, mirroring the
Notification Outbox KPI shape: buffered count (`Pending`+`Retrying`), parked count,
failed-last-interval, delivered-last-interval, oldest-pending age, and stuck count
(`Pending`/`Retrying` older than a configurable threshold, default 10 minutes —
display-only, no alerting).

## Central→site command path (Retry / Discard)

Parked operations live in the site's S&F buffer, so Retry/Discard from the Central UI
must travel down to the owning site:

- New ClusterClient command/control messages, central→site:
  `RetryParkedOperation(TrackedOperationId)` and
  `DiscardParkedOperation(TrackedOperationId)`, riding the existing per-site
  ClusterClient.
- The site applies the command to its S&F buffer / tracking table, then emits normal
  telemetry reflecting the new state (`Retrying`, or `Discarded`).
- Central never directly mutates the `SiteCalls` row. It sends the command and lets
  the resulting telemetry update the audit row — the site stays the single source of
  truth.
- If the site is offline, the command fails fast and the UI surfaces a
  "site unreachable" message.

## Central UI

New page — **Site Calls** — in the same nav group as the Notification Outbox page:

- Covers cached calls only: `ExternalCall` + `DatabaseWrite`. Notifications keep their
  existing dedicated Notification Outbox page.
- Queryable list filtered by site, kind, status, and time range. Columns: timestamp,
  site, kind, target summary, status badge, retry count, last error.
- Retry / Discard actions on `Parked` rows, issuing the central→site commands above.
- Headline KPI tiles on the Health dashboard alongside the existing Notification
  Outbox tiles. Stuck rows get a display-only badge — no escalation.
- Custom Blazor Server + Bootstrap components, consistent with the rest of the
  Central UI.

## Error handling & edge cases

- **Telemetry loss** — reconciliation pull self-heals; central is explicitly
  eventually-consistent.
- **Out-of-order / duplicate telemetry** — monotonic-status upsert keyed on
  `TrackedOperationId` makes both harmless.
- **Permanent failure on a cached call** — error returned synchronously to the script
  (unchanged) and recorded as terminal `Failed`.
- **Site offline during Retry/Discard** — command fails fast; UI says so; the audit
  row is unchanged until confirming telemetry arrives.
- **Cached-call double-delivery** — still the caller's responsibility; the idempotency
  note stays in the ESG doc.
- **Instance deletion** — tracking rows and buffered messages survive, per the
  existing S&F rule.

## Affected documents

- **New**: `docs/requirements/Component-SiteCallAudit.md`
- `Component-ExternalSystemGateway.md` — `CachedCall`/`CachedWrite` return
  `TrackedOperationId`; `Failed` state; `Tracking.Status`.
- `Component-StoreAndForward.md` — site-local tracking table, telemetry emission,
  reconciliation, `TrackedOperationId` on buffer entries.
- `Component-SiteRuntime.md` — Script Runtime API: return types and
  `Tracking.Status(id)`.
- `Component-Communication.md` — telemetry channel and
  `RetryParkedOperation`/`DiscardParkedOperation` commands.
- `Component-Commons.md` — `TrackedOperationId`, unified status enum, telemetry
  message contracts.
- `Component-ConfigurationDatabase.md` — `SiteCalls` table, EF mapping, migration.
- `Component-CentralUI.md` — new Site Calls page.
- `Component-HealthMonitoring.md` — KPI tiles on the dashboard.
- `Component-NotificationService.md` / `Component-NotificationOutbox.md` — note the
  shared `TrackedOperationId` model and `Notify.Status` alias.
- `README.md` — component table updated to 22 components.
- `CLAUDE.md` — component list and Key Design Decisions.

## Out of scope

- A CLI surface for site-local Retry/Discard (can be added later if needed).
- Merging notifications into the Site Calls page or a unified outbox component.
- Routing cached-call delivery through central.