# Component: Store-and-Forward Engine

## Purpose

The Store-and-Forward Engine provides reliable message delivery for outbound communications from site clusters. It buffers messages when the target system is unavailable, retries them according to configured policies, and parks messages that exhaust retries for manual review.

## Location

Site clusters only. The central cluster does not buffer messages.

## Responsibilities

- Buffer outbound messages when the target system is unavailable.
- Manage three categories of buffered messages:
  - External system API calls.
  - Notifications forwarded to the central cluster.
  - Cached database writes.
- Retry delivery per message according to the configured retry policy.
- Park messages that exhaust their retry limit (dead-letter).
- Persist buffered messages to local SQLite for durability.
- Maintain a site-local **operation tracking table** holding one row per `TrackedOperationId` for cached calls (`ExternalCall` and `DatabaseWrite`) — the authoritative status record consulted by `Tracking.Status(id)`.
- Emit cached-call lifecycle telemetry to the central Site Call Audit component on every status transition.
- Replicate buffered messages to the standby node via application-level replication over Akka.NET remoting.
- On failover, the standby node takes over delivery from its replicated copy.
- Respond to remote queries from central for parked message management (list, retry, discard), including central-driven Retry/Discard of parked cached calls.

## Message Lifecycle

```
Script submits message
    │
    ▼
Attempt immediate delivery
    │
    ├── Success → Remove from buffer
    │
    └── Failure → Buffer message
                    │
                    ▼
              Retry loop (per retry policy)
                    │
                    ├── Success → Remove from buffer + notify standby
                    │
                    └── Max retries exhausted → Park message
```

For notifications, "delivery" means forwarding the message to the central cluster via Central–Site Communication; "success" is central's ack, on which the message is cleared. Notifications do not park — they are retried at the fixed forward interval until central acks. Parking applies only to the external-system-call and cached-database-write categories.

For the cached-call categories (`ExternalCall` and `DatabaseWrite`), the operation tracking table is the status record and the S&F buffer is purely the retry mechanism. A cached call that succeeds on its first immediate attempt is written directly as a terminal `Delivered` tracking row and never enters the S&F buffer. When immediate delivery fails transiently, the message is buffered and its tracking row moves to `Pending`/`Retrying`; the buffered message carries its `TrackedOperationId` so the tracking row and the retry record stay linked. When immediate delivery fails **permanently** (e.g. HTTP 4xx), the message is not buffered — the error is returned synchronously to the calling script as before — but the tracking row is written directly as a terminal `Failed` row capturing the error. On every tracking-table status transition the site emits `CachedCallTelemetry` to central.

Every cached-call outcome maps to a tracking-table state: immediate success → `Delivered`; transient failure → `Pending`/`Retrying`, eventually `Delivered` or `Parked`; permanent failure → terminal `Failed`; operator discard of a parked row → terminal `Discarded`.

## Retry Policy

For the external-system-call and cached-database-write categories, retry settings are defined on the **source entity** (not per-message):
- **External systems**: Each external system definition includes max retry count and time between retries.
- **Cached database writes**: Each database connection definition includes max retry count and time between retries.

The **notification** category retries differently: it has no source-entity setting. The site→central forward uses a single fixed retry interval configured in the host `appsettings.json`. This interval is infrastructure config for reaching the central cluster, not a per-notification-list setting. It applies uniformly to every buffered notification regardless of its target list. A buffered notification is retried until central acks it; it is not parked on a retry limit (central, once reachable, owns delivery, retry, and parking from that point on).

The retry interval is **fixed** (not exponential backoff). Fixed interval is sufficient for the expected use cases.

**Note**: Only **transient failures** are eligible for store-and-forward buffering. For external system calls, transient failures are connection errors, timeouts, and HTTP 5xx responses. Permanent failures (HTTP 4xx) are returned directly to the calling script and are **not** queued for retry. This prevents the buffer from accumulating requests that will never succeed. For the cached-call categories, a permanent failure additionally sets the operation's tracking-table row to terminal `Failed`, capturing the error — so even a never-buffered cached call has an authoritative status record. `Failed` rows are not operator-actionable: a permanent failure would only fail again, and the error was already returned to the script.

## Buffer Size

There is **no maximum buffer size**. Messages accumulate in the buffer until delivery succeeds or retries are exhausted and the message is parked. Storage is bounded only by available disk space on the site node.

## Persistence

- Buffered messages are persisted to a **local SQLite database** on each site node.
- The active node persists locally and forwards each buffer operation (add, remove, park) to the standby node **asynchronously** via Akka.NET remoting. The active node does not wait for standby acknowledgment — this avoids adding latency to every script that buffers a message.
- The standby node applies the same operations to its own local SQLite database.
- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but remove not replicated) or a few **missed retries** (message added but not replicated). Both are acceptable trade-offs for the latency benefit.
- On failover, the new active node resumes delivery from its local copy.

### Operation Tracking Table

Alongside the S&F buffer DB, each site node holds a **site-local operation tracking table** in SQLite. It carries one row per `TrackedOperationId` for cached calls (`ExternalCall` and `DatabaseWrite`), created the moment the script issues the cached call and kept regardless of outcome.

- This table is the **status record**; the S&F buffer remains purely the **retry mechanism**. A buffered cached-call message references its `TrackedOperationId` back to its tracking row.
- Each row records the operation kind (`TrackedOperationKind`), a target summary (external system + method, or database connection name), the unified `TrackedOperationStatus`, retry count, last error, source provenance (instance / script), and the created/updated/terminal UTC timestamps.
- `Tracking.Status(id)` reads this table. For cached calls the **site is the authoritative source of truth** for status — the query is always answered site-locally, even when central is unreachable. The central Site Call Audit `SiteCalls` table is an eventually-consistent mirror.
- A cached call that succeeds on its first immediate attempt writes a terminal `Delivered` row directly here, with nothing placed in the S&F buffer.
- Terminal rows are purged after a configurable retention window (default 7 days) — the site holds live operational state; central holds long-term audit.

Notifications are unaffected: they have no tracking table. Their `NotificationId` and status are owned by the central `Notifications` table, and their lifecycle continues to forward to central exactly as before.

### Telemetry to Central

On every tracking-table status transition, the site emits a `CachedCallTelemetry` message to the central Site Call Audit component over the site→central channel. Emission is best-effort, at-least-once, and idempotent on `TrackedOperationId`. Because telemetry is best-effort, the site also responds to `CachedCallReconcileRequest` reconciliation pulls — cursor-based per-site reads of tracking rows changed since a cursor — so any missed telemetry self-heals. The site never depends on central; central converges to the site.

## Parked Message Management

- Parked messages remain stored at the site in SQLite.
- The central UI can query sites for parked messages via the Communication Layer.
- Operators can:
  - **Retry** a parked message (moves it back to the retry queue).
  - **Discard** a parked message (removes it permanently).
- For parked cached calls, Retry/Discard can be driven centrally: the Site Call Audit component relays `RetryParkedOperation` / `DiscardParkedOperation` commands (keyed by `TrackedOperationId`) down to the owning site. The site applies the command to its S&F buffer and tracking table, then emits `CachedCallTelemetry` reflecting the new state (`Retrying` or `Discarded`) — central never mutates its mirror row directly.
- Store-and-forward messages are **not** automatically cleared when an instance is deleted. Pending and parked messages, and their tracking rows, continue to exist and can be managed via the central UI.

## Message Format

Each buffered message stores:
- **Message ID**: Unique identifier.
- **Category**: External system call, notification, or cached database write.
- **Tracked Operation ID**: For the cached-call categories, the `TrackedOperationId` linking the buffered message to its row in the operation tracking table. Not used by the notification category, which is tracked centrally via its `NotificationId`.
- **Target**: External system name, the central cluster (for notifications), or database connection name.
- **Payload**: Serialized message content (API method + parameters; notification list name + subject + body plus the locally generated `NotificationId` and source provenance; SQL + parameters).
- **Retry Count**: Number of attempts so far.
- **Created At**: Timestamp when the message was first queued.
- **Last Attempt At**: Timestamp of the most recent delivery attempt.
- **Status**: Pending, retrying, or parked. This is the **buffer message's** retry state, distinct from the operation's `TrackedOperationStatus` lifecycle in the operation tracking table. A buffer message exists only while a cached call is mid-retry, so it never carries the terminal `Delivered`, `Failed`, or `Discarded` states — those live solely on the tracking row.

## Dependencies

- **SQLite**: Local persistence on each node.
- **Communication Layer**: Application-level replication to standby node; remote query handling from central; carries buffered notifications to the central cluster (ClusterClient) and receives central's acks.
- **External System Gateway**: Delivers external system API calls.
- **Central–Site Communication**: The delivery target for the notification category — a buffered notification is forwarded to the central cluster over Central–Site Communication and cleared on central's ack. Also carries `CachedCallTelemetry` and reconciliation responses to central, and receives `RetryParkedOperation` / `DiscardParkedOperation` commands.
- **Site Call Audit**: The central audit mirror for cached calls — receives this engine's cached-call telemetry and reconciliation responses, and relays operator Retry/Discard of parked cached calls back as commands.
- **Database Connections**: Delivers cached database writes.
- **Site Event Logging**: Logs store-and-forward activity (queued, delivered, retried, parked).

## Interactions

- **Site Runtime (Script Actors)**: Scripts submit messages to the buffer (external calls, notifications, cached DB writes).
- **Communication Layer**: Handles parked message queries/commands from central; carries buffered notifications to the central cluster.
- **Notification Outbox**: The central destination for the notification category — central ingests each forwarded notification into the `Notifications` table and acks the site, on which the engine clears the buffered message.
- **Site Call Audit**: The central observability sibling for the cached-call categories — this engine emits `CachedCallTelemetry` on every tracking-table transition, answers `CachedCallReconcileRequest` pulls, and executes the `RetryParkedOperation` / `DiscardParkedOperation` commands it relays.
- **Health Monitoring**: Reports buffer depth metrics, including the notification backlog covering the site→central forward leg.
