# Component: Store-and-Forward Engine

## Purpose

The Store-and-Forward Engine provides reliable message delivery for outbound communications from site clusters. It buffers messages when the target system is unavailable, retries them according to configured policies, and parks messages that exhaust retries for manual review.

## Location

Site clusters only. The central cluster does not buffer messages.

## Responsibilities

- Buffer outbound messages when the target system is unavailable.
- Manage three categories of buffered messages:
  - External system API calls.
  - Notifications forwarded to the central cluster.
  - Cached database writes.
- Retry delivery per message according to the configured retry policy.
- Park messages that exhaust their retry limit (dead-letter).
- Persist buffered messages to local SQLite for durability.
- Emit cached-call lifecycle telemetry to the central Site Call Audit component via the `ICachedCallLifecycleObserver` hook (one notification per attempt outcome) so the audit pipeline can record each status transition.
- Replicate buffered messages to the standby node via application-level replication over Akka.NET remoting.
- On failover, the standby node takes over delivery from its replicated copy.
- Respond to remote queries from central for parked message management (list, retry, discard), including central-driven Retry/Discard of parked cached calls.

## Message Lifecycle

```mermaid
%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    A([Script submits message]) --> B[Attempt immediate delivery]
    B --> C{Delivered?}
    C -->|Success| D([Remove from buffer])
    C -->|Failure| E[Buffer message]
    E --> F[Retry loop<br/>per retry policy]
    F --> G{Retry outcome}
    G -->|Success| H([Remove from buffer<br/>+ notify standby])
    G -->|Max retries exhausted| I([Park message<br/>dead-letter])

    classDef ok fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef buf fill:#ffe6cc,stroke:#d79b00,color:#111111;
    classDef bad fill:#f8cecc,stroke:#b85450,color:#111111;
    class A,D,H ok
    class B,F proc
    class C,G dec
    class E buf
    class I bad
```

For notifications, "delivery" means forwarding the message to the central cluster via Central–Site Communication; "success" is central's ack, on which the message is cleared. `Notify.Send` is **enqueue-only**: it buffers the notification into the local SQLite S&F store and returns the `NotificationId` immediately — it never runs the forwarder's central Ask inline on the script thread (it passes `deferToSweep: true`, which buffers the row due-immediately and kicks a background sweep). Its worst-case latency is therefore the local SQLite insert, whether central is up or down; the actual forward happens on the sweep. `Notify.Send` also enqueues with `maxRetries: 0` — the documented "no limit" escape hatch (`StoreAndForward-015`) — so notifications are retried at the fixed forward interval until central acks and are **never parked for retry exhaustion**: a sustained central outage can no longer strand a notification behind per-message operator unparking. ("do not park" is thus the real behaviour for retry-exhaustion, not merely a happy-path aspiration — but see the two remaining parking causes below.)

There are now exactly two ways a notification parks. First, a **corrupt buffered payload** (a row whose stored JSON no longer deserialises to a `NotificationSubmit`, or deserialises to null): the forwarder returns the delivery-handler contract's permanent-failure signal (`false`) so the engine parks the row, preserving the payload for operator forensics. This **supersedes `StoreAndForward-018`**, which discarded such rows by reporting them as delivered — a silent data loss with no parked row, no central `Notifications` row, and no audit trail. Retrying a corrupt payload is pointless (it can never deserialise), so parking, not retrying, is the correct home for it. Second, retry exhaustion under the `DefaultMaxRetries` cap — but because `Notify.Send` enqueues with `maxRetries: 0` (its unbounded-retry contract, above), that second cause is **off for script-produced notifications**, leaving corrupt payload as their only parking cause. (Retry-exhaustion parking remains reachable only for a hypothetical notification enqueued with a positive `maxRetries`.)

For the cached-call categories (`ExternalCall` and `DatabaseWrite`), the operation tracking table is the status record and the S&F buffer is purely the retry mechanism. A cached call that succeeds on its first immediate attempt is written directly as a terminal `Delivered` tracking row and never enters the S&F buffer. When immediate delivery fails transiently, the message is buffered and its tracking row moves to `Pending`/`Retrying`; the buffered message carries its `TrackedOperationId` so the tracking row and the retry record stay linked. When immediate delivery fails **permanently** (e.g. HTTP 4xx), the message is not buffered — the error is returned synchronously to the calling script as before — but the tracking row is written directly as a terminal `Failed` row capturing the error. On every tracking-table status transition the site emits `CachedCallTelemetry` to central.

Every cached-call outcome maps to a tracking-table state: immediate success → `Delivered`; transient failure → `Pending`/`Retrying`, eventually `Delivered` or `Parked`; permanent failure → terminal `Failed`; operator discard of a parked row → terminal `Discarded`.

## Retry Policy

For the external-system-call and cached-database-write categories, retry settings are defined on the **source entity** (not per-message):
- **External systems**: Each external system definition includes max retry count and time between retries.
- **Cached database writes**: Each database connection definition includes max retry count and time between retries.

The **notification** category retries differently: it has no source-entity setting. The site→central forward uses a single fixed retry interval configured in the host `appsettings.json`. This interval is infrastructure config for reaching the central cluster, not a per-notification-list setting. It applies uniformly to every buffered notification regardless of its target list. A buffered notification is retried at that interval until central acks it; the engine's `DefaultMaxRetries` cap still applies (matching the cached-call categories) and a notification whose retries are exhausted under a sustained central outage parks like any other buffered message. The cap is sized so the normal central-recovery window stays well inside it — central, once reachable, owns delivery, retry, and parking from the ack point on.

The retry interval is **fixed** (not exponential backoff). Fixed interval is sufficient for the expected use cases.

**Note**: Only **transient failures** are eligible for store-and-forward buffering. For external system calls, transient failures are connection errors, timeouts, and HTTP 5xx responses. Permanent failures (HTTP 4xx) are returned directly to the calling script and are **not** queued for retry. This prevents the buffer from accumulating requests that will never succeed. For the cached-call categories, a permanent failure additionally sets the operation's tracking-table row to terminal `Failed`, capturing the error — so even a never-buffered cached call has an authoritative status record. `Failed` rows are not operator-actionable: a permanent failure would only fail again, and the error was already returned to the script.

## Buffer Size

There is **no maximum buffer size**. Messages accumulate in the buffer until delivery succeeds or retries are exhausted and the message is parked. Storage is bounded only by available disk space on the site node.

## Persistence

- Buffered messages are persisted to a **local SQLite database** on each site node.
- The active node persists locally and forwards each buffer operation (add, remove, park) to the standby node **asynchronously** via Akka.NET remoting. The active node does not wait for standby acknowledgment — this avoids adding latency to every script that buffers a message.
- The standby node applies the same operations to its own local SQLite database but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby applies replicated operations purely to keep its copy warm for a future failover.
- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its `Remove` not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** — an in-flight delivery whose `Remove` had not yet replicated — and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit.
- On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy.
- **Peer-join anti-entropy resync (LocalDb CDC).** *(Rewritten for LocalDb Phase 2, 2026-07-20. The previous specification of a chunked, ack-confirmed `SfBufferSnapshotChunk` protocol described the bespoke `ReplicationService`, which Phase 2 deleted. The discussion is rewritten rather than removed, because the failure modes it reasoned about still exist — they are simply bounded differently now.)* The buffer lives in the consolidated LocalDb database as the replicated `sf_messages` table, and both nodes exchange changes over a gRPC sync stream rather than Akka remoting. A node that was down for an extended period no longer requests a full-buffer snapshot and **replaces** its local buffer; LocalDb's snapshot resync merges **per row under last-writer-wins and never deletes**. Several of the old hazards are therefore structurally gone rather than guarded against: **(a) the 128 000-byte Akka frame limit no longer applies** — the transport is gRPC, whose successor ceiling is the 4 MB default receive limit, managed by bounding `LocalDb:Replication:MaxBatchSize` (set to 16 on the rig, sized against a ~70 KB worst-case `config_json`; see the Phase 2 plan, D6). Chunking, `ResyncId` assembly, assembly timeouts, and the truncation flag are all retired with it. **(b) The N1 directional-authority hazard is gone.** That guard existed because the bespoke resync applied a destructive delete-all-then-insert-all, so running it in the wrong direction wiped a live buffer. With a non-destructive merge there is no wipe to gate, and replication is symmetric — either node may write. `ActiveNodeEvaluator` survives, but only for the **delivery** gate and the heartbeat, which still genuinely need a single active node. **(c) The N5 orphan-row race is gone.** A `Remove` ordered before a re-add can no longer resurrect a row: deletes are tombstoned with an HLC, and a tombstone beats any older write for the same key. **The duplicate-delivery bound, stated explicitly.** Delivery remains single-node: only the primary runs the sweep (`IClusterNodeProvider.SelfIsPrimary`). A message can therefore be delivered twice only when the OLD primary delivered it and the resulting status change had not yet replicated at the instant the gate flipped. The window is one replication flush interval plus the in-flight ack, not an unbounded divergence — and unlike the old model it does not grow with backlog depth or with how long a node was absent. **One new bound replaces the old ones:** a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) may resurrect deleted rows on rejoin, because the tombstones that would have suppressed them have been pruned. Stop-and-start a site pair together, and do not leave one node of a pair offline across that horizon.

### Operation Tracking Table (lives in Site Runtime, not here)

> **StoreAndForward-021:** the operation tracking table is **not** owned by
> this component. The `IOperationTrackingStore` interface lives in
> `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/`, and the SQLite-backed
> implementation (`OperationTrackingStore`, alongside `OperationTrackingOptions`)
> lives in [`src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/`](../../src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/).
> See [`Component-SiteRuntime.md`](Component-SiteRuntime.md) for the table's
> semantics, lifecycle, and central-mirror coordination — it is summarised here
> only because the S&F retry loop carries the `TrackedOperationId` linking a
> buffered cached-call row to its tracking entry.

For context: each site node also holds a site-local operation tracking table in SQLite (owned by Site Runtime) carrying one row per `TrackedOperationId` for cached calls (`ExternalCall` and `DatabaseWrite`), created the moment the script issues the cached call and kept regardless of outcome.

- That table is the **status record**; the S&F buffer remains purely the **retry mechanism**. A buffered cached-call message references its `TrackedOperationId` back to its tracking row.
- Each row records the operation kind (`TrackedOperationKind`), a target summary (external system + method, or database connection name), the unified `TrackedOperationStatus`, retry count, last error, source provenance (instance / script), and the created/updated/terminal UTC timestamps.
- `Tracking.Status(id)` reads that table. For cached calls the **site is the authoritative source of truth** for status — the query is always answered site-locally, even when central is unreachable. The central Site Call Audit `SiteCalls` table is an eventually-consistent mirror.
- A cached call that succeeds on its first immediate attempt writes a terminal `Delivered` row directly there, with nothing placed in the S&F buffer.
- Terminal rows are purged after a configurable retention window (default 7 days) — the site holds live operational state; central holds long-term audit.

Notifications are unaffected: they have no tracking table. Their `NotificationId` and status are owned by the central `Notifications` table, and their lifecycle continues to forward to central exactly as before.

### Telemetry to Central

On every tracking-table status transition, the site emits a `CachedCallTelemetry` message to the central Site Call Audit component over the site→central channel. Emission is best-effort, at-least-once, and idempotent on `TrackedOperationId`. Because telemetry is best-effort, the site also responds to `CachedCallReconcileRequest` reconciliation pulls — cursor-based per-site reads of tracking rows changed since a cursor — so any missed telemetry self-heals. The site never depends on central; central converges to the site.

The telemetry-emitting `ICachedCallLifecycleObserver` hook is dispatched through a bounded, single-reader **observer queue** (`StoreAndForwardOptions.ObserverQueueCapacity`, default 10,000, `DropOldest`) — the one unbounded `Channel<T>` left in the system before this bound was added (arch-review WP2.6c). A slow or stuck observer (e.g. a SQLite audit write wedged behind disk contention) can no longer grow this queue without limit; it instead sheds the oldest unprocessed notification, incrementing an `ObserverQueueDroppedCount` counter that counts every drop. **Logging is separately rate-limited (adversarial review finding F3):** a Warning fires for the first drop of an episode, then at most one rollup Warning per minute while drops keep happening — never one Warning-per-dropped-item, which would itself have been a log-flood risk under the exact sustained-drop condition the bound exists to survive. This bounds memory, not delivery: telemetry loss here is covered by the reconciliation pull above, same as any other missed telemetry.

### Retry Sweep Indexing

`GetMessagesForRetryAsync` orders candidates `ORDER BY created_at ASC` within the due `status`. The existing `idx_sf_messages_status_due (status, last_attempt_at_ms)` matches the status filter and due-time predicate but not this ordering, forcing a sort/scan on a large sweep. A second covering index, `idx_sf_messages_status_created (status, created_at)`, lets the sweep walk matching rows already in `created_at` order and stop at the batch limit without re-sorting or touching non-pending rows (arch-review WP1.4). Both indexes are retained — `idx_sf_messages_status_due` still backs status+due-time lookups that don't order by `created_at`.

## Parked Message Management

- Parked messages remain stored at the site in SQLite.
- The central UI can query sites for parked messages via the Communication Layer.
- Operators can:
  - **Retry** a parked message (moves it back to the retry queue).
  - **Discard** a parked message (removes it permanently).
- For parked cached calls, Retry/Discard can be driven centrally: the Site Call Audit component relays `RetryParkedOperation` / `DiscardParkedOperation` commands (keyed by `TrackedOperationId`) down to the owning site. The site applies the command to its S&F buffer and tracking table, then emits `CachedCallTelemetry` reflecting the new state (`Retrying` or `Discarded`) — central never mutates its mirror row directly.
- Store-and-forward messages are **not** automatically cleared when an instance is deleted. Pending and parked messages, and their tracking rows, continue to exist and can be managed via the central UI.

## Message Format

Each buffered message stores:
- **Message ID**: Unique identifier.
- **Category**: External system call, notification, or cached database write.
- **Tracked Operation ID**: For the cached-call categories, the `TrackedOperationId` linking the buffered message to its row in the operation tracking table. Not used by the notification category, which is tracked centrally via its `NotificationId`.
- **Target**: External system name, the central cluster (for notifications), or database connection name.
- **Payload**: Serialized message content (API method + parameters; notification list name + subject + body plus the locally generated `NotificationId` and source provenance; SQL + parameters).
- **Retry Count**: Number of attempts so far.
- **Created At**: Timestamp when the message was first queued.
- **Last Attempt At**: Timestamp of the most recent delivery attempt.
- **Status**: Pending, retrying, or parked. This is the **buffer message's** retry state, distinct from the operation's `TrackedOperationStatus` lifecycle in the operation tracking table. A buffer message exists only while a cached call is mid-retry, so it never carries the terminal `Delivered`, `Failed`, or `Discarded` states — those live solely on the tracking row.

## Dependencies

- **SQLite**: Local persistence on each node.
- **Communication Layer**: Application-level replication to standby node; remote query handling from central; carries buffered notifications to the central cluster (gRPC to `CentralControlService`) and receives central's acks.
- **External System Gateway**: Delivers external system API calls.
- **Central–Site Communication**: The delivery target for the notification category — a buffered notification is forwarded to the central cluster over Central–Site Communication and cleared on central's ack. Also carries `CachedCallTelemetry` and reconciliation responses to central, and receives `RetryParkedOperation` / `DiscardParkedOperation` commands.
- **Site Call Audit**: The central audit mirror for cached calls — receives this engine's cached-call telemetry and reconciliation responses, and relays operator Retry/Discard of parked cached calls back as commands.
- **Database Connections**: Delivers cached database writes.
- **Site Event Logging**: Logs store-and-forward activity (queued, delivered, retried, parked).

## Interactions

- **Site Runtime (Script Actors)**: Scripts submit messages to the buffer (external calls, notifications, cached DB writes).
- **Communication Layer**: Handles parked message queries/commands from central; carries buffered notifications to the central cluster.
- **Notification Outbox**: The central destination for the notification category — central ingests each forwarded notification into the `Notifications` table and acks the site, on which the engine clears the buffered message.
- **Site Call Audit**: The central observability sibling for the cached-call categories — this engine emits `CachedCallTelemetry` on every tracking-table transition, answers `CachedCallReconcileRequest` pulls, and executes the `RetryParkedOperation` / `DiscardParkedOperation` commands it relays.
- **Health Monitoring**: Reports buffer depth metrics, including the notification backlog covering the site→central forward leg.
