ScadaBridge/docs/requirements/Component-StoreAndForward.md

Component: Store-and-Forward Engine

Purpose

The Store-and-Forward Engine provides reliable message delivery for outbound communications from site clusters. It buffers messages when the target system is unavailable, retries them according to configured policies, and parks messages that exhaust retries for manual review.

Location

Site clusters only. The central cluster does not buffer messages.

Responsibilities

• Buffer outbound messages when the target system is unavailable.
• Manage three categories of buffered messages:
  • External system API calls.
  • Notifications forwarded to the central cluster.
  • Cached database writes.
• Retry delivery per message according to the configured retry policy.
• Park messages that exhaust their retry limit (dead-letter).
• Persist buffered messages to local SQLite for durability.
• Emit cached-call lifecycle telemetry to the central Site Call Audit component via the ICachedCallLifecycleObserver hook (one notification per attempt outcome) so the audit pipeline can record each status transition.
• Replicate buffered messages to the standby node via application-level replication over Akka.NET remoting.
• On failover, the standby node takes over delivery from its replicated copy.
• Respond to remote queries from central for parked message management (list, retry, discard), including central-driven Retry/Discard of parked cached calls.

Message Lifecycle

%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    A([Script submits message]) --> B[Attempt immediate delivery]
    B --> C{Delivered?}
    C -->|Success| D([Remove from buffer])
    C -->|Failure| E[Buffer message]
    E --> F[Retry loop<br/>per retry policy]
    F --> G{Retry outcome}
    G -->|Success| H([Remove from buffer<br/>+ notify standby])
    G -->|Max retries exhausted| I([Park message<br/>dead-letter])

    classDef ok fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef buf fill:#ffe6cc,stroke:#d79b00,color:#111111;
    classDef bad fill:#f8cecc,stroke:#b85450,color:#111111;
    class A,D,H ok
    class B,F proc
    class C,G dec
    class E buf
    class I bad

For notifications, "delivery" means forwarding the message to the central cluster via Central–Site Communication; "success" is central's ack, on which the message is cleared. Notify.Send is enqueue-only: it buffers the notification into the local SQLite S&F store and returns the NotificationId immediately — it never runs the forwarder's central Ask inline on the script thread (it passes deferToSweep: true, which buffers the row due-immediately and kicks a background sweep). Its worst-case latency is therefore the local SQLite insert, whether central is up or down; the actual forward happens on the sweep. Notify.Send also enqueues with maxRetries: 0 — the documented "no limit" escape hatch (StoreAndForward-015) — so notifications are retried at the fixed forward interval until central acks and are never parked for retry exhaustion: a sustained central outage can no longer strand a notification behind per-message operator unparking. ("do not park" is thus the real behaviour for retry-exhaustion, not merely a happy-path aspiration — but see the two remaining parking causes below.)

There are now exactly two ways a notification parks. First, a corrupt buffered payload (a row whose stored JSON no longer deserialises to a NotificationSubmit, or deserialises to null): the forwarder returns the delivery-handler contract's permanent-failure signal (false) so the engine parks the row, preserving the payload for operator forensics. This supersedes StoreAndForward-018, which discarded such rows by reporting them as delivered — a silent data loss with no parked row, no central Notifications row, and no audit trail. Retrying a corrupt payload is pointless (it can never deserialise), so parking, not retrying, is the correct home for it. Second, retry exhaustion under the DefaultMaxRetries cap — but because Notify.Send enqueues with maxRetries: 0 (its unbounded-retry contract, above), that second cause is off for script-produced notifications, leaving corrupt payload as their only parking cause. (Retry-exhaustion parking remains reachable only for a hypothetical notification enqueued with a positive maxRetries.)

For the cached-call categories (ExternalCall and DatabaseWrite), the operation tracking table is the status record and the S&F buffer is purely the retry mechanism. A cached call that succeeds on its first immediate attempt is written directly as a terminal Delivered tracking row and never enters the S&F buffer. When immediate delivery fails transiently, the message is buffered and its tracking row moves to Pending/Retrying; the buffered message carries its TrackedOperationId so the tracking row and the retry record stay linked. When immediate delivery fails permanently (e.g. HTTP 4xx), the message is not buffered — the error is returned synchronously to the calling script as before — but the tracking row is written directly as a terminal Failed row capturing the error. On every tracking-table status transition the site emits CachedCallTelemetry to central.

Every cached-call outcome maps to a tracking-table state: immediate success → Delivered; transient failure → Pending/Retrying, eventually Delivered or Parked; permanent failure → terminal Failed; operator discard of a parked row → terminal Discarded.

Retry Policy

For the external-system-call and cached-database-write categories, retry settings are defined on the source entity (not per-message):

• External systems: Each external system definition includes max retry count and time between retries.
• Cached database writes: Each database connection definition includes max retry count and time between retries.

The notification category retries differently: it has no source-entity setting. The site→central forward uses a single fixed retry interval configured in the host appsettings.json. This interval is infrastructure config for reaching the central cluster, not a per-notification-list setting. It applies uniformly to every buffered notification regardless of its target list. A buffered notification is retried at that interval until central acks it; the engine's DefaultMaxRetries cap still applies (matching the cached-call categories) and a notification whose retries are exhausted under a sustained central outage parks like any other buffered message. The cap is sized so the normal central-recovery window stays well inside it — central, once reachable, owns delivery, retry, and parking from the ack point on.

The retry interval is fixed (not exponential backoff). Fixed interval is sufficient for the expected use cases.

Note: Only transient failures are eligible for store-and-forward buffering. For external system calls, transient failures are connection errors, timeouts, and HTTP 5xx responses. Permanent failures (HTTP 4xx) are returned directly to the calling script and are not queued for retry. This prevents the buffer from accumulating requests that will never succeed. For the cached-call categories, a permanent failure additionally sets the operation's tracking-table row to terminal Failed, capturing the error — so even a never-buffered cached call has an authoritative status record. Failed rows are not operator-actionable: a permanent failure would only fail again, and the error was already returned to the script.

Buffer Size

There is no maximum buffer size. Messages accumulate in the buffer until delivery succeeds or retries are exhausted and the message is parked. Storage is bounded only by available disk space on the site node.

Persistence

• Buffered messages are persisted to a local SQLite database on each site node.
• The active node persists locally and forwards each buffer operation (add, remove, park) to the standby node asynchronously via Akka.NET remoting. The active node does not wait for standby acknowledgment — this avoids adding latency to every script that buffers a message.
• The standby node applies the same operations to its own local SQLite database but is passive: it never runs the delivery sweep. The retry sweep is gated to the active node (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby applies replicated operations purely to keep its copy warm for a future failover.
• On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few duplicate deliveries (message delivered but its Remove not yet replicated) or a few missed retries (message added but not replicated). Duplicate deliveries are therefore confined to the failover window — an in-flight delivery whose Remove had not yet replicated — and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit.
• On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy.
• Peer-join anti-entropy resync. Asynchronous, no-ack replication keeps the standby warm in steady state, but a standby that was down for an extended period (a crash, a long maintenance window) misses every operation replicated while it was gone and would otherwise diverge from the active node's buffer forever. To close that gap, whenever a node (re)tracks its peer, a standby requests a full-buffer snapshot (RequestSfBufferResync); the active node answers with up to MaxResyncRows (10 000) of its oldest rows (SfBufferSnapshot), and the standby replaces its entire local buffer with that snapshot (ReplaceAllAsync, one transaction). Only the active node answers; only a standby applies (each side checks the repo-standard leader+Up active-node predicate, safe-by-default to standby). Because replicated applies are upserts (see the replication apply path), any Add/Remove/Park that lands after the resync merges cleanly onto the resynced state — no primary-key conflict, no lost delta. If the buffer exceeds the 10 000-row cap the snapshot is flagged Truncated and the standby logs a Warning; the residual divergence beyond the cap drains naturally as the active node delivers.

Operation Tracking Table (lives in Site Runtime, not here)

StoreAndForward-021: the operation tracking table is not owned by this component. The IOperationTrackingStore interface lives in src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Services/, and the SQLite-backed implementation (OperationTrackingStore, alongside OperationTrackingOptions) lives in src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/. See Component-SiteRuntime.md for the table's semantics, lifecycle, and central-mirror coordination — it is summarised here only because the S&F retry loop carries the TrackedOperationId linking a buffered cached-call row to its tracking entry.

For context: each site node also holds a site-local operation tracking table in SQLite (owned by Site Runtime) carrying one row per TrackedOperationId for cached calls (ExternalCall and DatabaseWrite), created the moment the script issues the cached call and kept regardless of outcome.

• That table is the status record; the S&F buffer remains purely the retry mechanism. A buffered cached-call message references its TrackedOperationId back to its tracking row.
• Each row records the operation kind (TrackedOperationKind), a target summary (external system + method, or database connection name), the unified TrackedOperationStatus, retry count, last error, source provenance (instance / script), and the created/updated/terminal UTC timestamps.
• Tracking.Status(id) reads that table. For cached calls the site is the authoritative source of truth for status — the query is always answered site-locally, even when central is unreachable. The central Site Call Audit SiteCalls table is an eventually-consistent mirror.
• A cached call that succeeds on its first immediate attempt writes a terminal Delivered row directly there, with nothing placed in the S&F buffer.
• Terminal rows are purged after a configurable retention window (default 7 days) — the site holds live operational state; central holds long-term audit.

Notifications are unaffected: they have no tracking table. Their NotificationId and status are owned by the central Notifications table, and their lifecycle continues to forward to central exactly as before.

Telemetry to Central

On every tracking-table status transition, the site emits a CachedCallTelemetry message to the central Site Call Audit component over the site→central channel. Emission is best-effort, at-least-once, and idempotent on TrackedOperationId. Because telemetry is best-effort, the site also responds to CachedCallReconcileRequest reconciliation pulls — cursor-based per-site reads of tracking rows changed since a cursor — so any missed telemetry self-heals. The site never depends on central; central converges to the site.

Parked Message Management

• Parked messages remain stored at the site in SQLite.
• The central UI can query sites for parked messages via the Communication Layer.
• Operators can:
  • Retry a parked message (moves it back to the retry queue).
  • Discard a parked message (removes it permanently).
• For parked cached calls, Retry/Discard can be driven centrally: the Site Call Audit component relays RetryParkedOperation / DiscardParkedOperation commands (keyed by TrackedOperationId) down to the owning site. The site applies the command to its S&F buffer and tracking table, then emits CachedCallTelemetry reflecting the new state (Retrying or Discarded) — central never mutates its mirror row directly.
• Store-and-forward messages are not automatically cleared when an instance is deleted. Pending and parked messages, and their tracking rows, continue to exist and can be managed via the central UI.

Message Format

Each buffered message stores:

• Message ID: Unique identifier.
• Category: External system call, notification, or cached database write.
• Tracked Operation ID: For the cached-call categories, the TrackedOperationId linking the buffered message to its row in the operation tracking table. Not used by the notification category, which is tracked centrally via its NotificationId.
• Target: External system name, the central cluster (for notifications), or database connection name.
• Payload: Serialized message content (API method + parameters; notification list name + subject + body plus the locally generated NotificationId and source provenance; SQL + parameters).
• Retry Count: Number of attempts so far.
• Created At: Timestamp when the message was first queued.
• Last Attempt At: Timestamp of the most recent delivery attempt.
• Status: Pending, retrying, or parked. This is the buffer message's retry state, distinct from the operation's TrackedOperationStatus lifecycle in the operation tracking table. A buffer message exists only while a cached call is mid-retry, so it never carries the terminal Delivered, Failed, or Discarded states — those live solely on the tracking row.

Dependencies

• SQLite: Local persistence on each node.
• Communication Layer: Application-level replication to standby node; remote query handling from central; carries buffered notifications to the central cluster (ClusterClient) and receives central's acks.
• External System Gateway: Delivers external system API calls.
• Central–Site Communication: The delivery target for the notification category — a buffered notification is forwarded to the central cluster over Central–Site Communication and cleared on central's ack. Also carries CachedCallTelemetry and reconciliation responses to central, and receives RetryParkedOperation / DiscardParkedOperation commands.
• Site Call Audit: The central audit mirror for cached calls — receives this engine's cached-call telemetry and reconciliation responses, and relays operator Retry/Discard of parked cached calls back as commands.
• Database Connections: Delivers cached database writes.
• Site Event Logging: Logs store-and-forward activity (queued, delivered, retried, parked).

Interactions

• Site Runtime (Script Actors): Scripts submit messages to the buffer (external calls, notifications, cached DB writes).
• Communication Layer: Handles parked message queries/commands from central; carries buffered notifications to the central cluster.
• Notification Outbox: The central destination for the notification category — central ingests each forwarded notification into the Notifications table and acks the site, on which the engine clears the buffered message.
• Site Call Audit: The central observability sibling for the cached-call categories — this engine emits CachedCallTelemetry on every tracking-table transition, answers CachedCallReconcileRequest pulls, and executes the RetryParkedOperation / DiscardParkedOperation commands it relays.
• Health Monitoring: Reports buffer depth metrics, including the notification backlog covering the site→central forward leg.