ScadaBridge/docs/requirements/Component-NotificationOutbox.md

Component: Notification Outbox

Purpose

The Notification Outbox is the central component that receives store-and-forwarded notifications from site clusters, logs every one to the Notifications table in the central configuration database, and delivers them through per-type delivery adapters. The Notifications table is the single source of audit truth: every notification — successfully delivered, parked, or discarded — has exactly one durable row. The outbox provides delivery retry, parking of failures, per-notification status tracking, and KPIs for delivery health.

This inverts where notification delivery happens. Sites no longer send notifications directly via SMTP; a site script's notification is store-and-forwarded to central, and the central outbox owns dispatch and delivery.

Location

Central cluster. The NotificationOutboxActor is a singleton on the active central node. It is the first outbox component to live centrally — the Store-and-Forward Engine remains site-only.

Responsibilities

• Own the durable central queue — the Notifications table in the central MS SQL database.
• Ingest store-and-forwarded notifications from sites, insert-if-not-exists on NotificationId, and ack the site only after the row is persisted.
• Run the dispatcher loop: poll due rows, resolve the target notification list, and deliver via the matching adapter.
• Schedule retries for transient failures and park notifications on permanent failure or exhausted retries.
• Track per-notification status across the delivery lifecycle.
• Compute delivery KPIs from the Notifications table for the Health Monitoring dashboard and the Central UI.
• Purge terminal rows daily after a configurable retention window.

SMTP and HTTP delivery is blocking I/O. A dispatch sweep runs off the actor thread on the thread pool (the sweep task pipes its completion back to the actor), so delivery never blocks the actor's message loop. Within a sweep, deliveries run with bounded per-sweep parallelism — a SemaphoreSlim caps the number of concurrent deliveries at MaxParallelDeliveries (default 4; set to 1 for strictly sequential delivery). Each concurrent delivery uses its own DI scope/repository, so there is no shared-DbContext contention.

End-to-End Flow

%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    SCRIPT(["Site script: Notify.To('list').Send(subject, body)<br/>generate NotificationId (GUID) locally;<br/>return it to the script immediately"])
    SNF["Site Store-and-Forward Engine<br/>(notification category, target = central)<br/>durably forwards to central via Central-Site Communication<br/>(ClusterClient); buffers/retries if central is unreachable"]
    INGEST[("Central ingest: insert-if-not-exists on NotificationId<br/>to Notifications table (Pending)<br/>ack the site, site S and F clears the message")]
    OUTBOX["Central Notification Outbox actor<br/>(singleton, active central node)<br/>polls due rows; resolves the list;<br/>delivers via the matching adapter"]
    D1{Delivery outcome}
    DELIVERED(["Delivered"])
    RETRYING["Retrying<br/>(schedule NextAttemptAt)"]
    PARKED(["Parked"])

    SCRIPT --> SNF
    SNF --> INGEST
    INGEST --> OUTBOX
    OUTBOX --> D1
    D1 -->|success| DELIVERED
    D1 -->|transient failure| RETRYING
    D1 -->|"permanent failure /<br/>retries exhausted"| PARKED
    RETRYING -.->|retry due| OUTBOX

    classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef warn fill:#ffe6cc,stroke:#d79b00,color:#111111;
    classDef bad fill:#f8cecc,stroke:#b85450,color:#111111;
    classDef alt fill:#e1d5e7,stroke:#9673a6,color:#111111;
    class SCRIPT,DELIVERED start
    class SNF warn
    class INGEST proc
    class OUTBOX alt
    class D1,RETRYING dec
    class PARKED bad

The site forwards only (listName, subject, body) plus provenance — recipient resolution happens at central, at delivery time. This keeps notification-list definitions in one place and removes the deploy-to-sites artifact entirely.

Notify.Status(notificationId) returns a small status record — status, retry count, last error, and key timestamps (enqueued, delivered). While the notification is still in the site S&F buffer the site answers the query locally (status Forwarding); once forwarded, the query round-trips to central and reads the Notifications table.

The Notifications Table

The table is type-agnostic so it can record any notification type the system supports — Email and Sms today; others can be added by registering a new INotificationDeliveryAdapter. One row per notification.

Field	Notes
NotificationId	GUID, primary key. Generated at the site; used as the idempotency key.
Type	Email / Sms / … discriminator. Stamped at ingest from the target list's Type (default Email if the list is not found — delivery parks on "list not found" regardless).
ListName	Target notification list.
Subject, Body	Plain-text content.
TypeData	JSON — extensibility hook for future per-type fields.
Status	Lifecycle state — one of Pending, Retrying, Delivered, Parked, Discarded. See Status Lifecycle below.
RetryCount	Delivery attempts so far.
LastError	Detail of the most recent failure.
ResolvedTargets	Who the notification actually went to — snapshotted by central at delivery time, for audit.
SourceSiteId, SourceInstanceId, SourceScript	Provenance.
SourceNode	The cluster node on which the notification was enqueued — node-a / node-b for site-originated rows (qualified by SourceSiteId). Nullable. Carried verbatim from the site through the S&F handoff.
SiteEnqueuedAt	When the script called Send() (carried from the site).
CreatedAt	When central ingested the row.
LastAttemptAt, NextAttemptAt, DeliveredAt	Delivery timestamps.

All timestamps are UTC.

Status Lifecycle

• Forwarding — in the site S&F buffer, not yet received by central. Site-local only — never stored in the central Notifications table; reported by Notify.Status while the site still holds the notification.
• Pending — ingested by central, awaiting first dispatch.
• Retrying — a transient failure occurred; NextAttemptAt schedules the next attempt.
• Delivered — terminal, success.
• Parked — terminal-not-delivered: a permanent failure, or retries exhausted. LastError distinguishes which.
• Discarded — terminal, reached only by operator action on a parked notification. The row is kept (not deleted) so the table remains a complete audit record.

The Notification Outbox and the central Site Call Audit component share the TrackedOperationId tracking model and this status lifecycle, but differ in delivery locality: the Notification Outbox delivers notifications itself (central SMTP), whereas Site Call Audit only audits cached calls delivered site-locally by the site Store-and-Forward Engine — it is not a dispatcher.

Retry Policy

Delivery retry reuses the central SMTP configuration's max-retry-count and fixed retry interval for Email notifications. SMS notifications reuse the SmsConfiguration's retry settings. The interval is fixed (no exponential backoff), consistent with the existing fixed-interval store-and-forward convention.

Retention

Terminal rows (Delivered, Parked, Discarded) are removed by a daily purge job after a configurable window (default 365 days). This preserves a strong audit trail while bounding table growth. Non-terminal rows are never purged.

Ingest & Idempotency

The site→central handoff is at-least-once. Central ingests an inbound notification submission with an insert-if-not-exists on NotificationId, then acks the site; the site S&F engine clears the message only on that ack. Because central acks only after the row is persisted (ack-after-persist), a lost ack causes the site to resend, and the GUID NotificationId idempotency key makes the resend harmless — the duplicate insert is a no-op.

A rare central failover mid-delivery could re-send one already-Delivered notification. This is an accepted trade-off, consistent with the duplicate-delivery trade-off the Store-and-Forward Engine already accepts.

Dispatcher

The dispatcher loop runs on a fixed interval. On each tick the NotificationOutboxActor:

1. Polls the Notifications table for due rows — Pending rows, and Retrying rows whose NextAttemptAt has passed.
2. Resolves the target notification list to its recipients/targets at central, at delivery time.
3. Hands the notification to the delivery adapter registered for its Type. Deliveries within the sweep run with bounded parallelism (MaxParallelDeliveries, default 4), each on its own DI scope/repository, off the actor thread.
4. Applies the result:
   • success → Delivered, set DeliveredAt, snapshot ResolvedTargets.
   • transient failure → Retrying, increment RetryCount, set NextAttemptAt, record LastError; once retries are exhausted → Parked.
   • permanent failure → Parked, record LastError.

Each delivery attempt also writes a Notification.Attempt row to the central AuditLog via ICentralAuditWriter; a transition to a terminal status (Delivered / Parked / Discarded) writes a Notification.Terminal row. Audit writes are direct (no telemetry — the dispatcher runs at central), insert-if-not-exists on EventId. The site-emitted Notification.Enqueued row arrives separately via the standard audit telemetry channel from the site's SQLite write-buffer, so the full per-notification audit trail is Enqueued (site-originated) → Attempt × N (central direct-write) → Terminal (central direct-write). See Component-AuditLog.md, Central direct-write (central-originated events).

The operational Notifications table remains the source of truth for the dispatcher and for Retry/Discard actions; the AuditLog rows are immutable shadows. Operator Retry/Discard still mutates only the Notifications row, and each transition emits the corresponding audit row carrying the operator identity as Actor so an operator action is attributable: an operator Retry on a parked notification emits a Notification-channel NotifyDeliver row with status Submitted (recording who un-parked it — otherwise the trail reads Parked → Attempt → Delivered with no operator on record); a Discard emits the Notification.Terminal row with the operator as Actor. The username is captured at the Central UI and flows in on RetryNotificationRequest / DiscardNotificationRequest (RequestedBy); it is null (and the Actor falls back to system) only where no authenticated identity exists. These operator-attributed rows are best-effort like every other audit write — a writer failure never aborts the Retry/Discard.

Audit-write failure never affects delivery. If the ICentralAuditWriter direct-write fails (transient DB error, schema lock, etc.) the dispatcher logs the failure and increments the CentralAuditWriteFailures health metric (see Health Monitoring #11), but the delivery attempt's outcome on the Notifications row stands. The audit row is recovered by re-emission on the next dispatcher tick or by the on-startup reconciliation sweep; central never aborts a notification because audit failed.

Delivery Adapters

A delivery adapter implementing INotificationDeliveryAdapter is registered per Type. Each Deliver(...) call returns one of success | transient failure | permanent failure, mirroring the External System Gateway error-classification pattern.

• Email adapter. The existing SMTP composition/send logic, relocated to the central cluster. Sends a single BCC email to all list recipients.
• SMS adapter (Twilio REST). Added in T9/T10 (2026-06-19). Sends one Twilio REST request per recipient phone number. Per-recipient results are rolled up: all-accepted → Success; any-transient → retry/park; mix of accepted + permanent-bad → Success with bad numbers noted in LastError; all-permanent → Permanent (Park). See Component-NotificationService.md, SMS Delivery Adapter.

The outbox dispatches by looking up the adapter registered for the notification's Type. If no adapter is registered for a given Type, the notification is parked with a "no adapter" error — the seam is open for future delivery channels.

Delivery adapters are provided by the Notification Service, which manages notification-list, SMTP, and SMS definitions and supplies the stateless per-type "deliver one notification" implementations. The SMS SmsConfiguration (including encrypted Auth Token) travels in Transport bundles alongside SMTP config.

Active/Standby Behavior

The NotificationOutboxActor is a singleton on the active central node. All outbox state lives in MS SQL, which is already the central HA store, so no Akka-level replication is needed (unlike the site S&F engine). On central failover the new active node resumes dispatch directly from the Notifications table — Pending rows and due Retrying rows are picked up on the next dispatcher tick.

Monitoring

KPIs

KPIs are central-computed from the Notifications table — global, with a per-source-site breakdown:

• Queue depth — count of Pending + Retrying.
• Stuck count — Pending / Retrying rows older than the configurable stuck-age threshold.
• Parked count — count of Parked.
• Delivered (last interval) — count of Delivered since the previous sample.
• Oldest pending age — age of the oldest non-terminal notification.

KPIs are point-in-time, computed on demand from the table. The configurable row retention (default 365 days) answers historical questions directly, so no separate time-series store is added.

Stuck Detection

A notification is stuck if it is Pending or Retrying and older than a configurable age threshold (default 10 minutes). Detection is display-only — a count KPI and a row badge. There is no automated escalation or alerting, consistent with the system-wide no-alerting policy.

Surfacing

• Health Monitoring dashboard — headline KPI tiles: queue depth, stuck count, parked count. These are central-computed and are not part of the site health report. The site S&F notification backlog remains a separate site health metric covering the site→central leg.
• Central UI "Notification Outbox" page — KPI tiles plus a queryable notification list: filter by status, type, source site, list, and time range; a stuck-only toggle; keyword search on subject. Parked notifications offer Retry (→ Pending, reset RetryCount / NextAttemptAt) and Discard (→ Discarded) actions. Stuck rows are badged.

Both operator actions are read-then-write against a row the daily retention purge may delete in between, so the write reports whether it matched a row and the actor answers Success: false / "notification not found" when it did not — never a success against a notification that no longer exists (and no audit row is emitted for the action that did not happen). The dispatcher's own delivery-state write takes the opposite stance on the same signal: the delivery has already happened, there is nothing left to retry, so a vanished row is logged as a warning and the audit rows stand as the durable record.

Configuration

The component is configured via NotificationOutboxOptions, bound from an appsettings.json section on the central host (Options pattern):

• Dispatch interval — how often the dispatcher loop polls for due rows.
• Stuck-age threshold — age beyond which a non-terminal notification is counted as stuck (default 10 minutes).
• Terminal-row retention window — age after which terminal rows are removed by the daily purge job (default 365 days).

Delivery max-retry-count and retry interval are not part of NotificationOutboxOptions — they are reused from the central SMTP configuration.

Dependencies

• Notification Service: Provides notification-list, SMTP, and SMS definitions, and the per-type delivery adapters the outbox invokes (Email + Twilio SMS).
• Configuration Database: Hosts the Notifications table; provides the entity POCO, repository, and EF migration for outbox persistence.
• Central–Site Communication: Carries inbound notification submissions and acks between sites and central.
• Audit Log (#23): The dispatcher direct-writes Notification.Attempt and Notification.Terminal rows to the central AuditLog via ICentralAuditWriter (insert-if-not-exists on EventId); the site-emitted Notification.Enqueued row arrives via the standard audit telemetry channel. See Component-AuditLog.md, Central direct-write (central-originated events).
• Health Monitoring: Consumes the outbox KPIs as central-computed headline metrics.
• Central UI: Hosts the Notification Outbox page.

Interactions

• Site Store-and-Forward Engine: Forwards notifications to central via Central–Site Communication; the outbox ingests them and acks once persisted.
• Notification Service: Supplies delivery adapters (Email + SMS) and resolves notification lists at delivery time.
• Central UI: Queries the Notifications table for the Notification Outbox page and issues operator Retry/Discard actions on parked notifications.
• Health Monitoring: Polls the outbox for KPI tiles on the health dashboard.
• KPI History (#26): Emits IKpiSampleSource (NotificationOutboxKpiSampleSource, Global + per-Site + per-Node) consumed by the KpiHistory recorder (#26), reusing the existing Compute…KpisAsync reads; the resulting queueDepth / stuckCount / parkedCount / deliveredLastInterval / oldestPendingAgeSeconds series render as trends on the Notification Outbox page via KpiTrendChart. See Component-KpiHistory.md.