ScadaBridge/docs/requirements/Component-NotificationOutbox.md

# Component: Notification Outbox

## Purpose

The Notification Outbox is the central component that receives store-and-forwarded notifications from site clusters, logs every one to the `Notifications` table in the central configuration database, and delivers them through per-type delivery adapters. The `Notifications` table is the single source of audit truth: every notification — successfully delivered, parked, or discarded — has exactly one durable row. The outbox provides delivery retry, parking of failures, per-notification status tracking, and KPIs for delivery health.

This inverts where notification delivery happens. Sites no longer send notifications directly via SMTP; a site script's notification is store-and-forwarded to central, and the central outbox owns dispatch and delivery.

## Location

Central cluster. The `NotificationOutboxActor` is a **singleton on the active central node**. It is the first outbox component to live centrally — the Store-and-Forward Engine remains site-only.

## Responsibilities

- Own the durable central queue — the `Notifications` table in the central MS SQL database.
- Ingest store-and-forwarded notifications from sites, insert-if-not-exists on `NotificationId`, and ack the site only after the row is persisted.
- Run the dispatcher loop: poll due rows, resolve the target notification list, and deliver via the matching adapter.
- Schedule retries for transient failures and park notifications on permanent failure or exhausted retries.
- Track per-notification status across the delivery lifecycle.
- Compute delivery KPIs from the `Notifications` table for the Health Monitoring dashboard and the Central UI.
- Purge terminal rows daily after a configurable retention window.

SMTP and HTTP delivery is blocking I/O. A dispatch sweep runs **off the actor thread** on the thread pool (the sweep task pipes its completion back to the actor), so delivery never blocks the actor's message loop. Within a sweep, deliveries run with **bounded per-sweep parallelism** — a `SemaphoreSlim` caps the number of concurrent deliveries at `MaxParallelDeliveries` (default 4; set to 1 for strictly sequential delivery). Each concurrent delivery uses its own DI scope/repository, so there is no shared-`DbContext` contention.

## End-to-End Flow

```mermaid
%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    SCRIPT(["Site script: Notify.To('list').Send(subject, body)<br/>generate NotificationId (GUID) locally;<br/>return it to the script immediately"])
    SNF["Site Store-and-Forward Engine<br/>(notification category, target = central)<br/>durably forwards to central via Central-Site Communication<br/>(ClusterClient); buffers/retries if central is unreachable"]
    INGEST[("Central ingest: insert-if-not-exists on NotificationId<br/>to Notifications table (Pending)<br/>ack the site, site S and F clears the message")]
    OUTBOX["Central Notification Outbox actor<br/>(singleton, active central node)<br/>polls due rows; resolves the list;<br/>delivers via the matching adapter"]
    D1{Delivery outcome}
    DELIVERED(["Delivered"])
    RETRYING["Retrying<br/>(schedule NextAttemptAt)"]
    PARKED(["Parked"])

    SCRIPT --> SNF
    SNF --> INGEST
    INGEST --> OUTBOX
    OUTBOX --> D1
    D1 -->|success| DELIVERED
    D1 -->|transient failure| RETRYING
    D1 -->|"permanent failure /<br/>retries exhausted"| PARKED
    RETRYING -.->|retry due| OUTBOX

    classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef warn fill:#ffe6cc,stroke:#d79b00,color:#111111;
    classDef bad fill:#f8cecc,stroke:#b85450,color:#111111;
    classDef alt fill:#e1d5e7,stroke:#9673a6,color:#111111;
    class SCRIPT,DELIVERED start
    class SNF warn
    class INGEST proc
    class OUTBOX alt
    class D1,RETRYING dec
    class PARKED bad
```

The site forwards only `(listName, subject, body)` plus provenance — recipient resolution happens at central, at delivery time. This keeps notification-list definitions in one place and removes the deploy-to-sites artifact entirely.

`Notify.Status(notificationId)` returns a small status record — status, retry count, last error, and key timestamps (enqueued, delivered). While the notification is still in the site S&F buffer the site answers the query **locally** (status `Forwarding`); once forwarded, the query round-trips to central and reads the `Notifications` table.

## The `Notifications` Table

The table is type-agnostic so it can record any notification type the system supports — `Email` and `Sms` today; others can be added by registering a new `INotificationDeliveryAdapter`. One row per notification.

| Field | Notes |
|---|---|
| `NotificationId` | GUID, primary key. Generated at the **site**; used as the idempotency key. |
| `Type` | `Email` / `Sms` / … discriminator. Stamped at ingest from the target list's `Type` (default `Email` if the list is not found — delivery parks on "list not found" regardless). |
| `ListName` | Target notification list. |
| `Subject`, `Body` | Plain-text content. |
| `TypeData` | JSON — extensibility hook for future per-type fields. |
| `Status` | Lifecycle state — one of `Pending`, `Retrying`, `Delivered`, `Parked`, `Discarded`. See Status Lifecycle below. |
| `RetryCount` | Delivery attempts so far. |
| `LastError` | Detail of the most recent failure. |
| `ResolvedTargets` | Who the notification actually went to — snapshotted by central at delivery time, for audit. |
| `SourceSiteId`, `SourceInstanceId`, `SourceScript` | Provenance. |
| `SourceNode` | The cluster node on which the notification was enqueued — `node-a` / `node-b` for site-originated rows (qualified by `SourceSiteId`). Nullable. Carried verbatim from the site through the S&F handoff. |
| `SiteEnqueuedAt` | When the script called `Send()` (carried from the site). |
| `CreatedAt` | When central ingested the row. |
| `LastAttemptAt`, `NextAttemptAt`, `DeliveredAt` | Delivery timestamps. |

All timestamps are UTC.

### Status Lifecycle

- `Forwarding` — in the site S&F buffer, not yet received by central. **Site-local only** — never stored in the central `Notifications` table; reported by `Notify.Status` while the site still holds the notification.
- `Pending` — ingested by central, awaiting first dispatch.
- `Retrying` — a transient failure occurred; `NextAttemptAt` schedules the next attempt.
- `Delivered` — terminal, success.
- `Parked` — terminal-not-delivered: a permanent failure, or retries exhausted. `LastError` distinguishes which.
- `Discarded` — terminal, reached **only by operator action** on a parked notification. The row is kept (not deleted) so the table remains a complete audit record.

The Notification Outbox and the central [`Site Call Audit`](Component-SiteCallAudit.md) component share the `TrackedOperationId` tracking model and this status lifecycle, but differ in delivery locality: the Notification Outbox **delivers** notifications itself (central SMTP), whereas Site Call Audit only **audits** cached calls delivered site-locally by the site Store-and-Forward Engine — it is not a dispatcher.

### Retry Policy

Delivery retry reuses the central SMTP configuration's max-retry-count and fixed retry interval for Email notifications. SMS notifications reuse the `SmsConfiguration`'s retry settings. The interval is fixed (no exponential backoff), consistent with the existing fixed-interval store-and-forward convention.

### Retention

Terminal rows (`Delivered`, `Parked`, `Discarded`) are removed by a **daily purge job** after a configurable window (default 365 days). This preserves a strong audit trail while bounding table growth. Non-terminal rows are never purged.

## Ingest & Idempotency

The site→central handoff is **at-least-once**. Central ingests an inbound notification submission with an insert-if-not-exists on `NotificationId`, then acks the site; the site S&F engine clears the message only on that ack. Because central acks only after the row is persisted (ack-after-persist), a lost ack causes the site to resend, and the GUID `NotificationId` idempotency key makes the resend harmless — the duplicate insert is a no-op.

A rare central failover mid-delivery could re-send one already-`Delivered` notification. This is an accepted trade-off, consistent with the duplicate-delivery trade-off the Store-and-Forward Engine already accepts.

## Dispatcher

The dispatcher loop runs on a fixed interval. On each tick the `NotificationOutboxActor`:

1. Polls the `Notifications` table for **due rows** — `Pending` rows, and `Retrying` rows whose `NextAttemptAt` has passed.
2. Resolves the target notification list to its recipients/targets at central, at delivery time.
3. Hands the notification to the delivery adapter registered for its `Type`. Deliveries within the sweep run with bounded parallelism (`MaxParallelDeliveries`, default 4), each on its own DI scope, off the actor thread. **That per-delivery scope owns the channel adapter as well as the outbox repository** — see Delivery Adapters below.
4. Applies the result:
   - **success** → `Delivered`, set `DeliveredAt`, snapshot `ResolvedTargets`.
   - **transient failure** → `Retrying`, increment `RetryCount`, set `NextAttemptAt`, record `LastError`; once retries are exhausted → `Parked`.
   - **permanent failure** → `Parked`, record `LastError`.

Each delivery attempt also writes a `Notification.Attempt` row to the central `AuditLog` via `ICentralAuditWriter`; a transition to a terminal status (`Delivered` / `Parked` / `Discarded`) writes a `Notification.Terminal` row. Audit writes are **direct** (no telemetry — the dispatcher runs at central), insert-if-not-exists on `EventId`. The site-emitted `Notification.Enqueued` row arrives separately via the standard audit telemetry channel from the site's SQLite write-buffer, so the full per-notification audit trail is `Enqueued` (site-originated) → `Attempt` × N (central direct-write) → `Terminal` (central direct-write). See [Component-AuditLog.md](Component-AuditLog.md), Central direct-write (central-originated events).

The operational `Notifications` table remains the **source of truth** for the dispatcher and for Retry/Discard actions; the `AuditLog` rows are immutable shadows. Operator Retry/Discard still mutates only the `Notifications` row, and each transition emits the corresponding audit row **carrying the operator identity as `Actor`** so an operator action is attributable: an operator **Retry** on a parked notification emits a `Notification`-channel `NotifyDeliver` row with status `Submitted` (recording *who un-parked it* — otherwise the trail reads `Parked → Attempt → Delivered` with no operator on record); a **Discard** emits the `Notification.Terminal` row with the operator as `Actor`. The username is captured at the Central UI and flows in on `RetryNotificationRequest` / `DiscardNotificationRequest` (`RequestedBy`); it is `null` (and the `Actor` falls back to `system`) only where no authenticated identity exists. These operator-attributed rows are best-effort like every other audit write — a writer failure never aborts the Retry/Discard.

**Audit-write failure never affects delivery.** If the `ICentralAuditWriter` direct-write fails (transient DB error, schema lock, etc.) the dispatcher logs the failure and increments the `CentralAuditWriteFailures` health metric (see Health Monitoring #11), but the delivery attempt's outcome on the `Notifications` row stands. The audit row is recovered by re-emission on the next dispatcher tick or by the on-startup reconciliation sweep; central never aborts a notification because audit failed.

## Delivery Adapters

A delivery adapter implementing `INotificationDeliveryAdapter` is registered per `Type`. Each `Deliver(...)` call returns one of `success | transient failure | permanent failure`, mirroring the External System Gateway error-classification pattern.

- **Email adapter.** The existing SMTP composition/send logic, relocated to the central cluster. Sends a single BCC email to all list recipients.
- **SMS adapter (Twilio REST).** Added in T9/T10 (2026-06-19). Sends one Twilio REST request per recipient phone number. Per-recipient results are rolled up: all-accepted → `Success`; any-transient → retry/park; mix of accepted + permanent-bad → `Success` with bad numbers noted in `LastError`; all-permanent → `Permanent` (Park). See [Component-NotificationService.md](Component-NotificationService.md), SMS Delivery Adapter.

The outbox dispatches by looking up the adapter registered for the notification's `Type`. If no adapter is registered for a given `Type`, the notification is parked with a "no adapter" error — the seam is open for future delivery channels. Where more than one adapter declares the same `Type`, the last registered wins, mirroring DI's own last-wins resolution.

**Adapters are scoped services, resolved once per delivery.** Both shipped adapters hold a scoped `INotificationRepository` over the EF `DbContext`, so an adapter instance is bound to exactly one DbContext. The dispatcher therefore resolves the matching adapter *inside the per-delivery DI scope* it already opens for the outbox repository — never once per sweep, and never cached on the singleton actor. Sharing one adapter across the `MaxParallelDeliveries` concurrent deliveries would put several deliveries on one DbContext, which fails with `InvalidOperationException: A second operation was started on this context instance before a previous operation completed`; the affected delivery's outcome is lost, its row is left unwritten, and the wasted attempt eats into the row's retry budget toward `Parked`. An adapter must accordingly carry **no state across deliveries**: anything genuinely shared (connection pools, token caches) belongs in a singleton or factory the adapter depends on — as the Email adapter's SMTP client factory and the SMS adapter's `IHttpClientFactory` already are — not in adapter fields.

The Central UI's channel catalog is a separate consumer of the same registrations: it enumerates the registered adapters' `Type` values to populate the notification-list Type selector, so the set of offerable channels can never drift from the set the central node can actually deliver.

Delivery adapters are provided by the Notification Service, which manages notification-list, SMTP, and SMS definitions and supplies the stateless per-type "deliver one notification" implementations. The SMS `SmsConfiguration` (including encrypted Auth Token) travels in Transport bundles alongside SMTP config.

## Active/Standby Behavior

The `NotificationOutboxActor` is a singleton on the active central node. All outbox state lives in MS SQL, which is already the central HA store, so **no Akka-level replication is needed** (unlike the site S&F engine). On central failover the new active node resumes dispatch directly from the `Notifications` table — `Pending` rows and due `Retrying` rows are picked up on the next dispatcher tick.

## Monitoring

### KPIs

KPIs are central-computed from the `Notifications` table — global, with a per-source-site breakdown:

- **Queue depth** — count of `Pending` + `Retrying`.
- **Stuck count** — `Pending` / `Retrying` rows older than the configurable stuck-age threshold.
- **Parked count** — count of `Parked`.
- **Delivered (last interval)** — count of `Delivered` since the previous sample.
- **Oldest pending age** — age of the oldest non-terminal notification.

KPIs are point-in-time, computed on demand from the table. The configurable row retention (default 365 days) answers historical questions directly, so no separate time-series store is added.

### Stuck Detection

A notification is **stuck** if it is `Pending` or `Retrying` and older than a configurable age threshold (default 10 minutes). Detection is **display-only** — a count KPI and a row badge. There is no automated escalation or alerting, consistent with the system-wide no-alerting policy.

### Surfacing

- **Health Monitoring dashboard** — headline KPI tiles: queue depth, stuck count, parked count. These are central-computed and are not part of the site health report. The site S&F notification backlog remains a separate site health metric covering the site→central leg.
- **Central UI "Notification Outbox" page** — KPI tiles plus a queryable notification list: filter by status, type, source site, list, and time range; a stuck-only toggle; keyword search on subject. Parked notifications offer **Retry** (→ `Pending`, reset `RetryCount` / `NextAttemptAt`) and **Discard** (→ `Discarded`) actions. Stuck rows are badged.

Both operator actions are **read-then-write against a row the daily retention purge may delete in between**, so the write reports whether it matched a row and the actor answers `Success: false` / `"notification not found"` when it did not — never a success against a notification that no longer exists (and no audit row is emitted for the action that did not happen). The dispatcher's own delivery-state write takes the opposite stance on the same signal: the delivery has already happened, there is nothing left to retry, so a vanished row is logged as a warning and the audit rows stand as the durable record.

## Configuration

The component is configured via `NotificationOutboxOptions`, bound from an `appsettings.json` section on the central host (Options pattern):

- **Dispatch interval** — how often the dispatcher loop polls for due rows.
- **Stuck-age threshold** — age beyond which a non-terminal notification is counted as stuck (default 10 minutes).
- **Terminal-row retention window** — age after which terminal rows are removed by the daily purge job (default 365 days).

Delivery max-retry-count and retry interval are not part of `NotificationOutboxOptions` — they are reused from the central SMTP configuration.

## Dependencies

- **Notification Service**: Provides notification-list, SMTP, and SMS definitions, and the per-type delivery adapters the outbox invokes (Email + Twilio SMS).
- **Configuration Database**: Hosts the `Notifications` table; provides the entity POCO, repository, and EF migration for outbox persistence.
- **Central–Site Communication**: Carries inbound notification submissions and acks between sites and central.
- **Audit Log (#23)**: The dispatcher direct-writes `Notification.Attempt` and `Notification.Terminal` rows to the central `AuditLog` via `ICentralAuditWriter` (insert-if-not-exists on `EventId`); the site-emitted `Notification.Enqueued` row arrives via the standard audit telemetry channel. See [Component-AuditLog.md](Component-AuditLog.md), Central direct-write (central-originated events).
- **Health Monitoring**: Consumes the outbox KPIs as central-computed headline metrics.
- **Central UI**: Hosts the Notification Outbox page.

## Interactions

- **Site Store-and-Forward Engine**: Forwards notifications to central via Central–Site Communication; the outbox ingests them and acks once persisted.
- **Notification Service**: Supplies delivery adapters (Email + SMS) and resolves notification lists at delivery time.
- **Central UI**: Queries the `Notifications` table for the Notification Outbox page and issues operator Retry/Discard actions on parked notifications.
- **Health Monitoring**: Polls the outbox for KPI tiles on the health dashboard.
- **KPI History (#26)**: Emits `IKpiSampleSource` (`NotificationOutboxKpiSampleSource`, Global + per-Site + per-Node) consumed by the KpiHistory recorder (#26), reusing the existing `Compute…KpisAsync` reads; the resulting `queueDepth` / `stuckCount` / `parkedCount` / `deliveredLastInterval` / `oldestPendingAgeSeconds` series render as trends on the Notification Outbox page via `KpiTrendChart`. See [Component-KpiHistory.md](Component-KpiHistory.md).