Joseph Doherty
5d075f1374
Six adversarial-review findings in the central SQL/ingest layer. F1 (AuditLogRepository.InsertChunkAsync) — the set-based ingest declared each string parameter at its COLUMN width (Actor/Target 256, Action 64, Outcome 16, Category 32, SourceNode 64), so SqlClient truncated an over-long value at bind time and committed the mutilated row — silent, in an append-only store, with no PayloadTruncated flag — while the per-row and reconciliation paths sent the same value in full and let the server reject it with 2628. Bind at the value's own length instead; explicit SqlDbType is kept (it fixes the VALUES constructor's derived column types and datetime2 precision). Design: reject everywhere, truncate nowhere — matching today's per-row behaviour. F2 (SiteCallAuditRepository.UpsertAsync) — the single-statement upsert ran the monotonic UPDATE first and INSERTed only if nothing matched. Two writers racing the first packet of one TrackedOperationId (the cached dual-write and the reconciliation pull carry DIFFERENT lifecycle states) both matched nothing, and the loser then skipped its INSERT or swallowed a 2627 — dropping its Status/RetryCount/HttpStatus/TerminalAtUtc. Legs swapped to `IF NOT EXISTS … INSERT; UPDATE <monotonic>` — still one round trip, and the loser's UPDATE now lands on the winner's row. The duplicate-key catch re-runs the monotonic UPDATE for the same reason. Moved to raw SQL with explicitly-typed parameters so the intricate rank predicate exists in exactly one place (an untyped DateTime would bind as `datetime` and round the freshness tiebreaker). F3 (docs/plans/sql/*.sql) — filtered-index DDL failed with error 1934 under the documented `docker exec … sqlcmd` path, which defaults QUOTED_IDENTIFIER OFF; once IX_Notifications_Delivered exists, QI-OFF DML on Notifications fails too. All four scripts now open with `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO` (own batch, so it is in force when the next batch parses), and the migration convention in Component-ConfigurationDatabase.md documents `sqlcmd -I`. Verified live: the pre-fix script fails 1934 without -I, the fixed one applies. F4 (SiteCallAuditActor) — the off-mailbox reconciliation/purge passes reuse the injected repository, so tests drove one DbContext from the pass and a mailbox handler concurrently. Serialized at the CALL via a private SerializedRepository wrapper applied only by the test constructors, rather than running the pass on-mailbox: production keeps its PipeTo shape untouched, and the existing "a blocked drain does not stall ingest/query/KPI" regression tests stay meaningful (they would have been invalidated by suspending the mailbox). F5 (AuditLogIngestActor) — when the batch failed because the 20 s IngestBudget expired, the per-row fallback reused the same expired token: N instant failures, N counter bumps, zero accepted. The fallback now gets a fresh 5 s budget (inside the 30 s outer Ask), and a blown budget bumps the failure counter ONCE for the batch instead of once per row. F6 (NotificationOutboxRepository.UpdateAsync) — ExecuteUpdate's row count was discarded, so an operator Retry/Discard of a notification the retention purge had already deleted reported success (the pre-ExecuteUpdate code threw DbUpdateConcurrencyException). UpdateAsync now returns whether a row matched; the operator one-shots answer "notification not found" and emit no audit row for the action that did not happen, while the dispatcher logs a warning (its delivery already happened; nothing to retry). GetByIdAsync switched to AsNoTracking since the write is out-of-band. Tests: 5 new SQL-backed regressions (over-long Target rejected on both paths + boundary round-trip; concurrent first-write and already-created-by-another-writer upserts; vanished-row UpdateAsync), a token-identity pin on the ingest fallback, a repository-concurrency detector for the SiteCallAudit passes, and vanished-row operator-path tests. The F1/F2/F4 regressions were each confirmed failing against the pre-fix code. Suites: ConfigurationDatabase 369, AuditLog 378, SiteCallAudit 66, NotificationOutbox 152 — all green, solution builds with 0 warnings.
198 lines
18 KiB
Markdown
198 lines
18 KiB
Markdown
# Component: Notification Outbox
|
||
|
||
## Purpose
|
||
|
||
The Notification Outbox is the central component that receives store-and-forwarded notifications from site clusters, logs every one to the `Notifications` table in the central configuration database, and delivers them through per-type delivery adapters. The `Notifications` table is the single source of audit truth: every notification — successfully delivered, parked, or discarded — has exactly one durable row. The outbox provides delivery retry, parking of failures, per-notification status tracking, and KPIs for delivery health.
|
||
|
||
This inverts where notification delivery happens. Sites no longer send notifications directly via SMTP; a site script's notification is store-and-forwarded to central, and the central outbox owns dispatch and delivery.
|
||
|
||
## Location
|
||
|
||
Central cluster. The `NotificationOutboxActor` is a **singleton on the active central node**. It is the first outbox component to live centrally — the Store-and-Forward Engine remains site-only.
|
||
|
||
## Responsibilities
|
||
|
||
- Own the durable central queue — the `Notifications` table in the central MS SQL database.
|
||
- Ingest store-and-forwarded notifications from sites, insert-if-not-exists on `NotificationId`, and ack the site only after the row is persisted.
|
||
- Run the dispatcher loop: poll due rows, resolve the target notification list, and deliver via the matching adapter.
|
||
- Schedule retries for transient failures and park notifications on permanent failure or exhausted retries.
|
||
- Track per-notification status across the delivery lifecycle.
|
||
- Compute delivery KPIs from the `Notifications` table for the Health Monitoring dashboard and the Central UI.
|
||
- Purge terminal rows daily after a configurable retention window.
|
||
|
||
SMTP and HTTP delivery is blocking I/O. A dispatch sweep runs **off the actor thread** on the thread pool (the sweep task pipes its completion back to the actor), so delivery never blocks the actor's message loop. Within a sweep, deliveries run with **bounded per-sweep parallelism** — a `SemaphoreSlim` caps the number of concurrent deliveries at `MaxParallelDeliveries` (default 4; set to 1 for strictly sequential delivery). Each concurrent delivery uses its own DI scope/repository, so there is no shared-`DbContext` contention.
|
||
|
||
## End-to-End Flow
|
||
|
||
```mermaid
|
||
%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
|
||
flowchart TD
|
||
SCRIPT(["Site script: Notify.To('list').Send(subject, body)<br/>generate NotificationId (GUID) locally;<br/>return it to the script immediately"])
|
||
SNF["Site Store-and-Forward Engine<br/>(notification category, target = central)<br/>durably forwards to central via Central-Site Communication<br/>(ClusterClient); buffers/retries if central is unreachable"]
|
||
INGEST[("Central ingest: insert-if-not-exists on NotificationId<br/>to Notifications table (Pending)<br/>ack the site, site S and F clears the message")]
|
||
OUTBOX["Central Notification Outbox actor<br/>(singleton, active central node)<br/>polls due rows; resolves the list;<br/>delivers via the matching adapter"]
|
||
D1{Delivery outcome}
|
||
DELIVERED(["Delivered"])
|
||
RETRYING["Retrying<br/>(schedule NextAttemptAt)"]
|
||
PARKED(["Parked"])
|
||
|
||
SCRIPT --> SNF
|
||
SNF --> INGEST
|
||
INGEST --> OUTBOX
|
||
OUTBOX --> D1
|
||
D1 -->|success| DELIVERED
|
||
D1 -->|transient failure| RETRYING
|
||
D1 -->|"permanent failure /<br/>retries exhausted"| PARKED
|
||
RETRYING -.->|retry due| OUTBOX
|
||
|
||
classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
|
||
classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
|
||
classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
|
||
classDef warn fill:#ffe6cc,stroke:#d79b00,color:#111111;
|
||
classDef bad fill:#f8cecc,stroke:#b85450,color:#111111;
|
||
classDef alt fill:#e1d5e7,stroke:#9673a6,color:#111111;
|
||
class SCRIPT,DELIVERED start
|
||
class SNF warn
|
||
class INGEST proc
|
||
class OUTBOX alt
|
||
class D1,RETRYING dec
|
||
class PARKED bad
|
||
```
|
||
|
||
The site forwards only `(listName, subject, body)` plus provenance — recipient resolution happens at central, at delivery time. This keeps notification-list definitions in one place and removes the deploy-to-sites artifact entirely.
|
||
|
||
`Notify.Status(notificationId)` returns a small status record — status, retry count, last error, and key timestamps (enqueued, delivered). While the notification is still in the site S&F buffer the site answers the query **locally** (status `Forwarding`); once forwarded, the query round-trips to central and reads the `Notifications` table.
|
||
|
||
## The `Notifications` Table
|
||
|
||
The table is type-agnostic so it can record any notification type the system supports — `Email` and `Sms` today; others can be added by registering a new `INotificationDeliveryAdapter`. One row per notification.
|
||
|
||
| Field | Notes |
|
||
|---|---|
|
||
| `NotificationId` | GUID, primary key. Generated at the **site**; used as the idempotency key. |
|
||
| `Type` | `Email` / `Sms` / … discriminator. Stamped at ingest from the target list's `Type` (default `Email` if the list is not found — delivery parks on "list not found" regardless). |
|
||
| `ListName` | Target notification list. |
|
||
| `Subject`, `Body` | Plain-text content. |
|
||
| `TypeData` | JSON — extensibility hook for future per-type fields. |
|
||
| `Status` | Lifecycle state — one of `Pending`, `Retrying`, `Delivered`, `Parked`, `Discarded`. See Status Lifecycle below. |
|
||
| `RetryCount` | Delivery attempts so far. |
|
||
| `LastError` | Detail of the most recent failure. |
|
||
| `ResolvedTargets` | Who the notification actually went to — snapshotted by central at delivery time, for audit. |
|
||
| `SourceSiteId`, `SourceInstanceId`, `SourceScript` | Provenance. |
|
||
| `SourceNode` | The cluster node on which the notification was enqueued — `node-a` / `node-b` for site-originated rows (qualified by `SourceSiteId`). Nullable. Carried verbatim from the site through the S&F handoff. |
|
||
| `SiteEnqueuedAt` | When the script called `Send()` (carried from the site). |
|
||
| `CreatedAt` | When central ingested the row. |
|
||
| `LastAttemptAt`, `NextAttemptAt`, `DeliveredAt` | Delivery timestamps. |
|
||
|
||
All timestamps are UTC.
|
||
|
||
### Status Lifecycle
|
||
|
||
- `Forwarding` — in the site S&F buffer, not yet received by central. **Site-local only** — never stored in the central `Notifications` table; reported by `Notify.Status` while the site still holds the notification.
|
||
- `Pending` — ingested by central, awaiting first dispatch.
|
||
- `Retrying` — a transient failure occurred; `NextAttemptAt` schedules the next attempt.
|
||
- `Delivered` — terminal, success.
|
||
- `Parked` — terminal-not-delivered: a permanent failure, or retries exhausted. `LastError` distinguishes which.
|
||
- `Discarded` — terminal, reached **only by operator action** on a parked notification. The row is kept (not deleted) so the table remains a complete audit record.
|
||
|
||
The Notification Outbox and the central [`Site Call Audit`](Component-SiteCallAudit.md) component share the `TrackedOperationId` tracking model and this status lifecycle, but differ in delivery locality: the Notification Outbox **delivers** notifications itself (central SMTP), whereas Site Call Audit only **audits** cached calls delivered site-locally by the site Store-and-Forward Engine — it is not a dispatcher.
|
||
|
||
### Retry Policy
|
||
|
||
Delivery retry reuses the central SMTP configuration's max-retry-count and fixed retry interval for Email notifications. SMS notifications reuse the `SmsConfiguration`'s retry settings. The interval is fixed (no exponential backoff), consistent with the existing fixed-interval store-and-forward convention.
|
||
|
||
### Retention
|
||
|
||
Terminal rows (`Delivered`, `Parked`, `Discarded`) are removed by a **daily purge job** after a configurable window (default 365 days). This preserves a strong audit trail while bounding table growth. Non-terminal rows are never purged.
|
||
|
||
## Ingest & Idempotency
|
||
|
||
The site→central handoff is **at-least-once**. Central ingests an inbound notification submission with an insert-if-not-exists on `NotificationId`, then acks the site; the site S&F engine clears the message only on that ack. Because central acks only after the row is persisted (ack-after-persist), a lost ack causes the site to resend, and the GUID `NotificationId` idempotency key makes the resend harmless — the duplicate insert is a no-op.
|
||
|
||
A rare central failover mid-delivery could re-send one already-`Delivered` notification. This is an accepted trade-off, consistent with the duplicate-delivery trade-off the Store-and-Forward Engine already accepts.
|
||
|
||
## Dispatcher
|
||
|
||
The dispatcher loop runs on a fixed interval. On each tick the `NotificationOutboxActor`:
|
||
|
||
1. Polls the `Notifications` table for **due rows** — `Pending` rows, and `Retrying` rows whose `NextAttemptAt` has passed.
|
||
2. Resolves the target notification list to its recipients/targets at central, at delivery time.
|
||
3. Hands the notification to the delivery adapter registered for its `Type`. Deliveries within the sweep run with bounded parallelism (`MaxParallelDeliveries`, default 4), each on its own DI scope/repository, off the actor thread.
|
||
4. Applies the result:
|
||
- **success** → `Delivered`, set `DeliveredAt`, snapshot `ResolvedTargets`.
|
||
- **transient failure** → `Retrying`, increment `RetryCount`, set `NextAttemptAt`, record `LastError`; once retries are exhausted → `Parked`.
|
||
- **permanent failure** → `Parked`, record `LastError`.
|
||
|
||
Each delivery attempt also writes a `Notification.Attempt` row to the central `AuditLog` via `ICentralAuditWriter`; a transition to a terminal status (`Delivered` / `Parked` / `Discarded`) writes a `Notification.Terminal` row. Audit writes are **direct** (no telemetry — the dispatcher runs at central), insert-if-not-exists on `EventId`. The site-emitted `Notification.Enqueued` row arrives separately via the standard audit telemetry channel from the site's SQLite write-buffer, so the full per-notification audit trail is `Enqueued` (site-originated) → `Attempt` × N (central direct-write) → `Terminal` (central direct-write). See [Component-AuditLog.md](Component-AuditLog.md), Central direct-write (central-originated events).
|
||
|
||
The operational `Notifications` table remains the **source of truth** for the dispatcher and for Retry/Discard actions; the `AuditLog` rows are immutable shadows. Operator Retry/Discard still mutates only the `Notifications` row, and each transition emits the corresponding audit row **carrying the operator identity as `Actor`** so an operator action is attributable: an operator **Retry** on a parked notification emits a `Notification`-channel `NotifyDeliver` row with status `Submitted` (recording *who un-parked it* — otherwise the trail reads `Parked → Attempt → Delivered` with no operator on record); a **Discard** emits the `Notification.Terminal` row with the operator as `Actor`. The username is captured at the Central UI and flows in on `RetryNotificationRequest` / `DiscardNotificationRequest` (`RequestedBy`); it is `null` (and the `Actor` falls back to `system`) only where no authenticated identity exists. These operator-attributed rows are best-effort like every other audit write — a writer failure never aborts the Retry/Discard.
|
||
|
||
**Audit-write failure never affects delivery.** If the `ICentralAuditWriter` direct-write fails (transient DB error, schema lock, etc.) the dispatcher logs the failure and increments the `CentralAuditWriteFailures` health metric (see Health Monitoring #11), but the delivery attempt's outcome on the `Notifications` row stands. The audit row is recovered by re-emission on the next dispatcher tick or by the on-startup reconciliation sweep; central never aborts a notification because audit failed.
|
||
|
||
## Delivery Adapters
|
||
|
||
A delivery adapter implementing `INotificationDeliveryAdapter` is registered per `Type`. Each `Deliver(...)` call returns one of `success | transient failure | permanent failure`, mirroring the External System Gateway error-classification pattern.
|
||
|
||
- **Email adapter.** The existing SMTP composition/send logic, relocated to the central cluster. Sends a single BCC email to all list recipients.
|
||
- **SMS adapter (Twilio REST).** Added in T9/T10 (2026-06-19). Sends one Twilio REST request per recipient phone number. Per-recipient results are rolled up: all-accepted → `Success`; any-transient → retry/park; mix of accepted + permanent-bad → `Success` with bad numbers noted in `LastError`; all-permanent → `Permanent` (Park). See [Component-NotificationService.md](Component-NotificationService.md), SMS Delivery Adapter.
|
||
|
||
The outbox dispatches by looking up the adapter registered for the notification's `Type`. If no adapter is registered for a given `Type`, the notification is parked with a "no adapter" error — the seam is open for future delivery channels.
|
||
|
||
Delivery adapters are provided by the Notification Service, which manages notification-list, SMTP, and SMS definitions and supplies the stateless per-type "deliver one notification" implementations. The SMS `SmsConfiguration` (including encrypted Auth Token) travels in Transport bundles alongside SMTP config.
|
||
|
||
## Active/Standby Behavior
|
||
|
||
The `NotificationOutboxActor` is a singleton on the active central node. All outbox state lives in MS SQL, which is already the central HA store, so **no Akka-level replication is needed** (unlike the site S&F engine). On central failover the new active node resumes dispatch directly from the `Notifications` table — `Pending` rows and due `Retrying` rows are picked up on the next dispatcher tick.
|
||
|
||
## Monitoring
|
||
|
||
### KPIs
|
||
|
||
KPIs are central-computed from the `Notifications` table — global, with a per-source-site breakdown:
|
||
|
||
- **Queue depth** — count of `Pending` + `Retrying`.
|
||
- **Stuck count** — `Pending` / `Retrying` rows older than the configurable stuck-age threshold.
|
||
- **Parked count** — count of `Parked`.
|
||
- **Delivered (last interval)** — count of `Delivered` since the previous sample.
|
||
- **Oldest pending age** — age of the oldest non-terminal notification.
|
||
|
||
KPIs are point-in-time, computed on demand from the table. The configurable row retention (default 365 days) answers historical questions directly, so no separate time-series store is added.
|
||
|
||
### Stuck Detection
|
||
|
||
A notification is **stuck** if it is `Pending` or `Retrying` and older than a configurable age threshold (default 10 minutes). Detection is **display-only** — a count KPI and a row badge. There is no automated escalation or alerting, consistent with the system-wide no-alerting policy.
|
||
|
||
### Surfacing
|
||
|
||
- **Health Monitoring dashboard** — headline KPI tiles: queue depth, stuck count, parked count. These are central-computed and are not part of the site health report. The site S&F notification backlog remains a separate site health metric covering the site→central leg.
|
||
- **Central UI "Notification Outbox" page** — KPI tiles plus a queryable notification list: filter by status, type, source site, list, and time range; a stuck-only toggle; keyword search on subject. Parked notifications offer **Retry** (→ `Pending`, reset `RetryCount` / `NextAttemptAt`) and **Discard** (→ `Discarded`) actions. Stuck rows are badged.
|
||
|
||
Both operator actions are **read-then-write against a row the daily retention purge may delete in between**, so the write reports whether it matched a row and the actor answers `Success: false` / `"notification not found"` when it did not — never a success against a notification that no longer exists (and no audit row is emitted for the action that did not happen). The dispatcher's own delivery-state write takes the opposite stance on the same signal: the delivery has already happened, there is nothing left to retry, so a vanished row is logged as a warning and the audit rows stand as the durable record.
|
||
|
||
## Configuration
|
||
|
||
The component is configured via `NotificationOutboxOptions`, bound from an `appsettings.json` section on the central host (Options pattern):
|
||
|
||
- **Dispatch interval** — how often the dispatcher loop polls for due rows.
|
||
- **Stuck-age threshold** — age beyond which a non-terminal notification is counted as stuck (default 10 minutes).
|
||
- **Terminal-row retention window** — age after which terminal rows are removed by the daily purge job (default 365 days).
|
||
|
||
Delivery max-retry-count and retry interval are not part of `NotificationOutboxOptions` — they are reused from the central SMTP configuration.
|
||
|
||
## Dependencies
|
||
|
||
- **Notification Service**: Provides notification-list, SMTP, and SMS definitions, and the per-type delivery adapters the outbox invokes (Email + Twilio SMS).
|
||
- **Configuration Database**: Hosts the `Notifications` table; provides the entity POCO, repository, and EF migration for outbox persistence.
|
||
- **Central–Site Communication**: Carries inbound notification submissions and acks between sites and central.
|
||
- **Audit Log (#23)**: The dispatcher direct-writes `Notification.Attempt` and `Notification.Terminal` rows to the central `AuditLog` via `ICentralAuditWriter` (insert-if-not-exists on `EventId`); the site-emitted `Notification.Enqueued` row arrives via the standard audit telemetry channel. See [Component-AuditLog.md](Component-AuditLog.md), Central direct-write (central-originated events).
|
||
- **Health Monitoring**: Consumes the outbox KPIs as central-computed headline metrics.
|
||
- **Central UI**: Hosts the Notification Outbox page.
|
||
|
||
## Interactions
|
||
|
||
- **Site Store-and-Forward Engine**: Forwards notifications to central via Central–Site Communication; the outbox ingests them and acks once persisted.
|
||
- **Notification Service**: Supplies delivery adapters (Email + SMS) and resolves notification lists at delivery time.
|
||
- **Central UI**: Queries the `Notifications` table for the Notification Outbox page and issues operator Retry/Discard actions on parked notifications.
|
||
- **Health Monitoring**: Polls the outbox for KPI tiles on the health dashboard.
|
||
- **KPI History (#26)**: Emits `IKpiSampleSource` (`NotificationOutboxKpiSampleSource`, Global + per-Site + per-Node) consumed by the KpiHistory recorder (#26), reusing the existing `Compute…KpisAsync` reads; the resulting `queueDepth` / `stuckCount` / `parkedCount` / `deliveredLastInterval` / `oldestPendingAgeSeconds` series render as trends on the Notification Outbox page via `KpiTrendChart`. See [Component-KpiHistory.md](Component-KpiHistory.md).
|