fix(notification-outbox): per-attempt timestamps, Pending/Retrying split claim + sweep budget, SMTP MaxConcurrentConnections clamp (review 2026-10-21 06/COR-1, 06/PERF-5, 06/IMPL-2)
This commit is contained in:
@@ -115,10 +115,11 @@ Delivery has no row-level claim or lease: dispatchers read due rows and persist
|
||||
|
||||
The dispatcher loop runs on a fixed interval. On each tick the `NotificationOutboxActor`:
|
||||
|
||||
1. Polls the `Notifications` table for **due rows** — `Pending` rows, and `Retrying` rows whose `NextAttemptAt` has passed.
|
||||
1. Polls the `Notifications` table for **due rows** — `Pending` rows, and `Retrying` rows whose `NextAttemptAt` has passed — capped at `DispatchBatchSize` (default 100). The batch is **shared between the two statuses**: due `Retrying` rows take at most half of it and `Pending` rows the rest, each oldest-first, with a share the other status cannot fill back-filled. A single oldest-first claim used to let a retrying backlog (always older than fresh traffic) monopolise every sweep during a partial relay outage, so a new alarm email was not even attempted until the backlog parked.
|
||||
2. Resolves the target notification list to its recipients/targets at central, at delivery time.
|
||||
3. Hands the notification to the delivery adapter registered for its `Type`. Deliveries within the sweep run with bounded parallelism (`MaxParallelDeliveries`, default 4), each on its own DI scope, off the actor thread. **That per-delivery scope owns the channel adapter as well as the outbox repository** — see Delivery Adapters below.
|
||||
3. Hands the notification to the delivery adapter registered for its `Type`. Deliveries within the sweep run with bounded parallelism (`MaxParallelDeliveries`, default 4), each on its own DI scope, off the actor thread. **That per-delivery scope owns the channel adapter as well as the outbox repository** — see Delivery Adapters below. Email over the **SMTP** transport is further clamped per sweep to the active (lowest-Id) SMTP row's `MaxConcurrentConnections` when that value is positive, so a relay that limits connections is honoured; the EWS transport (which does not use the field) and SMS keep the full `MaxParallelDeliveries`. A sweep also has a **wall-clock budget** (`DispatchSweepBudget`, default 2 min): once it is spent, claimed notifications that have not yet started an attempt are skipped — left untouched and still due, so the next sweep re-claims them alongside newer rows — while attempts already in flight run to completion.
|
||||
4. Applies the result:
|
||||
Every timestamp written for an attempt (`LastAttemptAt`, `DeliveredAt`, `NextAttemptAt` = attempt instant + `RetryDelay`, and the attempt/terminal audit rows' occurrence time) is the **attempt instant**, captured after the adapter returns; the sweep-start instant is used only for the due-claim predicate. Stamping sweep-start skewed `DeliveredAt` for the tail of a slow sweep and scheduled `NextAttemptAt` in the past, collapsing the retry delay.
|
||||
- **success** → `Delivered`, set `DeliveredAt`, snapshot `ResolvedTargets`.
|
||||
- **transient failure** → `Retrying`, increment `RetryCount`, set `NextAttemptAt`, record `LastError`; once retries are exhausted → `Parked`.
|
||||
- **permanent failure** → `Parked`, record `LastError`.
|
||||
|
||||
@@ -51,7 +51,7 @@ The SMTP configuration is defined centrally and used by the central Email delive
|
||||
- **TLS mode**: None, StartTLS, or SSL.
|
||||
- **From address**: The sender email address for all notifications (e.g., `scada-notifications@company.com`).
|
||||
- **Connection timeout**: Maximum time to wait for SMTP connection (default: 30 seconds).
|
||||
- **Max concurrent connections**: Maximum simultaneous SMTP connections from the central cluster (default: 5).
|
||||
- **Max concurrent connections**: Maximum simultaneous SMTP connections from the central cluster (default: 5). Enforced by the Notification Outbox dispatcher, which clamps each sweep's Email parallelism to `min(MaxParallelDeliveries, MaxConcurrentConnections)` for the SMTP transport (a non-positive value applies no clamp).
|
||||
- **Retry settings**: Max retry count, fixed time between retries. The Notification Outbox reuses these for delivery retry of transient failures.
|
||||
- **Transport**: The submission mechanism — `Smtp` (default) or `Ews`. Null/empty means `Smtp`, so every pre-existing row keeps its behavior. See EWS Transport below.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user