fix(notification-outbox): per-attempt timestamps, Pending/Retrying split claim + sweep budget, SMTP MaxConcurrentConnections clamp (review 2026-10-21 06/COR-1, 06/PERF-5, 06/IMPL-2)

This commit is contained in:
Joseph Doherty
2026-10-22 09:22:10 -04:00
parent ebf4b96789
commit 811a561885
10 changed files with 547 additions and 37 deletions
@@ -115,10 +115,11 @@ Delivery has no row-level claim or lease: dispatchers read due rows and persist
The dispatcher loop runs on a fixed interval. On each tick the `NotificationOutboxActor`:
1. Polls the `Notifications` table for **due rows** — `Pending` rows, and `Retrying` rows whose `NextAttemptAt` has passed.
1. Polls the `Notifications` table for **due rows** — `Pending` rows, and `Retrying` rows whose `NextAttemptAt` has passed — capped at `DispatchBatchSize` (default 100). The batch is **shared between the two statuses**: due `Retrying` rows take at most half of it and `Pending` rows the rest, each oldest-first, with a share the other status cannot fill back-filled. A single oldest-first claim used to let a retrying backlog (always older than fresh traffic) monopolise every sweep during a partial relay outage, so a new alarm email was not even attempted until the backlog parked.
2. Resolves the target notification list to its recipients/targets at central, at delivery time.
3. Hands the notification to the delivery adapter registered for its `Type`. Deliveries within the sweep run with bounded parallelism (`MaxParallelDeliveries`, default 4), each on its own DI scope, off the actor thread. **That per-delivery scope owns the channel adapter as well as the outbox repository** — see Delivery Adapters below.
3. Hands the notification to the delivery adapter registered for its `Type`. Deliveries within the sweep run with bounded parallelism (`MaxParallelDeliveries`, default 4), each on its own DI scope, off the actor thread. **That per-delivery scope owns the channel adapter as well as the outbox repository** — see Delivery Adapters below. Email over the **SMTP** transport is further clamped per sweep to the active (lowest-Id) SMTP row's `MaxConcurrentConnections` when that value is positive, so a relay that limits connections is honoured; the EWS transport (which does not use the field) and SMS keep the full `MaxParallelDeliveries`. A sweep also has a **wall-clock budget** (`DispatchSweepBudget`, default 2 min): once it is spent, claimed notifications that have not yet started an attempt are skipped — left untouched and still due, so the next sweep re-claims them alongside newer rows — while attempts already in flight run to completion.
4. Applies the result:
Every timestamp written for an attempt (`LastAttemptAt`, `DeliveredAt`, `NextAttemptAt` = attempt instant + `RetryDelay`, and the attempt/terminal audit rows' occurrence time) is the **attempt instant**, captured after the adapter returns; the sweep-start instant is used only for the due-claim predicate. Stamping sweep-start skewed `DeliveredAt` for the tail of a slow sweep and scheduled `NextAttemptAt` in the past, collapsing the retry delay.
- **success** → `Delivered`, set `DeliveredAt`, snapshot `ResolvedTargets`.
- **transient failure** → `Retrying`, increment `RetryCount`, set `NextAttemptAt`, record `LastError`; once retries are exhausted → `Parked`.
- **permanent failure** → `Parked`, record `LastError`.