fix(outbox): resolve the delivery adapter per delivery, not per actor lifetime

Register row 45. NotificationOutboxActor.ResolveAdapters() cached adapter
instances in an actor-lifetime IServiceScope (_adaptersScope), but both
EmailNotificationDeliveryAdapter and SmsNotificationDeliveryAdapter are
AddScoped and hold a scoped INotificationRepository over one EF DbContext.
MaxParallelDeliveries (default 4) then drove up to four concurrent deliveries
through that single adapter and single context, which central-b logged
continuously on the rig as "InvalidOperationException: A second operation was
started on this context instance before a previous operation completed". The
collided delivery's row is never written and the wasted attempt counts against
its retry budget, so sustained contention could park a legitimate notification.
The per-delivery scope DeliverGatedAsync already opened for the outbox
repository — added for exactly this hazard — did not cover the adapter.

_adaptersCache, _adaptersScope and ResolveAdapters() are deleted. Delivery
adapters are now resolved inside the per-delivery scope via a static
ResolveAdapter(IServiceProvider, NotificationType), preserving DI's last-wins
resolution semantics (the enumeration runs to completion rather than
short-circuiting) and the null-adapter park path. Nothing shared is lost: the
Email adapter builds a fresh SMTP client per attempt from an injected factory
and the SMS adapter draws pooled connections from the singleton
IHttpClientFactory, so no adapter carried state worth preserving across
deliveries. The Central UI's NotificationChannelCatalog is an independent
consumer of the same scoped registrations and is untouched, so the
adapter-gated Type selector still enumerates exactly the deliverable channels.

Tests: Dispatch_ResolvesAdaptersOnce_AcrossMultipleSweeps (which pinned the
superseded caching behaviour) is replaced by
ParallelDeliveries_EachGetTheirOwnAdapterAndDbContext — two parallel deliveries
through a scoped repository whose simulated DbContext throws EF's exact
second-operation exception on overlap, with a rendezvous that guarantees the
overlap window — and Dispatch_ResolvesAdapterPerDelivery_NotOncePerActorLifetime.
Both fail against the pre-fix actor (stash-verified: 1 of 2 rows persisted; 1
distinct adapter instead of 4) and pass after. Parallel-delivery
ordering/throughput semantics are unchanged.

Docs: Component-NotificationOutbox.md gains the scoped-adapter contract under
Delivery Adapters (resolve per delivery; adapters carry no cross-delivery
state; genuinely shared resources belong in a singleton/factory), and the
NotificationOutbox-006 finding is marked SUPERSEDED with the reasoning.

Solution build clean; NotificationOutbox (153), AuditLog (383), integration
NotificationOutbox flow (2) and CentralUI notification (14) suites green.
This commit is contained in:
Joseph Doherty
2026-08-15 05:42:24 -04:00
parent 9cf24a4496
commit 9e7697359e
5 changed files with 320 additions and 160 deletions
@@ -117,7 +117,7 @@ The dispatcher loop runs on a fixed interval. On each tick the `NotificationOutb
1. Polls the `Notifications` table for **due rows** — `Pending` rows, and `Retrying` rows whose `NextAttemptAt` has passed.
2. Resolves the target notification list to its recipients/targets at central, at delivery time.
3. Hands the notification to the delivery adapter registered for its `Type`. Deliveries within the sweep run with bounded parallelism (`MaxParallelDeliveries`, default 4), each on its own DI scope/repository, off the actor thread.
3. Hands the notification to the delivery adapter registered for its `Type`. Deliveries within the sweep run with bounded parallelism (`MaxParallelDeliveries`, default 4), each on its own DI scope, off the actor thread. **That per-delivery scope owns the channel adapter as well as the outbox repository** — see Delivery Adapters below.
4. Applies the result:
- **success** → `Delivered`, set `DeliveredAt`, snapshot `ResolvedTargets`.
- **transient failure** → `Retrying`, increment `RetryCount`, set `NextAttemptAt`, record `LastError`; once retries are exhausted → `Parked`.
@@ -136,7 +136,11 @@ A delivery adapter implementing `INotificationDeliveryAdapter` is registered per
- **Email adapter.** The existing SMTP composition/send logic, relocated to the central cluster. Sends a single BCC email to all list recipients.
- **SMS adapter (Twilio REST).** Added in T9/T10 (2026-06-19). Sends one Twilio REST request per recipient phone number. Per-recipient results are rolled up: all-accepted → `Success`; any-transient → retry/park; mix of accepted + permanent-bad → `Success` with bad numbers noted in `LastError`; all-permanent → `Permanent` (Park). See [Component-NotificationService.md](Component-NotificationService.md), SMS Delivery Adapter.
The outbox dispatches by looking up the adapter registered for the notification's `Type`. If no adapter is registered for a given `Type`, the notification is parked with a "no adapter" error — the seam is open for future delivery channels.
The outbox dispatches by looking up the adapter registered for the notification's `Type`. If no adapter is registered for a given `Type`, the notification is parked with a "no adapter" error — the seam is open for future delivery channels. Where more than one adapter declares the same `Type`, the last registered wins, mirroring DI's own last-wins resolution.
**Adapters are scoped services, resolved once per delivery.** Both shipped adapters hold a scoped `INotificationRepository` over the EF `DbContext`, so an adapter instance is bound to exactly one DbContext. The dispatcher therefore resolves the matching adapter *inside the per-delivery DI scope* it already opens for the outbox repository — never once per sweep, and never cached on the singleton actor. Sharing one adapter across the `MaxParallelDeliveries` concurrent deliveries would put several deliveries on one DbContext, which fails with `InvalidOperationException: A second operation was started on this context instance before a previous operation completed`; the affected delivery's outcome is lost, its row is left unwritten, and the wasted attempt eats into the row's retry budget toward `Parked`. An adapter must accordingly carry **no state across deliveries**: anything genuinely shared (connection pools, token caches) belongs in a singleton or factory the adapter depends on — as the Email adapter's SMTP client factory and the SMS adapter's `IHttpClientFactory` already are — not in adapter fields.
The Central UI's channel catalog is a separate consumer of the same registrations: it enumerates the registered adapters' `Type` values to populate the notification-list Type selector, so the set of offerable channels can never drift from the set the central node can actually deliver.
Delivery adapters are provided by the Notification Service, which manages notification-list, SMTP, and SMS definitions and supplies the stateless per-type "deliver one notification" implementations. The SMS `SmsConfiguration` (including encrypted Auth Token) travels in Transport bundles alongside SMTP config.