dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(outbox): resolve the delivery adapter per delivery, not per actor lifetime

Browse Source 
Register row 45. NotificationOutboxActor.ResolveAdapters() cached adapter
instances in an actor-lifetime IServiceScope (_adaptersScope), but both
EmailNotificationDeliveryAdapter and SmsNotificationDeliveryAdapter are
AddScoped and hold a scoped INotificationRepository over one EF DbContext.
MaxParallelDeliveries (default 4) then drove up to four concurrent deliveries
through that single adapter and single context, which central-b logged
continuously on the rig as "InvalidOperationException: A second operation was
started on this context instance before a previous operation completed". The
collided delivery's row is never written and the wasted attempt counts against
its retry budget, so sustained contention could park a legitimate notification.
The per-delivery scope DeliverGatedAsync already opened for the outbox
repository — added for exactly this hazard — did not cover the adapter.

_adaptersCache, _adaptersScope and ResolveAdapters() are deleted. Delivery
adapters are now resolved inside the per-delivery scope via a static
ResolveAdapter(IServiceProvider, NotificationType), preserving DI's last-wins
resolution semantics (the enumeration runs to completion rather than
short-circuiting) and the null-adapter park path. Nothing shared is lost: the
Email adapter builds a fresh SMTP client per attempt from an injected factory
and the SMS adapter draws pooled connections from the singleton
IHttpClientFactory, so no adapter carried state worth preserving across
deliveries. The Central UI's NotificationChannelCatalog is an independent
consumer of the same scoped registrations and is untouched, so the
adapter-gated Type selector still enumerates exactly the deliverable channels.

Tests: Dispatch_ResolvesAdaptersOnce_AcrossMultipleSweeps (which pinned the
superseded caching behaviour) is replaced by
ParallelDeliveries_EachGetTheirOwnAdapterAndDbContext — two parallel deliveries
through a scoped repository whose simulated DbContext throws EF's exact
second-operation exception on overlap, with a rendezvous that guarantees the
overlap window — and Dispatch_ResolvesAdapterPerDelivery_NotOncePerActorLifetime.
Both fail against the pre-fix actor (stash-verified: 1 of 2 rows persisted; 1
distinct adapter instead of 4) and pass after. Parallel-delivery
ordering/throughput semantics are unchanged.

Docs: Component-NotificationOutbox.md gains the scoped-adapter contract under
Delivery Adapters (resolve per delivery; adapters carry no cross-delivery
state; genuinely shared resources belong in a singleton/factory), and the
NotificationOutbox-006 finding is marked SUPERSEDED with the reasoning.

Solution build clean; NotificationOutbox (153), AuditLog (383), integration
NotificationOutbox flow (2) and CentralUI notification (14) suites green.
This commit is contained in:
Joseph Doherty
2026-08-15 05:42:24 -04:00
parent 9cf24a4496
commit 9e7697359e
5 changed files with 320 additions and 160 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/NotificationOutboxActor.cs
+60 -79
View File
@@ -50,34 +50,6 @@ public class NotificationOutboxActor : ReceiveActor, IWithTimers
/// </summary>
private bool _dispatching;
/// <summary>
/// Cached <see cref="NotificationType"/> → adapter lookup, built
/// lazily on the first dispatch sweep and reused for the lifetime of the actor. The
/// adapter registration is decided at startup by <c>AddNotificationOutbox</c> (the set is
/// keyed by <see cref="NotificationType"/> and is static per process lifetime), so
/// rebuilding this dictionary on every sweep was pure allocation waste.
/// </summary>
/// <remarks>
/// The cache is paired with <see cref="_adaptersScope"/>, an actor-lifetime
/// <see cref="IServiceScope"/> created on first use so the cached scoped adapter
/// instances and their dependencies live as long as the cache itself. The scope is
/// disposed in <see cref="PostStop"/>. The adapters are stateless wrappers that
/// resolve their per-call collaborators (e.g. <see cref="INotificationRepository"/>'s
/// underlying DbContext) through their own injected dependencies; holding them for
/// the actor's lifetime is consistent with the actor's own singleton lifetime on the
/// active central node.
/// </remarks>
private IReadOnlyDictionary<NotificationType, INotificationDeliveryAdapter>? _adaptersCache;
/// <summary>
/// Actor-lifetime DI scope that owns the cached
/// <see cref="_adaptersCache"/> adapter instances. Created lazily on the first
/// dispatch sweep that needs adapters; disposed in <see cref="PostStop"/> so the
/// scoped adapter graph (and any disposable dependencies it transitively holds) is
/// torn down with the actor.
/// </summary>
private IServiceScope? _adaptersScope;
/// <summary>
/// Lifecycle-scoped cancellation source, cancelled in <see cref="PostStop"/> so
/// any in-flight dispatch sweep — including a long-running SMTP send via the channel
@@ -156,13 +128,9 @@ public class NotificationOutboxActor : ReceiveActor, IWithTimers
_shutdownCts?.Dispose();
_shutdownCts = null;
// Dispose the actor-lifetime adapter scope so the cached
// scoped adapter instances and their disposable dependencies are torn down with
// the actor (e.g. on a CoordinatedShutdown / failover that stops the singleton).
_adaptersScope?.Dispose();
_adaptersScope = null;
_adaptersCache = null;
// No adapter state to tear down: delivery adapters are resolved from — and
// disposed with — the per-delivery DI scope opened in DeliverGatedAsync, so the
// actor holds no scoped service beyond the lifetime of a single delivery.
base.PostStop();
}
@@ -278,12 +246,13 @@ public class NotificationOutboxActor : ReceiveActor, IWithTimers
/// and retry-policy resolution can all throw, and a faulted task would otherwise leave
/// the dispatcher's in-flight guard stuck and wedge the loop permanently.
///
/// The per-sweep DI scope still owns the repository graph
/// (<see cref="INotificationOutboxRepository"/> + <see cref="INotificationRepository"/>),
/// which is correct because those services back a fresh DbContext per sweep. The
/// channel delivery adapters, however, are cached for the actor's lifetime via
/// <see cref="ResolveAdapters"/> — see <see cref="_adaptersCache"/> for the
/// rationale.
/// The per-sweep DI scope owns only the sweep-level repository graph
/// (<see cref="INotificationOutboxRepository"/> + <see cref="INotificationRepository"/>)
/// used to claim the due batch and resolve the retry policies — both of which run
/// before any delivery starts, on the sweep's own DbContext. The channel delivery
/// adapters are NOT resolved here: each is resolved inside its own per-delivery scope
/// by <see cref="DeliverGatedAsync"/>, so parallel deliveries never share an adapter
/// or the DbContext behind it.
/// </summary>
private async Task RunDispatchPass(DateTimeOffset now, CancellationToken cancellationToken)
{
@@ -292,7 +261,6 @@ public class NotificationOutboxActor : ReceiveActor, IWithTimers
using var scope = _serviceProvider.CreateScope();
var outboxRepository = scope.ServiceProvider.GetRequiredService<INotificationOutboxRepository>();
var notificationRepository = scope.ServiceProvider.GetRequiredService<INotificationRepository>();
var adapters = ResolveAdapters();
IReadOnlyList<Notification> due;
try
@@ -322,18 +290,18 @@ public class NotificationOutboxActor : ReceiveActor, IWithTimers
// (multi-second SMTP/Twilio round trips), so a strictly sequential sweep caps
// throughput at ~1/attempt-latency — an alarm-storm backlog behind one slow
// host would drain far too slowly (arch-review 04). A SemaphoreSlim caps the
// number of in-flight deliveries; each delivery runs on its OWN DI scope/
// repository (the EF DbContext is not thread-safe and rows are distinct per
// notification, so concurrent deliveries never contend). The cached adapters
// and the lifetime-scoped audit writer are already safe for concurrent use.
// MaxParallelDeliveries = 1 preserves the strictly sequential behaviour.
// number of in-flight deliveries; each delivery runs on its OWN DI scope —
// repository AND channel adapter alike (the EF DbContext is not thread-safe
// and rows are distinct per notification, so concurrent deliveries never
// contend). The lifetime-scoped audit writer is already safe for concurrent
// use. MaxParallelDeliveries = 1 preserves the strictly sequential behaviour.
using var deliveryGate = new SemaphoreSlim(
_options.ResolvedMaxParallelDeliveries, _options.ResolvedMaxParallelDeliveries);
var deliveries = new List<Task>(due.Count);
foreach (var notification in due)
{
deliveries.Add(DeliverGatedAsync(
notification, now, policies, adapters, deliveryGate, cancellationToken));
notification, now, policies, deliveryGate, cancellationToken));
}
// Each DeliverGatedAsync isolates its own faults, so WhenAll never throws.
@@ -355,11 +323,24 @@ public class NotificationOutboxActor : ReceiveActor, IWithTimers
/// pre-attempt state for the next active sweep) and any other exception is logged — so
/// one bad delivery never aborts the rest of the batch and <c>Task.WhenAll</c> never faults.
/// </summary>
/// <remarks>
/// The per-delivery scope covers the channel delivery adapter as well as the outbox
/// repository. Both <see cref="EmailNotificationDeliveryAdapter"/> and
/// <see cref="SmsNotificationDeliveryAdapter"/> are registered <em>scoped</em> and hold
/// a scoped <see cref="INotificationRepository"/> backed by the EF DbContext, so an
/// adapter resolved outside this scope and reused across concurrent deliveries put
/// several deliveries on ONE DbContext — which fails with
/// <c>InvalidOperationException: A second operation was started on this context
/// instance…</c> and burns the row's retries. Resolving the adapter here binds its
/// lifetime to exactly one delivery. Neither adapter carries state worth sharing: the
/// Email adapter builds a fresh SMTP client per attempt from an injected factory and
/// the SMS adapter takes its pooled connections from <c>IHttpClientFactory</c>, which
/// is a singleton and shares the handler pool across scopes regardless.
/// </remarks>
private async Task DeliverGatedAsync(
Notification notification,
DateTimeOffset now,
RetryPolicies policies,
IReadOnlyDictionary<NotificationType, INotificationDeliveryAdapter> adapters,
SemaphoreSlim gate,
CancellationToken cancellationToken)
{
@@ -381,13 +362,14 @@ public class NotificationOutboxActor : ReceiveActor, IWithTimers
return;
}
// Each delivery owns its DI scope + repository so parallel UpdateAsync calls
// never share a DbContext. The adapters map + audit writer are already safe
// for concurrent use.
// Each delivery owns its DI scope, repository AND channel adapter so parallel
// deliveries never share a DbContext. The audit writer is already safe for
// concurrent use.
using var scope = _serviceProvider.CreateScope();
var outboxRepository = scope.ServiceProvider.GetRequiredService<INotificationOutboxRepository>();
var adapter = ResolveAdapter(scope.ServiceProvider, notification.Type);
await DeliverOneAsync(
notification, now, policies, outboxRepository, adapters, cancellationToken)
notification, now, policies, outboxRepository, adapter, cancellationToken)
.ConfigureAwait(false);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
@@ -522,37 +504,36 @@ public class NotificationOutboxActor : ReceiveActor, IWithTimers
}
/// <summary>
/// Returns the <see cref="NotificationType"/> → adapter lookup, building it lazily on
/// the first call and caching it on <see cref="_adaptersCache"/> for the actor's
/// lifetime. The last adapter registered for a given type wins, mirroring DI's
/// last-wins resolution semantics.
/// Resolves the channel delivery adapter registered for <paramref name="type"/> from
/// <paramref name="scopedServices"/> — the caller's per-delivery DI scope — or null
/// when no adapter handles that type (the caller then parks the notification). The
/// LAST adapter registered for a given type wins, mirroring DI's last-wins resolution
/// semantics, which is why the enumeration runs to completion rather than short-
/// circuiting on the first match.
/// </summary>
/// <remarks>
/// The lookup used to be rebuilt on every dispatch sweep
/// from the per-sweep DI scope. Adapter registration is static per process
/// lifetime, so the dict is now built ONCE — on the first sweep that needs it —
/// and reused. To respect each adapter's scoped lifetime
/// (<see cref="EmailNotificationDeliveryAdapter"/> takes a scoped
/// <see cref="INotificationRepository"/>), the cache is paired with
/// <see cref="_adaptersScope"/>, an actor-lifetime <see cref="IServiceScope"/> that
/// owns the cached adapter instances and is disposed in <see cref="PostStop"/>.
/// This deliberately resolves per delivery rather than caching an actor-lifetime
/// lookup: the adapters are scoped services holding a scoped
/// <see cref="INotificationRepository"/>/DbContext, so a cached instance shared by up
/// to <c>MaxParallelDeliveries</c> concurrent deliveries drove a second-operation-on-
/// this-context failure. See <see cref="DeliverGatedAsync"/>. Enumerating
/// <c>GetServices</c> constructs every registered adapter (two today), which is a
/// negligible cost next to the SMTP/HTTPS round trip that follows and keeps the
/// last-wins contract identical to the previous lookup.
/// </remarks>
private IReadOnlyDictionary<NotificationType, INotificationDeliveryAdapter> ResolveAdapters()
private static INotificationDeliveryAdapter? ResolveAdapter(
IServiceProvider scopedServices, NotificationType type)
{
if (_adaptersCache is not null)
INotificationDeliveryAdapter? match = null;
foreach (var adapter in scopedServices.GetServices<INotificationDeliveryAdapter>())
{
return _adaptersCache;
if (adapter.Type == type)
{
match = adapter;
}
}
_adaptersScope = _serviceProvider.CreateScope();
var adapters = new Dictionary<NotificationType, INotificationDeliveryAdapter>();
foreach (var adapter in _adaptersScope.ServiceProvider.GetServices<INotificationDeliveryAdapter>())
{
adapters[adapter.Type] = adapter;
}
_adaptersCache = adapters;
return _adaptersCache;
return match;
}
/// <summary>
@@ -585,7 +566,7 @@ public class NotificationOutboxActor : ReceiveActor, IWithTimers
DateTimeOffset now,
RetryPolicies policies,
INotificationOutboxRepository outboxRepository,
IReadOnlyDictionary<NotificationType, INotificationDeliveryAdapter> adapters,
INotificationDeliveryAdapter? adapter,
CancellationToken cancellationToken)
{
// Select the retry policy by delivery channel: SMS honors SmsConfiguration's own
@@ -594,7 +575,7 @@ public class NotificationOutboxActor : ReceiveActor, IWithTimers
? policies.SmsPolicy
: policies.SmtpPolicy;
if (!adapters.TryGetValue(notification.Type, out var adapter))
if (adapter is null)
{
// Missing-adapter park: from the dispatcher's perspective this is an
// attempt that resolved to a terminal park. Emit Attempted then the
src/ZB.MOM.WW.ScadaBridge.NotificationOutbox/ServiceCollectionExtensions.cs
+5 -3
View File
@@ -37,9 +37,11 @@ public static class ServiceCollectionExtensions
///
/// <see cref="EmailNotificationDeliveryAdapter"/> is registered <em>scoped</em> because it
/// takes a scoped <see cref="ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories.INotificationRepository"/>
/// directly. The <see cref="NotificationOutboxActor"/> resolves the adapters from a fresh
/// scope per dispatch sweep rather than holding them, so no scoped adapter is captured by
/// the singleton actor.
/// directly. The <see cref="NotificationOutboxActor"/> resolves the matching adapter from
/// the fresh DI scope it opens <em>per delivery</em> — not per sweep and never held on the
/// actor — so no scoped adapter is captured by the singleton actor and no two of the
/// dispatcher's concurrent deliveries (<c>MaxParallelDeliveries</c>) can land on one
/// DbContext.
///
/// The <see cref="SmsNotificationDeliveryAdapter"/> (Twilio-REST) is registered the same
/// way alongside the Email adapter, so both resolve from