fix(notifications): list sort index + CAS operator Retry/Discard (P5-3, P5-4)

P5-3: add IX_Notifications_Created_NotificationId (CreatedAt DESC,
NotificationId DESC) — the Central UI Notification Outbox list page sorts on
exactly that pair, but no existing index leads unconditionally on CreatedAt,
so an unfiltered/lightly filtered page fell back to a full clustered-index
scan + sort on every COUNT + page request. Mirrors the Medium wave's
IX_SiteCalls_Created_TrackedOperationId migration + SQL twin pattern
faithfully, including the handwritten docs/plans/sql script.

P5-4: operator Retry/Discard was read-check-write keyed on NotificationId
only — a detached PipeTo task interleaving with the dispatcher's own write
(or a second concurrent operator action) could land between the read and
the write and get silently clobbered by the unconditional UpdateAsync,
stamping a Delivered row Discarded with stale fields. Adds
INotificationOutboxRepository.TryUpdateFromParkedAsync, a single-statement
compare-and-swap (WHERE Status == Parked) mirroring
SecuredWriteRepository.TryMarkApprovedAsync's CAS shape. NotificationOutboxActor's
RetryAsync/DiscardAsync now use the CAS write and, on a lost race, re-read
to report "notification already progressed to <Status>" instead of silently
overwriting whatever is there now.

Proof of failure: reverting only the actor's CAS usage while keeping the
repository addition reproduces the silent-overwrite bug — 7 tests fail,
including two new deterministic (non-sleep) race-proof tests that mock the
exact GetByIdAsync -> lost-CAS -> re-read sequence the finding describes.
P5-3 is perf-only; verified via migration/SQL-twin parity instead (no
runtime proof-of-failure is meaningful for an index).
This commit is contained in:
Joseph Doherty
2026-08-19 16:20:18 -04:00
parent 11675285af
commit 1cea01d924
12 changed files with 2607 additions and 32 deletions
@@ -0,0 +1,51 @@
-- arch-review 2026-08-19 P5-3 (register row-35 class): the Notifications list
-- page's sort key gets its own index.
--
-- NotificationOutboxRepository.QueryAsync orders (CreatedAt DESC, NotificationId
-- DESC) (NotificationId breaks CreatedAt ties for deterministic OFFSET paging).
-- None of the existing Notifications indexes lead unconditionally on CreatedAt:
-- - IX_Notifications_Status_NextAttemptAt requires an equality on Status first.
-- - IX_Notifications_SourceSiteId_CreatedAt requires an equality on SourceSiteId first.
-- - IX_Notifications_Delivered is filtered to Status = 'Delivered' only.
-- So an unfiltered (or lightly filtered) page fell back to a full clustered-index
-- scan + sort. IX_Notifications_Created_NotificationId (CreatedAt DESC,
-- NotificationId DESC) makes a forward scan of the index deliver rows in the
-- requested order directly, so a page reads a Top-N off the index instead of
-- sorting the whole table on every COUNT + page request.
--
-- Notifications is small enough today that a plain (offline) index build is
-- fine. On an Enterprise edition instance where the table has grown large and
-- the Central UI must stay responsive, add WITH (ONLINE = ON) to the CREATE
-- INDEX below.
--
-- Run with QUOTED_IDENTIFIER ON. Several tables in this database carry filtered
-- indexes and persisted computed columns, and SQL Server refuses DDL/DML against
-- them with error 1934 when the session has QUOTED_IDENTIFIER OFF -- which is
-- exactly what `sqlcmd` gives you by default (pass -I, or keep this header). The
-- SETs sit in their own batch so they are in force when the statements after the
-- GO parse.
SET QUOTED_IDENTIFIER ON;
SET ANSI_NULLS ON;
GO
BEGIN TRANSACTION;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20260819201210_AddNotificationsListSortIndex'
)
BEGIN
EXEC(N'CREATE INDEX [IX_Notifications_Created_NotificationId]
ON [Notifications] ([CreatedAt] DESC, [NotificationId] DESC)');
END;
IF NOT EXISTS (
SELECT * FROM [__EFMigrationsHistory]
WHERE [MigrationId] = N'20260819201210_AddNotificationsListSortIndex'
)
BEGIN
INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
VALUES (N'20260819201210_AddNotificationsListSortIndex', N'10.0.7');
END;
COMMIT;
GO
@@ -1,4 +1,5 @@
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Notifications;
namespace ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
@@ -83,6 +84,43 @@ public interface INotificationOutboxRepository
/// </returns>
Task<bool> UpdateAsync(Notification n, CancellationToken cancellationToken = default);
/// <summary>
/// Compare-and-swap variant of <see cref="UpdateAsync"/> for the operator
/// Retry/Discard one-shots: persists <paramref name="n"/>'s delivery-state
/// columns (same narrow column set as <see cref="UpdateAsync"/>) ONLY if the
/// row's current <c>Status</c> in the database is still
/// <see cref="NotificationStatus.Parked"/>. Commits internally — this call
/// is its own transaction.
/// </summary>
/// <remarks>
/// <b>Why this exists (arch-review 2026-08-19 P5-4).</b> The operator retry/
/// discard handlers are read-check-write across two round trips:
/// <see cref="GetByIdAsync"/> reads the row, the handler checks its
/// in-memory <c>Status</c>, then a write persists the transition. A detached
/// <c>PipeTo</c> task can interleave with the dispatcher's own write (or a
/// second concurrent operator action) between those two round trips, so the
/// in-memory check is a fast-path rejection, not a correctness guard — a
/// blind <see cref="UpdateAsync"/> in that window can stamp a row that has
/// already moved on (e.g. the dispatcher delivered it) with the operator's
/// stale fields, silently overwriting a <c>Delivered</c> row as
/// <c>Discarded</c>. The <c>WHERE Status = Parked</c> predicate below makes
/// the <c>Parked-&gt;(Pending|Discarded)</c> transition atomic at the row
/// level, mirroring <c>SecuredWriteRepository.TryMarkApprovedAsync</c>'s
/// single-statement compare-and-swap. A row that has already progressed
/// (dispatcher won the race, or a second operator action won it) and a row
/// that has vanished (retention purge) both report zero rows affected;
/// callers cannot tell those apart from the return value alone and must
/// re-read to report which honestly.
/// </remarks>
/// <param name="n">The notification whose delivery state should be persisted, if still Parked.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>
/// <see langword="true"/> when a row with that <c>NotificationId</c> was
/// found with <c>Status == Parked</c> and its delivery state was persisted;
/// <see langword="false"/> otherwise (not found, or no longer Parked).
/// </returns>
Task<bool> TryUpdateFromParkedAsync(Notification n, CancellationToken cancellationToken = default);
/// <summary>
/// Returns a page of notifications matching <paramref name="filter"/>, ordered by
/// <c>CreatedAt</c> descending, together with the total matching count.
@@ -83,5 +83,23 @@ public class NotificationOutboxConfiguration : IEntityTypeConfiguration<Notifica
builder.HasIndex(n => n.DeliveredAt)
.HasDatabaseName("IX_Notifications_Delivered")
.HasFilter("[Status] = 'Delivered'");
// Created_NotificationId backs the Central UI Notification Outbox list
// page's sort key — NotificationOutboxRepository.QueryAsync orders
// (CreatedAt DESC, NotificationId DESC) (NotificationId breaks CreatedAt
// ties for deterministic OFFSET paging). None of the indexes above lead
// unconditionally on CreatedAt: Status_NextAttemptAt requires an equality
// on Status first, SourceSiteId_CreatedAt requires an equality on
// SourceSiteId first, and IX_Notifications_Delivered is filtered to
// Status='Delivered' only. So an unfiltered (or lightly filtered) page
// fell back to a full clustered-index scan + sort on every COUNT + page
// request (arch-review 2026-08-19 P5-3, register row-35 class). With both
// key columns descending, a forward scan of this index already delivers
// rows in the requested order. Exact mirror of
// SiteCallEntityTypeConfiguration's IX_SiteCalls_Created_TrackedOperationId
// (2026-08-19, arch-review P5-2).
builder.HasIndex(n => new { n.CreatedAt, n.NotificationId })
.IsDescending(true, true)
.HasDatabaseName("IX_Notifications_Created_NotificationId");
}
}
@@ -0,0 +1,46 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations
{
/// <summary>
/// Adds <c>IX_Notifications_Created_NotificationId (CreatedAt DESC,
/// NotificationId DESC)</c> — arch-review 2026-08-19 P5-3, register
/// row-35 class.
/// </summary>
/// <remarks>
/// <see cref="ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories.NotificationOutboxRepository.QueryAsync"/>
/// orders <c>(CreatedAt DESC, NotificationId DESC)</c> for the Central UI
/// Notification Outbox list page. None of the existing Notifications indexes
/// lead unconditionally on <c>CreatedAt</c> — <c>IX_Notifications_Status_NextAttemptAt</c>
/// requires an equality on <c>Status</c> first, <c>IX_Notifications_SourceSiteId_CreatedAt</c>
/// requires an equality on <c>SourceSiteId</c> first, and
/// <c>IX_Notifications_Delivered</c> is filtered to <c>Status = 'Delivered'</c>
/// only — so an unfiltered (or lightly filtered) list page fell back to a full
/// clustered-index scan + sort on every COUNT + page request. With both key
/// columns descending, a forward scan of this index already delivers rows in
/// the requested order. See <c>docs/plans/sql/AddNotificationsListSortIndex.sql</c>
/// for the handwritten production-parity script.
/// </remarks>
public partial class AddNotificationsListSortIndex : Migration
{
/// <inheritdoc />
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.CreateIndex(
name: "IX_Notifications_Created_NotificationId",
table: "Notifications",
columns: new[] { "CreatedAt", "NotificationId" },
descending: new bool[0]);
}
/// <inheritdoc />
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropIndex(
name: "IX_Notifications_Created_NotificationId",
table: "Notifications");
}
}
}
@@ -913,6 +913,10 @@ namespace ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Migrations
.HasDatabaseName("IX_Notifications_Delivered")
.HasFilter("[Status] = 'Delivered'");
b.HasIndex("CreatedAt", "NotificationId")
.IsDescending()
.HasDatabaseName("IX_Notifications_Created_NotificationId");
b.HasIndex("SourceSiteId", "CreatedAt");
b.HasIndex("Status", "NextAttemptAt");
@@ -205,6 +205,48 @@ VALUES
return rowsAffected > 0;
}
/// <inheritdoc />
public async Task<bool> TryUpdateFromParkedAsync(Notification n, CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(n);
// Same seven mutable delivery-state columns as UpdateAsync, but the WHERE
// clause below ALSO requires the row's current Status to still be Parked
// -- a single-statement compare-and-swap (arch-review 2026-08-19 P5-4).
// Two round trips (GetByIdAsync then this write) with a status check in
// between is not a correctness guard by itself: a detached PipeTo task
// can race the dispatcher's own write (or a second concurrent operator
// action) in that window. Mirrors SecuredWriteRepository.TryMarkApprovedAsync's
// conditional ExecuteUpdate/ExecuteSql pattern, expressed via LINQ
// ExecuteUpdateAsync (like UpdateAsync above) so this stays portable
// across the SQL Server production provider and the SQLite provider the
// repository tests run against.
var status = n.Status;
var retryCount = n.RetryCount;
var lastError = n.LastError;
var resolvedTargets = n.ResolvedTargets;
var lastAttemptAt = n.LastAttemptAt;
var nextAttemptAt = n.NextAttemptAt;
var deliveredAt = n.DeliveredAt;
var notificationId = n.NotificationId;
var rowsAffected = await _context.Notifications
.Where(row => row.NotificationId == notificationId && row.Status == NotificationStatus.Parked)
.ExecuteUpdateAsync(
setters => setters
.SetProperty(row => row.Status, status)
.SetProperty(row => row.RetryCount, retryCount)
.SetProperty(row => row.LastError, lastError)
.SetProperty(row => row.ResolvedTargets, resolvedTargets)
.SetProperty(row => row.LastAttemptAt, lastAttemptAt)
.SetProperty(row => row.NextAttemptAt, nextAttemptAt)
.SetProperty(row => row.DeliveredAt, deliveredAt),
cancellationToken);
return rowsAffected > 0;
}
/// <inheritdoc />
public async Task<Notification?> GetByIdAsync(string notificationId, CancellationToken cancellationToken = default)
{
@@ -1109,14 +1109,25 @@ public class NotificationOutboxActor : ReceiveActor, IWithTimers
notification.NextAttemptAt = null;
notification.LastError = null;
// Zero rows updated means the row was purged between the read above and
// this write. Answer the operator honestly instead of reporting a
// re-queue that never happened — and emit no un-park audit row, because
// there is nothing to attribute it to.
if (!await repository.UpdateAsync(notification))
// Compare-and-swap on Status == Parked (arch-review 2026-08-19 P5-4). The
// read above and this write are two separate round trips; a detached
// PipeTo task can race the dispatcher's own write (or a second
// concurrent operator action) in between, so the in-memory check above
// is a fast-path rejection, not the correctness guard — a blind
// UpdateAsync here could stamp a row that has already moved on (e.g. the
// dispatcher delivered it) back to Pending with stale fields. A lost CAS
// means either the row vanished (purge) or it progressed past Parked
// under us; re-read to report which honestly instead of silently
// overwriting whatever is there now, and emit no un-park audit row in
// either case, because there is nothing to attribute it to.
if (!await repository.TryUpdateFromParkedAsync(notification))
{
var current = await repository.GetByIdAsync(request.NotificationId);
return new RetryNotificationResponse(
request.CorrelationId, Success: false, ErrorMessage: "notification not found");
request.CorrelationId, Success: false,
ErrorMessage: current is null
? "notification not found"
: $"notification already progressed to {current.Status}; retry not applied");
}
// Operator re-queued a parked notification. Emit a Submitted NotifyDeliver
@@ -1190,13 +1201,22 @@ public class NotificationOutboxActor : ReceiveActor, IWithTimers
notification.Status = NotificationStatus.Discarded;
// Same vanished-row honesty as the retry path: a purge between the read
// and the write leaves nothing to discard, so report not-found rather
// than success (and skip the terminal audit row below).
if (!await repository.UpdateAsync(notification))
// Same compare-and-swap on Status == Parked as the retry path (arch-review
// 2026-08-19 P5-4): a purge between the read and the write leaves nothing
// to discard, and a dispatcher/other-operator write landing in that same
// window (e.g. a delivery succeeding) must not be silently clobbered by a
// blind write of this stale in-memory copy. Re-read on a lost CAS to
// report which case it was, and skip the terminal audit row below either
// way — there is nothing to attribute a discard of a row that was never
// actually discarded.
if (!await repository.TryUpdateFromParkedAsync(notification))
{
var current = await repository.GetByIdAsync(request.NotificationId);
return new DiscardNotificationResponse(
request.CorrelationId, Success: false, ErrorMessage: "notification not found");
request.CorrelationId, Success: false,
ErrorMessage: current is null
? "notification not found"
: $"notification already progressed to {current.Status}; discard not applied");
}
// A manual discard is the OTHER code path that produces
@@ -573,6 +573,154 @@ public class NotificationOutboxRepositoryTests : IDisposable
Assert.False(await _repository.UpdateAsync(loaded));
}
[Fact]
public async Task TryUpdateFromParkedAsync_StatusIsParked_UpdatesAndReturnsTrue()
{
var id = Guid.NewGuid().ToString();
_context.Notifications.Add(MakeNotification(id, NotificationStatus.Parked));
await _context.SaveChangesAsync();
_context.ChangeTracker.Clear();
var loaded = await _repository.GetByIdAsync(id);
loaded!.Status = NotificationStatus.Pending;
loaded.RetryCount = 0;
loaded.NextAttemptAt = null;
loaded.LastError = null;
Assert.True(await _repository.TryUpdateFromParkedAsync(loaded));
_context.ChangeTracker.Clear();
var reloaded = await _context.Notifications.FindAsync(id);
Assert.Equal(NotificationStatus.Pending, reloaded!.Status);
}
/// <summary>
/// Direct CAS-miss: the row is currently Delivered (never Parked in this
/// scenario), so the conditional WHERE Status == Parked matches nothing and
/// the row is left completely untouched — arch-review 2026-08-19 P5-4.
/// </summary>
[Fact]
public async Task TryUpdateFromParkedAsync_StatusIsNotParked_ReturnsFalse_AndLeavesRowUnchanged()
{
var id = Guid.NewGuid().ToString();
var deliveredAt = new DateTimeOffset(2026, 5, 19, 9, 0, 0, TimeSpan.Zero);
_context.Notifications.Add(MakeNotification(
id, NotificationStatus.Delivered, deliveredAt: deliveredAt));
await _context.SaveChangesAsync();
_context.ChangeTracker.Clear();
var staleDiscard = MakeNotification(id, NotificationStatus.Discarded);
Assert.False(await _repository.TryUpdateFromParkedAsync(staleDiscard));
_context.ChangeTracker.Clear();
var reloaded = await _context.Notifications.FindAsync(id);
Assert.Equal(NotificationStatus.Delivered, reloaded!.Status);
Assert.Equal(deliveredAt, reloaded.DeliveredAt);
}
[Fact]
public async Task TryUpdateFromParkedAsync_RowPurgedBeforeWrite_ReturnsFalse()
{
var id = Guid.NewGuid().ToString();
_context.Notifications.Add(MakeNotification(id, NotificationStatus.Parked));
await _context.SaveChangesAsync();
_context.ChangeTracker.Clear();
var loaded = await _repository.GetByIdAsync(id);
Assert.NotNull(loaded);
await _context.Notifications.Where(n => n.NotificationId == id).ExecuteDeleteAsync();
_context.ChangeTracker.Clear();
loaded!.Status = NotificationStatus.Discarded;
Assert.False(await _repository.TryUpdateFromParkedAsync(loaded));
}
/// <summary>
/// Deterministic (non-sleep) proof of the arch-review 2026-08-19 P5-4 race:
/// the operator reads a Parked row, but before its write lands the
/// dispatcher (or a concurrent second operator action) has already moved
/// the row to Delivered. <see cref="TryUpdateFromParkedAsync"/>'s CAS must
/// refuse the stale write and leave the Delivered row exactly as it is.
/// The companion fact below proves the general-purpose
/// <see cref="NotificationOutboxRepository.UpdateAsync"/> — the method the
/// operator path used PRE-FIX — has no such guard, which is the mechanism
/// the finding describes.
/// </summary>
[Fact]
public async Task TryUpdateFromParkedAsync_ConcurrentlyDelivered_ReturnsFalse_AndDoesNotOverwrite()
{
var id = Guid.NewGuid().ToString();
_context.Notifications.Add(MakeNotification(id, NotificationStatus.Parked));
await _context.SaveChangesAsync();
_context.ChangeTracker.Clear();
// The operator's read (this snapshot is carried forward stale below).
var operatorRead = await _repository.GetByIdAsync(id);
Assert.NotNull(operatorRead);
// The dispatcher's own write lands first, in the window between the
// operator's read and write.
var deliveredAt = new DateTimeOffset(2026, 5, 19, 9, 0, 0, TimeSpan.Zero);
var dispatcherRead = await _repository.GetByIdAsync(id);
dispatcherRead!.Status = NotificationStatus.Delivered;
dispatcherRead.DeliveredAt = deliveredAt;
Assert.True(await _repository.UpdateAsync(dispatcherRead));
_context.ChangeTracker.Clear();
// The operator's stale in-memory copy still says Parked; it attempts a Discard.
operatorRead!.Status = NotificationStatus.Discarded;
Assert.False(await _repository.TryUpdateFromParkedAsync(operatorRead));
_context.ChangeTracker.Clear();
var reloaded = await _context.Notifications.FindAsync(id);
Assert.Equal(NotificationStatus.Delivered, reloaded!.Status);
Assert.Equal(deliveredAt, reloaded.DeliveredAt);
}
/// <summary>
/// Companion to <see cref="TryUpdateFromParkedAsync_ConcurrentlyDelivered_ReturnsFalse_AndDoesNotOverwrite"/>:
/// runs the IDENTICAL interleaving through the unconditional
/// <see cref="NotificationOutboxRepository.UpdateAsync"/> instead — the
/// method the operator Retry/Discard path called pre-fix — and shows it
/// silently clobbers the Delivered row. This documents the exact defect
/// arch-review 2026-08-19 P5-4 fixed by routing the operator path through
/// <see cref="TryUpdateFromParkedAsync"/>: UpdateAsync itself is unchanged
/// (the dispatcher still needs an unconditional write), so this scenario
/// remains "true" by design — it is the operator actor's call-site choice
/// that was the bug, proven fixed by the actor-level tests in
/// NotificationOutbox.Tests (Discard_ConcurrentlyDelivered_...).
/// </summary>
[Fact]
public async Task UpdateAsync_Unconditional_WouldOverwriteConcurrentlyDeliveredRow_DemonstratingP5_4Mechanism()
{
var id = Guid.NewGuid().ToString();
_context.Notifications.Add(MakeNotification(id, NotificationStatus.Parked));
await _context.SaveChangesAsync();
_context.ChangeTracker.Clear();
var operatorRead = await _repository.GetByIdAsync(id);
Assert.NotNull(operatorRead);
var deliveredAt = new DateTimeOffset(2026, 5, 19, 9, 0, 0, TimeSpan.Zero);
var dispatcherRead = await _repository.GetByIdAsync(id);
dispatcherRead!.Status = NotificationStatus.Delivered;
dispatcherRead.DeliveredAt = deliveredAt;
Assert.True(await _repository.UpdateAsync(dispatcherRead));
_context.ChangeTracker.Clear();
operatorRead!.Status = NotificationStatus.Discarded;
// The unconditional UPDATE has no status guard, so it "succeeds" here —
// this is the bug. It clobbers the Delivered row to Discarded and drops
// DeliveredAt, which is exactly the silent-overwrite hazard P5-4 closes
// by moving the operator path to TryUpdateFromParkedAsync.
Assert.True(await _repository.UpdateAsync(operatorRead));
_context.ChangeTracker.Clear();
var reloaded = await _context.Notifications.FindAsync(id);
Assert.Equal(NotificationStatus.Discarded, reloaded!.Status);
Assert.Null(reloaded.DeliveredAt);
}
[Fact]
public async Task QueryAsync_AppliesFilters_OrdersByCreatedAtDescending_AndPaginates()
{
@@ -258,7 +258,7 @@ public class NotificationOutboxActorQueryTests : TestKit
Assert.True(response.Success);
Assert.Null(response.ErrorMessage);
_repository.Received(1).UpdateAsync(
_repository.Received(1).TryUpdateFromParkedAsync(
Arg.Is<Notification>(n =>
n.NotificationId == row.NotificationId &&
n.Status == NotificationStatus.Pending &&
@@ -280,7 +280,7 @@ public class NotificationOutboxActorQueryTests : TestKit
var response = ExpectMsg<RetryNotificationResponse>();
Assert.False(response.Success);
Assert.NotNull(response.ErrorMessage);
_repository.DidNotReceive().UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>());
_repository.DidNotReceive().TryUpdateFromParkedAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>());
}
[Fact]
@@ -300,17 +300,19 @@ public class NotificationOutboxActorQueryTests : TestKit
/// <summary>
/// The row is read successfully but the retention purge deletes it before the
/// write lands, so the targeted UPDATE matches nothing. The operator must be
/// told the notification is gone — reporting a re-queue that never happened
/// is what the ExecuteUpdate rewrite silently introduced (the predecessor
/// threw DbUpdateConcurrencyException here).
/// write lands, so the CAS write matches nothing. The re-read that follows a
/// lost CAS (arch-review 2026-08-19 P5-4) finds nothing either, and the
/// operator must be told the notification is gone — reporting a re-queue
/// that never happened is what a blind ExecuteUpdate would silently do (the
/// pre-ExecuteUpdate predecessor threw DbUpdateConcurrencyException here).
/// </summary>
[Fact]
public void Retry_RowPurgedBeforeWrite_RepliesNotFound()
{
var row = MakeNotification(status: NotificationStatus.Parked, retryCount: 10, lastError: "gave up");
_repository.GetByIdAsync(row.NotificationId, Arg.Any<CancellationToken>()).Returns(row);
_repository.UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>()).Returns(false);
_repository.GetByIdAsync(row.NotificationId, Arg.Any<CancellationToken>())
.Returns(row, (Notification?)null);
_repository.TryUpdateFromParkedAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>()).Returns(false);
var actor = CreateActor();
actor.Tell(new RetryNotificationRequest("corr-vanished-retry", row.NotificationId), TestActor);
@@ -327,8 +329,9 @@ public class NotificationOutboxActorQueryTests : TestKit
public void Discard_RowPurgedBeforeWrite_RepliesNotFound()
{
var row = MakeNotification(status: NotificationStatus.Parked);
_repository.GetByIdAsync(row.NotificationId, Arg.Any<CancellationToken>()).Returns(row);
_repository.UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>()).Returns(false);
_repository.GetByIdAsync(row.NotificationId, Arg.Any<CancellationToken>())
.Returns(row, (Notification?)null);
_repository.TryUpdateFromParkedAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>()).Returns(false);
var actor = CreateActor();
actor.Tell(new DiscardNotificationRequest("corr-vanished-discard", row.NotificationId), TestActor);
@@ -340,6 +343,75 @@ public class NotificationOutboxActorQueryTests : TestKit
Assert.Contains("not found", response.ErrorMessage);
}
/// <summary>
/// PROOF OF FAILURE for arch-review 2026-08-19 P5-4. Models the race the
/// finding describes: the operator's read sees Parked, but by the time the
/// write lands the dispatcher has already delivered the row (a detached
/// PipeTo task interleaving with the dispatcher's own write). The mechanism
/// is proven deterministically via a mocked call sequence, not a wall-clock
/// sleep (per the "no new sleep-then-assert" constraint) — the first
/// <see cref="INotificationOutboxRepository.GetByIdAsync"/> call (the
/// operator's read) returns the Parked row, the CAS write reports a lost
/// race, and the SECOND <c>GetByIdAsync</c> call (the fix's re-read used to
/// build an honest reply) returns the row now Delivered.
/// <para>
/// Against the pre-fix actor — which calls the unconditional
/// <c>UpdateAsync</c> exactly once and never re-reads — this test fails:
/// the single-read code path only ever consumes the first queued value
/// (Parked), the in-memory check passes, <c>UpdateAsync</c> is stubbed
/// <c>Healthy()</c> (returns true), and the response comes back
/// <c>Success = true</c> — i.e. the Delivered row gets silently clobbered
/// to Discarded. Post-fix, the CAS is lost, the actor re-reads and reports
/// "already progressed to Delivered" instead of overwriting it.
/// </para>
/// </summary>
[Fact]
public void Discard_ConcurrentlyDelivered_RepliesAlreadyProgressed_DoesNotOverwrite()
{
var parkedRow = MakeNotification(status: NotificationStatus.Parked, retryCount: 5, lastError: "gave up");
var deliveredRow = MakeNotification(status: NotificationStatus.Delivered);
_repository.GetByIdAsync(parkedRow.NotificationId, Arg.Any<CancellationToken>())
.Returns(parkedRow, deliveredRow);
_repository.TryUpdateFromParkedAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>()).Returns(false);
var actor = CreateActor();
actor.Tell(new DiscardNotificationRequest("corr-race-discard", parkedRow.NotificationId), TestActor);
var response = ExpectMsg<DiscardNotificationResponse>();
Assert.Equal("corr-race-discard", response.CorrelationId);
Assert.False(response.Success);
Assert.NotNull(response.ErrorMessage);
Assert.Contains("already progressed", response.ErrorMessage);
Assert.Contains("Delivered", response.ErrorMessage);
// The critical invariant: the old unconditional UpdateAsync must NEVER
// be reached by the operator path — that call is what silently
// clobbered the Delivered row pre-fix.
_repository.DidNotReceive().UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>());
}
/// <summary>Retry half of <see cref="Discard_ConcurrentlyDelivered_RepliesAlreadyProgressed_DoesNotOverwrite"/>.</summary>
[Fact]
public void Retry_ConcurrentlyDelivered_RepliesAlreadyProgressed_DoesNotOverwrite()
{
var parkedRow = MakeNotification(status: NotificationStatus.Parked, retryCount: 5, lastError: "gave up");
var deliveredRow = MakeNotification(status: NotificationStatus.Delivered);
_repository.GetByIdAsync(parkedRow.NotificationId, Arg.Any<CancellationToken>())
.Returns(parkedRow, deliveredRow);
_repository.TryUpdateFromParkedAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>()).Returns(false);
var actor = CreateActor();
actor.Tell(new RetryNotificationRequest("corr-race-retry", parkedRow.NotificationId), TestActor);
var response = ExpectMsg<RetryNotificationResponse>();
Assert.Equal("corr-race-retry", response.CorrelationId);
Assert.False(response.Success);
Assert.NotNull(response.ErrorMessage);
Assert.Contains("already progressed", response.ErrorMessage);
Assert.Contains("Delivered", response.ErrorMessage);
_repository.DidNotReceive().UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>());
}
[Fact]
public void Discard_ParkedNotification_MarksDiscarded_AndSucceeds()
{
@@ -353,7 +425,7 @@ public class NotificationOutboxActorQueryTests : TestKit
Assert.Equal("corr-8", response.CorrelationId);
Assert.True(response.Success);
_repository.Received(1).UpdateAsync(
_repository.Received(1).TryUpdateFromParkedAsync(
Arg.Is<Notification>(n => n.Status == NotificationStatus.Discarded),
Arg.Any<CancellationToken>());
}
@@ -370,7 +442,7 @@ public class NotificationOutboxActorQueryTests : TestKit
var response = ExpectMsg<DiscardNotificationResponse>();
Assert.False(response.Success);
Assert.NotNull(response.ErrorMessage);
_repository.DidNotReceive().UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>());
_repository.DidNotReceive().TryUpdateFromParkedAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>());
}
[Fact]
@@ -147,7 +147,7 @@ public class NotificationOutboxActorRetryEmissionTests : TestKit
ExpectMsg<RetryNotificationResponse>(r => r.Success);
AwaitAssert(() =>
{
_outboxRepository.Received(1).UpdateAsync(
_outboxRepository.Received(1).TryUpdateFromParkedAsync(
Arg.Is<Notification>(n => n.Status == NotificationStatus.Pending),
Arg.Any<CancellationToken>());
});
@@ -8,14 +8,16 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests;
/// Factory for the outbox repository substitute used across these tests.
/// </summary>
/// <remarks>
/// <see cref="INotificationOutboxRepository.UpdateAsync"/> returns whether the
/// targeted UPDATE actually matched a row — <c>false</c> is the "this
/// notification no longer exists" signal the operator retry/discard handlers
/// turn into a not-found reply. NSubstitute's default for <c>Task&lt;bool&gt;</c>
/// is <c>false</c>, i.e. the VANISHED-row answer, so an unconfigured substitute
/// would silently put every test on the failure path. Tests that want a healthy
/// store take one from here; the vanished-row tests configure
/// <c>Returns(false)</c> for themselves.
/// <see cref="INotificationOutboxRepository.UpdateAsync"/> and
/// <see cref="INotificationOutboxRepository.TryUpdateFromParkedAsync"/> both
/// return whether the targeted UPDATE actually matched a row — <c>false</c> is
/// the "this notification no longer exists (or, for the CAS variant, is no
/// longer Parked)" signal the operator retry/discard handlers turn into a
/// not-found / already-progressed reply. NSubstitute's default for
/// <c>Task&lt;bool&gt;</c> is <c>false</c>, i.e. the failure answer, so an
/// unconfigured substitute would silently put every test on the failure path.
/// Tests that want a healthy store take one from here; the vanished-row and
/// lost-CAS tests configure <c>Returns(false)</c> for themselves.
/// </remarks>
internal static class OutboxRepositorySubstitute
{
@@ -26,6 +28,9 @@ internal static class OutboxRepositorySubstitute
repository
.UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>())
.Returns(true);
repository
.TryUpdateFromParkedAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>())
.Returns(true);
return repository;
}
}