fix(central): review findings — no client-side audit truncation, insert-first upsert, QI-safe scripts, honest operator replies

Six adversarial-review findings in the central SQL/ingest layer. F1 (AuditLogRepository.InsertChunkAsync) — the set-based ingest declared each string parameter at its COLUMN width (Actor/Target 256, Action 64, Outcome 16, Category 32, SourceNode 64), so SqlClient truncated an over-long value at bind time and committed the mutilated row — silent, in an append-only store, with no PayloadTruncated flag — while the per-row and reconciliation paths sent the same value in full and let the server reject it with 2628. Bind at the value's own length instead; explicit SqlDbType is kept (it fixes the VALUES constructor's derived column types and datetime2 precision). Design: reject everywhere, truncate nowhere — matching today's per-row behaviour. F2 (SiteCallAuditRepository.UpsertAsync) — the single-statement upsert ran the monotonic UPDATE first and INSERTed only if nothing matched. Two writers racing the first packet of one TrackedOperationId (the cached dual-write and the reconciliation pull carry DIFFERENT lifecycle states) both matched nothing, and the loser then skipped its INSERT or swallowed a 2627 — dropping its Status/RetryCount/HttpStatus/TerminalAtUtc. Legs swapped to `IF NOT EXISTS … INSERT; UPDATE <monotonic>` — still one round trip, and the loser's UPDATE now lands on the winner's row. The duplicate-key catch re-runs the monotonic UPDATE for the same reason. Moved to raw SQL with explicitly-typed parameters so the intricate rank predicate exists in exactly one place (an untyped DateTime would bind as `datetime` and round the freshness tiebreaker). F3 (docs/plans/sql/*.sql) — filtered-index DDL failed with error 1934 under the documented `docker exec … sqlcmd` path, which defaults QUOTED_IDENTIFIER OFF; once IX_Notifications_Delivered exists, QI-OFF DML on Notifications fails too. All four scripts now open with `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO` (own batch, so it is in force when the next batch parses), and the migration convention in Component-ConfigurationDatabase.md documents `sqlcmd -I`. Verified live: the pre-fix script fails 1934 without -I, the fixed one applies. F4 (SiteCallAuditActor) — the off-mailbox reconciliation/purge passes reuse the injected repository, so tests drove one DbContext from the pass and a mailbox handler concurrently. Serialized at the CALL via a private SerializedRepository wrapper applied only by the test constructors, rather than running the pass on-mailbox: production keeps its PipeTo shape untouched, and the existing "a blocked drain does not stall ingest/query/KPI" regression tests stay meaningful (they would have been invalidated by suspending the mailbox). F5 (AuditLogIngestActor) — when the batch failed because the 20 s IngestBudget expired, the per-row fallback reused the same expired token: N instant failures, N counter bumps, zero accepted. The fallback now gets a fresh 5 s budget (inside the 30 s outer Ask), and a blown budget bumps the failure counter ONCE for the batch instead of once per row. F6 (NotificationOutboxRepository.UpdateAsync) — ExecuteUpdate's row count was discarded, so an operator Retry/Discard of a notification the retention purge had already deleted reported success (the pre-ExecuteUpdate code threw DbUpdateConcurrencyException). UpdateAsync now returns whether a row matched; the operator one-shots answer "notification not found" and emit no audit row for the action that did not happen, while the dispatcher logs a warning (its delivery already happened; nothing to retry). GetByIdAsync switched to AsNoTracking since the write is out-of-band. Tests: 5 new SQL-backed regressions (over-long Target rejected on both paths + boundary round-trip; concurrent first-write and already-created-by-another-writer upserts; vanished-row UpdateAsync), a token-identity pin on the ingest fallback, a repository-concurrency detector for the SiteCallAudit passes, and vanished-row operator-path tests. The F1/F2/F4 regressions were each confirmed failing against the pre-fix code. Suites: ConfigurationDatabase 369, AuditLog 378, SiteCallAudit 66, NotificationOutbox 152 — all green, solution builds with 0 warnings.
2026-08-14 23:46:28 -04:00
parent b1de9dfdd4
commit 5d075f1374
30 changed files with 1042 additions and 126 deletions
@@ -21,7 +21,7 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests.Ingest;
 public class NotificationIngestTypeStampingTests : TestKit
 {
    private readonly INotificationOutboxRepository _outboxRepository =
-        Substitute.For<INotificationOutboxRepository>();
+        OutboxRepositorySubstitute.Healthy();

    private readonly INotificationRepository _listRepository =
        Substitute.For<INotificationRepository>();
@@ -26,7 +26,7 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests;
 public class NotificationOutboxActorAttemptEmissionTests : TestKit
 {
    private readonly INotificationOutboxRepository _outboxRepository =
-        Substitute.For<INotificationOutboxRepository>();
+        OutboxRepositorySubstitute.Healthy();

    private readonly INotificationRepository _notificationRepository =
        Substitute.For<INotificationRepository>();
@@ -21,7 +21,7 @@ public class NotificationOutboxActorAuditInjectionTests : TestKit
    private static IServiceProvider BuildEmptyProvider()
    {
        var services = new ServiceCollection();
-        services.AddScoped(_ => Substitute.For<INotificationOutboxRepository>());
+        services.AddScoped(_ => OutboxRepositorySubstitute.Healthy());
        services.AddScoped(_ => Substitute.For<INotificationRepository>());
        return services.BuildServiceProvider();
    }
@@ -22,7 +22,7 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests;
 public class NotificationOutboxActorDispatchTests : TestKit
 {
    private readonly INotificationOutboxRepository _outboxRepository =
-        Substitute.For<INotificationOutboxRepository>();
+        OutboxRepositorySubstitute.Healthy();

    private readonly INotificationRepository _notificationRepository =
        Substitute.For<INotificationRepository>();
@@ -491,7 +491,7 @@ public class NotificationOutboxActorDispatchTests : TestKit
        // INotificationOutboxRepository registration with a private counting factory,
        // so we don't mutate the shared _outboxRepository field that other tests in
        // this class configure differently.
-        var outboxRepository = Substitute.For<INotificationOutboxRepository>();
+        var outboxRepository = OutboxRepositorySubstitute.Healthy();
        // De-race (S11): hand out a fresh due notification for the FIRST THREE claims, then
        // an empty batch forever. This caps the deliverable work — and therefore the
        // UpdateAsync count — at exactly three, no matter how many dispatch ticks the
@@ -21,7 +21,7 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests;
 public class NotificationOutboxActorIngestTests : TestKit
 {
    private readonly INotificationOutboxRepository _repository =
-        Substitute.For<INotificationOutboxRepository>();
+        OutboxRepositorySubstitute.Healthy();

    private IServiceProvider BuildServiceProvider()
    {
@@ -21,7 +21,7 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests;
 public class NotificationOutboxActorPurgeTests : TestKit
 {
    private readonly INotificationOutboxRepository _outboxRepository =
-        Substitute.For<INotificationOutboxRepository>();
+        OutboxRepositorySubstitute.Healthy();

    private readonly INotificationRepository _notificationRepository =
        Substitute.For<INotificationRepository>();
@@ -22,7 +22,7 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests;
 public class NotificationOutboxActorQueryTests : TestKit
 {
    private readonly INotificationOutboxRepository _repository =
-        Substitute.For<INotificationOutboxRepository>();
+        OutboxRepositorySubstitute.Healthy();

    private IServiceProvider BuildServiceProvider()
    {
@@ -298,6 +298,48 @@ public class NotificationOutboxActorQueryTests : TestKit
        Assert.Contains("not found", response.ErrorMessage);
    }

+    /// <summary>
+    /// The row is read successfully but the retention purge deletes it before the
+    /// write lands, so the targeted UPDATE matches nothing. The operator must be
+    /// told the notification is gone — reporting a re-queue that never happened
+    /// is what the ExecuteUpdate rewrite silently introduced (the predecessor
+    /// threw DbUpdateConcurrencyException here).
+    /// </summary>
+    [Fact]
+    public void Retry_RowPurgedBeforeWrite_RepliesNotFound()
+    {
+        var row = MakeNotification(status: NotificationStatus.Parked, retryCount: 10, lastError: "gave up");
+        _repository.GetByIdAsync(row.NotificationId, Arg.Any<CancellationToken>()).Returns(row);
+        _repository.UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>()).Returns(false);
+        var actor = CreateActor();
+
+        actor.Tell(new RetryNotificationRequest("corr-vanished-retry", row.NotificationId), TestActor);
+
+        var response = ExpectMsg<RetryNotificationResponse>();
+        Assert.Equal("corr-vanished-retry", response.CorrelationId);
+        Assert.False(response.Success);
+        Assert.NotNull(response.ErrorMessage);
+        Assert.Contains("not found", response.ErrorMessage);
+    }
+
+    /// <summary>Discard half of <see cref="Retry_RowPurgedBeforeWrite_RepliesNotFound"/>.</summary>
+    [Fact]
+    public void Discard_RowPurgedBeforeWrite_RepliesNotFound()
+    {
+        var row = MakeNotification(status: NotificationStatus.Parked);
+        _repository.GetByIdAsync(row.NotificationId, Arg.Any<CancellationToken>()).Returns(row);
+        _repository.UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>()).Returns(false);
+        var actor = CreateActor();
+
+        actor.Tell(new DiscardNotificationRequest("corr-vanished-discard", row.NotificationId), TestActor);
+
+        var response = ExpectMsg<DiscardNotificationResponse>();
+        Assert.Equal("corr-vanished-discard", response.CorrelationId);
+        Assert.False(response.Success);
+        Assert.NotNull(response.ErrorMessage);
+        Assert.Contains("not found", response.ErrorMessage);
+    }
+
    [Fact]
    public void Discard_ParkedNotification_MarksDiscarded_AndSucceeds()
    {
@@ -25,7 +25,7 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests;
 public class NotificationOutboxActorRetryEmissionTests : TestKit
 {
    private readonly INotificationOutboxRepository _outboxRepository =
-        Substitute.For<INotificationOutboxRepository>();
+        OutboxRepositorySubstitute.Healthy();

    private readonly RecordingCentralAuditWriter _auditWriter = new();

@@ -27,7 +27,7 @@ namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests;
 public class NotificationOutboxActorTerminalEmissionTests : TestKit
 {
    private readonly INotificationOutboxRepository _outboxRepository =
-        Substitute.For<INotificationOutboxRepository>();
+        OutboxRepositorySubstitute.Healthy();

    private readonly INotificationRepository _notificationRepository =
        Substitute.For<INotificationRepository>();
@@ -0,0 +1,31 @@
+using NSubstitute;
+using ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications;
+using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
+
+namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests;
+
+/// <summary>
+/// Factory for the outbox repository substitute used across these tests.
+/// </summary>
+/// <remarks>
+/// <see cref="INotificationOutboxRepository.UpdateAsync"/> returns whether the
+/// targeted UPDATE actually matched a row — <c>false</c> is the "this
+/// notification no longer exists" signal the operator retry/discard handlers
+/// turn into a not-found reply. NSubstitute's default for <c>Task&lt;bool&gt;</c>
+/// is <c>false</c>, i.e. the VANISHED-row answer, so an unconfigured substitute
+/// would silently put every test on the failure path. Tests that want a healthy
+/// store take one from here; the vanished-row tests configure
+/// <c>Returns(false)</c> for themselves.
+/// </remarks>
+internal static class OutboxRepositorySubstitute
+{
+    /// <summary>Creates a substitute whose writes report that the row was found.</summary>
+    public static INotificationOutboxRepository Healthy()
+    {
+        var repository = Substitute.For<INotificationOutboxRepository>();
+        repository
+            .UpdateAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>())
+            .Returns(true);
+        return repository;
+    }
+}