Files
ScadaBridge/tests/ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests/NotificationOutboxActorIngestTests.cs
T
Joseph Doherty 5d075f1374 fix(central): review findings — no client-side audit truncation, insert-first upsert, QI-safe scripts, honest operator replies
Six adversarial-review findings in the central SQL/ingest layer.

F1 (AuditLogRepository.InsertChunkAsync) — the set-based ingest declared each
string parameter at its COLUMN width (Actor/Target 256, Action 64, Outcome 16,
Category 32, SourceNode 64), so SqlClient truncated an over-long value at bind
time and committed the mutilated row — silent, in an append-only store, with no
PayloadTruncated flag — while the per-row and reconciliation paths sent the same
value in full and let the server reject it with 2628. Bind at the value's own
length instead; explicit SqlDbType is kept (it fixes the VALUES constructor's
derived column types and datetime2 precision). Design: reject everywhere,
truncate nowhere — matching today's per-row behaviour.

F2 (SiteCallAuditRepository.UpsertAsync) — the single-statement upsert ran the
monotonic UPDATE first and INSERTed only if nothing matched. Two writers racing
the first packet of one TrackedOperationId (the cached dual-write and the
reconciliation pull carry DIFFERENT lifecycle states) both matched nothing, and
the loser then skipped its INSERT or swallowed a 2627 — dropping its
Status/RetryCount/HttpStatus/TerminalAtUtc. Legs swapped to
`IF NOT EXISTS … INSERT; UPDATE <monotonic>` — still one round trip, and the
loser's UPDATE now lands on the winner's row. The duplicate-key catch re-runs
the monotonic UPDATE for the same reason. Moved to raw SQL with explicitly-typed
parameters so the intricate rank predicate exists in exactly one place (an
untyped DateTime would bind as `datetime` and round the freshness tiebreaker).

F3 (docs/plans/sql/*.sql) — filtered-index DDL failed with error 1934 under the
documented `docker exec … sqlcmd` path, which defaults QUOTED_IDENTIFIER OFF;
once IX_Notifications_Delivered exists, QI-OFF DML on Notifications fails too.
All four scripts now open with `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO`
(own batch, so it is in force when the next batch parses), and the migration
convention in Component-ConfigurationDatabase.md documents `sqlcmd -I`. Verified
live: the pre-fix script fails 1934 without -I, the fixed one applies.

F4 (SiteCallAuditActor) — the off-mailbox reconciliation/purge passes reuse the
injected repository, so tests drove one DbContext from the pass and a mailbox
handler concurrently. Serialized at the CALL via a private SerializedRepository
wrapper applied only by the test constructors, rather than running the pass
on-mailbox: production keeps its PipeTo shape untouched, and the existing
"a blocked drain does not stall ingest/query/KPI" regression tests stay
meaningful (they would have been invalidated by suspending the mailbox).

F5 (AuditLogIngestActor) — when the batch failed because the 20 s IngestBudget
expired, the per-row fallback reused the same expired token: N instant failures,
N counter bumps, zero accepted. The fallback now gets a fresh 5 s budget (inside
the 30 s outer Ask), and a blown budget bumps the failure counter ONCE for the
batch instead of once per row.

F6 (NotificationOutboxRepository.UpdateAsync) — ExecuteUpdate's row count was
discarded, so an operator Retry/Discard of a notification the retention purge had
already deleted reported success (the pre-ExecuteUpdate code threw
DbUpdateConcurrencyException). UpdateAsync now returns whether a row matched; the
operator one-shots answer "notification not found" and emit no audit row for the
action that did not happen, while the dispatcher logs a warning (its delivery
already happened; nothing to retry). GetByIdAsync switched to AsNoTracking since
the write is out-of-band.

Tests: 5 new SQL-backed regressions (over-long Target rejected on both paths +
boundary round-trip; concurrent first-write and already-created-by-another-writer
upserts; vanished-row UpdateAsync), a token-identity pin on the ingest fallback,
a repository-concurrency detector for the SiteCallAudit passes, and vanished-row
operator-path tests. The F1/F2/F4 regressions were each confirmed failing against
the pre-fix code. Suites: ConfigurationDatabase 369, AuditLog 378, SiteCallAudit
66, NotificationOutbox 152 — all green, solution builds with 0 warnings.
2026-08-14 23:46:28 -04:00

237 lines
9.2 KiB
C#

using Akka.Actor;
using Akka.TestKit.Xunit2;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging.Abstractions;
using NSubstitute;
using NSubstitute.ExceptionExtensions;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Notifications;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.NotificationOutbox.Delivery;
using ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.NotificationOutbox.Tests;
/// <summary>
/// Task 13: Tests for the <see cref="NotificationOutboxActor"/> ingest path — building a
/// <see cref="Notification"/> from a <see cref="NotificationSubmit"/>, persisting it via
/// <see cref="INotificationOutboxRepository.InsertIfNotExistsAsync"/>, and acking the sender.
/// </summary>
public class NotificationOutboxActorIngestTests : TestKit
{
private readonly INotificationOutboxRepository _repository =
OutboxRepositorySubstitute.Healthy();
private IServiceProvider BuildServiceProvider()
{
var services = new ServiceCollection();
services.AddScoped(_ => _repository);
return services.BuildServiceProvider();
}
private IActorRef CreateActor()
{
return Sys.ActorOf(Props.Create(() => new NotificationOutboxActor(
BuildServiceProvider(),
new NotificationOutboxOptions(),
new NoOpCentralAuditWriter(),
NullLogger<NotificationOutboxActor>.Instance)));
}
private static NotificationSubmit MakeSubmit(
string? notificationId = null,
Guid? originExecutionId = null,
Guid? originParentExecutionId = null,
string? sourceNode = null)
{
return new NotificationSubmit(
NotificationId: notificationId ?? Guid.NewGuid().ToString(),
ListName: "ops-team",
Subject: "Tank overflow",
Body: "Tank 3 level critical",
SourceSiteId: "site-1",
SourceInstanceId: "instance-42",
SourceScript: "AlarmScript",
SiteEnqueuedAt: new DateTimeOffset(2026, 5, 19, 8, 30, 0, TimeSpan.Zero),
OriginExecutionId: originExecutionId,
OriginParentExecutionId: originParentExecutionId,
SourceNode: sourceNode);
}
[Fact]
public void NotificationSubmit_PersistsMappedNotification_AndAcksAccepted()
{
_repository.InsertIfNotExistsAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>())
.Returns(true);
var submit = MakeSubmit();
var actor = CreateActor();
actor.Tell(submit, TestActor);
var ack = ExpectMsg<NotificationSubmitAck>();
Assert.Equal(submit.NotificationId, ack.NotificationId);
Assert.True(ack.Accepted);
Assert.Null(ack.Error);
_repository.Received(1).InsertIfNotExistsAsync(
Arg.Is<Notification>(n =>
n.NotificationId == submit.NotificationId &&
n.Type == NotificationType.Email &&
n.ListName == submit.ListName &&
n.Subject == submit.Subject &&
n.Body == submit.Body &&
n.SourceSiteId == submit.SourceSiteId &&
n.SourceInstanceId == submit.SourceInstanceId &&
n.SourceScript == submit.SourceScript &&
n.SiteEnqueuedAt == submit.SiteEnqueuedAt &&
n.Status == NotificationStatus.Pending &&
n.CreatedAt != default),
Arg.Any<CancellationToken>());
}
[Fact]
public void NotificationSubmit_CopiesOriginExecutionId_OntoPersistedNotification()
{
// Audit Log #23: the originating script execution's id rides on the
// NotificationSubmit and must be persisted on the Notification row so
// the dispatcher can later echo it onto NotifyDeliver audit rows.
_repository.InsertIfNotExistsAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>())
.Returns(true);
var executionId = Guid.NewGuid();
var submit = MakeSubmit(originExecutionId: executionId);
var actor = CreateActor();
actor.Tell(submit, TestActor);
ExpectMsg<NotificationSubmitAck>();
_repository.Received(1).InsertIfNotExistsAsync(
Arg.Is<Notification>(n => n.OriginExecutionId == executionId),
Arg.Any<CancellationToken>());
}
[Fact]
public void NotificationSubmit_NullOriginExecutionId_PersistsNull()
{
_repository.InsertIfNotExistsAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>())
.Returns(true);
var submit = MakeSubmit(originExecutionId: null);
var actor = CreateActor();
actor.Tell(submit, TestActor);
ExpectMsg<NotificationSubmitAck>();
_repository.Received(1).InsertIfNotExistsAsync(
Arg.Is<Notification>(n => n.OriginExecutionId == null),
Arg.Any<CancellationToken>());
}
[Fact]
public void NotificationSubmit_CopiesOriginParentExecutionId_OntoPersistedNotification()
{
// Audit Log ParentExecutionId: the routed run's parent ExecutionId rides
// on the NotificationSubmit and must be persisted on the Notification row
// so the dispatcher can later echo it onto NotifyDeliver audit rows.
_repository.InsertIfNotExistsAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>())
.Returns(true);
var parentExecutionId = Guid.NewGuid();
var submit = MakeSubmit(originParentExecutionId: parentExecutionId);
var actor = CreateActor();
actor.Tell(submit, TestActor);
ExpectMsg<NotificationSubmitAck>();
_repository.Received(1).InsertIfNotExistsAsync(
Arg.Is<Notification>(n => n.OriginParentExecutionId == parentExecutionId),
Arg.Any<CancellationToken>());
}
[Fact]
public void NotificationSubmit_NullOriginParentExecutionId_PersistsNull()
{
_repository.InsertIfNotExistsAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>())
.Returns(true);
var submit = MakeSubmit(originParentExecutionId: null);
var actor = CreateActor();
actor.Tell(submit, TestActor);
ExpectMsg<NotificationSubmitAck>();
_repository.Received(1).InsertIfNotExistsAsync(
Arg.Is<Notification>(n => n.OriginParentExecutionId == null),
Arg.Any<CancellationToken>());
}
[Fact]
public void DuplicateSubmit_RepositoryReturnsFalse_StillAcksAccepted()
{
_repository.InsertIfNotExistsAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>())
.Returns(false);
var submit = MakeSubmit();
var actor = CreateActor();
actor.Tell(submit, TestActor);
var ack = ExpectMsg<NotificationSubmitAck>();
Assert.Equal(submit.NotificationId, ack.NotificationId);
Assert.True(ack.Accepted);
Assert.Null(ack.Error);
}
[Fact]
public void RepositoryThrows_AcksNotAcceptedWithError()
{
_repository.InsertIfNotExistsAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>())
.ThrowsAsync(new InvalidOperationException("database unavailable"));
var submit = MakeSubmit();
var actor = CreateActor();
actor.Tell(submit, TestActor);
var ack = ExpectMsg<NotificationSubmitAck>();
Assert.Equal(submit.NotificationId, ack.NotificationId);
Assert.False(ack.Accepted);
Assert.NotNull(ack.Error);
Assert.Contains("database unavailable", ack.Error);
}
[Fact]
public void NotificationSubmit_CopiesSourceNode_OntoPersistedNotification()
{
// SourceNode-stamping (Task 13): the originating site's node name (node-a/node-b)
// rides on the NotificationSubmit and must be persisted on the Notification row so
// central observers (KPIs, audit drill-ins, ops dashboards) can see which node
// emitted the notification.
_repository.InsertIfNotExistsAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>())
.Returns(true);
var submit = MakeSubmit(sourceNode: "node-a");
var actor = CreateActor();
actor.Tell(submit, TestActor);
ExpectMsg<NotificationSubmitAck>();
_repository.Received(1).InsertIfNotExistsAsync(
Arg.Is<Notification>(n => n.SourceNode == "node-a"),
Arg.Any<CancellationToken>());
}
[Fact]
public void NotificationSubmit_NullSourceNode_PersistsNull()
{
// Submissions from a host that didn't wire INodeIdentityProvider, or from
// pre-SourceNode-stamping clients, carry null SourceNode — the central row must
// persist NULL rather than fall back to a placeholder.
_repository.InsertIfNotExistsAsync(Arg.Any<Notification>(), Arg.Any<CancellationToken>())
.Returns(true);
var submit = MakeSubmit(sourceNode: null);
var actor = CreateActor();
actor.Tell(submit, TestActor);
ExpectMsg<NotificationSubmitAck>();
_repository.Received(1).InsertIfNotExistsAsync(
Arg.Is<Notification>(n => n.SourceNode == null),
Arg.Any<CancellationToken>());
}
}