scadalink-design/docs/plans/2026-05-19-notification-outbox-implementation.md

Notification Outbox — Code Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

Goal: Build the central Notification Outbox feature in the ScadaLink src/ codebase — sites store-and-forward notifications to the central cluster, which logs each to a Notifications table and delivers it via per-type adapters with retry, parking, status handles, and KPIs.

Architecture: A new ScadaLink.NotificationOutbox project hosts a NotificationOutboxActor cluster singleton on the active central node. Sites enqueue notifications into the existing site Store-and-Forward Engine (notification category, retargeted from SMTP to "central"); the S&F engine forwards them to central via ClusterClient; the CentralCommunicationActor routes each NotificationSubmit to the outbox singleton, which inserts a row into the central MS SQL Notifications table (insert-if-not-exists on a site-generated NotificationId GUID) and acks. A timer-driven dispatcher polls due rows and delivers them through an INotificationDeliveryAdapter (Email adapter now; Teams later). A Blazor page surfaces KPIs and a queryable list with Retry/Discard.

Tech Stack: .NET 10, Akka.NET (cluster singletons, ClusterClient, TestKit), EF Core (MS SQL; SQLite in-memory for tests), Blazor Server + Bootstrap, xUnit + NSubstitute + bUnit. Solution: ScadaLink.slnx.

Authoritative design: docs/plans/notif.md and docs/requirements/Component-NotificationOutbox.md. Read both before starting.

---

Conventions (read once, applies to every task)

These were confirmed by exploring the existing codebase. Follow them in every task.

• Entities (Commons): POCOs in src/ScadaLink.Commons/Entities/<Area>/. Auto-properties, parameterized constructor with null checks, navigation collections initialised to new List<T>(). No data annotations.
• EF mapping (ConfigurationDatabase): Fluent IEntityTypeConfiguration<T> classes in src/ScadaLink.ConfigurationDatabase/Configurations/, auto-applied by ApplyConfigurationsFromAssembly. Enums stored as strings via .HasConversion<string>(). Add a DbSet<T> to ScadaLinkDbContext.
• Repositories: Interface in src/ScadaLink.Commons/Interfaces/Repositories/, implementation in src/ScadaLink.ConfigurationDatabase/Repositories/. Inject ScadaLinkDbContext, use _context.Set<T>(), expose explicit SaveChangesAsync. Register in ConfigurationDatabase/ServiceCollectionExtensions.cs with AddScoped.
• Migrations: dotnet ef migrations add <Name> --project src/ScadaLink.ConfigurationDatabase — timestamp-named. Applied via MigrationHelper.ApplyOrValidateMigrationsAsync (auto in dev).
• Message contracts (Commons): record types in src/ScadaLink.Commons/Messages/<Area>/, named positional params, additive-only evolution.
• Options pattern: <Component>Options class owned by the component project; component's ServiceCollectionExtensions.Add<Component>() calls services.AddOptions<T>().BindConfiguration("ScadaLink:<Section>"); Host also services.Configure<T>(...). Config lives in appsettings.Central.json / appsettings.Site.json.
• Actors: No Akka.DI framework. Dependencies passed via Props.Create(() => new XActor(...)). Actors that need scoped services take IServiceProvider and call CreateScope(). Cluster singletons use ClusterSingletonManager.Props + ClusterSingletonProxy.Props, created in src/ScadaLink.Host/Actors/AkkaHostedService.cs.
• Tests: xUnit, NSubstitute, built-in Assert. One tests/ScadaLink.<Component>.Tests/ project per src/ project. Actor tests inherit Akka.TestKit.Xunit2.TestKit. Repository tests use SQLite in-memory (DataSource=:memory:, OpenConnection() + EnsureCreated(), IDisposable). Blazor tests inherit bUnit BunitContext. Test naming: Method_Scenario_Result.
• Run tests: whole suite dotnet test ScadaLink.slnx; single project dotnet test tests/ScadaLink.<X>.Tests/ScadaLink.<X>.Tests.csproj; single test --filter "FullyQualifiedName~<Class>.<Method>".
• Build: dotnet build ScadaLink.slnx.
• TDD: every task writes the failing test first, runs it red, implements, runs it green, commits. Use the superpowers-extended-cc:test-driven-development discipline.
• Commits: one per task, message feat(notification-outbox): <task summary>.

Status lifecycle (central Notifications table — Forwarding is site-local, never stored centrally): Pending → Retrying → Delivered | Parked, plus Discarded (operator action only).

---

Phase A — Data layer (Commons + ConfigurationDatabase)

Task 1: Notification enums

Files:

• Create: src/ScadaLink.Commons/Types/Enums/NotificationType.cs
• Create: src/ScadaLink.Commons/Types/Enums/NotificationStatus.cs
• Test: tests/ScadaLink.Commons.Tests/Types/NotificationEnumTests.cs (create if the test project lacks a Types/ folder)

Step 1 — failing test. Assert the enums expose exactly the expected members:

[Fact]
public void NotificationStatus_HasExactlyTheCentralStates()
{
    var names = Enum.GetNames<NotificationStatus>();
    Assert.Equal(
        new[] { "Pending", "Retrying", "Delivered", "Parked", "Discarded" },
        names);
}

[Fact]
public void NotificationType_HasEmail()
{
    Assert.True(Enum.IsDefined(NotificationType.Email));
}

Note: Forwarding is intentionally NOT a NotificationStatus member — it is a site-local concept (Task 19), never persisted centrally.

Step 2 — run red: dotnet test tests/ScadaLink.Commons.Tests/ScadaLink.Commons.Tests.csproj --filter "FullyQualifiedName~NotificationEnumTests" → FAIL (types don't exist).

Step 3 — implement.

// NotificationType.cs  — namespace ScadaLink.Commons.Types.Enums
public enum NotificationType { Email }   // Teams and others added later

// NotificationStatus.cs — namespace ScadaLink.Commons.Types.Enums
public enum NotificationStatus { Pending, Retrying, Delivered, Parked, Discarded }

Step 4 — run green. Same filter → PASS.

Step 5 — commit:

git add src/ScadaLink.Commons/Types/Enums/NotificationType.cs src/ScadaLink.Commons/Types/Enums/NotificationStatus.cs tests/ScadaLink.Commons.Tests/Types/NotificationEnumTests.cs
git commit -m "feat(notification-outbox): add NotificationType and NotificationStatus enums"

---

Task 2: Notification entity POCO

Files:

• Create: src/ScadaLink.Commons/Entities/Notifications/Notification.cs
• Test: tests/ScadaLink.Commons.Tests/Entities/NotificationEntityTests.cs

Step 1 — failing test. Verify the constructor sets required fields, defaults Status to Pending and RetryCount to 0, and rejects nulls:

[Fact]
public void Constructor_SetsDefaults()
{
    var n = new Notification("id-1", NotificationType.Email, "ops-team", "subj", "body", "SiteA");
    Assert.Equal(NotificationStatus.Pending, n.Status);
    Assert.Equal(0, n.RetryCount);
    Assert.Equal("id-1", n.NotificationId);
}

[Fact]
public void Constructor_NullListName_Throws()
    => Assert.Throws<ArgumentNullException>(
        () => new Notification("id", NotificationType.Email, null!, "s", "b", "SiteA"));

Step 2 — run red.

Step 3 — implement. Match the Notifications table schema in notif.md:

namespace ScadaLink.Commons.Entities.Notifications;

public class Notification
{
    public string NotificationId { get; set; }            // GUID PK, generated at site
    public NotificationType Type { get; set; }
    public string ListName { get; set; }
    public string Subject { get; set; }
    public string Body { get; set; }
    public string? TypeData { get; set; }                 // JSON extensibility hook
    public NotificationStatus Status { get; set; } = NotificationStatus.Pending;
    public int RetryCount { get; set; }
    public string? LastError { get; set; }
    public string? ResolvedTargets { get; set; }          // snapshotted at delivery, for audit
    public string SourceSiteId { get; set; }
    public string? SourceInstanceId { get; set; }
    public string? SourceScript { get; set; }
    public DateTimeOffset SiteEnqueuedAt { get; set; }
    public DateTimeOffset CreatedAt { get; set; }          // central ingest time
    public DateTimeOffset? LastAttemptAt { get; set; }
    public DateTimeOffset? NextAttemptAt { get; set; }
    public DateTimeOffset? DeliveredAt { get; set; }

    public Notification(string notificationId, NotificationType type, string listName,
        string subject, string body, string sourceSiteId)
    {
        NotificationId = notificationId ?? throw new ArgumentNullException(nameof(notificationId));
        Type = type;
        ListName = listName ?? throw new ArgumentNullException(nameof(listName));
        Subject = subject ?? throw new ArgumentNullException(nameof(subject));
        Body = body ?? throw new ArgumentNullException(nameof(body));
        SourceSiteId = sourceSiteId ?? throw new ArgumentNullException(nameof(sourceSiteId));
    }
}

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add Notification entity).

---

Task 3: Type field on NotificationList

Files:

• Modify: src/ScadaLink.Commons/Entities/Notifications/NotificationList.cs
• Modify: src/ScadaLink.ConfigurationDatabase/Configurations/NotificationConfiguration.cs (NotificationListConfiguration)
• Test: tests/ScadaLink.ConfigurationDatabase.Tests/RepositoryTests.cs (add a test to the notification repository tests)

Step 1 — failing test. A NotificationList round-trips its Type through the repository:

[Fact]
public async Task NotificationList_PersistsType()
{
    var list = new NotificationList("ops") { Type = NotificationType.Email };
    await _notificationRepo.AddNotificationListAsync(list);
    await _notificationRepo.SaveChangesAsync();
    _context.ChangeTracker.Clear();
    var loaded = await _notificationRepo.GetListByNameAsync("ops");
    Assert.Equal(NotificationType.Email, loaded!.Type);
}

Step 2 — run red.

Step 3 — implement. Add to NotificationList: public NotificationType Type { get; set; } = NotificationType.Email;. In NotificationListConfiguration.Configure, add builder.Property(n => n.Type).HasConversion<string>().HasMaxLength(32).IsRequired();.

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add Type field to NotificationList).

---

Task 4: Notification EF configuration + DbSet

Files:

• Create: src/ScadaLink.ConfigurationDatabase/Configurations/NotificationOutboxConfiguration.cs
• Modify: src/ScadaLink.ConfigurationDatabase/ScadaLinkDbContext.cs (add DbSet<Notification>)
• Test: tests/ScadaLink.ConfigurationDatabase.Tests/RepositoryTests.cs

Step 1 — failing test. A Notification round-trips all fields through the DbContext (use the SQLite in-memory fixture pattern). Assert the Status/Type enums persist as strings and the row is found by NotificationId.

Step 2 — run red.

Step 3 — implement. Configuration:

public class NotificationOutboxConfiguration : IEntityTypeConfiguration<Notification>
{
    public void Configure(EntityTypeBuilder<Notification> builder)
    {
        builder.ToTable("Notifications");
        builder.HasKey(n => n.NotificationId);
        builder.Property(n => n.NotificationId).HasMaxLength(64);
        builder.Property(n => n.Type).HasConversion<string>().HasMaxLength(32).IsRequired();
        builder.Property(n => n.Status).HasConversion<string>().HasMaxLength(32).IsRequired();
        builder.Property(n => n.ListName).HasMaxLength(200).IsRequired();
        builder.Property(n => n.Subject).HasMaxLength(1000).IsRequired();
        builder.Property(n => n.Body).IsRequired();                       // nvarchar(max)
        builder.Property(n => n.TypeData);                                // nvarchar(max), nullable
        builder.Property(n => n.ResolvedTargets);                         // nvarchar(max), nullable
        builder.Property(n => n.LastError).HasMaxLength(4000);
        builder.Property(n => n.SourceSiteId).HasMaxLength(100).IsRequired();
        builder.Property(n => n.SourceInstanceId).HasMaxLength(200);
        builder.Property(n => n.SourceScript).HasMaxLength(200);
        builder.HasIndex(n => new { n.Status, n.NextAttemptAt });         // dispatcher polling
        builder.HasIndex(n => new { n.SourceSiteId, n.CreatedAt });       // KPIs / UI query
    }
}

Add public DbSet<Notification> Notifications => Set<Notification>(); to ScadaLinkDbContext.

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add Notification EF configuration and DbSet).

---

Task 5: INotificationOutboxRepository + implementation

Files:

• Create: src/ScadaLink.Commons/Interfaces/Repositories/INotificationOutboxRepository.cs
• Create: src/ScadaLink.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs
• Modify: src/ScadaLink.ConfigurationDatabase/ServiceCollectionExtensions.cs (register AddScoped)
• Test: tests/ScadaLink.ConfigurationDatabase.Tests/RepositoryTests.cs

Step 1 — failing tests. Cover the operations the outbox actor needs:

• InsertIfNotExistsAsync inserts a new row and returns true; a second call with the same NotificationId returns false and does not duplicate (idempotency key).
• GetDueAsync(now, batchSize) returns Pending rows and Retrying rows with NextAttemptAt <= now, ordered by CreatedAt, capped at batchSize.
• UpdateAsync persists status transitions.
• GetByIdAsync returns a row or null.
• QueryAsync(filter, page, pageSize) filters by status/type/source site and paginates.
• DeleteTerminalOlderThanAsync(cutoff) bulk-deletes Delivered/Parked/Discarded rows older than cutoff and returns the count; leaves non-terminal rows.
• ComputeKpisAsync returns queue depth, stuck count, parked count, delivered-last-window, oldest-pending age.

Step 2 — run red.

Step 3 — implement. Interface:

public interface INotificationOutboxRepository
{
    Task<bool> InsertIfNotExistsAsync(Notification n, CancellationToken ct = default);
    Task<IReadOnlyList<Notification>> GetDueAsync(DateTimeOffset now, int batchSize, CancellationToken ct = default);
    Task<Notification?> GetByIdAsync(string notificationId, CancellationToken ct = default);
    Task UpdateAsync(Notification n, CancellationToken ct = default);
    Task<(IReadOnlyList<Notification> Rows, int TotalCount)> QueryAsync(
        NotificationOutboxFilter filter, int pageNumber, int pageSize, CancellationToken ct = default);
    Task<int> DeleteTerminalOlderThanAsync(DateTimeOffset cutoff, CancellationToken ct = default);
    Task<NotificationKpiSnapshot> ComputeKpisAsync(DateTimeOffset stuckCutoff, DateTimeOffset deliveredSince, CancellationToken ct = default);
    Task<int> SaveChangesAsync(CancellationToken ct = default);
}

NotificationOutboxFilter (a record in Commons Types/) and NotificationKpiSnapshot (a record) are created in this task alongside the interface. InsertIfNotExistsAsync: check await _context.Notifications.FindAsync(...), if present return false, else AddAsync + SaveChangesAsync, return true. DeleteTerminalOlderThanAsync: use ExecuteDeleteAsync with a Where on terminal statuses and CreatedAt < cutoff. Register in ServiceCollectionExtensions.AddConfigurationDatabase.

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add NotificationOutbox repository).

---

Task 6: EF migration

Files:

• Create: src/ScadaLink.ConfigurationDatabase/Migrations/<timestamp>_AddNotificationsTable.cs (generated)

Step 1 — generate:

dotnet ef migrations add AddNotificationsTable --project src/ScadaLink.ConfigurationDatabase

This also picks up the NotificationList.Type column from Task 3.

Step 2 — verify. Inspect the generated migration: confirm a Notifications table with the columns and two indexes from Task 4, and an AlterColumn/AddColumn for NotificationLists.Type. Run the ConfigurationDatabase test project — the SQLite EnsureCreated() fixture builds from the model, and dotnet build ScadaLink.slnx must succeed.

Step 3 — run: dotnet test tests/ScadaLink.ConfigurationDatabase.Tests/ScadaLink.ConfigurationDatabase.Tests.csproj → PASS.

Step 4 — commit (feat(notification-outbox): add Notifications table migration).

---

Phase B — Message contracts (Commons)

Task 7: Site↔central notification message contracts

Files:

• Create: src/ScadaLink.Commons/Messages/Notification/NotificationMessages.cs
• Test: tests/ScadaLink.Commons.Tests/Messages/NotificationMessagesTests.cs

Step 1 — failing test. A trivial construction/round-trip test (these are records — assert positional construction and value equality; if the project has a serialization test helper, round-trip through it).

Step 2 — run red.

Step 3 — implement. Namespace ScadaLink.Commons.Messages.Notification:

// Site → Central: submit a notification for central delivery (fire-and-forget with ack).
public record NotificationSubmit(
    string NotificationId, string ListName, string Subject, string Body,
    string SourceSiteId, string? SourceInstanceId, string? SourceScript,
    DateTimeOffset SiteEnqueuedAt);

// Central → Site: ack after the row is persisted (idempotent — safe to re-send).
public record NotificationSubmitAck(string NotificationId, bool Accepted, string? Error);

// Site → Central: query delivery status for a NotificationId.
public record NotificationStatusQuery(string CorrelationId, string NotificationId);

public record NotificationStatusResponse(
    string CorrelationId, bool Found, string Status,
    int RetryCount, string? LastError,
    DateTimeOffset? DeliveredAt);

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add site/central notification message contracts).

---

Task 8: Outbox UI query/action contracts

Files:

• Create: src/ScadaLink.Commons/Messages/Notification/NotificationOutboxQueries.cs
• Test: tests/ScadaLink.Commons.Tests/Messages/NotificationOutboxQueriesTests.cs

Step 1 — failing test. Construction test as in Task 7.

Step 2 — run red.

Step 3 — implement. Records the Central UI / CommunicationService use to talk to the outbox actor:

public record NotificationOutboxQueryRequest(
    string CorrelationId, string? StatusFilter, string? TypeFilter, string? SourceSiteFilter,
    string? ListNameFilter, bool StuckOnly, string? SubjectKeyword,
    DateTimeOffset? From, DateTimeOffset? To, int PageNumber, int PageSize);

public record NotificationSummary(
    string NotificationId, string Type, string ListName, string Subject, string Status,
    int RetryCount, string? LastError, string SourceSiteId, string? SourceInstanceId,
    DateTimeOffset CreatedAt, DateTimeOffset? DeliveredAt, bool IsStuck);

public record NotificationOutboxQueryResponse(
    string CorrelationId, bool Success, string? ErrorMessage,
    IReadOnlyList<NotificationSummary> Notifications, int TotalCount);

public record RetryNotificationRequest(string CorrelationId, string NotificationId);
public record RetryNotificationResponse(string CorrelationId, bool Success, string? ErrorMessage);
public record DiscardNotificationRequest(string CorrelationId, string NotificationId);
public record DiscardNotificationResponse(string CorrelationId, bool Success, string? ErrorMessage);

public record NotificationKpiRequest(string CorrelationId);
public record NotificationKpiResponse(
    string CorrelationId, int QueueDepth, int StuckCount, int ParkedCount,
    int DeliveredLastInterval, TimeSpan? OldestPendingAge);

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add outbox query and action contracts).

---

Phase C — NotificationOutbox project + delivery

Task 9: Create the ScadaLink.NotificationOutbox project

Files:

• Create: src/ScadaLink.NotificationOutbox/ScadaLink.NotificationOutbox.csproj
• Create: tests/ScadaLink.NotificationOutbox.Tests/ScadaLink.NotificationOutbox.Tests.csproj
• Modify: ScadaLink.slnx (add both projects)

Step 1 — create the projects. Copy the .csproj shape from src/ScadaLink.NotificationService/ScadaLink.NotificationService.csproj (same TargetFramework, central-managed package versions via Directory.Packages.props). The src project references ScadaLink.Commons and Akka packages (Akka, Akka.Cluster.Tools). The test project mirrors tests/ScadaLink.NotificationService.Tests/ (xUnit, NSubstitute, Akka.TestKit.Xunit2) and references the new src project. Add both <Project> entries to ScadaLink.slnx.

Step 2 — add a placeholder test so the test project is non-empty:

public class ProjectSmokeTest { [Fact] public void ProjectCompiles() => Assert.True(true); }

Step 3 — verify: dotnet build ScadaLink.slnx succeeds; dotnet test tests/ScadaLink.NotificationOutbox.Tests/ScadaLink.NotificationOutbox.Tests.csproj → PASS.

Step 4 — commit (feat(notification-outbox): scaffold ScadaLink.NotificationOutbox project).

---

Task 10: NotificationOutboxOptions

Files:

• Create: src/ScadaLink.NotificationOutbox/NotificationOutboxOptions.cs
• Test: tests/ScadaLink.NotificationOutbox.Tests/NotificationOutboxOptionsTests.cs

Step 1 — failing test. Assert the defaults.

Step 2 — run red.

Step 3 — implement.

public class NotificationOutboxOptions
{
    public TimeSpan DispatchInterval { get; set; } = TimeSpan.FromSeconds(10);
    public int DispatchBatchSize { get; set; } = 100;
    public TimeSpan StuckAgeThreshold { get; set; } = TimeSpan.FromMinutes(10);
    public TimeSpan TerminalRetention { get; set; } = TimeSpan.FromDays(365);
    public TimeSpan PurgeInterval { get; set; } = TimeSpan.FromDays(1);
    public TimeSpan DeliveredKpiWindow { get; set; } = TimeSpan.FromMinutes(1);
}

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add NotificationOutboxOptions).

---

Task 11: INotificationDeliveryAdapter + DeliveryOutcome

Files:

• Create: src/ScadaLink.NotificationOutbox/Delivery/INotificationDeliveryAdapter.cs
• Create: src/ScadaLink.NotificationOutbox/Delivery/DeliveryOutcome.cs
• Test: tests/ScadaLink.NotificationOutbox.Tests/Delivery/DeliveryOutcomeTests.cs

Step 1 — failing test. Assert DeliveryOutcome factory methods produce the right classification.

Step 2 — run red.

Step 3 — implement. Mirror the External System Gateway error-classification pattern:

public enum DeliveryResult { Success, TransientFailure, PermanentFailure }

public record DeliveryOutcome(DeliveryResult Result, string? ResolvedTargets, string? Error)
{
    public static DeliveryOutcome Success(string resolvedTargets) => new(DeliveryResult.Success, resolvedTargets, null);
    public static DeliveryOutcome Transient(string error) => new(DeliveryResult.TransientFailure, null, error);
    public static DeliveryOutcome Permanent(string error) => new(DeliveryResult.PermanentFailure, null, error);
}

public interface INotificationDeliveryAdapter
{
    NotificationType Type { get; }
    Task<DeliveryOutcome> DeliverAsync(Notification notification, CancellationToken ct = default);
}

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add delivery adapter abstraction).

---

Task 12: EmailNotificationDeliveryAdapter

Files:

• Create: src/ScadaLink.NotificationOutbox/Delivery/EmailNotificationDeliveryAdapter.cs
• Modify: src/ScadaLink.NotificationOutbox/ScadaLink.NotificationOutbox.csproj (reference ScadaLink.NotificationService for ISmtpClientWrapper)
• Test: tests/ScadaLink.NotificationOutbox.Tests/Delivery/EmailNotificationDeliveryAdapterTests.cs

Step 1 — failing tests. Using NSubstitute mocks of INotificationOutboxRepository-resolved data and a substituted ISmtpClientWrapper:

• list resolved + send succeeds → DeliveryResult.Success, ResolvedTargets lists the recipient addresses.
• list not found / no recipients → PermanentFailure.
• SMTP throws SmtpPermanentException → PermanentFailure.
• SMTP throws a transient error (socket/timeout) → TransientFailure.

Step 2 — run red.

Step 3 — implement. The adapter resolves the list + recipients + SMTP config from INotificationRepository (the existing notification-list repo — recipients are resolved centrally at delivery time), composes and sends via the existing ISmtpClientWrapper (Func<ISmtpClientWrapper> injected, same as NotificationService), classifies errors identically to NotificationDeliveryService. Reuse the SMTP composition logic from src/ScadaLink.NotificationService/NotificationDeliveryService.cs (BCC delivery, plain text, address validation, the SmtpPermanentException → permanent mapping). On success return DeliveryOutcome.Success(<comma-joined recipient addresses>). Type => NotificationType.Email.

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add Email delivery adapter).

---

Task 13: NotificationOutboxActor — ingest

Files:

• Create: src/ScadaLink.NotificationOutbox/NotificationOutboxActor.cs
• Create: src/ScadaLink.NotificationOutbox/Messages/InternalMessages.cs (actor-internal tick messages)
• Test: tests/ScadaLink.NotificationOutbox.Tests/NotificationOutboxActorIngestTests.cs

Step 1 — failing tests (TestKit). The actor takes IServiceProvider, NotificationOutboxOptions, ILogger. Use a mocked INotificationOutboxRepository registered in the test ServiceProvider:

• Send NotificationSubmit → actor calls InsertIfNotExistsAsync with a Notification whose fields map from the message, Status = Pending, CreatedAt set; replies NotificationSubmitAck(NotificationId, Accepted: true, null) to Sender.
• Send the same NotificationSubmit twice → second InsertIfNotExistsAsync returns false; actor still replies Accepted: true (idempotent — the row already exists, ack so the site clears its buffer).
• Repository throws → actor replies Accepted: false with the error (site will retry the forward).

Step 2 — run red.

Step 3 — implement. ReceiveActor. On NotificationSubmit: build a Notification, CreateScope() to resolve INotificationOutboxRepository, call InsertIfNotExistsAsync, PipeTo the result back so the reply preserves Sender. Reply NotificationSubmitAck. Keep dispatch (Task 14) out of this task — ingest only.

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add NotificationOutboxActor ingest).

---

Task 14: NotificationOutboxActor — dispatcher loop

Files:

• Modify: src/ScadaLink.NotificationOutbox/NotificationOutboxActor.cs
• Test: tests/ScadaLink.NotificationOutbox.Tests/NotificationOutboxActorDispatchTests.cs

Step 1 — failing tests (TestKit, with a registered set of INotificationDeliveryAdapter and a mocked repo):

• On a DispatchTick, the actor calls GetDueAsync, and for each row invokes the adapter for its Type.
• adapter Success → row updated to Delivered, DeliveredAt/ResolvedTargets/LastAttemptAt set, UpdateAsync called.
• adapter TransientFailure → Retrying, RetryCount incremented, NextAttemptAt = now + retry interval, LastError set.
• adapter TransientFailure when RetryCount already at the SMTP-config max → Parked.
• adapter PermanentFailure → Parked immediately, LastError set.
• no adapter for the row's Type → Parked with an explanatory error.

Step 2 — run red.

Step 3 — implement. IWithTimers; in PreStart start a periodic DispatchTick every options.DispatchInterval. On DispatchTick: scope-resolve the repo, GetDueAsync(now, options.DispatchBatchSize), and for each notification resolve the adapter from a Dictionary<NotificationType, INotificationDeliveryAdapter> (injected), await DeliverAsync, apply the status transition, UpdateAsync. Retry count/interval come from the central SMTP config (SmtpConfiguration.MaxRetries / RetryDelay via INotificationRepository). Run delivery on a blocking-safe path (the actor PipeTos the async work; do not block the actor thread). Guard against overlapping ticks (ignore a new tick while one is in flight).

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add dispatcher loop to NotificationOutboxActor).

---

Task 15: NotificationOutboxActor — query, retry, discard, KPIs

Files:

• Modify: src/ScadaLink.NotificationOutbox/NotificationOutboxActor.cs
• Test: tests/ScadaLink.NotificationOutbox.Tests/NotificationOutboxActorQueryTests.cs

Step 1 — failing tests (TestKit):

• NotificationOutboxQueryRequest → actor calls QueryAsync, replies NotificationOutboxQueryResponse with mapped NotificationSummary rows; IsStuck true when Status is Pending/Retrying and CreatedAt older than options.StuckAgeThreshold.
• NotificationStatusQuery → replies NotificationStatusResponse (Found:false when the id is unknown).
• RetryNotificationRequest on a Parked row → row reset to Pending, RetryCount 0, NextAttemptAt cleared; replies success. On a non-Parked row → Success:false.
• DiscardNotificationRequest on a Parked row → Status = Discarded; replies success.
• NotificationKpiRequest → replies NotificationKpiResponse from ComputeKpisAsync (stuck cutoff = now − StuckAgeThreshold; delivered window = now − DeliveredKpiWindow).

Step 2 — run red.

Step 3 — implement the additional Receive<> handlers, each scope-resolving the repo and PipeTo-ing the reply.

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add query, retry, discard, and KPI handlers).

---

Task 16: Daily purge job

Files:

• Modify: src/ScadaLink.NotificationOutbox/NotificationOutboxActor.cs
• Test: tests/ScadaLink.NotificationOutbox.Tests/NotificationOutboxActorPurgeTests.cs

Step 1 — failing test. On a PurgeTick, the actor calls DeleteTerminalOlderThanAsync(now − options.TerminalRetention).

Step 2 — run red.

Step 3 — implement. In PreStart start a second periodic timer PurgeTick every options.PurgeInterval. Handler scope-resolves the repo and calls DeleteTerminalOlderThanAsync; log the deleted count.

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add daily terminal-row purge).

---

Task 17: AddNotificationOutbox DI extension

Files:

• Create: src/ScadaLink.NotificationOutbox/ServiceCollectionExtensions.cs
• Test: tests/ScadaLink.NotificationOutbox.Tests/ServiceRegistrationTests.cs

Step 1 — failing test. Build a ServiceCollection, call AddNotificationOutbox, and assert NotificationOutboxOptions, the EmailNotificationDeliveryAdapter, and the adapter dictionary resolve.

Step 2 — run red.

Step 3 — implement. public const string OptionsSection = "ScadaLink:NotificationOutbox"; plus AddNotificationOutbox(this IServiceCollection) registering AddOptions<NotificationOutboxOptions>().BindConfiguration(OptionsSection), the SMTP client Func<ISmtpClientWrapper> (reuse NotificationService's registration or register here), EmailNotificationDeliveryAdapter, and a registration that exposes IReadOnlyDictionary<NotificationType, INotificationDeliveryAdapter> built from all registered adapters.

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add AddNotificationOutbox DI registration).

---

Phase D — Site retarget + central wiring

Task 18: Retarget the site S&F notification handler to forward to central

Files:

• Modify: src/ScadaLink.StoreAndForward/StoreAndForwardService.cs and/or the site registration that wires the Notification category delivery handler
• Modify: src/ScadaLink.Host/SiteServiceRegistration.cs (where the notification handler is registered)
• Test: tests/ScadaLink.StoreAndForward.Tests/ (a test that the registered notification handler forwards to the communication actor and treats an ack as success)

Step 1 — investigate + failing test. Currently the Notification category handler calls NotificationDeliveryService.DeliverBufferedAsync. The new handler must instead send a NotificationSubmit to central via the site's communication actor (ClusterClient.Send("/user/central-communication", submit)) and treat a NotificationSubmitAck(Accepted:true) as delivered (true), a non-ack/timeout as transient (throw), so S&F retries the forward. Write a test with a TestProbe standing in for the central client: handler invoked → probe receives NotificationSubmit; reply NotificationSubmitAck(Accepted:true) → handler result true; timeout → handler throws (transient).

Step 2 — run red.

Step 3 — implement. Add a NotificationForwarder (small class or the handler lambda) that holds the site communication actor ref and does Ask<NotificationSubmitAck> with the host-configured forward-retry timeout. Register it as the StoreAndForwardCategory.Notification delivery handler in SiteServiceRegistration, replacing the NotificationDeliveryService handler. The S&F engine already buffers/retries on a thrown (transient) result — no S&F core change needed.

Step 4 — run green. Step 5 — commit (feat(notification-outbox): forward site S&F notifications to central).

---

Task 19: Notify.Send async + Notify.Status (SiteRuntime)

Files:

• Modify: src/ScadaLink.SiteRuntime/Scripts/ScriptRuntimeContext.cs (NotifyHelper, NotifyTarget)
• Test: tests/ScadaLink.SiteRuntime.Tests/ (Notify API tests)

Step 1 — failing tests.

• Notify.To("list").Send("subj","body") generates a GUID NotificationId, enqueues a StoreAndForwardCategory.Notification message into StoreAndForwardService (target "central", payload = serialized NotificationSubmit), and returns the NotificationId string immediately.
• Notify.Status(id) issues a NotificationStatusQuery to central and returns the mapped status record; while the notification is still in the site S&F buffer (central has no row / query says Found:false but the S&F buffer still holds the id) it reports Forwarding.

Step 2 — run red.

Step 3 — implement. Change NotifyTarget.Send to return Task<string> (the NotificationId): create the GUID, build a NotificationSubmit (with SourceSiteId, SourceInstanceId = _instanceName, SiteEnqueuedAt = UtcNow), EnqueueAsync(Notification, "central", payloadJson). Add NotifyHelper.Status(string notificationId) returning a status record: query central via the site communication actor; if central returns Found:false and the id is still buffered in S&F, return status Forwarding. Keep the script-facing surface minimal (Send, Status).

Step 2 note: the Notify API is consumed by compiled scripts — confirm the script trust model / compilation still accepts the changed signature; update any script-API surface tests.

Step 4 — run green. Step 5 — commit (feat(notification-outbox): async Notify.Send with status handle).

---

Task 20: Central ingest routing

Files:

• Modify: src/ScadaLink.Communication/Actors/CentralCommunicationActor.cs
• Test: tests/ScadaLink.Communication.Tests/CentralCommunicationActorTests.cs

Step 1 — failing test. When CentralCommunicationActor receives a NotificationSubmit (sent site→central via ClusterClient to /user/central-communication), it forwards it to the notification-outbox singleton proxy and the ack flows back to the original Sender. Use a TestProbe for the outbox proxy.

Step 2 — run red.

Step 3 — implement. CentralCommunicationActor takes an optional outbox-proxy IActorRef (passed at construction by the Host, Task 21). Receive<NotificationSubmit>(m => _outboxProxy.Forward(m)) — Forward preserves the original sender so the NotificationSubmitAck returns to the site's ClusterClient.

Step 4 — run green. Step 5 — commit (feat(notification-outbox): route NotificationSubmit to the outbox actor).

---

Task 21: Host registration + appsettings

Files:

• Modify: src/ScadaLink.Host/Actors/AkkaHostedService.cs (RegisterCentralActors)
• Modify: src/ScadaLink.Host/Program.cs (call AddNotificationOutbox; Configure<NotificationOutboxOptions>)
• Modify: src/ScadaLink.Host/appsettings.Central.json (ScadaLink:NotificationOutbox section)
• Modify: src/ScadaLink.Host/appsettings.Site.json (site→central notification forward-retry interval, if not already covered by S&F config)
• Test: tests/ScadaLink.Host.Tests/ if present, else verify via build + the integration test in Task 25

Step 1 — implement. In RegisterCentralActors: create the NotificationOutboxActor as a cluster singleton (ClusterSingletonManager.Props + ClusterSingletonProxy.Props, singleton name "notification-outbox", no explicit role — central nodes only run this role), passing IServiceProvider, NotificationOutboxOptions, the adapter dictionary, and a logger. Pass the singleton proxy ref into CentralCommunicationActor's Props.Create. In Program.cs central path, call builder.Services.AddNotificationOutbox() and services.Configure<NotificationOutboxOptions>(...GetSection(ServiceCollectionExtensions.OptionsSection)). Add the ScadaLink:NotificationOutbox block to appsettings.Central.json with the Task 10 defaults.

Step 2 — verify: dotnet build ScadaLink.slnx succeeds.

Step 3 — commit (feat(notification-outbox): register NotificationOutbox singleton in Host).

---

Phase E — Central UI

Task 22: CommunicationService outbox methods

Files:

• Modify: src/ScadaLink.Communication/CommunicationService.cs
• Test: tests/ScadaLink.Communication.Tests/CommunicationServiceTests.cs (or the existing service test file)

Step 1 — failing tests. New methods QueryNotificationOutboxAsync, RetryNotificationAsync, DiscardNotificationAsync, GetNotificationKpisAsync each Ask the central outbox proxy and return the typed response. (These are central-side and do not go through SiteEnvelope — they talk to the local outbox proxy directly.) Test with a TestProbe for the proxy.

Step 2 — run red.

Step 3 — implement. Add an outbox-proxy IActorRef to CommunicationService (set by the Host like SetCommunicationActor). Each method Ask<TResponse>(request, _options.QueryTimeout).

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add CommunicationService outbox methods).

---

Task 23: Notification Outbox Blazor page + nav entry

Files:

• Create: src/ScadaLink.CentralUI/Components/Pages/Monitoring/NotificationOutbox.razor
• Modify: src/ScadaLink.CentralUI/Components/Layout/NavMenu.razor
• Test: tests/ScadaLink.CentralUI.Tests/Pages/NotificationOutboxPageTests.cs (bUnit)

Step 1 — failing test (bUnit). Render the page with a substituted CommunicationService returning a fixed KPI response and a page of NotificationSummary rows; assert the KPI tiles show the values and the table renders the rows; assert clicking Retry on a Parked row calls RetryNotificationAsync.

Step 2 — run red.

Step 3 — implement. Model the page on Components/Pages/Monitoring/ParkedMessages.razor: @page "/monitoring/notification-outbox", @attribute [Authorize(Policy = AuthorizationPolicies.RequireDeployment)]. KPI tile row (Bootstrap card tiles like Health.razor) bound to GetNotificationKpisAsync; a filter card (status, type, source site, list, time range, stuck-only toggle, subject keyword); a table of NotificationSummary with stuck rows badged; Retry/Discard buttons on Parked rows using IDialogService.ConfirmAsync + ToastNotification. Add a NavLink to NavMenu.razor under the Deployment-role Monitoring section (href="/monitoring/notification-outbox").

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add Notification Outbox UI page).

---

Task 24: Health dashboard outbox KPI tiles

Files:

• Modify: src/ScadaLink.CentralUI/Components/Pages/Monitoring/Health.razor
• Test: tests/ScadaLink.CentralUI.Tests/Pages/HealthPageTests.cs (extend if present)

Step 1 — failing test (bUnit). With a substituted CommunicationService.GetNotificationKpisAsync, the Health page renders three headline outbox tiles: queue depth, stuck count, parked count.

Step 2 — run red.

Step 3 — implement. Add a "Notification Outbox" tile row to Health.razor, fetched on init / on the existing 10s polling timer, styled like the existing overview cards.

Step 4 — run green. Step 5 — commit (feat(notification-outbox): add outbox KPI tiles to Health dashboard).

---

Phase F — Integration & verification

Task 25: End-to-end integration test

Files:

• Create: tests/ScadaLink.IntegrationTests/NotificationOutboxFlowTests.cs

Step 1 — failing test. Following the patterns in tests/ScadaLink.IntegrationTests/, exercise the flow with an in-memory/SQLite-backed ScadaLinkDbContext and a real NotificationOutboxActor: submit a NotificationSubmit → assert a Notifications row exists (Pending) → trigger a DispatchTick with a stub adapter that returns Success → assert the row is Delivered. Add a second case: stub adapter returns PermanentFailure → row Parked; then a RetryNotificationRequest → row back to Pending.

Step 2 — run red. Step 3 — make it pass (it should, if Phases A–D are correct; fix any wiring gaps found). Step 4 — commit (test(notification-outbox): end-to-end outbox flow integration test).

---

Task 26: Full build + suite verification

Files: none (verification only).

Step 1: dotnet build ScadaLink.slnx → must succeed with no errors.

Step 2: dotnet test ScadaLink.slnx → the whole suite must pass. Investigate and fix any regressions (notably in ScadaLink.NotificationService.Tests, ScadaLink.StoreAndForward.Tests, ScadaLink.SiteRuntime.Tests, ScadaLink.Communication.Tests — the docs/design changed the notification path and existing tests may assert old behavior; update them to the new design).

Step 3: If the docker cluster is used for smoke testing, note that bash docker/deploy.sh rebuilds the image — out of scope for this plan unless the user asks.

Step 4 — commit any test fixes (test(notification-outbox): update existing tests for the central-delivery model).

---

Done

The Notification Outbox feature is implemented end to end: site scripts enqueue notifications that store-and-forward to central, the NotificationOutboxActor singleton ingests them into the Notifications table and delivers them via the Email adapter with retry/parking, operators see KPIs and manage notifications from the Central UI, and the full test suite passes. Teams and other delivery adapters can be added later by implementing INotificationDeliveryAdapter and registering it — no other change required.