# Notification Outbox — Code Implementation Plan > **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. **Goal:** Build the central Notification Outbox feature in the ScadaLink `src/` codebase — sites store-and-forward notifications to the central cluster, which logs each to a `Notifications` table and delivers it via per-type adapters with retry, parking, status handles, and KPIs. **Architecture:** A new `ScadaLink.NotificationOutbox` project hosts a `NotificationOutboxActor` cluster singleton on the active central node. Sites enqueue notifications into the existing site Store-and-Forward Engine (notification category, retargeted from SMTP to "central"); the S&F engine forwards them to central via `ClusterClient`; the `CentralCommunicationActor` routes each `NotificationSubmit` to the outbox singleton, which inserts a row into the central MS SQL `Notifications` table (insert-if-not-exists on a site-generated `NotificationId` GUID) and acks. A timer-driven dispatcher polls due rows and delivers them through an `INotificationDeliveryAdapter` (Email adapter now; Teams later). A Blazor page surfaces KPIs and a queryable list with Retry/Discard. **Tech Stack:** .NET 10, Akka.NET (cluster singletons, ClusterClient, TestKit), EF Core (MS SQL; SQLite in-memory for tests), Blazor Server + Bootstrap, xUnit + NSubstitute + bUnit. Solution: `ScadaLink.slnx`. **Authoritative design:** `docs/plans/notif.md` and `docs/requirements/Component-NotificationOutbox.md`. Read both before starting. --- ## Conventions (read once, applies to every task) These were confirmed by exploring the existing codebase. Follow them in every task. - **Entities (Commons):** POCOs in `src/ScadaLink.Commons/Entities//`. Auto-properties, parameterized constructor with null checks, navigation collections initialised to `new List()`. No data annotations. - **EF mapping (ConfigurationDatabase):** Fluent `IEntityTypeConfiguration` classes in `src/ScadaLink.ConfigurationDatabase/Configurations/`, auto-applied by `ApplyConfigurationsFromAssembly`. Enums stored as strings via `.HasConversion()`. Add a `DbSet` to `ScadaLinkDbContext`. - **Repositories:** Interface in `src/ScadaLink.Commons/Interfaces/Repositories/`, implementation in `src/ScadaLink.ConfigurationDatabase/Repositories/`. Inject `ScadaLinkDbContext`, use `_context.Set()`, expose explicit `SaveChangesAsync`. Register in `ConfigurationDatabase/ServiceCollectionExtensions.cs` with `AddScoped`. - **Migrations:** `dotnet ef migrations add --project src/ScadaLink.ConfigurationDatabase` — timestamp-named. Applied via `MigrationHelper.ApplyOrValidateMigrationsAsync` (auto in dev). - **Message contracts (Commons):** `record` types in `src/ScadaLink.Commons/Messages//`, named positional params, additive-only evolution. - **Options pattern:** `Options` class owned by the component project; component's `ServiceCollectionExtensions.Add()` calls `services.AddOptions().BindConfiguration("ScadaLink:
")`; Host also `services.Configure(...)`. Config lives in `appsettings.Central.json` / `appsettings.Site.json`. - **Actors:** No Akka.DI framework. Dependencies passed via `Props.Create(() => new XActor(...))`. Actors that need scoped services take `IServiceProvider` and call `CreateScope()`. Cluster singletons use `ClusterSingletonManager.Props` + `ClusterSingletonProxy.Props`, created in `src/ScadaLink.Host/Actors/AkkaHostedService.cs`. - **Tests:** xUnit, NSubstitute, built-in `Assert`. One `tests/ScadaLink..Tests/` project per `src/` project. Actor tests inherit `Akka.TestKit.Xunit2.TestKit`. Repository tests use SQLite in-memory (`DataSource=:memory:`, `OpenConnection()` + `EnsureCreated()`, `IDisposable`). Blazor tests inherit bUnit `BunitContext`. Test naming: `Method_Scenario_Result`. - **Run tests:** whole suite `dotnet test ScadaLink.slnx`; single project `dotnet test tests/ScadaLink..Tests/ScadaLink..Tests.csproj`; single test `--filter "FullyQualifiedName~."`. - **Build:** `dotnet build ScadaLink.slnx`. - **TDD:** every task writes the failing test first, runs it red, implements, runs it green, commits. Use the superpowers-extended-cc:test-driven-development discipline. - **Commits:** one per task, message `feat(notification-outbox): `. **Status lifecycle** (central `Notifications` table — `Forwarding` is site-local, never stored centrally): `Pending → Retrying → Delivered | Parked`, plus `Discarded` (operator action only). --- ## Phase A — Data layer (Commons + ConfigurationDatabase) ### Task 1: Notification enums **Files:** - Create: `src/ScadaLink.Commons/Types/Enums/NotificationType.cs` - Create: `src/ScadaLink.Commons/Types/Enums/NotificationStatus.cs` - Test: `tests/ScadaLink.Commons.Tests/Types/NotificationEnumTests.cs` (create if the test project lacks a `Types/` folder) **Step 1 — failing test.** Assert the enums expose exactly the expected members: ```csharp [Fact] public void NotificationStatus_HasExactlyTheCentralStates() { var names = Enum.GetNames(); Assert.Equal( new[] { "Pending", "Retrying", "Delivered", "Parked", "Discarded" }, names); } [Fact] public void NotificationType_HasEmail() { Assert.True(Enum.IsDefined(NotificationType.Email)); } ``` Note: `Forwarding` is intentionally NOT a `NotificationStatus` member — it is a site-local concept (Task 19), never persisted centrally. **Step 2 — run red:** `dotnet test tests/ScadaLink.Commons.Tests/ScadaLink.Commons.Tests.csproj --filter "FullyQualifiedName~NotificationEnumTests"` → FAIL (types don't exist). **Step 3 — implement.** ```csharp // NotificationType.cs — namespace ScadaLink.Commons.Types.Enums public enum NotificationType { Email } // Teams and others added later // NotificationStatus.cs — namespace ScadaLink.Commons.Types.Enums public enum NotificationStatus { Pending, Retrying, Delivered, Parked, Discarded } ``` **Step 4 — run green.** Same filter → PASS. **Step 5 — commit:** ```bash git add src/ScadaLink.Commons/Types/Enums/NotificationType.cs src/ScadaLink.Commons/Types/Enums/NotificationStatus.cs tests/ScadaLink.Commons.Tests/Types/NotificationEnumTests.cs git commit -m "feat(notification-outbox): add NotificationType and NotificationStatus enums" ``` --- ### Task 2: `Notification` entity POCO **Files:** - Create: `src/ScadaLink.Commons/Entities/Notifications/Notification.cs` - Test: `tests/ScadaLink.Commons.Tests/Entities/NotificationEntityTests.cs` **Step 1 — failing test.** Verify the constructor sets required fields, defaults `Status` to `Pending` and `RetryCount` to 0, and rejects nulls: ```csharp [Fact] public void Constructor_SetsDefaults() { var n = new Notification("id-1", NotificationType.Email, "ops-team", "subj", "body", "SiteA"); Assert.Equal(NotificationStatus.Pending, n.Status); Assert.Equal(0, n.RetryCount); Assert.Equal("id-1", n.NotificationId); } [Fact] public void Constructor_NullListName_Throws() => Assert.Throws( () => new Notification("id", NotificationType.Email, null!, "s", "b", "SiteA")); ``` **Step 2 — run red.** **Step 3 — implement.** Match the `Notifications` table schema in `notif.md`: ```csharp namespace ScadaLink.Commons.Entities.Notifications; public class Notification { public string NotificationId { get; set; } // GUID PK, generated at site public NotificationType Type { get; set; } public string ListName { get; set; } public string Subject { get; set; } public string Body { get; set; } public string? TypeData { get; set; } // JSON extensibility hook public NotificationStatus Status { get; set; } = NotificationStatus.Pending; public int RetryCount { get; set; } public string? LastError { get; set; } public string? ResolvedTargets { get; set; } // snapshotted at delivery, for audit public string SourceSiteId { get; set; } public string? SourceInstanceId { get; set; } public string? SourceScript { get; set; } public DateTimeOffset SiteEnqueuedAt { get; set; } public DateTimeOffset CreatedAt { get; set; } // central ingest time public DateTimeOffset? LastAttemptAt { get; set; } public DateTimeOffset? NextAttemptAt { get; set; } public DateTimeOffset? DeliveredAt { get; set; } public Notification(string notificationId, NotificationType type, string listName, string subject, string body, string sourceSiteId) { NotificationId = notificationId ?? throw new ArgumentNullException(nameof(notificationId)); Type = type; ListName = listName ?? throw new ArgumentNullException(nameof(listName)); Subject = subject ?? throw new ArgumentNullException(nameof(subject)); Body = body ?? throw new ArgumentNullException(nameof(body)); SourceSiteId = sourceSiteId ?? throw new ArgumentNullException(nameof(sourceSiteId)); } } ``` **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add Notification entity`). --- ### Task 3: `Type` field on `NotificationList` **Files:** - Modify: `src/ScadaLink.Commons/Entities/Notifications/NotificationList.cs` - Modify: `src/ScadaLink.ConfigurationDatabase/Configurations/NotificationConfiguration.cs` (`NotificationListConfiguration`) - Test: `tests/ScadaLink.ConfigurationDatabase.Tests/RepositoryTests.cs` (add a test to the notification repository tests) **Step 1 — failing test.** A `NotificationList` round-trips its `Type` through the repository: ```csharp [Fact] public async Task NotificationList_PersistsType() { var list = new NotificationList("ops") { Type = NotificationType.Email }; await _notificationRepo.AddNotificationListAsync(list); await _notificationRepo.SaveChangesAsync(); _context.ChangeTracker.Clear(); var loaded = await _notificationRepo.GetListByNameAsync("ops"); Assert.Equal(NotificationType.Email, loaded!.Type); } ``` **Step 2 — run red.** **Step 3 — implement.** Add to `NotificationList`: `public NotificationType Type { get; set; } = NotificationType.Email;`. In `NotificationListConfiguration.Configure`, add `builder.Property(n => n.Type).HasConversion().HasMaxLength(32).IsRequired();`. **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add Type field to NotificationList`). --- ### Task 4: `Notification` EF configuration + DbSet **Files:** - Create: `src/ScadaLink.ConfigurationDatabase/Configurations/NotificationOutboxConfiguration.cs` - Modify: `src/ScadaLink.ConfigurationDatabase/ScadaLinkDbContext.cs` (add `DbSet`) - Test: `tests/ScadaLink.ConfigurationDatabase.Tests/RepositoryTests.cs` **Step 1 — failing test.** A `Notification` round-trips all fields through the `DbContext` (use the SQLite in-memory fixture pattern). Assert the `Status`/`Type` enums persist as strings and the row is found by `NotificationId`. **Step 2 — run red.** **Step 3 — implement.** Configuration: ```csharp public class NotificationOutboxConfiguration : IEntityTypeConfiguration { public void Configure(EntityTypeBuilder builder) { builder.ToTable("Notifications"); builder.HasKey(n => n.NotificationId); builder.Property(n => n.NotificationId).HasMaxLength(64); builder.Property(n => n.Type).HasConversion().HasMaxLength(32).IsRequired(); builder.Property(n => n.Status).HasConversion().HasMaxLength(32).IsRequired(); builder.Property(n => n.ListName).HasMaxLength(200).IsRequired(); builder.Property(n => n.Subject).HasMaxLength(1000).IsRequired(); builder.Property(n => n.Body).IsRequired(); // nvarchar(max) builder.Property(n => n.TypeData); // nvarchar(max), nullable builder.Property(n => n.ResolvedTargets); // nvarchar(max), nullable builder.Property(n => n.LastError).HasMaxLength(4000); builder.Property(n => n.SourceSiteId).HasMaxLength(100).IsRequired(); builder.Property(n => n.SourceInstanceId).HasMaxLength(200); builder.Property(n => n.SourceScript).HasMaxLength(200); builder.HasIndex(n => new { n.Status, n.NextAttemptAt }); // dispatcher polling builder.HasIndex(n => new { n.SourceSiteId, n.CreatedAt }); // KPIs / UI query } } ``` Add `public DbSet Notifications => Set();` to `ScadaLinkDbContext`. **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add Notification EF configuration and DbSet`). --- ### Task 5: `INotificationOutboxRepository` + implementation **Files:** - Create: `src/ScadaLink.Commons/Interfaces/Repositories/INotificationOutboxRepository.cs` - Create: `src/ScadaLink.ConfigurationDatabase/Repositories/NotificationOutboxRepository.cs` - Modify: `src/ScadaLink.ConfigurationDatabase/ServiceCollectionExtensions.cs` (register `AddScoped`) - Test: `tests/ScadaLink.ConfigurationDatabase.Tests/RepositoryTests.cs` **Step 1 — failing tests.** Cover the operations the outbox actor needs: - `InsertIfNotExistsAsync` inserts a new row and returns `true`; a second call with the same `NotificationId` returns `false` and does not duplicate (idempotency key). - `GetDueAsync(now, batchSize)` returns `Pending` rows and `Retrying` rows with `NextAttemptAt <= now`, ordered by `CreatedAt`, capped at `batchSize`. - `UpdateAsync` persists status transitions. - `GetByIdAsync` returns a row or null. - `QueryAsync(filter, page, pageSize)` filters by status/type/source site and paginates. - `DeleteTerminalOlderThanAsync(cutoff)` bulk-deletes `Delivered`/`Parked`/`Discarded` rows older than `cutoff` and returns the count; leaves non-terminal rows. - `ComputeKpisAsync` returns queue depth, stuck count, parked count, delivered-last-window, oldest-pending age. **Step 2 — run red.** **Step 3 — implement.** Interface: ```csharp public interface INotificationOutboxRepository { Task InsertIfNotExistsAsync(Notification n, CancellationToken ct = default); Task> GetDueAsync(DateTimeOffset now, int batchSize, CancellationToken ct = default); Task GetByIdAsync(string notificationId, CancellationToken ct = default); Task UpdateAsync(Notification n, CancellationToken ct = default); Task<(IReadOnlyList Rows, int TotalCount)> QueryAsync( NotificationOutboxFilter filter, int pageNumber, int pageSize, CancellationToken ct = default); Task DeleteTerminalOlderThanAsync(DateTimeOffset cutoff, CancellationToken ct = default); Task ComputeKpisAsync(DateTimeOffset stuckCutoff, DateTimeOffset deliveredSince, CancellationToken ct = default); Task SaveChangesAsync(CancellationToken ct = default); } ``` `NotificationOutboxFilter` (a `record` in Commons `Types/`) and `NotificationKpiSnapshot` (a `record`) are created in this task alongside the interface. `InsertIfNotExistsAsync`: check `await _context.Notifications.FindAsync(...)`, if present return false, else `AddAsync` + `SaveChangesAsync`, return true. `DeleteTerminalOlderThanAsync`: use `ExecuteDeleteAsync` with a `Where` on terminal statuses and `CreatedAt < cutoff`. Register in `ServiceCollectionExtensions.AddConfigurationDatabase`. **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add NotificationOutbox repository`). --- ### Task 6: EF migration **Files:** - Create: `src/ScadaLink.ConfigurationDatabase/Migrations/_AddNotificationsTable.cs` (generated) **Step 1 — generate:** ```bash dotnet ef migrations add AddNotificationsTable --project src/ScadaLink.ConfigurationDatabase ``` This also picks up the `NotificationList.Type` column from Task 3. **Step 2 — verify.** Inspect the generated migration: confirm a `Notifications` table with the columns and two indexes from Task 4, and an `AlterColumn`/`AddColumn` for `NotificationLists.Type`. Run the ConfigurationDatabase test project — the SQLite `EnsureCreated()` fixture builds from the model, and `dotnet build ScadaLink.slnx` must succeed. **Step 3 — run:** `dotnet test tests/ScadaLink.ConfigurationDatabase.Tests/ScadaLink.ConfigurationDatabase.Tests.csproj` → PASS. **Step 4 — commit** (`feat(notification-outbox): add Notifications table migration`). --- ## Phase B — Message contracts (Commons) ### Task 7: Site↔central notification message contracts **Files:** - Create: `src/ScadaLink.Commons/Messages/Notification/NotificationMessages.cs` - Test: `tests/ScadaLink.Commons.Tests/Messages/NotificationMessagesTests.cs` **Step 1 — failing test.** A trivial construction/round-trip test (these are records — assert positional construction and value equality; if the project has a serialization test helper, round-trip through it). **Step 2 — run red.** **Step 3 — implement.** Namespace `ScadaLink.Commons.Messages.Notification`: ```csharp // Site → Central: submit a notification for central delivery (fire-and-forget with ack). public record NotificationSubmit( string NotificationId, string ListName, string Subject, string Body, string SourceSiteId, string? SourceInstanceId, string? SourceScript, DateTimeOffset SiteEnqueuedAt); // Central → Site: ack after the row is persisted (idempotent — safe to re-send). public record NotificationSubmitAck(string NotificationId, bool Accepted, string? Error); // Site → Central: query delivery status for a NotificationId. public record NotificationStatusQuery(string CorrelationId, string NotificationId); public record NotificationStatusResponse( string CorrelationId, bool Found, string Status, int RetryCount, string? LastError, DateTimeOffset? DeliveredAt); ``` **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add site/central notification message contracts`). --- ### Task 8: Outbox UI query/action contracts **Files:** - Create: `src/ScadaLink.Commons/Messages/Notification/NotificationOutboxQueries.cs` - Test: `tests/ScadaLink.Commons.Tests/Messages/NotificationOutboxQueriesTests.cs` **Step 1 — failing test.** Construction test as in Task 7. **Step 2 — run red.** **Step 3 — implement.** Records the Central UI / `CommunicationService` use to talk to the outbox actor: ```csharp public record NotificationOutboxQueryRequest( string CorrelationId, string? StatusFilter, string? TypeFilter, string? SourceSiteFilter, string? ListNameFilter, bool StuckOnly, string? SubjectKeyword, DateTimeOffset? From, DateTimeOffset? To, int PageNumber, int PageSize); public record NotificationSummary( string NotificationId, string Type, string ListName, string Subject, string Status, int RetryCount, string? LastError, string SourceSiteId, string? SourceInstanceId, DateTimeOffset CreatedAt, DateTimeOffset? DeliveredAt, bool IsStuck); public record NotificationOutboxQueryResponse( string CorrelationId, bool Success, string? ErrorMessage, IReadOnlyList Notifications, int TotalCount); public record RetryNotificationRequest(string CorrelationId, string NotificationId); public record RetryNotificationResponse(string CorrelationId, bool Success, string? ErrorMessage); public record DiscardNotificationRequest(string CorrelationId, string NotificationId); public record DiscardNotificationResponse(string CorrelationId, bool Success, string? ErrorMessage); public record NotificationKpiRequest(string CorrelationId); public record NotificationKpiResponse( string CorrelationId, int QueueDepth, int StuckCount, int ParkedCount, int DeliveredLastInterval, TimeSpan? OldestPendingAge); ``` **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add outbox query and action contracts`). --- ## Phase C — NotificationOutbox project + delivery ### Task 9: Create the `ScadaLink.NotificationOutbox` project **Files:** - Create: `src/ScadaLink.NotificationOutbox/ScadaLink.NotificationOutbox.csproj` - Create: `tests/ScadaLink.NotificationOutbox.Tests/ScadaLink.NotificationOutbox.Tests.csproj` - Modify: `ScadaLink.slnx` (add both projects) **Step 1 — create the projects.** Copy the `.csproj` shape from `src/ScadaLink.NotificationService/ScadaLink.NotificationService.csproj` (same `TargetFramework`, central-managed package versions via `Directory.Packages.props`). The src project references `ScadaLink.Commons` and Akka packages (`Akka`, `Akka.Cluster.Tools`). The test project mirrors `tests/ScadaLink.NotificationService.Tests/` (xUnit, NSubstitute, `Akka.TestKit.Xunit2`) and references the new src project. Add both `` entries to `ScadaLink.slnx`. **Step 2 — add a placeholder test** so the test project is non-empty: ```csharp public class ProjectSmokeTest { [Fact] public void ProjectCompiles() => Assert.True(true); } ``` **Step 3 — verify:** `dotnet build ScadaLink.slnx` succeeds; `dotnet test tests/ScadaLink.NotificationOutbox.Tests/ScadaLink.NotificationOutbox.Tests.csproj` → PASS. **Step 4 — commit** (`feat(notification-outbox): scaffold ScadaLink.NotificationOutbox project`). --- ### Task 10: `NotificationOutboxOptions` **Files:** - Create: `src/ScadaLink.NotificationOutbox/NotificationOutboxOptions.cs` - Test: `tests/ScadaLink.NotificationOutbox.Tests/NotificationOutboxOptionsTests.cs` **Step 1 — failing test.** Assert the defaults. **Step 2 — run red.** **Step 3 — implement.** ```csharp public class NotificationOutboxOptions { public TimeSpan DispatchInterval { get; set; } = TimeSpan.FromSeconds(10); public int DispatchBatchSize { get; set; } = 100; public TimeSpan StuckAgeThreshold { get; set; } = TimeSpan.FromMinutes(10); public TimeSpan TerminalRetention { get; set; } = TimeSpan.FromDays(365); public TimeSpan PurgeInterval { get; set; } = TimeSpan.FromDays(1); public TimeSpan DeliveredKpiWindow { get; set; } = TimeSpan.FromMinutes(1); } ``` **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add NotificationOutboxOptions`). --- ### Task 11: `INotificationDeliveryAdapter` + `DeliveryOutcome` **Files:** - Create: `src/ScadaLink.NotificationOutbox/Delivery/INotificationDeliveryAdapter.cs` - Create: `src/ScadaLink.NotificationOutbox/Delivery/DeliveryOutcome.cs` - Test: `tests/ScadaLink.NotificationOutbox.Tests/Delivery/DeliveryOutcomeTests.cs` **Step 1 — failing test.** Assert `DeliveryOutcome` factory methods produce the right classification. **Step 2 — run red.** **Step 3 — implement.** Mirror the External System Gateway error-classification pattern: ```csharp public enum DeliveryResult { Success, TransientFailure, PermanentFailure } public record DeliveryOutcome(DeliveryResult Result, string? ResolvedTargets, string? Error) { public static DeliveryOutcome Success(string resolvedTargets) => new(DeliveryResult.Success, resolvedTargets, null); public static DeliveryOutcome Transient(string error) => new(DeliveryResult.TransientFailure, null, error); public static DeliveryOutcome Permanent(string error) => new(DeliveryResult.PermanentFailure, null, error); } public interface INotificationDeliveryAdapter { NotificationType Type { get; } Task DeliverAsync(Notification notification, CancellationToken ct = default); } ``` **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add delivery adapter abstraction`). --- ### Task 12: `EmailNotificationDeliveryAdapter` **Files:** - Create: `src/ScadaLink.NotificationOutbox/Delivery/EmailNotificationDeliveryAdapter.cs` - Modify: `src/ScadaLink.NotificationOutbox/ScadaLink.NotificationOutbox.csproj` (reference `ScadaLink.NotificationService` for `ISmtpClientWrapper`) - Test: `tests/ScadaLink.NotificationOutbox.Tests/Delivery/EmailNotificationDeliveryAdapterTests.cs` **Step 1 — failing tests.** Using NSubstitute mocks of `INotificationOutboxRepository`-resolved data and a substituted `ISmtpClientWrapper`: - list resolved + send succeeds → `DeliveryResult.Success`, `ResolvedTargets` lists the recipient addresses. - list not found / no recipients → `PermanentFailure`. - SMTP throws `SmtpPermanentException` → `PermanentFailure`. - SMTP throws a transient error (socket/timeout) → `TransientFailure`. **Step 2 — run red.** **Step 3 — implement.** The adapter resolves the list + recipients + SMTP config from `INotificationRepository` (the existing notification-list repo — recipients are resolved centrally at delivery time), composes and sends via the existing `ISmtpClientWrapper` (`Func` injected, same as `NotificationService`), classifies errors identically to `NotificationDeliveryService`. Reuse the SMTP composition logic from `src/ScadaLink.NotificationService/NotificationDeliveryService.cs` (BCC delivery, plain text, address validation, the `SmtpPermanentException` → permanent mapping). On success return `DeliveryOutcome.Success()`. `Type => NotificationType.Email`. **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add Email delivery adapter`). --- ### Task 13: `NotificationOutboxActor` — ingest **Files:** - Create: `src/ScadaLink.NotificationOutbox/NotificationOutboxActor.cs` - Create: `src/ScadaLink.NotificationOutbox/Messages/InternalMessages.cs` (actor-internal tick messages) - Test: `tests/ScadaLink.NotificationOutbox.Tests/NotificationOutboxActorIngestTests.cs` **Step 1 — failing tests** (TestKit). The actor takes `IServiceProvider`, `NotificationOutboxOptions`, `ILogger`. Use a mocked `INotificationOutboxRepository` registered in the test `ServiceProvider`: - Send `NotificationSubmit` → actor calls `InsertIfNotExistsAsync` with a `Notification` whose fields map from the message, `Status = Pending`, `CreatedAt` set; replies `NotificationSubmitAck(NotificationId, Accepted: true, null)` to `Sender`. - Send the same `NotificationSubmit` twice → second `InsertIfNotExistsAsync` returns false; actor still replies `Accepted: true` (idempotent — the row already exists, ack so the site clears its buffer). - Repository throws → actor replies `Accepted: false` with the error (site will retry the forward). **Step 2 — run red.** **Step 3 — implement.** `ReceiveActor`. On `NotificationSubmit`: build a `Notification`, `CreateScope()` to resolve `INotificationOutboxRepository`, call `InsertIfNotExistsAsync`, `PipeTo` the result back so the reply preserves `Sender`. Reply `NotificationSubmitAck`. Keep dispatch (Task 14) out of this task — ingest only. **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add NotificationOutboxActor ingest`). --- ### Task 14: `NotificationOutboxActor` — dispatcher loop **Files:** - Modify: `src/ScadaLink.NotificationOutbox/NotificationOutboxActor.cs` - Test: `tests/ScadaLink.NotificationOutbox.Tests/NotificationOutboxActorDispatchTests.cs` **Step 1 — failing tests** (TestKit, with a registered set of `INotificationDeliveryAdapter` and a mocked repo): - On a `DispatchTick`, the actor calls `GetDueAsync`, and for each row invokes the adapter for its `Type`. - adapter `Success` → row updated to `Delivered`, `DeliveredAt`/`ResolvedTargets`/`LastAttemptAt` set, `UpdateAsync` called. - adapter `TransientFailure` → `Retrying`, `RetryCount` incremented, `NextAttemptAt = now + retry interval`, `LastError` set. - adapter `TransientFailure` when `RetryCount` already at the SMTP-config max → `Parked`. - adapter `PermanentFailure` → `Parked` immediately, `LastError` set. - no adapter for the row's `Type` → `Parked` with an explanatory error. **Step 2 — run red.** **Step 3 — implement.** `IWithTimers`; in `PreStart` start a periodic `DispatchTick` every `options.DispatchInterval`. On `DispatchTick`: scope-resolve the repo, `GetDueAsync(now, options.DispatchBatchSize)`, and for each notification resolve the adapter from a `Dictionary` (injected), `await DeliverAsync`, apply the status transition, `UpdateAsync`. Retry count/interval come from the central SMTP config (`SmtpConfiguration.MaxRetries` / `RetryDelay` via `INotificationRepository`). Run delivery on a blocking-safe path (the actor `PipeTo`s the async work; do not block the actor thread). Guard against overlapping ticks (ignore a new tick while one is in flight). **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add dispatcher loop to NotificationOutboxActor`). --- ### Task 15: `NotificationOutboxActor` — query, retry, discard, KPIs **Files:** - Modify: `src/ScadaLink.NotificationOutbox/NotificationOutboxActor.cs` - Test: `tests/ScadaLink.NotificationOutbox.Tests/NotificationOutboxActorQueryTests.cs` **Step 1 — failing tests** (TestKit): - `NotificationOutboxQueryRequest` → actor calls `QueryAsync`, replies `NotificationOutboxQueryResponse` with mapped `NotificationSummary` rows; `IsStuck` true when `Status` is `Pending`/`Retrying` and `CreatedAt` older than `options.StuckAgeThreshold`. - `NotificationStatusQuery` → replies `NotificationStatusResponse` (`Found:false` when the id is unknown). - `RetryNotificationRequest` on a `Parked` row → row reset to `Pending`, `RetryCount` 0, `NextAttemptAt` cleared; replies success. On a non-`Parked` row → `Success:false`. - `DiscardNotificationRequest` on a `Parked` row → `Status = Discarded`; replies success. - `NotificationKpiRequest` → replies `NotificationKpiResponse` from `ComputeKpisAsync` (stuck cutoff = now − `StuckAgeThreshold`; delivered window = now − `DeliveredKpiWindow`). **Step 2 — run red.** **Step 3 — implement** the additional `Receive<>` handlers, each scope-resolving the repo and `PipeTo`-ing the reply. **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add query, retry, discard, and KPI handlers`). --- ### Task 16: Daily purge job **Files:** - Modify: `src/ScadaLink.NotificationOutbox/NotificationOutboxActor.cs` - Test: `tests/ScadaLink.NotificationOutbox.Tests/NotificationOutboxActorPurgeTests.cs` **Step 1 — failing test.** On a `PurgeTick`, the actor calls `DeleteTerminalOlderThanAsync(now − options.TerminalRetention)`. **Step 2 — run red.** **Step 3 — implement.** In `PreStart` start a second periodic timer `PurgeTick` every `options.PurgeInterval`. Handler scope-resolves the repo and calls `DeleteTerminalOlderThanAsync`; log the deleted count. **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add daily terminal-row purge`). --- ### Task 17: `AddNotificationOutbox` DI extension **Files:** - Create: `src/ScadaLink.NotificationOutbox/ServiceCollectionExtensions.cs` - Test: `tests/ScadaLink.NotificationOutbox.Tests/ServiceRegistrationTests.cs` **Step 1 — failing test.** Build a `ServiceCollection`, call `AddNotificationOutbox`, and assert `NotificationOutboxOptions`, the `EmailNotificationDeliveryAdapter`, and the adapter dictionary resolve. **Step 2 — run red.** **Step 3 — implement.** `public const string OptionsSection = "ScadaLink:NotificationOutbox";` plus `AddNotificationOutbox(this IServiceCollection)` registering `AddOptions().BindConfiguration(OptionsSection)`, the SMTP client `Func` (reuse `NotificationService`'s registration or register here), `EmailNotificationDeliveryAdapter`, and a registration that exposes `IReadOnlyDictionary` built from all registered adapters. **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add AddNotificationOutbox DI registration`). --- ## Phase D — Site retarget + central wiring ### Task 18: Retarget the site S&F notification handler to forward to central **Files:** - Modify: `src/ScadaLink.StoreAndForward/StoreAndForwardService.cs` and/or the site registration that wires the `Notification` category delivery handler - Modify: `src/ScadaLink.Host/SiteServiceRegistration.cs` (where the notification handler is registered) - Test: `tests/ScadaLink.StoreAndForward.Tests/` (a test that the registered notification handler forwards to the communication actor and treats an ack as success) **Step 1 — investigate + failing test.** Currently the `Notification` category handler calls `NotificationDeliveryService.DeliverBufferedAsync`. The new handler must instead send a `NotificationSubmit` to central via the site's communication actor (`ClusterClient.Send("/user/central-communication", submit)`) and treat a `NotificationSubmitAck(Accepted:true)` as delivered (`true`), a non-ack/timeout as transient (throw), so S&F retries the forward. Write a test with a `TestProbe` standing in for the central client: handler invoked → probe receives `NotificationSubmit`; reply `NotificationSubmitAck(Accepted:true)` → handler result `true`; timeout → handler throws (transient). **Step 2 — run red.** **Step 3 — implement.** Add a `NotificationForwarder` (small class or the handler lambda) that holds the site communication actor ref and does `Ask` with the host-configured forward-retry timeout. Register it as the `StoreAndForwardCategory.Notification` delivery handler in `SiteServiceRegistration`, replacing the `NotificationDeliveryService` handler. The S&F engine already buffers/retries on a thrown (transient) result — no S&F core change needed. **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): forward site S&F notifications to central`). --- ### Task 19: `Notify.Send` async + `Notify.Status` (SiteRuntime) **Files:** - Modify: `src/ScadaLink.SiteRuntime/Scripts/ScriptRuntimeContext.cs` (`NotifyHelper`, `NotifyTarget`) - Test: `tests/ScadaLink.SiteRuntime.Tests/` (Notify API tests) **Step 1 — failing tests.** - `Notify.To("list").Send("subj","body")` generates a GUID `NotificationId`, enqueues a `StoreAndForwardCategory.Notification` message into `StoreAndForwardService` (target `"central"`, payload = serialized `NotificationSubmit`), and returns the `NotificationId` string immediately. - `Notify.Status(id)` issues a `NotificationStatusQuery` to central and returns the mapped status record; while the notification is still in the site S&F buffer (central has no row / query says `Found:false` but the S&F buffer still holds the id) it reports `Forwarding`. **Step 2 — run red.** **Step 3 — implement.** Change `NotifyTarget.Send` to return `Task` (the `NotificationId`): create the GUID, build a `NotificationSubmit` (with `SourceSiteId`, `SourceInstanceId = _instanceName`, `SiteEnqueuedAt = UtcNow`), `EnqueueAsync(Notification, "central", payloadJson)`. Add `NotifyHelper.Status(string notificationId)` returning a status record: query central via the site communication actor; if central returns `Found:false` and the id is still buffered in S&F, return status `Forwarding`. Keep the script-facing surface minimal (`Send`, `Status`). **Step 2 note:** the `Notify` API is consumed by compiled scripts — confirm the script trust model / compilation still accepts the changed signature; update any script-API surface tests. **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): async Notify.Send with status handle`). --- ### Task 20: Central ingest routing **Files:** - Modify: `src/ScadaLink.Communication/Actors/CentralCommunicationActor.cs` - Test: `tests/ScadaLink.Communication.Tests/CentralCommunicationActorTests.cs` **Step 1 — failing test.** When `CentralCommunicationActor` receives a `NotificationSubmit` (sent site→central via ClusterClient to `/user/central-communication`), it forwards it to the notification-outbox singleton proxy and the ack flows back to the original `Sender`. Use a `TestProbe` for the outbox proxy. **Step 2 — run red.** **Step 3 — implement.** `CentralCommunicationActor` takes an optional outbox-proxy `IActorRef` (passed at construction by the Host, Task 21). `Receive(m => _outboxProxy.Forward(m))` — `Forward` preserves the original sender so the `NotificationSubmitAck` returns to the site's ClusterClient. **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): route NotificationSubmit to the outbox actor`). --- ### Task 21: Host registration + appsettings **Files:** - Modify: `src/ScadaLink.Host/Actors/AkkaHostedService.cs` (`RegisterCentralActors`) - Modify: `src/ScadaLink.Host/Program.cs` (call `AddNotificationOutbox`; `Configure`) - Modify: `src/ScadaLink.Host/appsettings.Central.json` (`ScadaLink:NotificationOutbox` section) - Modify: `src/ScadaLink.Host/appsettings.Site.json` (site→central notification forward-retry interval, if not already covered by S&F config) - Test: `tests/ScadaLink.Host.Tests/` if present, else verify via build + the integration test in Task 25 **Step 1 — implement.** In `RegisterCentralActors`: create the `NotificationOutboxActor` as a **cluster singleton** (`ClusterSingletonManager.Props` + `ClusterSingletonProxy.Props`, singleton name `"notification-outbox"`, no explicit role — central nodes only run this role), passing `IServiceProvider`, `NotificationOutboxOptions`, the adapter dictionary, and a logger. Pass the singleton **proxy** ref into `CentralCommunicationActor`'s `Props.Create`. In `Program.cs` central path, call `builder.Services.AddNotificationOutbox()` and `services.Configure(...GetSection(ServiceCollectionExtensions.OptionsSection))`. Add the `ScadaLink:NotificationOutbox` block to `appsettings.Central.json` with the Task 10 defaults. **Step 2 — verify:** `dotnet build ScadaLink.slnx` succeeds. **Step 3 — commit** (`feat(notification-outbox): register NotificationOutbox singleton in Host`). --- ## Phase E — Central UI ### Task 22: `CommunicationService` outbox methods **Files:** - Modify: `src/ScadaLink.Communication/CommunicationService.cs` - Test: `tests/ScadaLink.Communication.Tests/CommunicationServiceTests.cs` (or the existing service test file) **Step 1 — failing tests.** New methods `QueryNotificationOutboxAsync`, `RetryNotificationAsync`, `DiscardNotificationAsync`, `GetNotificationKpisAsync` each `Ask` the central outbox proxy and return the typed response. (These are central-side and do not go through `SiteEnvelope` — they talk to the local outbox proxy directly.) Test with a `TestProbe` for the proxy. **Step 2 — run red.** **Step 3 — implement.** Add an outbox-proxy `IActorRef` to `CommunicationService` (set by the Host like `SetCommunicationActor`). Each method `Ask(request, _options.QueryTimeout)`. **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add CommunicationService outbox methods`). --- ### Task 23: Notification Outbox Blazor page + nav entry **Files:** - Create: `src/ScadaLink.CentralUI/Components/Pages/Monitoring/NotificationOutbox.razor` - Modify: `src/ScadaLink.CentralUI/Components/Layout/NavMenu.razor` - Test: `tests/ScadaLink.CentralUI.Tests/Pages/NotificationOutboxPageTests.cs` (bUnit) **Step 1 — failing test** (bUnit). Render the page with a substituted `CommunicationService` returning a fixed KPI response and a page of `NotificationSummary` rows; assert the KPI tiles show the values and the table renders the rows; assert clicking Retry on a `Parked` row calls `RetryNotificationAsync`. **Step 2 — run red.** **Step 3 — implement.** Model the page on `Components/Pages/Monitoring/ParkedMessages.razor`: `@page "/monitoring/notification-outbox"`, `@attribute [Authorize(Policy = AuthorizationPolicies.RequireDeployment)]`. KPI tile row (Bootstrap `card` tiles like `Health.razor`) bound to `GetNotificationKpisAsync`; a filter card (status, type, source site, list, time range, stuck-only toggle, subject keyword); a table of `NotificationSummary` with stuck rows badged; Retry/Discard buttons on `Parked` rows using `IDialogService.ConfirmAsync` + `ToastNotification`. Add a `NavLink` to `NavMenu.razor` under the Deployment-role Monitoring section (`href="/monitoring/notification-outbox"`). **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add Notification Outbox UI page`). --- ### Task 24: Health dashboard outbox KPI tiles **Files:** - Modify: `src/ScadaLink.CentralUI/Components/Pages/Monitoring/Health.razor` - Test: `tests/ScadaLink.CentralUI.Tests/Pages/HealthPageTests.cs` (extend if present) **Step 1 — failing test** (bUnit). With a substituted `CommunicationService.GetNotificationKpisAsync`, the Health page renders three headline outbox tiles: queue depth, stuck count, parked count. **Step 2 — run red.** **Step 3 — implement.** Add a "Notification Outbox" tile row to `Health.razor`, fetched on init / on the existing 10s polling timer, styled like the existing overview cards. **Step 4 — run green. Step 5 — commit** (`feat(notification-outbox): add outbox KPI tiles to Health dashboard`). --- ## Phase F — Integration & verification ### Task 25: End-to-end integration test **Files:** - Create: `tests/ScadaLink.IntegrationTests/NotificationOutboxFlowTests.cs` **Step 1 — failing test.** Following the patterns in `tests/ScadaLink.IntegrationTests/`, exercise the flow with an in-memory/SQLite-backed `ScadaLinkDbContext` and a real `NotificationOutboxActor`: submit a `NotificationSubmit` → assert a `Notifications` row exists (`Pending`) → trigger a `DispatchTick` with a stub adapter that returns `Success` → assert the row is `Delivered`. Add a second case: stub adapter returns `PermanentFailure` → row `Parked`; then a `RetryNotificationRequest` → row back to `Pending`. **Step 2 — run red. Step 3 — make it pass** (it should, if Phases A–D are correct; fix any wiring gaps found). **Step 4 — commit** (`test(notification-outbox): end-to-end outbox flow integration test`). --- ### Task 26: Full build + suite verification **Files:** none (verification only). **Step 1:** `dotnet build ScadaLink.slnx` → must succeed with no errors. **Step 2:** `dotnet test ScadaLink.slnx` → the whole suite must pass. Investigate and fix any regressions (notably in `ScadaLink.NotificationService.Tests`, `ScadaLink.StoreAndForward.Tests`, `ScadaLink.SiteRuntime.Tests`, `ScadaLink.Communication.Tests` — the docs/design changed the notification path and existing tests may assert old behavior; update them to the new design). **Step 3:** If the docker cluster is used for smoke testing, note that `bash docker/deploy.sh` rebuilds the image — out of scope for this plan unless the user asks. **Step 4 — commit** any test fixes (`test(notification-outbox): update existing tests for the central-delivery model`). --- ## Follow-ups (post-merge, not blocking) - **Remove the now-dead site-side `AddNotificationService()` (from Task 19 review).** After Task 19, the site script runtime no longer resolves `INotificationDeliveryService` (it enqueues into the Store-and-Forward engine instead). `src/ScadaLink.Host/SiteServiceRegistration.cs` still calls `AddNotificationService()`. Task 21 (Host registration) should drop it from the site path — `NotificationService` is now central-only. - **Re-align the Central UI script sandbox `Notify` API (from Task 19 review).** `SandboxNotifyTarget.Send` in `src/ScadaLink.CentralUI/ScriptAnalysis/` still returns `Task` and has no `Status` method, while the production `NotifyTarget.Send` now returns `Task` plus `Notify.Status`. A script that test-runs cleanly in the sandbox would not compile against the real runtime. The sandbox `Notify` surface should be rewritten to match production so the test-run feature stays faithful. - **Populate `SourceScript` on outbound notifications (from Task 19 review).** `NotifyTarget.Send` currently passes `SourceScript: null` — the executing script name is not threaded down to the `NotifyHelper`. The payload field and the forwarder already carry it end to end; only the enqueue side needs the wiring. - **Share the SMTP helpers (from Task 12 review).** `EmailNotificationDeliveryAdapter` reimplements `ClassifySmtpError`/`SmtpErrorClass`, `ValidateAddresses`, and a `ScrubCredentials` helper because the originals are `internal` to `ScadaLink.NotificationService`. To avoid divergence (especially in the security-relevant credential redaction and the SMTP 4xx/5xx classification policy), promote `CredentialRedactor` to `public`, extract a `public static SmtpErrorClassifier`, and make `ValidateAddresses` shared — then have the adapter call them and delete the duplicates. The project reference already exists, so this is low-cost. ## Done The Notification Outbox feature is implemented end to end: site scripts enqueue notifications that store-and-forward to central, the `NotificationOutboxActor` singleton ingests them into the `Notifications` table and delivers them via the Email adapter with retry/parking, operators see KPIs and manage notifications from the Central UI, and the full test suite passes. Teams and other delivery adapters can be added later by implementing `INotificationDeliveryAdapter` and registering it — no other change required.