diff --git a/docs/requirements/Component-Commons.md b/docs/requirements/Component-Commons.md index 4f3967c..0662da9 100644 --- a/docs/requirements/Component-Commons.md +++ b/docs/requirements/Component-Commons.md @@ -65,12 +65,14 @@ Entity classes are organized by domain area: - **Shared Scripts**: `SharedScript`. - **Sites & Data Connections**: `Site`, `DataConnection`. - **External Systems & Database Connections**: `ExternalSystemDefinition`, `ExternalSystemMethod`, `DatabaseConnectionDefinition`. -- **Notifications**: `NotificationList`, `NotificationRecipient`, `SmtpConfiguration`. +- **Notifications**: `NotificationList` (carries a `Type` field — `NotificationType` enum — selecting the list's notification type and its type-specific targets), `NotificationRecipient`, `SmtpConfiguration`, `Notification` (the durable central-queue row — see below). - **Inbound API**: `ApiKey`, `ApiMethod`. - **Security**: `LdapGroupMapping`, `SiteScopeRule`. - **Deployment**: `DeploymentRecord`, `SystemArtifactDeploymentRecord`, `DeployedConfigSnapshot`. - **Audit**: `AuditLogEntry`. +The **`Notification`** entity is the persistence-ignorant POCO for a row of the central `Notifications` table — the durable notification queue owned by the Notification Outbox. It is a plain class with properties for `NotificationId` (GUID, the idempotency key), `Type` (`NotificationType` enum discriminator), `ListName`, `Subject`, `Body`, `TypeData` (a JSON string — the type-agnostic extensibility hook), `Status` (`NotificationStatus` enum), `RetryCount`, `LastError`, `ResolvedTargets`, the provenance fields `SourceSiteId` / `SourceInstanceId` / `SourceScript`, and the UTC timestamps `SiteEnqueuedAt`, `CreatedAt`, `LastAttemptAt`, `NextAttemptAt`, `DeliveredAt`. As with every entity class it has no EF dependency; the Configuration Database component supplies the Fluent API mapping, value conversions, and indexes. The `Type` and `Status` enums (`NotificationType`: `Email`, `Teams`, …; `NotificationStatus`: `Pending`, `Retrying`, `Delivered`, `Parked`, `Discarded`) are defined under `Types/Enums/` per REQ-COM-1. + ### REQ-COM-4: Per-Component Repository Interfaces Commons must define repository interfaces that consuming components use for data access. Each interface is tailored to the data needs of its consuming component: @@ -80,7 +82,8 @@ Commons must define repository interfaces that consuming components use for data - `ISecurityRepository` — LDAP group mappings, site scoping rules. - `IInboundApiRepository` — API keys, API method definitions. - `IExternalSystemRepository` — External system definitions, method definitions, database connection definitions. -- `INotificationRepository` — Notification lists, recipients, SMTP configuration. +- `INotificationRepository` — Notification lists (including the `Type` field), recipients, SMTP configuration. +- `INotificationOutboxRepository` — The `Notifications` table: insert-if-not-exists ingest on `NotificationId`, due-row polling (`Pending` rows and `Retrying` rows past `NextAttemptAt`), status transitions, KPI aggregate queries, and the daily purge of terminal rows. - `ISiteRepository` — Sites, data connections, and their site assignments. - `ICentralUiRepository` — Read-oriented queries spanning multiple domain areas for display purposes. @@ -115,6 +118,7 @@ Commons must define the shared DTOs and message contracts used for inter-compone - **Debug View DTOs**: Subscribe/unsubscribe requests, one-shot snapshot request (`DebugSnapshotRequest`), initial snapshot, stream filter criteria. - **Script Execution DTOs**: Script call requests (with recursion depth), return values, error results. - **System-Wide Artifact DTOs**: Shared script packages, external system definitions, database connection definitions, notification list definitions. +- **Notification DTOs**: `NotificationSubmit` — the site→central notification submission carrying `NotificationId` (GUID), `ListName`, `Subject`, `Body`, provenance (`SourceSiteId`, `SourceInstanceId`, `SourceScript`), and `SiteEnqueuedAt` — and `NotificationSubmitAck`, the central acknowledgement returned after the `Notifications` row is persisted (ack-after-persist), which the site Store-and-Forward Engine waits on before clearing the buffered message. Recipient resolution is *not* part of the contract — the site forwards only `(listName, subject, body)` and central resolves the list at delivery time. `Notify.Status` also round-trips a small status-query request and a status-record response (status, retry count, last error, key timestamps) once a notification has been forwarded. These contracts are `record` types subject to the additive-only evolution rules in REQ-COM-5a — relevant because a notification submission can cross the site→central version skew boundary; future per-type fields are carried by adding optional fields (or via the `Notifications` table's `TypeData` JSON hook) rather than by changing existing ones. All message types must be `record` types or immutable classes suitable for use as Akka.NET messages (though Commons itself must not depend on Akka.NET). @@ -144,7 +148,8 @@ ScadaLink.Commons/ │ ├── Enums/ # InstanceState, DeploymentStatus, AlarmState, │ │ # AlarmLevel, AlarmTriggerType, ConnectionHealth, │ │ # DataType, StoreAndForwardCategory, -│ │ # StoreAndForwardMessageStatus +│ │ # StoreAndForwardMessageStatus, +│ │ # NotificationType, NotificationStatus │ ├── DataConnections/ # OPC UA endpoint config value objects + enums │ ├── Flattening/ # FlattenedConfiguration, ConfigurationDiff, │ │ # DeploymentPackage, ValidationResult @@ -158,6 +163,7 @@ ScadaLink.Commons/ │ │ ├── IInboundApiRepository.cs │ │ ├── IExternalSystemRepository.cs │ │ ├── INotificationRepository.cs +│ │ ├── INotificationOutboxRepository.cs │ │ ├── ISiteRepository.cs │ │ └── ICentralUiRepository.cs │ └── Services/ # REQ-COM-4a: Cross-cutting service interfaces @@ -174,7 +180,8 @@ ScadaLink.Commons/ │ ├── Sites/ # Site, DataConnection │ ├── ExternalSystems/ # ExternalSystemDefinition, ExternalSystemMethod, │ │ # DatabaseConnectionDefinition -│ ├── Notifications/ # NotificationList, NotificationRecipient, SmtpConfiguration +│ ├── Notifications/ # NotificationList, NotificationRecipient, SmtpConfiguration, +│ │ # Notification (central Notifications-table row) │ ├── InboundApi/ # ApiKey, ApiMethod │ ├── Security/ # LdapGroupMapping, SiteScopeRule │ ├── Deployment/ # DeploymentRecord, SystemArtifactDeploymentRecord, @@ -193,6 +200,7 @@ ScadaLink.Commons/ │ ├── DataConnection/ # data-connection subscribe/write/health messages │ ├── Instance/ # attribute get/set request/command messages │ ├── Integration/ # external-integration call request/response +│ ├── Notification/ # NotificationSubmit + ack, Notify.Status query/response │ ├── InboundApi/ # Route.To() request messages │ ├── RemoteQuery/ # event-log and parked-message query messages │ └── Management/ # HTTP/ClusterClient management commands + registry diff --git a/docs/requirements/Component-ConfigurationDatabase.md b/docs/requirements/Component-ConfigurationDatabase.md index 5bcb3a4..c08bf54 100644 --- a/docs/requirements/Component-ConfigurationDatabase.md +++ b/docs/requirements/Component-ConfigurationDatabase.md @@ -52,9 +52,10 @@ The configuration database stores all central system data, organized by domain a - **Database Connection Definitions**: Named database connections (name, connection details, retry settings). ### Notifications -- **Notification Lists**: List definitions (name). +- **Notification Lists**: List definitions (name, `Type` discriminator — `Email` / `Teams` / … — selecting the list's notification type and type-specific targets). - **Notification Recipients**: Recipients per list (name, email address). - **SMTP Configuration**: Email server settings. +- **Notifications**: The durable central notification queue owned by the Notification Outbox — one row per notification, the single source of audit truth. The schema is **type-agnostic** so it records any notification type the system supports (email today, Microsoft Teams and others later): a `Type` discriminator selects the type, and a `TypeData` JSON column (`nvarchar(max)`) carries any future per-type fields without a schema change. Columns: `NotificationId` (GUID, primary key — generated at the site, used as the idempotency key), `Type`, `ListName`, `Subject`, `Body`, `TypeData`, `Status`, `RetryCount`, `LastError`, `ResolvedTargets`, `SourceSiteId`, `SourceInstanceId`, `SourceScript`, `SiteEnqueuedAt`, `CreatedAt`, `LastAttemptAt`, `NextAttemptAt`, `DeliveredAt`. `Status` is a `NotificationStatus` enum stored with values `Pending`, `Retrying`, `Delivered`, `Parked`, `Discarded` (the site-local `Forwarding` state is never persisted centrally). Indexed on `Status` and `NextAttemptAt` for efficient dispatcher polling of due rows, and on `SourceSiteId` and `CreatedAt` for KPI computation and the Central UI query page. Terminal rows (`Delivered`, `Parked`, `Discarded`) are removed by a daily purge job after a configurable retention window (default 365 days); non-terminal rows are never purged. See Component-NotificationOutbox.md for the full lifecycle. ### Inbound API - **API Keys**: Key definitions (name/label, key value, enabled flag). @@ -94,7 +95,8 @@ Repository interfaces are defined in **Commons** alongside the POCO entity class | `ISecurityRepository` | Security & Auth | LDAP group mappings, site scoping rules | | `IInboundApiRepository` | Inbound API | API keys, API method definitions | | `IExternalSystemRepository` | External System Gateway | External system definitions, method definitions, database connection definitions | -| `INotificationRepository` | Notification Service | Notification lists, recipients, SMTP configuration | +| `INotificationRepository` | Notification Service | Notification lists (including the `Type` field), recipients, SMTP configuration | +| `INotificationOutboxRepository` | Notification Outbox | The `Notifications` table — insert-if-not-exists ingest, due-row polling, status transitions, KPI aggregate queries, and daily purge of terminal rows | | `IHealthMonitoringRepository` | Health Monitoring | (Minimal — health data is in-memory; repository needed only if connectivity history is persisted in the future) | | `ICentralUiRepository` | Central UI | Read-oriented queries spanning multiple domain areas for display purposes | @@ -226,6 +228,7 @@ Results are returned in reverse chronological order (most recent first) with pag - Schema changes are managed via EF Core Migrations (`dotnet ef migrations add`, `dotnet ef migrations script`). - Each migration is a versioned, incremental schema change. +- New tables are introduced as their own migration — for example, the `Notifications` table for the Notification Outbox ships as a dedicated EF Core migration that creates the table, its `Type`/`Status` value conversions, and its dispatcher and KPI indexes. ### Development Environment - Migrations are **auto-applied** at application startup using `dbContext.Database.MigrateAsync()`. @@ -265,6 +268,14 @@ The Configuration Database supports seeding initial data required for the system --- +## Scheduled Maintenance + +### Notifications Table Purge + +The `Notifications` table grows one row per notification and is never trimmed by normal operation — `Discarded` rows are deliberately retained for audit. To bound table growth while preserving a strong audit trail, a **daily purge job** deletes terminal rows (`Delivered`, `Parked`, `Discarded`) whose `CreatedAt` is older than a configurable retention window (default 365 days). Non-terminal rows (`Pending`, `Retrying`) are never purged. The purge is a bulk `DELETE` against `INotificationOutboxRepository`; it is owned and scheduled by the Notification Outbox component (see Component-NotificationOutbox.md), which supplies the retention window from `NotificationOutboxOptions`. The Configuration Database component provides only the repository operation and the table. + +--- + ## Connection Management - Connection strings are provided via the Host's `DatabaseConfiguration` options (bound from `appsettings.json`). @@ -289,6 +300,7 @@ The Configuration Database supports seeding initial data required for the system - **Inbound API**: Uses `IInboundApiRepository` for API keys and method definitions. - **External System Gateway**: Uses `IExternalSystemRepository` for external system and database connection definitions. - **Notification Service**: Uses `INotificationRepository` for notification lists and SMTP configuration. +- **Notification Outbox**: Uses `INotificationOutboxRepository` for all access to the `Notifications` table — ingest, dispatch polling, status updates, KPI queries, and the daily purge of terminal rows. - **Central UI**: Uses `ICentralUiRepository` for read-oriented queries across domain areas, including audit log queries for the audit log viewer. - **All central components that modify state**: Call `IAuditService.LogAsync()` after successful operations to record audit entries within the same transaction. - **Host**: Provides database connection configuration. Registers DbContext, repository implementations, and `IAuditService` implementation in the DI container. Triggers auto-migration in development or validates schema version in production.