docs(notification-outbox): fix stale cross-references after outbox addition

docs/requirements/Component-CentralUI.md

@@ -98,7 +98,7 @@ Central cluster only. Sites have no user interface.
 - Track deployment status (pending, in-progress, success, failed).
 
 ### System-Wide Artifact Deployment (Deployment Role)
-- Explicitly deploy shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, and SMTP configuration to all sites or to an individual site.
+- Explicitly deploy shared scripts, external system definitions, database connection definitions, and data connection definitions to all sites or to an individual site. (Notification lists and SMTP configuration are central-only and are not deployed.)
 - **Per-site deployment**: A "Deploy Artifacts" button on the Sites admin page allows deploying all artifacts to an individual site.
 - **Deploy all**: A bulk action deploys artifacts to all sites at once.
 - This is a **separate action** from instance deployment — system-wide artifacts are not automatically pushed when definitions change.
@@ -115,7 +115,7 @@ Central cluster only. Sites have no user interface.
 - Subscribe-on-demand — stream starts when opened, stops when closed.
 
 ### Parked Message Management (Deployment Role)
-- Query sites for parked messages (external system calls, notifications, cached DB writes).
+- Query sites for parked messages (external system calls, cached DB writes). (Parked notifications are managed centrally on the Notification Outbox page, not here.)
 - View message details (target, payload, retry count, timestamps).
 - Retry or discard individual parked messages.
 

docs/requirements/Component-Communication.md

@@ -36,7 +36,7 @@ Both central and site clusters. Each side has communication actors that handle m
 
 ### 3. System-Wide Artifact Deployment (Central → Site(s))
 - **Pattern**: Broadcast with per-site acknowledgment (deploy to all sites), or targeted to a single site (per-site deployment).
-- When shared scripts, external system definitions, database connections, data connections, notification lists, or SMTP configuration are explicitly deployed, central sends them to the target site(s).
+- When shared scripts, external system definitions, database connections, or data connections are explicitly deployed, central sends them to the target site(s). (Notification lists and SMTP configuration are central-only and are not deployed to sites.)
 - Each site acknowledges receipt and reports success/failure independently.
 - **Shared script deployment triggers immediate recompilation on the site** — the site's `SharedScriptLibrary` replaces its in-memory compiled code, making updated shared scripts available to all running instances without redeployment. Other artifact types (external systems, database connections, etc.) are stored but do not require recompilation.
 

docs/requirements/Component-ConfigurationDatabase.md

@@ -200,7 +200,7 @@ Since only the after-state is stored, change history for an entity is reconstruc
 | Alarms | Create, edit, delete alarm definitions |
 | Instances | Create, override values, bind connections, area assignment, disable, enable, delete |
 | Deployments | Deploy to instance (who, what, which instance, success/failure) |
-| System-Wide Artifact Deployments | Deploy shared scripts / external system definitions / DB connections / data connections / notification lists / SMTP config to site(s) (who, what, which site(s), result) |
+| System-Wide Artifact Deployments | Deploy shared scripts / external system definitions / DB connections / data connections to site(s) (who, what, which site(s), result) |
 | External Systems | Create, edit, delete definitions |
 | Database Connections | Create, edit, delete definitions |
 | Notification Lists | Create, edit, delete lists and recipients |

docs/requirements/Component-DeploymentManager.md

@@ -17,7 +17,7 @@ Central cluster only. The site-side deployment responsibilities (receiving confi
 - Track deployment status (pending, in-progress, success, failed).
 - Handle deployment failures gracefully — if a site is unreachable or the deployment fails, report the failure. No retry or buffering at central.
 - If a central failover occurs during deployment, the deployment is treated as failed and must be re-initiated.
-- Deploy system-wide artifacts (shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, SMTP configuration) to all sites or to an individual site on explicit request.
+- Deploy system-wide artifacts (shared scripts, external system definitions, database connection definitions, data connection definitions) to all sites or to an individual site on explicit request.
 - Send instance lifecycle commands (disable, enable, delete) to sites via the Communication Layer.
 
 ## Deployment Flow
@@ -106,30 +106,13 @@ A deployment to a site includes the flattened instance configuration plus any sy
 - External system definitions
 - Database connection definitions
 - Data connection definitions
-- Notification lists
-- SMTP configuration
 
 System-wide artifact deployment is a **separate action** from instance deployment, triggered explicitly by a user with the Deployment role. Artifacts can be deployed to all sites at once or to an individual site (per-site deployment via the Sites admin page).
 
-### Secret handling in artifacts
-
-The SMTP configuration artifact carries the SMTP credential (password or OAuth2
-client secret). This is a **conscious, accepted design decision**: SMTP
-configuration is a deployable artifact, so the credential is distributed to
-sites that need it. The credential is protected by the following controls:
-
-- **In transit** — artifact-deployment commands travel over the inter-cluster
-  transport, which is TLS-protected (see Cluster Infrastructure / Communication).
-- **Not logged** — the Deployment Manager never writes credential values to
-  logs; deployment log statements reference only site IDs/names, the deployment
-  ID, and exception messages.
-- **At rest on the site** — the credential is stored in the site's local SQLite
-  artifact store. At-rest encryption of that field is **not** currently applied;
-  it is treated as acceptable given the TLS-protected transport, the absence of
-  any logging leak, and the trust boundary of the site host. Encrypting the
-  credential field within the artifact payload would require a key-management
-  scheme (key location and distribution to sites) and is recorded here as a
-  possible future hardening item, not a current requirement.
+Notification lists and SMTP configuration are **not** deployable artifacts — they
+are central-only definitions managed by the Notification Service. Notification
+delivery happens on the central cluster (see Component-NotificationOutbox.md), so
+no notification artifact or SMTP credential is ever distributed to sites.
 
 ## Site-Side Apply Atomicity
 

docs/requirements/Component-Host.md

@@ -31,8 +31,8 @@ The same compiled binary must be deployable to both central and site nodes. The
 
 At startup the Host must inspect the configured node role and register only the component services appropriate for that role:
 
-- **Shared** (both Central and Site): ClusterInfrastructure, Communication, HealthMonitoring, ExternalSystemGateway, NotificationService.
-- **Central only**: TemplateEngine, DeploymentManager, Security, AuditLogging, CentralUI, InboundAPI, ManagementService.
+- **Shared** (both Central and Site): ClusterInfrastructure, Communication, HealthMonitoring, ExternalSystemGateway.
+- **Central only**: TemplateEngine, DeploymentManager, Security, AuditLogging, CentralUI, InboundAPI, ManagementService, NotificationService, NotificationOutbox.
 - **Site only**: SiteRuntime, DataConnectionLayer, StoreAndForward, SiteEventLogging.
 
 Components not applicable to the current role must not be registered in the DI container or the Akka.NET actor system.
@@ -60,7 +60,8 @@ The Host must bind configuration sections from `appsettings.json` to strongly-ty
 | `ScadaLink:Communication` | `CommunicationOptions` | Communication | DeploymentTimeout, LifecycleTimeout, QueryTimeout, TransportHeartbeatInterval, TransportFailureThreshold |
 | `ScadaLink:Security` | `SecurityOptions` | Security & Auth | LdapServer, LdapPort, LdapUseTls, JwtSigningKey, JwtExpiryMinutes, IdleTimeoutMinutes |
 | `ScadaLink:InboundApi` | `InboundApiOptions` | Inbound API | DefaultMethodTimeout |
-| `ScadaLink:Notification` | `NotificationOptions` | Notification Service | (SMTP config is stored in config DB and deployed to sites, not in appsettings) |
+| `ScadaLink:Notification` | `NotificationOptions` | Notification Service | (SMTP config is stored in the central config DB, not in appsettings) |
+| `ScadaLink:NotificationOutbox` | `NotificationOutboxOptions` | Notification Outbox | Dispatcher poll interval, stuck-age threshold, retention window (delivery retry settings reuse the central SMTP configuration) |
 | `ScadaLink:ManagementService` | `ManagementServiceOptions` | Management Service | (Reserved for future configuration) |
 | `ScadaLink:Logging` | `LoggingOptions` | Host | Serilog sink configuration, log level overrides |
 
@@ -176,7 +177,8 @@ The Host's `Program.cs` calls these extension methods; the component libraries o
 | Communication | Yes | Yes | Yes | Yes | No |
 | HealthMonitoring | Yes | Yes | Yes | Yes | No |
 | ExternalSystemGateway | Yes | Yes | Yes | Yes | No |
-| NotificationService | Yes | Yes | Yes | Yes | No |
+| NotificationService | Yes | No | Yes | Yes | No |
+| NotificationOutbox | Yes | No | Yes | Yes | No |
 | TemplateEngine | Yes | No | Yes | Yes | No |
 | DeploymentManager | Yes | No | Yes | Yes | No |
 | Security | Yes | No | Yes | Yes | No |

docs/requirements/Component-ManagementService.md

@@ -123,7 +123,7 @@ The endpoint performs LDAP authentication and role resolution server-side, colla
 ### Deployments
 
 - **DeployInstance**: Deploy configuration to a specific instance (includes pre-deployment validation).
-- **DeployArtifacts**: Deploy system-wide artifacts (shared scripts, external system definitions, DB connections, data connections, notification lists, SMTP config) to all sites or a specific site.
+- **DeployArtifacts**: Deploy system-wide artifacts (shared scripts, external system definitions, DB connections, data connections) to all sites or a specific site.
 - **GetDeploymentStatus**: Query deployment status.
 
 ### External Systems

docs/requirements/Component-Security.md

@@ -76,10 +76,11 @@ Central cluster. Sites do not have user-facing interfaces and do not perform ind
   - Create and manage instances (overrides, connection bindings, area assignment).
   - Disable, enable, and delete instances.
   - Deploy configurations to instances.
-  - Deploy system-wide artifacts (shared scripts, external system definitions, DB connections, notification lists) to all sites.
+  - Deploy system-wide artifacts (shared scripts, external system definitions, DB connections, data connections) to all sites.
   - View deployment diffs and status.
   - Use debug view.
   - Manage parked messages.
+  - Monitor and manage the Notification Outbox (retry and discard parked notifications).
   - View site event logs.
 - **Site scoping**: A user with site-scoped Deployment role can only perform these actions for instances at their permitted sites.
 

docs/requirements/Component-SiteRuntime.md

@@ -66,7 +66,7 @@ Deployment Manager Singleton (Cluster Singleton)
 - Reports deployment result (success/failure) back to central.
 
 ### System-Wide Artifact Handling
-- Receives updated shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, and SMTP configuration from central.
+- Receives updated shared scripts, external system definitions, database connection definitions, and data connection definitions from central. (Notification lists and SMTP configuration are central-only and are not deployed to sites — see Component-NotificationService.md.)
 - Stores all artifacts in local SQLite. After artifact deployment, the site is fully self-contained — all runtime configuration is read from local SQLite with no access to the central configuration database.
 - Recompiles shared scripts and makes updated code available to all Script Actors.
 
@@ -257,7 +257,8 @@ Available to all Script Execution Actors and Alarm Execution Actors:
 - `ExternalSystem.CachedCall("systemName", "methodName", params)` — Fire-and-forget with store-and-forward on transient failure. Use for outbound data pushes where deferred delivery is acceptable.
 
 ### Notifications
-- `Notify.To("listName").Send("subject", "message")` — Send an email notification via a named notification list.
+- `Notify.To("listName").Send("subject", "message")` — Send a notification via a named notification list. Generates a `NotificationId` (GUID) locally and returns it immediately; the notification is store-and-forwarded to the central cluster, which owns delivery. The script never contacts SMTP.
+- `Notify.Status("notificationId")` — Returns a status record (status, retry count, last error, key timestamps). While the notification is still in the site store-and-forward buffer the site answers locally (status `Forwarding`); once forwarded the query round-trips to central.
 
 ### Database Access
 - `Database.Connection("connectionName")` — Obtain a raw MS SQL client connection (ADO.NET) for synchronous read/write.
@@ -355,11 +356,11 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
 - **Data Connection Layer**: Provides tag value updates to Instance Actors. Receives write requests from Instance Actors.
 - **Store-and-Forward Engine**: Handles reliable delivery for external system calls, notifications, and cached database writes submitted by scripts.
 - **External System Gateway**: Provides external system method invocations for scripts.
-- **Notification Service**: Handles email delivery for scripts.
+- **Store-and-Forward Engine**: Carries script-generated notifications to the central cluster for delivery (the notification category targets central, not SMTP).
 - **Communication Layer**: Receives deployments and lifecycle commands from central. Handles debug view requests. Reports deployment results.
 - **Site Event Logging**: Records script executions, alarm events, deployment events, instance lifecycle events.
 - **Health Monitoring**: Reports script error rates and alarm evaluation error rates.
-- **Local SQLite**: Persists deployed configurations, system-wide artifacts (external system definitions, database connection definitions, data connection definitions, notification lists, SMTP configuration).
+- **Local SQLite**: Persists deployed configurations, system-wide artifacts (external system definitions, database connection definitions, data connection definitions).
 
 ## Interactions
 

docs/requirements/HighLevelReqs.md

@@ -33,7 +33,7 @@
 - **Pre-deployment validation**: Before any deployment is sent to a site, the central cluster performs comprehensive validation including flattening the configuration, test-compiling all scripts, verifying alarm trigger references, verifying script trigger references, and checking data connection binding completeness (see Section 3.11).
 
 ### 1.5 System-Wide Artifact Deployment
-- Changes to shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, and SMTP configuration are **not automatically propagated** to sites.
+- Changes to shared scripts, external system definitions, database connection definitions, and data connection definitions are **not automatically propagated** to sites. (Notification lists and SMTP configuration are central-only and are not deployed to sites — see Section 6.)
 - Deployment of system-wide artifacts requires **explicit action** by a user with the **Deployment** role.
 - Artifacts can be deployed to **all sites at once** or to an **individual site** (per-site deployment).
 - The Design role manages the definitions; the Deployment role triggers deployment to sites. A user may hold both roles.
@@ -55,7 +55,7 @@
 
 ### 2.3 Site-Level Storage & Interface
 - Sites have **no user interface** — they are headless collectors, forwarders, and script executors.
-- Sites require local storage for: the current deployed (flattened) configurations, deployed scripts, shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, and SMTP configuration.
+- Sites require local storage for: the current deployed (flattened) configurations, deployed scripts, shared scripts, external system definitions, database connection definitions, and data connection definitions. (Notification lists and SMTP configuration are central-only and are not stored at sites — see Section 6.)
 - After artifact deployment, sites are **fully self-contained** — all runtime configuration is read from local SQLite. Sites do **not** access the central configuration database at runtime.
 - Store-and-forward buffers are persisted to a **local SQLite database on each node** and replicated between nodes via application-level replication (see 1.3).
 
@@ -286,7 +286,7 @@ Scripts **cannot** access other instances' attributes or scripts.
 - Parked messages are **stored at the site** where they originated.
 - The **central UI** can **query sites** for parked messages and manage them remotely.
 - Operators can **retry** or **discard** parked messages from the central UI.
-- Parked message management covers **external system calls**, **notifications**, and **cached database writes**.
+- Parked message management covers **external system calls** and **cached database writes**. (Notifications are not parked at sites — they are store-and-forwarded to central, which owns delivery and parking; see Section 6.)
 
 ### 5.5 Database Connections
 - Database connections are **predefined, named resources** created by users with the **Design** role.
@@ -306,23 +306,23 @@ Scripts can interact with databases in two modes:
 
 ### 6.1 Notification Lists
 - Notification lists are **system-wide**, managed by users with the **Design** role.
-- Each list has a **name** and contains one or more **recipients**.
-- Each recipient has a **name** and an **email address**.
-- Notification lists are deployed to **all sites** (deployment requires explicit action by a user with the Deployment role).
-- At the site, notification lists and recipients are read from **local SQLite** (populated by artifact deployment), not from the central config DB.
+- Each list has a **name** and a **`Type`** (`Email` now; `Teams` and other types planned). The type determines the type-specific targets the list carries.
+- An `Email` list contains one or more **recipients**, each with a **name** and an **email address**.
+- Notification lists are defined and stored **centrally only** — they are **not deployed to sites** and have no local SQLite copy. Recipient resolution happens at central, at delivery time.
 
 ### 6.2 Email Support
 - The system has **predefined support for sending email** as the notification delivery mechanism.
-- Email server configuration (SMTP settings) is defined centrally and deployed to all sites as part of **artifact deployment** (see Section 1.5). Sites read SMTP configuration from **local SQLite**.
+- Email server configuration (SMTP settings) is defined and stored **centrally only**. Sites never talk to SMTP; all delivery happens on the central cluster.
 
 ### 6.3 Script API
 - Scripts send notifications using a simplified API: `Notify.To("list name").Send("subject", "message")`
 - This API is available to instance scripts, alarm on-trigger scripts, and shared scripts.
+- `Send()` generates a `NotificationId` (GUID) locally and returns it to the script immediately; `Notify.Status(notificationId)` returns a status record (status, retry count, last error, key timestamps).
 
-### 6.4 Store-and-Forward for Notifications
-- If the email server is unavailable, notifications are **buffered locally at the site**.
-- Follows the same retry pattern as external system calls: configurable **max retry count** and **time between retries** (fixed interval).
-- After max retries are exhausted, the notification is **parked** for manual review (managed via central UI alongside external system parked messages).
+### 6.4 Notification Delivery and Store-and-Forward
+- Notification delivery is **inverted to the central cluster**. A site script's notification is **store-and-forwarded to central** (target = central, not SMTP); the central **Notification Outbox** logs every notification to a `Notifications` audit table and owns dispatch, delivery, retry, and parking.
+- If the central cluster is unreachable, the notification is **buffered locally at the site** by the Store-and-Forward Engine and retried at a fixed forward interval until central acks it. The site→central forward does not park.
+- Once central holds the notification, delivery retry reuses the central SMTP configuration's **max retry count** and fixed **time between retries**. After retries are exhausted, or on a permanent failure, the notification is **parked** for review on the Central UI Notification Outbox page.
 - There is **no maximum buffer size** for notification messages.
 
 ## 7. Inbound API (Central)
@@ -370,10 +370,11 @@ The central cluster hosts a **configuration and management UI** (no live machine
 - **Site & Data Connection Management**: Define sites (including optional NodeAAddress and NodeBAddress fields for Akka remoting paths, and optional GrpcNodeAAddress and GrpcNodeBAddress fields for gRPC streaming endpoints), manage data connections and assign them to sites.
 - **Area Management**: Define hierarchical area structures per site for organizing instances.
 - **Deployment**: View diffs between deployed and current template-derived configurations, deploy updates to individual instances. Filter instances by area. Pre-deployment validation runs automatically before any deployment is sent.
-- **System-Wide Artifact Deployment**: Explicitly deploy shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, and SMTP configuration to all sites or to an individual site (requires Deployment role). Per-site deployment is available via the Sites admin page.
+- **System-Wide Artifact Deployment**: Explicitly deploy shared scripts, external system definitions, database connection definitions, and data connection definitions to all sites or to an individual site (requires Deployment role). Per-site deployment is available via the Sites admin page. (Notification lists and SMTP configuration are central-only and are not deployed.)
 - **Deployment Status Monitoring**: Track whether deployments were successfully applied at site level.
 - **Debug View**: On-demand real-time view of a specific instance's tag values and alarm states for troubleshooting (see 8.1).
-- **Parked Message Management**: Query sites for parked messages (external system calls, notifications, and cached database writes), retry or discard them.
+- **Parked Message Management**: Query sites for parked messages (external system calls and cached database writes), retry or discard them.
+- **Notification Outbox**: Monitor centrally-delivered notifications — KPI tiles and a queryable `Notifications` audit list with Retry/Discard actions on parked notifications (see Section 6).
 - **Health Monitoring Dashboard**: View site cluster health, node status, data connection health, script error rates, alarm evaluation errors, and store-and-forward buffer depths (see Section 11).
 - **Site Event Log Viewer**: Query and view operational event logs from site clusters (see Section 12).
 
@@ -422,7 +423,7 @@ All system-modifying actions are logged, including:
 - **Alarm changes**: Create, edit, delete alarm definitions.
 - **Instance changes**: Create, override values, bind connections, area assignment, disable, enable, delete.
 - **Deployments**: Who deployed what to which instance, and the result (success/failure).
-- **System-wide artifact deployments**: Who deployed shared scripts / external system definitions / DB connections / data connections / notification lists / SMTP config, to which site(s), and the result.
+- **System-wide artifact deployments**: Who deployed shared scripts / external system definitions / DB connections / data connections, to which site(s), and the result.
 - **External system definition changes**: Create, edit, delete.
 - **Database connection changes**: Create, edit, delete.
 - **Notification list changes**: Create, edit, delete lists and recipients.
@@ -448,7 +449,8 @@ The central cluster monitors the health of each site cluster, including:
 - **Data connection health**: Connected/disconnected status per data connection at the site.
 - **Script error rates**: Frequency of script failures at the site.
 - **Alarm evaluation errors**: Frequency of alarm evaluation failures at the site.
-- **Store-and-forward buffer depth**: Number of messages currently queued (broken down by external system calls, notifications, and cached database writes).
+- **Store-and-forward buffer depth**: Number of messages currently queued (broken down by external system calls, notifications, and cached database writes). The notification backlog covers the site→central forward leg.
+- **Notification Outbox KPIs**: Central-computed delivery-health metrics — queue depth, stuck count, and parked count — surfaced as headline tiles on the health dashboard. These are computed centrally from the `Notifications` table, not collected from sites.
 
 ### 11.2 Reporting
 - Site clusters **report health metrics to central** periodically.