dohertj2/scadalink-design
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: include data connections and SMTP in artifact deployment

Browse Source
This commit is contained in:
Joseph Doherty
2026-03-17 13:48:52 -04:00
parent e313eda9fd
commit 2f3e0ceecb
17 changed files with 151 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
Component-CentralUI.md
View File

@@ -96,7 +96,9 @@ No manual refresh or polling is required for any of these features.
- Track deployment status (pending, in-progress, success, failed).
### System-Wide Artifact Deployment (Deployment Role)
- Explicitly deploy shared scripts, external system definitions, database connection definitions, and notification lists to all sites.
- Explicitly deploy shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, and SMTP configuration to all sites or to an individual site.
- **Per-site deployment**: A "Deploy Artifacts" button on the Sites admin page allows deploying all artifacts to an individual site.
- **Deploy all**: A bulk action deploys artifacts to all sites at once.
- This is a **separate action** from instance deployment — system-wide artifacts are not automatically pushed when definitions change.
- Track per-site deployment status.

6
Component-Communication.md
View File

@@ -30,9 +30,9 @@ Both central and site clusters. Each side has communication actors that handle m
- Site Runtime processes the command and responds with success/failure.
- If the site is unreachable, the command fails immediately (no buffering).
### 3. System-Wide Artifact Deployment (Central → All Sites)
- **Pattern**: Broadcast with per-site acknowledgment.
- When shared scripts, external system definitions, database connections, or notification lists are explicitly deployed, central sends them to all sites.
### 3. System-Wide Artifact Deployment (Central → Site(s))
- **Pattern**: Broadcast with per-site acknowledgment (deploy to all sites), or targeted to a single site (per-site deployment).
- When shared scripts, external system definitions, database connections, data connections, notification lists, or SMTP configuration are explicitly deployed, central sends them to the target site(s).
- Each site acknowledges receipt and reports success/failure independently.
### 4. Integration Routing (External System → Central → Site → Central → External System)

4
Component-ConfigurationDatabase.md
View File

@@ -6,7 +6,7 @@ The Configuration Database component provides the centralized data access layer
## Location
Central cluster only. Site clusters do not access the configuration database (they receive deployed configurations via the Communication Layer).
Central cluster only. Site clusters do not access the configuration database — they receive all configuration via artifact deployment and instance deployment through the Communication Layer, and read it from local SQLite at runtime.
## Responsibilities
@@ -197,7 +197,7 @@ Since only the after-state is stored, change history for an entity is reconstruc
| Alarms | Create, edit, delete alarm definitions |
| Instances | Create, override values, bind connections, area assignment, disable, enable, delete |
| Deployments | Deploy to instance (who, what, which instance, success/failure) |
| System-Wide Artifact Deployments | Deploy shared scripts / external system definitions / DB connections / notification lists to sites (who, what, result) |
| System-Wide Artifact Deployments | Deploy shared scripts / external system definitions / DB connections / data connections / notification lists / SMTP config to site(s) (who, what, which site(s), result) |
| External Systems | Create, edit, delete definitions |
| Database Connections | Create, edit, delete definitions |
| Notification Lists | Create, edit, delete lists and recipients |

2
Component-DataConnectionLayer.md
View File

@@ -10,7 +10,7 @@ Site clusters only. Central does not interact with machines directly.
## Responsibilities
- Manage data connections defined at the site level (OPC UA servers, LmxProxy endpoints).
- Manage data connections defined centrally and deployed to sites as part of artifact deployment (OPC UA servers, LmxProxy endpoints). Data connection definitions are stored in local SQLite after deployment.
- Establish and maintain connections to data sources based on deployed instance configurations.
- Subscribe to tag paths as requested by Instance Actors (based on attribute data source references in the flattened configuration).
- Deliver tag value updates to the requesting Instance Actors.

10
Component-DeploymentManager.md
View File

@@ -17,7 +17,7 @@ Central cluster only. The site-side deployment responsibilities (receiving confi
- Track deployment status (pending, in-progress, success, failed).
- Handle deployment failures gracefully — if a site is unreachable or the deployment fails, report the failure. No retry or buffering at central.
- If a central failover occurs during deployment, the deployment is treated as failed and must be re-initiated.
- Deploy system-wide artifacts (shared scripts, external system definitions, database connection definitions, notification lists) to all sites on explicit request.
- Deploy system-wide artifacts (shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, SMTP configuration) to all sites or to an individual site on explicit request.
- Send instance lifecycle commands (disable, enable, delete) to sites via the Communication Layer.
## Deployment Flow
@@ -105,9 +105,11 @@ A deployment to a site includes the flattened instance configuration plus any sy
- Shared scripts
- External system definitions
- Database connection definitions
- Notification lists (and SMTP configuration)
- Data connection definitions
- Notification lists
- SMTP configuration
System-wide artifact deployment is a **separate action** from instance deployment, triggered explicitly by a user with the Deployment role.
System-wide artifact deployment is a **separate action** from instance deployment, triggered explicitly by a user with the Deployment role. Artifacts can be deployed to all sites at once or to an individual site (per-site deployment via the Sites admin page).
## Site-Side Apply Atomicity
@@ -119,7 +121,7 @@ Applying a deployment at the site is **all-or-nothing per instance**:
## System-Wide Artifact Version Compatibility
- Cross-site version skew for artifacts (shared scripts, external system definitions, etc.) is **supported** — sites can temporarily run different artifact versions after a partial deployment.
- Cross-site version skew for artifacts (shared scripts, external system definitions, data connection definitions, etc.) is **supported** — sites can temporarily run different artifact versions after a partial deployment.
- Artifacts are self-contained and site-independent. A site running an older version of shared scripts continues to operate correctly with its current instance configurations.
- The Central UI clearly indicates which sites have pending artifact updates so engineers can remediate.

5
Component-ExternalSystemGateway.md
View File

@@ -6,7 +6,7 @@ The External System Gateway manages predefined integrations with external system
## Location
Site clusters (executes calls directly to external systems). Central cluster (stores definitions, brokers inbound requests from external systems to sites).
Site clusters (executes calls directly to external systems, reads definitions from local SQLite). Central cluster (stores definitions in config DB, brokers inbound requests from external systems to sites).
## Responsibilities
@@ -112,7 +112,8 @@ Scripts choose between two call modes per invocation, mirroring the dual-mode da
## Dependencies
- **Configuration Database (MS SQL)**: Stores external system and database connection definitions.
- **Configuration Database (MS SQL)**: Stores external system and database connection definitions (central only).
- **Local SQLite**: At sites, external system and database connection definitions are read from local SQLite (populated by artifact deployment). Sites do not access the central config DB.
- **Store-and-Forward Engine**: Handles buffering for failed external system calls and cached database writes.
- **Communication Layer**: Routes inbound external system requests from central to sites.
- **Security & Auth**: Design role manages definitions.

2
Component-Host.md
View File

@@ -80,7 +80,7 @@ Before the Akka.NET actor system is created, the Host must validate all required
- `NodeConfiguration.RemotingPort` must be in valid port range (1–65535).
- Site nodes must have a non-empty `SiteId`.
- Central nodes must have non-empty `ConfigurationDb` and `MachineDataDb` connection strings.
- Site nodes must have non-empty SQLite path values.
- Site nodes must have non-empty SQLite path values. Site nodes do **not** require a `ConfigurationDb` connection string — all configuration is received via artifact deployment and read from local SQLite.
- At least two seed nodes must be configured.
### REQ-HOST-4a: Readiness Gating

9
Component-NotificationService.md
View File

@@ -6,7 +6,7 @@ The Notification Service provides email notification capabilities to scripts run
## Location
Central cluster (definition management). Site clusters (email delivery).
Central cluster (definition management, stores in config DB). Site clusters (email delivery, reads definitions from local SQLite).
## Responsibilities
@@ -17,8 +17,8 @@ Central cluster (definition management). Site clusters (email delivery).
- Managed by users with the Design role.
### Delivery (Site)
- Resolve notification list names to recipient lists.
- Compose and send emails via SMTP.
- Resolve notification list names to recipient lists from **local SQLite** (populated by artifact deployment). Sites do not access the central config DB.
- Compose and send emails via SMTP using locally stored SMTP configuration.
- On delivery failure, submit the notification to the Store-and-Forward Engine for buffered retry.
## Notification List Definition
@@ -72,7 +72,8 @@ Consistent with the External System Gateway pattern:
## Dependencies
- **Configuration Database (MS SQL)**: Stores notification list definitions and SMTP config.
- **Configuration Database (MS SQL)**: Stores notification list definitions and SMTP config (central only).
- **Local SQLite**: At sites, notification lists, recipients, and SMTP configuration are read from local SQLite (populated by artifact deployment). Sites do not access the central config DB.
- **Store-and-Forward Engine**: Handles buffering for failed email deliveries.
- **Security & Auth**: Design role manages notification lists.
- **Configuration Database (via IAuditService)**: Notification list changes are audit logged.

6
Component-SiteRuntime.md
View File

@@ -66,8 +66,8 @@ Deployment Manager Singleton (Cluster Singleton)
- Reports deployment result (success/failure) back to central.
### System-Wide Artifact Handling
- Receives updated shared scripts, external system definitions, database connection definitions, and notification lists from central.
- Stores artifacts in local SQLite/filesystem.
- Receives updated shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, and SMTP configuration from central.
- Stores all artifacts in local SQLite. After artifact deployment, the site is fully self-contained — all runtime configuration is read from local SQLite with no access to the central configuration database.
- Recompiles shared scripts and makes updated code available to all Script Actors.
### Instance Lifecycle Commands
@@ -308,7 +308,7 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
- **Communication Layer**: Receives deployments and lifecycle commands from central. Handles debug view requests. Reports deployment results.
- **Site Event Logging**: Records script executions, alarm events, deployment events, instance lifecycle events.
- **Health Monitoring**: Reports script error rates and alarm evaluation error rates.
- **Local SQLite**: Persists deployed configurations.
- **Local SQLite**: Persists deployed configurations, system-wide artifacts (external system definitions, database connection definitions, data connection definitions, notification lists, SMTP configuration).
## Interactions

17
HighLevelReqs.md
View File

@@ -33,8 +33,9 @@
- **Pre-deployment validation**: Before any deployment is sent to a site, the central cluster performs comprehensive validation including flattening the configuration, test-compiling all scripts, verifying alarm trigger references, verifying script trigger references, and checking data connection binding completeness (see Section 3.11).
### 1.5 System-Wide Artifact Deployment
- Changes to shared scripts, external system definitions, database connection definitions, and notification lists are **not automatically propagated** to sites.
- Changes to shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, and SMTP configuration are **not automatically propagated** to sites.
- Deployment of system-wide artifacts requires **explicit action** by a user with the **Deployment** role.
- Artifacts can be deployed to **all sites at once** or to an **individual site** (per-site deployment).
- The Design role manages the definitions; the Deployment role triggers deployment to sites. A user may hold both roles.
## 2. Data Storage & Data Flow
@@ -50,7 +51,8 @@
### 2.3 Site-Level Storage & Interface
- Sites have **no user interface** — they are headless collectors, forwarders, and script executors.
- Sites require local storage for: the current deployed (flattened) configurations, deployed scripts, shared scripts, external system definitions, database connection definitions, and notification lists.
- Sites require local storage for: the current deployed (flattened) configurations, deployed scripts, shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, and SMTP configuration.
- After artifact deployment, sites are **fully self-contained** — all runtime configuration is read from local SQLite. Sites do **not** access the central configuration database at runtime.
- Store-and-forward buffers are persisted to a **local SQLite database on each node** and replicated between nodes via application-level replication (see 1.3).
### 2.4 Data Connection Protocols
@@ -81,7 +83,7 @@ Each attribute carries the following metadata:
- **Data Source Reference** *(optional)*: A **relative path** within a data connection (e.g., `/Motor/Speed`). The template defines *what* to read — the path relative to a data connection. The template does **not** specify which data connection to use; that is an instance-level concern (see Section 3.3). Attributes without a data source reference are static configuration values.
### 3.3 Data Connections
- **Data connections** are reusable, named resources defined centrally and then **assigned to specific sites** (e.g., an OPC server, a PLC endpoint).
- **Data connections** are reusable, named resources defined centrally and then **assigned to specific sites** (e.g., an OPC server, a PLC endpoint). Data connection definitions are deployed to sites as part of **artifact deployment** (see Section 1.5) and stored in local SQLite.
- A data connection encapsulates the details needed to communicate with a data source (protocol, address, credentials, etc.).
- Attributes with a data source reference must be **bound to a data connection at instance creation** — the template defines *what* to read (the relative path), and the instance specifies *where* to read it from (the data connection assigned to the site).
- **Binding is per-attribute**: Each attribute with a data source reference individually selects its data connection. Different attributes on the same instance may use different data connections. The Central UI supports bulk assignment (selecting multiple attributes and assigning a data connection to all of them at once) to reduce tedium.
@@ -257,6 +259,7 @@ Scripts **cannot** access other instances' attributes or scripts.
- **Method definitions**: Available API methods with defined parameters and return types.
- Definitions are deployed **uniformly to all sites** — no per-site connection detail overrides.
- Deployment of definition changes requires **explicit action** by a user with the Deployment role.
- At the site, external system definitions are read from **local SQLite** (populated by artifact deployment), not from the central config DB.
### 5.2 Site-to-External-System Communication
- Sites communicate with external systems **directly** (not routed through central).
@@ -283,6 +286,7 @@ Scripts **cannot** access other instances' attributes or scripts.
- Each definition includes configurable retry settings (same pattern as external systems): **max retry count** and **time between retries** (fixed interval).
- Definitions are deployed **uniformly to all sites** — no per-site overrides.
- Deployment of definition changes requires **explicit action** by a user with the Deployment role.
- At the site, database connection definitions are read from **local SQLite** (populated by artifact deployment), not from the central config DB.
### 5.6 Database Access Modes
Scripts can interact with databases in two modes:
@@ -297,10 +301,11 @@ Scripts can interact with databases in two modes:
- Each list has a **name** and contains one or more **recipients**.
- Each recipient has a **name** and an **email address**.
- Notification lists are deployed to **all sites** (deployment requires explicit action by a user with the Deployment role).
- At the site, notification lists and recipients are read from **local SQLite** (populated by artifact deployment), not from the central config DB.
### 6.2 Email Support
- The system has **predefined support for sending email** as the notification delivery mechanism.
- Email server configuration (SMTP settings) is defined centrally and deployed to all sites.
- Email server configuration (SMTP settings) is defined centrally and deployed to all sites as part of **artifact deployment** (see Section 1.5). Sites read SMTP configuration from **local SQLite**.
### 6.3 Script API
- Scripts send notifications using a simplified API: `Notify.To("list name").Send("subject", "message")`
@@ -357,7 +362,7 @@ The central cluster hosts a **configuration and management UI** (no live machine
- **Site & Data Connection Management**: Define sites, manage data connections and assign them to sites.
- **Area Management**: Define hierarchical area structures per site for organizing instances.
- **Deployment**: View diffs between deployed and current template-derived configurations, deploy updates to individual instances. Filter instances by area. Pre-deployment validation runs automatically before any deployment is sent.
- **System-Wide Artifact Deployment**: Explicitly deploy shared scripts, external system definitions, database connection definitions, and notification lists to all sites (requires Deployment role).
- **System-Wide Artifact Deployment**: Explicitly deploy shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, and SMTP configuration to all sites or to an individual site (requires Deployment role). Per-site deployment is available via the Sites admin page.
- **Deployment Status Monitoring**: Track whether deployments were successfully applied at site level.
- **Debug View**: On-demand real-time view of a specific instance's tag values and alarm states for troubleshooting (see 8.1).
- **Parked Message Management**: Query sites for parked messages (external system calls, notifications, and cached database writes), retry or discard them.
@@ -409,7 +414,7 @@ All system-modifying actions are logged, including:
- **Alarm changes**: Create, edit, delete alarm definitions.
- **Instance changes**: Create, override values, bind connections, area assignment, disable, enable, delete.
- **Deployments**: Who deployed what to which instance, and the result (success/failure).
- **System-wide artifact deployments**: Who deployed shared scripts / external system definitions / DB connections / notification lists, and the result.
- **System-wide artifact deployments**: Who deployed shared scripts / external system definitions / DB connections / data connections / notification lists / SMTP config, to which site(s), and the result.
- **External system definition changes**: Create, edit, delete.
- **Database connection changes**: Create, edit, delete.
- **Notification list changes**: Create, edit, delete lists and recipients.

6
src/ScadaLink.Commons/Messages/Artifacts/DataConnectionArtifact.cs Normal file
View File

@@ -0,0 +1,6 @@
namespace ScadaLink.Commons.Messages.Artifacts;
public record DataConnectionArtifact(
string Name,
string Protocol,
string? ConfigurationJson);

2
src/ScadaLink.Commons/Messages/Artifacts/DeployArtifactsCommand.cs
View File

@@ -6,4 +6,6 @@ public record DeployArtifactsCommand(
IReadOnlyList<ExternalSystemArtifact>? ExternalSystems,
IReadOnlyList<DatabaseConnectionArtifact>? DatabaseConnections,
IReadOnlyList<NotificationListArtifact>? NotificationLists,
IReadOnlyList<DataConnectionArtifact>? DataConnections,
IReadOnlyList<SmtpConfigurationArtifact>? SmtpConfigurations,
DateTimeOffset Timestamp);

11
src/ScadaLink.Commons/Messages/Artifacts/SmtpConfigurationArtifact.cs Normal file
View File

@@ -0,0 +1,11 @@
namespace ScadaLink.Commons.Messages.Artifacts;
public record SmtpConfigurationArtifact(
string Name,
string Server,
int Port,
string AuthMode,
string FromAddress,
string? Username,
string? Password,
string? OAuthConfig);

21
src/ScadaLink.SiteRuntime/Actors/DeploymentManagerActor.cs
View File

@@ -479,6 +479,27 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
}
}
// Store data connection definitions (OPC UA endpoints, etc.)
if (command.DataConnections != null)
{
foreach (var dc in command.DataConnections)
{
await _storage.StoreDataConnectionDefinitionAsync(
dc.Name, dc.Protocol, dc.ConfigurationJson);
}
}
// Store SMTP configurations
if (command.SmtpConfigurations != null)
{
foreach (var smtp in command.SmtpConfigurations)
{
await _storage.StoreSmtpConfigurationAsync(
smtp.Name, smtp.Server, smtp.Port, smtp.AuthMode,
smtp.FromAddress, smtp.Username, smtp.Password, smtp.OAuthConfig);
}
}
return new ArtifactDeploymentResponse(
command.DeploymentId, "", true, null, DateTimeOffset.UtcNow);
}

71
src/ScadaLink.SiteRuntime/Persistence/SiteStorageService.cs
View File

@@ -78,6 +78,13 @@ public class SiteStorageService
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS data_connection_definitions (
name TEXT PRIMARY KEY,
protocol TEXT NOT NULL,
configuration TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS smtp_configurations (
name TEXT PRIMARY KEY,
server TEXT NOT NULL,
@@ -467,6 +474,60 @@ public class SiteStorageService
await command.ExecuteNonQueryAsync();
}
// ── Data Connection Definition CRUD ──
/// <summary>
/// Stores or updates a data connection definition (OPC UA endpoint, etc.).
/// </summary>
public async Task StoreDataConnectionDefinitionAsync(string name, string protocol, string? configJson)
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var command = connection.CreateCommand();
command.CommandText = @"
INSERT INTO data_connection_definitions (name, protocol, configuration, updated_at)
VALUES (@name, @protocol, @config, @updatedAt)
ON CONFLICT(name) DO UPDATE SET
protocol = excluded.protocol,
configuration = excluded.configuration,
updated_at = excluded.updated_at";
command.Parameters.AddWithValue("@name", name);
command.Parameters.AddWithValue("@protocol", protocol);
command.Parameters.AddWithValue("@config", (object?)configJson ?? DBNull.Value);
command.Parameters.AddWithValue("@updatedAt", DateTimeOffset.UtcNow.ToString("O"));
await command.ExecuteNonQueryAsync();
_logger.LogDebug("Stored data connection definition '{Name}' (protocol={Protocol})", name, protocol);
}
/// <summary>
/// Returns all stored data connection definitions.
/// </summary>
public async Task<List<StoredDataConnectionDefinition>> GetAllDataConnectionDefinitionsAsync()
{
await using var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var command = connection.CreateCommand();
command.CommandText = "SELECT name, protocol, configuration FROM data_connection_definitions";
var results = new List<StoredDataConnectionDefinition>();
await using var reader = await command.ExecuteReaderAsync();
while (await reader.ReadAsync())
{
results.Add(new StoredDataConnectionDefinition
{
Name = reader.GetString(0),
Protocol = reader.GetString(1),
ConfigurationJson = reader.IsDBNull(2) ? null : reader.GetString(2)
});
}
return results;
}
}
/// <summary>
@@ -492,3 +553,13 @@ public class StoredSharedScript
public string? ParameterDefinitions { get; init; }
public string? ReturnDefinition { get; init; }
}
/// <summary>
/// Represents a data connection definition stored locally in SQLite.
/// </summary>
public class StoredDataConnectionDefinition
{
public string Name { get; init; } = string.Empty;
public string Protocol { get; init; } = string.Empty;
public string? ConfigurationJson { get; init; }
}

2
tests/ScadaLink.Commons.Tests/Messages/MessageConventionTests.cs
View File

@@ -117,7 +117,7 @@ public class MessageConventionTests
{
new("script1", "code", null, null)
},
null, null, null,
null, null, null, null, null,
DateTimeOffset.UtcNow);
var json = JsonSerializer.Serialize(msg);

2
tests/ScadaLink.DeploymentManager.Tests/ArtifactDeploymentServiceTests.cs
View File

@@ -80,6 +80,6 @@ public class ArtifactDeploymentServiceTests
private static DeployArtifactsCommand CreateCommand()
{
return new DeployArtifactsCommand(
"dep1", null, null, null, null, DateTimeOffset.UtcNow);
"dep1", null, null, null, null, null, null, DateTimeOffset.UtcNow);
}
}