Files
scadalink-design/docs/requirements/Component-Commons.md

284 lines
27 KiB
Markdown

# Component: Commons
## Purpose
The Commons component provides the shared foundation of data types, interfaces, enums, message contracts, data transfer objects, and persistence-ignorant domain entity classes used across all other ScadaLink components. It ensures consistent type definitions for cross-component communication, data access, and eliminates duplication of common abstractions.
## Location
Referenced by all component libraries and the Host.
## Responsibilities
- Define shared data types (enums, value objects, result types) used across multiple components.
- Define **persistence-ignorant domain entity classes** (POCOs) representing all configuration database entities. These classes have no dependency on Entity Framework or any persistence framework — EF mapping is handled entirely by the Configuration Database component via Fluent API.
- Define **per-component repository interfaces** that consuming components use for data access. Repository implementations are owned by the Configuration Database component.
- Define protocol abstraction interfaces for the Data Connection Layer.
- Define cross-component message contracts and DTOs for deployment, health, communication, instance lifecycle, and other inter-component data flows.
- Contain **no business logic** — only data structures, interfaces, and enums.
- Maintain **minimal dependencies** — only core .NET libraries; no Akka.NET, no ASP.NET, no Entity Framework.
---
## Requirements
### REQ-COM-1: Shared Data Type System
Commons must define shared primitive and utility types used across multiple components, including but not limited to:
- **`DataType` enum**: Enumerates the data types supported by the system (e.g., Boolean, Int32, Float, Double, String, DateTime, Binary).
- **`RetryPolicy`**: A record or immutable class describing retry behavior (max retries, fixed delay between retries).
- **`Result<T>`**: A discriminated result type that represents either a success value or an error, enabling consistent error handling across component boundaries without exceptions.
- **`InstanceState` enum**: Enabled, Disabled.
- **`DeploymentStatus` enum**: Pending, InProgress, Success, Failed.
- **`AlarmState` enum**: Active, Normal.
- **`AlarmLevel` enum**: None, Low, LowLow, High, HighHigh. Severity level for an active alarm; always `None` for binary trigger types, set by `HiLo` triggers.
- **`AlarmTriggerType` enum**: ValueMatch, RangeViolation, RateOfChange, HiLo.
- **`ConnectionHealth` enum**: Connected, Disconnected, Connecting, Error.
- **`TrackedOperationId`**: A GUID identifying a tracked store-and-forward operation (`ExternalSystem.CachedCall`, `Database.CachedWrite`, `Notify.Send`). Generated caller-side at the site at call time, returned to the script as a tracking handle, and reused as the idempotency key for telemetry sent to central. The notification domain's existing `NotificationId` is the notification-specific name for this same concept.
- **`TrackedOperationKind` enum**: ExternalCall, DatabaseWrite. Discriminates the two cached-call kinds carried by a tracked operation (notifications are tracked separately via the `NotificationType` enum).
- **`TrackedOperationStatus` enum**: Pending, Retrying, Delivered, Parked, Failed, Discarded. The unified lifecycle state shared by all tracked store-and-forward operations. This is the operation's externally-observable lifecycle status in the site-local tracking table (the status record); it is related to but distinct from the S&F buffer's own `StoreAndForwardMessageStatus`, which tracks a buffered message's retry state within the buffer (the retry mechanism). `Failed` (permanent failure) has no notification analogue — notifications use only the other five states (the `NotificationStatus` enum omits `Failed`).
- **`AuditChannel` enum**: ApiOutbound, DbOutbound, Notification, ApiInbound. Discriminates the script-trust-boundary channel that produced an `AuditEvent`. Owned by the Audit Log component.
- **`AuditKind` enum**: SyncCall, CachedEnqueued, CachedAttempt, CachedTerminal, SyncWrite, SyncRead, Enqueued, Attempt, Terminal, Completed. Channel-specific event kind — the valid `Kind` values for each `AuditChannel` are listed in the Audit Log component design (`Component-AuditLog.md`).
- **`AuditStatus` enum**: Success, TransientFailure, PermanentFailure, Enqueued, Retrying, Delivered, Parked, Discarded. Outcome of a single audit event row; superset of `TrackedOperationStatus` to also cover one-shot sync calls.
- **`AuditEvent`**: A record carrying every column of the central `AuditLog` row — `EventId` (GUID, idempotency key), `OccurredAtUtc`, `IngestedAtUtc`, `Channel` (`AuditChannel`), `Kind` (`AuditKind`), `CorrelationId`, `SourceSiteId`, `SourceInstanceId`, `SourceScript`, `Actor`, `Target`, `Status` (`AuditStatus`), `HttpStatus`, `DurationMs`, `ErrorMessage`, `ErrorDetail`, `RequestSummary`, `ResponseSummary`, `PayloadTruncated`, `Extra` — plus a site-only `ForwardState` (`Pending` | `Forwarded` | `Reconciled`) used by the site SQLite write-buffer's telemetry/reconciliation loop. `IngestedAtUtc` is unset at the site and stamped on central ingest. See `Component-AuditLog.md` for the persistence schema and ingest semantics.
Types defined here must be immutable and thread-safe.
**Timestamp convention**: All timestamps throughout the system must use **UTC** (`DateTime` with `DateTimeKind.Utc` or `DateTimeOffset` with zero offset). This applies to all stored timestamps (SQLite, MS SQL, audit log entries), all message timestamps (attribute values, alarm state changes, health reports, event log entries, deployment records), and all wire-format timestamps (Akka remoting, Inbound API responses). Local time conversion, if needed, is a UI display concern only.
### REQ-COM-2: Protocol Abstraction
Commons must define the protocol abstraction interfaces that the Data Connection Layer implements and other components consume:
- **`IDataConnection`**: The common interface for reading, writing, and subscribing to device data regardless of the underlying protocol (OPC UA, custom legacy, etc.).
- **Related types**: Tag identifiers, read/write results, subscription callbacks, connection status enums, and quality codes.
These interfaces must not reference any specific protocol implementation.
### REQ-COM-3: Domain Entity Classes (POCOs)
Commons must define persistence-ignorant POCO entity classes for all configuration database entities. These classes:
- Are plain C# classes with properties — no EF attributes, no base classes from EF, no navigation property annotations.
- May include navigation properties (e.g., `Template.Attributes` as `ICollection<TemplateAttribute>`) defined as plain collections. The Configuration Database component configures the relationships via Fluent API.
- May include constructors that enforce invariants (e.g., required fields).
- Must have **no dependency** on Entity Framework Core or any persistence library.
Entity classes are organized by domain area:
- **Template & Modeling**: `Template`, `TemplateAttribute`, `TemplateAlarm`, `TemplateScript`, `TemplateComposition`, `TemplateFolder`.
- **Instances**: `Instance`, `InstanceAttributeOverride`, `InstanceConnectionBinding`, `InstanceAlarmOverride`, `Area`.
- **Shared Scripts**: `SharedScript`.
- **Sites & Data Connections**: `Site`, `DataConnection`.
- **External Systems & Database Connections**: `ExternalSystemDefinition`, `ExternalSystemMethod`, `DatabaseConnectionDefinition`.
- **Notifications**: `NotificationList` (carries a `Type` field — `NotificationType` enum — selecting the list's notification type and its type-specific targets), `NotificationRecipient`, `SmtpConfiguration`, `Notification` (the durable central-queue row — see below).
- **Inbound API**: `ApiKey`, `ApiMethod`.
- **Security**: `LdapGroupMapping`, `SiteScopeRule`.
- **Deployment**: `DeploymentRecord`, `SystemArtifactDeploymentRecord`, `DeployedConfigSnapshot`.
- **Audit**: `AuditLogEntry`.
The **`Notification`** entity is the persistence-ignorant POCO for a row of the central `Notifications` table — the durable notification queue owned by the Notification Outbox. It is a plain class with properties for `NotificationId` (GUID, the idempotency key), `Type` (`NotificationType` enum discriminator), `ListName`, `Subject`, `Body`, `TypeData` (a JSON string — the type-agnostic extensibility hook), `Status` (`NotificationStatus` enum), `RetryCount`, `LastError`, `ResolvedTargets`, the provenance fields `SourceSiteId` / `SourceInstanceId` / `SourceScript`, and the UTC timestamps `SiteEnqueuedAt`, `CreatedAt`, `LastAttemptAt`, `NextAttemptAt`, `DeliveredAt`. As with every entity class it has no EF dependency; the Configuration Database component supplies the Fluent API mapping, value conversions, and indexes. The `Type` and `Status` enums (`NotificationType`: `Email`, `Teams`, …; `NotificationStatus`: `Pending`, `Retrying`, `Delivered`, `Parked`, `Discarded`) are defined under `Types/Enums/` per REQ-COM-1.
### REQ-COM-4: Per-Component Repository Interfaces
Commons must define repository interfaces that consuming components use for data access. Each interface is tailored to the data needs of its consuming component:
- `ITemplateEngineRepository` — Templates, attributes, alarms, scripts, compositions, template folders, instances, overrides, alarm overrides, connection bindings, areas.
- `IDeploymentManagerRepository` — Deployment records, deployed configuration snapshots, system-wide artifact deployment records.
- `ISecurityRepository` — LDAP group mappings, site scoping rules.
- `IInboundApiRepository` — API keys, API method definitions.
- `IExternalSystemRepository` — External system definitions, method definitions, database connection definitions.
- `INotificationRepository` — Notification lists (including the `Type` field), recipients, SMTP configuration.
- `INotificationOutboxRepository` — The `Notifications` table: insert-if-not-exists ingest on `NotificationId`, due-row polling (`Pending` rows and `Retrying` rows past `NextAttemptAt`), status transitions, KPI aggregate queries, and the bulk delete of terminal rows used by the daily purge job.
- `ISiteCallAuditRepository` — The `SiteCalls` table: insert-if-not-exists ingest on `TrackedOperationId`, upsert-on-newer-status from telemetry and reconciliation pulls, KPI aggregate queries, and the bulk delete of terminal rows used by the daily purge job.
- `ISiteRepository` — Sites, data connections, and their site assignments.
- `ICentralUiRepository` — Read-oriented queries spanning multiple domain areas for display purposes.
All repository interfaces must:
- Accept and return the POCO entity classes defined in Commons.
- Include a `SaveChangesAsync()` method (or equivalent) to support unit-of-work commit.
- Have **no dependency** on Entity Framework Core — they are pure interfaces.
Implementations of these interfaces are owned by the Configuration Database component.
### REQ-COM-4a: Cross-Cutting Service Interfaces
Commons must define service interfaces for cross-cutting concerns that multiple components consume:
- **`IAuditService`**: Provides a single method for components to log audit entries: `LogAsync(user, action, entityType, entityId, entityName, afterState)`. The implementation (owned by the Configuration Database component) serializes the state as JSON and adds the audit entry to the current unit-of-work transaction. Defined in Commons so any central component can call it without depending on the Configuration Database component directly.
- **`IDatabaseGateway`**: Provides script-facing ADO.NET database access via named database connections. Implemented by the External System Gateway, consumed by the Site Runtime's script runtime context.
- **`IExternalSystemClient`**: Provides script-facing invocation of external system HTTP APIs (synchronous `Call` and store-and-forward `CachedCall`). Implemented by the External System Gateway, consumed by the script runtime context.
- **`IInstanceLocator`**: Resolves an instance unique name to its site identifier. Used by the Inbound API's `Route.To()` to determine the destination site.
- **`INotificationDeliveryService`**: Sends notifications to a named notification list, routing transient failures to store-and-forward. Implemented by the Notification Service, consumed by the script runtime context.
- **`IAuditWriter`**: Site-local hot-path interface for appending an `AuditEvent` to the site SQLite `AuditLog`: `Task WriteAsync(AuditEvent evt, CancellationToken ct)`. Single durable INSERT, `ForwardState = Pending`. Consumed by the script-trust-boundary call paths (External System Gateway, Database layer, Store-and-Forward Engine). Implementation lives in the Audit Log component.
- **`ICentralAuditWriter`**: Central direct-write interface for central-originated audit rows (Inbound API request completion, Notification Outbox dispatcher attempts/terminals): `Task WriteAsync(AuditEvent evt, CancellationToken ct)`, with insert-if-not-exists semantics on `EventId` so retried handlers cannot produce duplicates. Implementation lives in the Audit Log component.
These interfaces are defined in Commons so that consuming components depend only on the abstraction, not on the implementing component.
### REQ-COM-5: Cross-Component Message Contracts
Commons must define the shared DTOs and message contracts used for inter-component communication, including:
- **Deployment DTOs**: Configuration snapshots, deployment commands, deployment status, validation results.
- **Instance Lifecycle DTOs**: Disable, enable, delete commands and responses.
- **Health DTOs**: Health check results, site status reports, heartbeat messages. Includes script error rates and alarm evaluation error rates.
- **Communication DTOs**: Site identity, connection state, routing metadata.
- **Attribute Stream DTOs**: Attribute value change messages (instance name, attribute path, value, quality, timestamp) and alarm state change messages (instance name, alarm name, state, priority, timestamp) for the site-wide Akka stream.
- **Debug View DTOs**: Subscribe/unsubscribe requests, one-shot snapshot request (`DebugSnapshotRequest`), initial snapshot, stream filter criteria.
- **Script Execution DTOs**: Script call requests (with recursion depth), return values, error results.
- **System-Wide Artifact DTOs**: Shared script packages, external system definitions, database connection definitions, notification list definitions.
- **Notification DTOs**: `NotificationSubmit` (site→central submission: `NotificationId`, `ListName`, `Subject`, `Body`, provenance, `SiteEnqueuedAt`) and `NotificationSubmitAck` (central acknowledgement returned only after the `Notifications` row is persisted — ack-after-persist — which the site Store-and-Forward Engine waits on before clearing the buffered message). `NotificationStatusQuery` / `NotificationStatusResponse` back the `Notify.Status` script API, round-tripping a status record (status, retry count, last error, key timestamps) once a notification has been forwarded. Recipient resolution is *not* part of any contract — the site forwards only `(listName, subject, body)` and central resolves the list at delivery time. Subject to the additive-only evolution rules in REQ-COM-5a, since a submission can cross the site→central version-skew boundary.
- **Cached Call Tracking DTOs**: `CachedCallTelemetry` (site→central lifecycle telemetry for a tracked cached call: `TrackedOperationId`, source site, `Kind` — the `TrackedOperationKind` enum (`ExternalCall` / `DatabaseWrite`) — target summary, status, retry count, last error, key timestamps, and source instance / script provenance) and `CachedCallReconcileRequest` / `CachedCallReconcileResponse` (cursor-based per-site pull of tracking rows changed since a cursor, used so missed telemetry self-heals). All three live in the `Integration/` message folder and are subject to the additive-only evolution rules in REQ-COM-5a, since they cross the site→central version-skew boundary. `CachedCallTelemetry` is additively extended to also carry the `AuditEvent` content for the corresponding lifecycle transition (`CachedEnqueued` / `CachedAttempt` / `CachedTerminal`), so one packet drives both the `SiteCalls` operational upsert and the `AuditLog` insert-if-not-exists in a single central transaction — see [Component-AuditLog.md](Component-AuditLog.md), Cached Operations — Combined Telemetry.
- **Parked Operation Command DTOs**: `RetryParkedOperation` and `DiscardParkedOperation` (central→site command/control messages keyed by `TrackedOperationId`, instructing the owning site to retry or discard a parked store-and-forward operation). These generalize the existing parked-message retry/discard commands to also cover parked cached calls; they live in the `RemoteQuery/` message folder alongside the other parked-message management messages.
- **Audit Telemetry DTOs**: `AuditTelemetryEnvelope` (site→central gRPC message wrapping a batch of `AuditEvent` rows for the `IngestAuditEvents` telemetry call) and the matching reconciliation pull messages (`PullAuditEvents` request/response carrying a `sinceUtc` cursor and a batch of `AuditEvent` rows). Live in the `Integration/` message folder, subject to the additive-only evolution rules in REQ-COM-5a since they cross the site→central version-skew boundary. Cached-operation audit rows do **not** travel via `AuditTelemetryEnvelope` — they are folded into `CachedCallTelemetry` per the bullet above.
All message types must be `record` types or immutable classes suitable for use as Akka.NET messages (though Commons itself must not depend on Akka.NET).
### REQ-COM-5a: Message Contract Versioning
Since the system supports cross-site artifact version skew (sites may temporarily run different versions), message contracts must follow **additive-only evolution rules**:
- New fields may be added with default values. Existing fields must not be removed or have their types changed.
- Serialization must tolerate unknown fields (forward compatibility) and missing optional fields (backward compatibility).
- Breaking changes require a new message type and a coordinated deployment to all nodes.
- The Akka.NET serialization binding configuration (in the Host component) must explicitly map message types to serializers to prevent accidental binary serialization.
### REQ-COM-5b: Namespace & Folder Convention
All types in Commons are organized by **category** and **domain area** using a consistent namespace and folder hierarchy:
```
ScadaLink.Commons/
├── Types/ # REQ-COM-1: Shared data types
│ ├── Result.cs
│ ├── RetryPolicy.cs
│ ├── ScriptArgs.cs # script-call parameter normalization helper
│ ├── ScriptParameters.cs # typed script-parameter access helper
│ ├── StaleTagMonitor.cs # heartbeat staleness watchdog
│ ├── ValueFormatter.cs # culture-invariant value-to-string helper
│ ├── DynamicJsonElement.cs # dynamic JSON wrapper for scripts
│ ├── TrackedOperationId.cs # tracked store-and-forward operation ID (GUID)
│ ├── Enums/ # InstanceState, DeploymentStatus, AlarmState,
│ │ # AlarmLevel, AlarmTriggerType, ConnectionHealth,
│ │ # DataType, StoreAndForwardCategory,
│ │ # StoreAndForwardMessageStatus,
│ │ # NotificationType, NotificationStatus,
│ │ # TrackedOperationKind, TrackedOperationStatus,
│ │ # AuditChannel, AuditKind, AuditStatus
│ ├── Audit/ # AuditEvent record (site + central audit row)
│ ├── DataConnections/ # OPC UA endpoint config value objects + enums
│ ├── Flattening/ # FlattenedConfiguration, ConfigurationDiff,
│ │ # DeploymentPackage, ValidationResult
│ └── Scripts/ # AlarmContext, ScriptScope
├── Interfaces/ # Shared interfaces by concern
│ ├── Protocol/ # REQ-COM-2: Protocol abstraction (IDataConnection, etc.)
│ ├── Repositories/ # REQ-COM-4: Per-component repository interfaces
│ │ ├── ITemplateEngineRepository.cs
│ │ ├── IDeploymentManagerRepository.cs
│ │ ├── ISecurityRepository.cs
│ │ ├── IInboundApiRepository.cs
│ │ ├── IExternalSystemRepository.cs
│ │ ├── INotificationRepository.cs
│ │ ├── INotificationOutboxRepository.cs
│ │ ├── ISiteCallAuditRepository.cs
│ │ ├── ISiteRepository.cs
│ │ └── ICentralUiRepository.cs
│ └── Services/ # REQ-COM-4a: Cross-cutting service interfaces
│ ├── IAuditService.cs
│ ├── IAuditWriter.cs
│ ├── ICentralAuditWriter.cs
│ ├── IDatabaseGateway.cs
│ ├── IExternalSystemClient.cs
│ ├── IInstanceLocator.cs
│ └── INotificationDeliveryService.cs
├── Entities/ # REQ-COM-3: Domain entity POCOs, by domain area
│ ├── Templates/ # Template, TemplateAttribute, TemplateAlarm,
│ │ # TemplateScript, TemplateComposition, TemplateFolder
│ ├── Instances/ # Instance, InstanceAttributeOverride,
│ │ # InstanceConnectionBinding, InstanceAlarmOverride, Area
│ ├── Sites/ # Site, DataConnection
│ ├── ExternalSystems/ # ExternalSystemDefinition, ExternalSystemMethod,
│ │ # DatabaseConnectionDefinition
│ ├── Notifications/ # NotificationList, NotificationRecipient, SmtpConfiguration,
│ │ # Notification (central Notifications-table row)
│ ├── InboundApi/ # ApiKey, ApiMethod
│ ├── Security/ # LdapGroupMapping, SiteScopeRule
│ ├── Deployment/ # DeploymentRecord, SystemArtifactDeploymentRecord,
│ │ # DeployedConfigSnapshot
│ ├── Scripts/ # SharedScript
│ └── Audit/ # AuditLogEntry
├── Messages/ # REQ-COM-5: Cross-component message contracts, by concern
│ ├── Deployment/
│ ├── Lifecycle/
│ ├── Health/
│ ├── Communication/
│ ├── Streaming/
│ ├── DebugView/
│ ├── ScriptExecution/
│ ├── Artifacts/
│ ├── DataConnection/ # data-connection subscribe/write/health messages
│ ├── Instance/ # attribute get/set request/command messages
│ ├── Integration/ # external-integration call request/response,
│ │ # cached-call tracking telemetry + reconcile,
│ │ # audit telemetry envelope + reconcile
│ ├── Notification/ # NotificationSubmit + ack,
│ │ # NotificationStatusQuery/Response
│ ├── InboundApi/ # Route.To() request messages
│ ├── RemoteQuery/ # event-log and parked-message query messages,
│ │ # parked-operation retry/discard commands
│ └── Management/ # HTTP/ClusterClient management commands + registry
├── Serialization/ # OpcUaEndpointConfigSerializer (typed↔legacy JSON)
└── Validators/ # OpcUaEndpointConfigValidator
```
**Naming rules**:
- Namespaces mirror the folder structure: `ScadaLink.Commons.Entities.Templates`, `ScadaLink.Commons.Interfaces.Repositories`, etc.
- Interface names use the `I` prefix: `ITemplateEngineRepository`, `IAuditService`, `IDataConnection`.
- Entity classes are named after the domain concept (no suffixes like `Entity` or `Model`): `Template`, `Instance`, `Site`.
- Message contracts are named as commands, events, or responses: `DeployInstanceCommand`, `DeploymentStatusResponse`, `AttributeValueChanged`.
- Enums use singular names: `AlarmState`, not `AlarmStates`.
### REQ-COM-6: No Business Logic; Pure Helpers Permitted
Commons must contain only:
- Data structures (records, classes, structs)
- Interfaces
- Enums
- Constants
- **Pure, stateless helpers** — see the carve-out below.
It must **not** contain any business logic that orchestrates other components, service implementations that perform I/O (database, network, file system), actor definitions, or orchestration code.
**Pure-helper carve-out.** Commons *may* contain stateless, side-effect-free helper types whose behavior is confined to transforming, formatting, parsing, or validating the data types Commons already defines, provided they:
- have no dependency on Akka.NET, ASP.NET Core, EF Core, or any I/O surface (consistent with REQ-COM-7);
- hold no shared mutable state across calls (a self-contained instance helper such as `StaleTagMonitor`, which owns only its own timer, is acceptable);
- do not call into other components or perform orchestration.
Examples currently in Commons that fall under this carve-out: `Result<T>`, `ScriptParameters` and `ScriptArgs` (script-parameter shaping), `ValueFormatter` (value-to-string formatting), `DynamicJsonElement` (dynamic JSON access), `StaleTagMonitor` (a self-contained heartbeat watchdog), `OpcUaEndpointConfigSerializer` (typed↔legacy JSON conversion of a Commons value object) and `OpcUaEndpointConfigValidator` (rule checks over a Commons value object). These are intentionally placed in Commons so every consuming component shares one implementation rather than duplicating the logic. Anything that would require an I/O dependency, mutable cross-call state, or knowledge of another component's behavior does **not** qualify and must live in the owning component.
### REQ-COM-7: Minimal Dependencies
Commons must depend only on core .NET libraries (`System.*`, `Microsoft.Extensions.Primitives` if needed). It must **not** reference:
- Akka.NET or any Akka.* packages
- ASP.NET Core or any Microsoft.AspNetCore.* packages
- Entity Framework Core or any Microsoft.EntityFrameworkCore.* packages
- Any third-party libraries requiring paid licenses
This ensures Commons can be referenced by all components without introducing transitive dependency conflicts.
---
## Dependencies
- **None** — only core .NET SDK.
## Interactions
- **All component libraries**: Reference Commons for shared types, interfaces, entity classes, and contracts.
- **Configuration Database**: Implements the repository interfaces defined in Commons. Maps the POCO entity classes to the database via EF Core Fluent API.
- **Host**: References Commons transitively through the component libraries.