scadalink-design/docs/requirements/Component-ConfigurationDatabase.md

Component: Configuration Database

Purpose

The Configuration Database component provides the centralized data access layer for all system configuration data stored in MS SQL. It owns the database schema, Entity Framework DbContext, repository implementations, unit-of-work support, migration management, and audit logging. All central components access configuration data through this component — no other component interacts with the configuration database directly.

Location

Central cluster only. Site clusters do not access the configuration database — they receive all configuration via artifact deployment and instance deployment through the Communication Layer, and read it from local SQLite at runtime.

Responsibilities

• Define and own the complete database schema for the configuration MS SQL database via EF Core Fluent API mappings.
• Provide the Entity Framework Core DbContext as the single point of access to the configuration database.
• Implement the per-component repository interfaces defined in Commons. The interfaces and POCO entity classes live in Commons (persistence-ignorant); this component provides the EF Core implementations.
• Implement the IAuditService interface defined in Commons. Handles JSON serialization of entity state and writes audit entries within the same unit-of-work transaction as the change being audited.
• Provide unit-of-work support via EF Core's DbContext for transactional multi-entity operations.
• Manage schema migrations via EF Core Migrations with support for generating SQL scripts for manual execution in production.
• Support seed data for initial system setup.
• Manage connection pooling and connection lifecycle for the configuration database.

Note: This component does not manage the Machine Data Database. The Machine Data Database is a separate concern with different access patterns (direct ADO.NET connections from scripts via Database.Connection()).

---

Database Schema

The configuration database stores all central system data, organized by domain area:

Template & Modeling

• Templates: Template definitions (name, parent template reference, description).
• Template Attributes: Attribute definitions per template (name, value, data type, lock flag, description, data source reference).
• Template Alarms: Alarm definitions per template (name, description, priority, lock flag, trigger type, trigger configuration, on-trigger script reference).
• Template Scripts: Script definitions per template (name, lock flag, C# source code, trigger type, trigger configuration, minimum time between runs, parameter definitions, return value definitions).
• Template Compositions: Feature module composition relationships (composing template, composed template, module instance name).
• Instances: Instance definitions (template reference, site reference, area reference, enabled/disabled state).
• Instance Attribute Overrides: Per-instance attribute value overrides.
• Instance Connection Bindings: Per-attribute data connection binding for each instance.
• Areas: Hierarchical area definitions per site (name, parent area reference, site reference).

Shared Scripts

• Shared Scripts: System-wide reusable script definitions (name, C# source code, parameter definitions, return value definitions).

Sites & Data Connections

• Sites: Site definitions (name, identifier, description, NodeAAddress, NodeBAddress, GrpcNodeAAddress, GrpcNodeBAddress).
• Data Connections: Data connection definitions (name, protocol type, connection details) with site assignments.

External Systems & Database Connections

• External System Definitions: External system contracts (name, connection details, retry settings).
• External System Methods: API method definitions per external system (method name, parameter definitions, return type definitions).
• Database Connection Definitions: Named database connections (name, connection details, retry settings).

Notifications

• Notification Lists: List definitions (name).
• Notification Recipients: Recipients per list (name, email address).
• SMTP Configuration: Email server settings.

Inbound API

• API Keys: Key definitions (name/label, key value, enabled flag).
• API Methods: Method definitions (name, approved key references, parameter definitions, return value definitions, implementation script, timeout).

Security

• LDAP Group Mappings: Mappings between LDAP group names and system roles (Admin, Design, Deployment).
• Site Scoping Rules: Per-mapping site scope restrictions for Deployment role.

Deployment

• Deployment Records: Deployment history per instance (timestamp, user, status, deployed configuration snapshot).
• System-Wide Artifact Deployment Records: Deployment history for shared artifacts (timestamp, user, artifact type, status).

Audit Logging

• Audit Log Entries: Append-only audit trail (timestamp, user, action, entity type, entity ID, entity name, state as JSON). Stores only the after-state — change history is reconstructed by comparing consecutive entries. Entries are never modified or deleted. No retention policy — retained indefinitely. Indexed on timestamp, user, entity type, entity ID, and action for efficient filtering.

---

Data Access Architecture

DbContext

A single ScadaLinkDbContext (or a small number of bounded DbContexts if warranted) serves as the EF Core entry point. The DbContext:

• Maps the POCO entity classes defined in Commons to the database using Fluent API only — no data annotations on the entity classes.
• Configures relationships, indexes, constraints, and value conversions.
• Provides SaveChangesAsync() as the unit-of-work commit mechanism.

Per-Component Repository Implementations

Repository interfaces are defined in Commons alongside the POCO entity classes (see Component-Commons.md, REQ-COM-4). This component provides the EF Core implementations of those interfaces.

Repository Interface (in Commons)	Consuming Component	Scope
ITemplateEngineRepository	Template Engine	Templates, attributes, alarms, scripts, compositions, instances, overrides, connection bindings, areas
IDeploymentManagerRepository	Deployment Manager	Current deployment status per instance, deployed configuration snapshots, system-wide artifact deployment status per site (no deployment history — audit log provides historical traceability)
ISecurityRepository	Security & Auth	LDAP group mappings, site scoping rules
IInboundApiRepository	Inbound API	API keys, API method definitions
IExternalSystemRepository	External System Gateway	External system definitions, method definitions, database connection definitions
INotificationRepository	Notification Service	Notification lists, recipients, SMTP configuration
IHealthMonitoringRepository	Health Monitoring	(Minimal — health data is in-memory; repository needed only if connectivity history is persisted in the future)
ICentralUiRepository	Central UI	Read-oriented queries spanning multiple domain areas for display purposes

Each implementation class uses the DbContext internally and works with the POCO entity classes from Commons. Consuming components depend only on Commons (for interfaces and entities) — they never reference this component or EF Core directly. The DI container in the Host wires the implementations to the interfaces.

Unit of Work

EF Core's DbContext naturally provides unit-of-work semantics:

• Multiple entity modifications within a single request are tracked by the DbContext.
• SaveChangesAsync() commits all pending changes in a single database transaction.
• If any part fails, the entire transaction rolls back.
• Optimistic concurrency is used on deployment status records and instance lifecycle state via EF Core rowversion / concurrency tokens. This prevents stale deployment status transitions (e.g., two concurrent requests both trying to update the same instance's status). Template editing remains last-write-wins by design — optimistic concurrency is intentionally not applied to template content.
• For operations that span multiple repository calls (e.g., creating a template with attributes, alarms, and scripts), the consuming component uses a single DbContext instance (via DI scoping) to ensure atomicity.

Example Transactional Flow

Template Engine: Create Template
    │
    ├── repository.AddTemplate(template)      // template is a Commons POCO
    ├── repository.AddAttributes(attributes)  // attributes are Commons POCOs
    ├── repository.AddAlarms(alarms)          // alarms are Commons POCOs
    ├── repository.AddScripts(scripts)        // scripts are Commons POCOs
    └── repository.SaveChangesAsync()         // single transaction commits all

---

Audit Logging

The Configuration Database component implements the IAuditService interface (defined in Commons), providing audit logging as a built-in capability of the data access layer.

IAuditService Implementation

Components call IAuditService after a successful operation:

IAuditService.LogAsync(user, action, entityType, entityId, entityName, afterState)

• user: The authenticated AD user who performed the action.
• action: The type of operation (Create, Update, Delete, Deploy, Disable, Enable).
• entityType: What was changed (Template, Instance, SharedScript, Alarm, ExternalSystem, DatabaseConnection, NotificationList, ApiKey, ApiMethod, Area, Site, DataConnection, LdapGroupMapping).
• entityId: Unique identifier of the specific entity.
• entityName: Human-readable name of the entity.
• afterState: The entity's state after the change, which the implementation serializes as JSON. Null for deletes.

Transactional Guarantee

Audit entries are written synchronously within the same database transaction as the change. The IAuditService implementation adds an AuditLogEntry to the current DbContext. When the calling component calls SaveChangesAsync(), both the change and the audit entry commit together. This guarantees:

• If the change succeeds, the audit entry is always recorded.
• If the change fails and rolls back, the audit entry is also rolled back.
• No audit entries are lost due to process crashes between the change and the audit write.

Integration Example

Template Engine: Update Template
    │
    ├── repository.UpdateTemplate(template)
    ├── auditService.LogAsync(user, "Update", "Template", template.Id,
    │       template.Name, template)
    └── repository.SaveChangesAsync()  ← both the change and audit entry commit together

Audit Entry Schema

Field	Type	Description
Id	Long / GUID	Unique identifier for the audit entry.
Timestamp	DateTimeOffset	When the action occurred (UTC).
User	String	Authenticated AD username.
Action	String	The type of operation.
EntityType	String	What was changed.
EntityId	String	Unique identifier of the entity.
EntityName	String	Human-readable name (for display without deserializing state).
State	nvarchar(max)	Entity state after the change, serialized as JSON. Null for deletes.

State Serialization

• Entity state is serialized as JSON using the standard .NET JSON serializer.
• JSON is stored in nvarchar(max) and is queryable via SQL Server's JSON_VALUE and OPENJSON functions.
• For deletes, the state is null. The previous state can be found by querying the most recent prior entry for the same entity.

Granularity

• One audit entry per save operation. When a user edits a template and changes multiple attributes in a single save, one entry is created with the full entity state after the save.

Reconstructing Change History

Since only the after-state is stored, change history for an entity is reconstructed by querying all entries for that entity ordered by timestamp. Comparing consecutive entries reveals what changed at each step. This is a query-time concern handled by the Central UI.

Audited Actions

Category	Actions
Templates	Create, edit, delete templates
Scripts	Create, edit, delete template scripts and shared scripts
Alarms	Create, edit, delete alarm definitions
Instances	Create, override values, bind connections, area assignment, disable, enable, delete
Deployments	Deploy to instance (who, what, which instance, success/failure)
System-Wide Artifact Deployments	Deploy shared scripts / external system definitions / DB connections / data connections / notification lists / SMTP config to site(s) (who, what, which site(s), result)
External Systems	Create, edit, delete definitions
Database Connections	Create, edit, delete definitions
Notification Lists	Create, edit, delete lists and recipients
Inbound API	API key create, enable/disable, delete. API method create, edit, delete
Areas	Create, edit, delete area definitions
Sites & Data Connections	Create, edit, delete sites. Define and assign data connections to sites
Security/Admin	Role mapping changes, site permission changes

Query Capabilities

The Central UI audit log viewer can filter by:

• User: Who made the change.
• Entity type: What kind of entity was changed.
• Action type: What kind of operation was performed.
• Time range: When the change occurred.
• Specific entity ID/name: Changes to a particular entity.

Results are returned in reverse chronological order (most recent first) with pagination support.

---

Migration Management

Entity Framework Core Migrations

• Schema changes are managed via EF Core Migrations (dotnet ef migrations add, dotnet ef migrations script).
• Each migration is a versioned, incremental schema change.

Development Environment

• Migrations are auto-applied at application startup using dbContext.Database.MigrateAsync().
• This allows rapid iteration without manual SQL execution.

Production Environment

• Migrations are never auto-applied.
• SQL scripts are generated via dotnet ef migrations script --idempotent and reviewed by a DBA or engineer.
• Scripts are executed manually in SQL Server Management Studio (SSMS) or equivalent tooling.
• The Host startup in production validates that the database schema version matches the expected migration level and fails fast with a clear error if not.

Migration Script Generation

# Generate idempotent SQL script for all pending migrations
dotnet ef migrations script --idempotent --output migration.sql --project <ConfigDbProject>

# Generate script from a specific migration to another
dotnet ef migrations script FromMigration ToMigration --output migration.sql

Generated scripts are idempotent — they can be safely re-run without causing errors or duplicate changes.

---

Seed Data

The Configuration Database supports seeding initial data required for the system to be usable after a fresh installation. Seed data is applied as part of the migration pipeline.

Seed Data Includes

• Default system configuration values.
• Any baseline reference data required by the application.

Mechanism

• Seed data is defined using EF Core's HasData() in entity configurations or in dedicated seed migrations.
• Seed data is included in the generated SQL scripts, so it is applied alongside schema changes in both development and production.

---

Connection Management

• Connection strings are provided via the Host's DatabaseConfiguration options (bound from appsettings.json).
• EF Core manages connection pooling via the underlying ADO.NET SQL Server provider.
• The DbContext is registered as a scoped service in the DI container, ensuring each request/operation gets its own instance.
• No connection management for the Machine Data Database — that is handled separately by consumers (Inbound API scripts, external system gateway).

---

Dependencies

• Entity Framework Core: ORM, DbContext, migrations, change tracking.
• Microsoft.EntityFrameworkCore.SqlServer: SQL Server database provider.
• MS SQL Server: The configuration database instance.
• Commons: POCO entity classes and repository interfaces that this component maps and implements.

Interactions

• Template Engine: Uses ITemplateEngineRepository for all template, instance, and area data operations.
• Deployment Manager: Uses IDeploymentManagerRepository for deployment records and status tracking.
• Security & Auth: Uses ISecurityRepository for LDAP group mappings and site scoping.
• Inbound API: Uses IInboundApiRepository for API keys and method definitions.
• External System Gateway: Uses IExternalSystemRepository for external system and database connection definitions.
• Notification Service: Uses INotificationRepository for notification lists and SMTP configuration.
• Central UI: Uses ICentralUiRepository for read-oriented queries across domain areas, including audit log queries for the audit log viewer.
• All central components that modify state: Call IAuditService.LogAsync() after successful operations to record audit entries within the same transaction.
• Host: Provides database connection configuration. Registers DbContext, repository implementations, and IAuditService implementation in the DI container. Triggers auto-migration in development or validates schema version in production.