# SCADA System - High Level Requirements

## 1. Deployment Architecture

- **Site Clusters**: 2-node failover clusters deployed at each site, running on Windows.
- **Central Cluster**: A single 2-node failover cluster serving as the central hub.
- **Communication Topology**: Hub-and-spoke. Central cluster communicates with each site cluster. Site clusters do **not** communicate with one another.

### 1.1 Central vs. Site Responsibilities
- **Central cluster** is the single source of truth for all template authoring, configuration, and deployment decisions.
- **Site clusters** receive **flattened configurations** — fully resolved attribute sets with no template structure. Sites do not need to understand templates, inheritance, or composition.
- Sites **do not** support local/emergency configuration overrides. All configuration changes originate from central.

### 1.2 Failover
- Failover is managed at the **application level** using **Akka.NET** (not Windows Server Failover Clustering).
- Each cluster (central and site) runs an **active/standby** pair where Akka.NET manages node roles and failover detection.
- **Site failover**: The standby node takes over data collection and script execution seamlessly, including responsibility for the store-and-forward buffers. The Site Runtime Deployment Manager singleton is restarted on the new active node, which reads deployed configurations from local SQLite and re-creates the full Instance Actor hierarchy.
- **Central failover**: The standby node takes over central responsibilities. Deployments that are in-progress during a failover are treated as **failed** and must be re-initiated by the engineer.

### 1.3 Store-and-Forward Persistence (Site Clusters Only)
- Store-and-forward applies **only at site clusters** — the central cluster does **not** buffer messages. If a site is unreachable, operations from central fail and must be retried by the engineer.
- All site-level store-and-forward buffers (external system calls, notifications, and cached database writes) are **replicated between the two site cluster nodes** using **application-level replication** over Akka.NET remoting.
- The **active node** persists buffered messages to a **local SQLite database** and forwards them to the standby node, which maintains its own local SQLite copy.
- On failover, the standby node already has a replicated copy of the buffer and takes over delivery seamlessly.
- Successfully delivered messages are removed from both nodes' local stores.
- There is **no maximum buffer size** — messages accumulate until they either succeed or exhaust retries and are parked.
- Retry intervals are **fixed** (not exponential backoff). The fixed interval is sufficient for the expected use cases.

### 1.4 Deployment Behavior
- When central deploys a new configuration to a site instance, the site **applies it immediately** upon receipt — no local operator confirmation is required.
- If a site loses connectivity to central, it **continues operating** with its last received deployed configuration.
- The site reports back to central whether deployment was successfully applied.
- **Pre-deployment validation**: Before any deployment is sent to a site, the central cluster performs comprehensive validation including flattening the configuration, test-compiling all scripts, verifying alarm trigger references, verifying script trigger references, and checking data connection binding completeness (see Section 3.11).

### 1.5 System-Wide Artifact Deployment
- Changes to shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, and SMTP configuration are **not automatically propagated** to sites.
- Deployment of system-wide artifacts requires **explicit action** by a user with the **Deployment** role.
- Artifacts can be deployed to **all sites at once** or to an **individual site** (per-site deployment).
- The Design role manages the definitions; the Deployment role triggers deployment to sites. A user may hold both roles.

## 2. Data Storage & Data Flow

### 2.1 Central Databases (MS SQL)
- **Configuration Database**: A dedicated database for system-specific configuration data (e.g., templates, site definitions, instance configurations, system settings).
- **Machine Data Database**: A separate database for collected machine data (e.g., telemetry, measurements, events).

### 2.2 Communication: Central ↔ Site
- Two transport layers are used for central-site communication:
  - **Akka.NET ClusterClient/ClusterClientReceptionist**: Handles **command/control** messaging — deployments, instance lifecycle commands, subscribe/unsubscribe handshake, debug snapshots, health reports, remote queries, and integration routing. Provides automatic failover between contact points.
  - **gRPC server-streaming (site→central)**: Handles **real-time data streaming** — attribute value updates and alarm state changes. Each site node hosts a **SiteStreamGrpcServer** on a dedicated HTTP/2 port (Kestrel, default port 8083). Central creates per-site **SiteStreamGrpcClient** instances to subscribe to site streams. gRPC provides HTTP/2 flow control and per-stream backpressure that ClusterClient lacks.
- **Site addressing**: Site Akka base addresses (NodeA and NodeB) and gRPC endpoints (GrpcNodeAAddress and GrpcNodeBAddress) are stored in the **Sites database table** and configured via the Central UI or CLI. Central creates a ClusterClient per site using both Akka addresses as contact points, and per-site gRPC clients using the gRPC addresses.
- **Central contact points**: Sites configure **multiple central contact points** (both central node addresses) for redundancy. ClusterClient handles failover between central nodes automatically.
- **Central as integration hub**: Central brokers requests between external systems and sites. For example, a recipe manager sends a recipe to central, which routes it to the appropriate site. MES requests machine values from central, which routes the request to the site and returns the response.
- **Real-time data streaming** is not continuous for all machine data. The only real-time stream is an **on-demand debug view** — an engineer in the central UI can open a live view of a specific instance's tag values and alarm states for troubleshooting purposes. This is session-based and temporary. The debug view subscribes via gRPC to the site's SiteStreamManager filtered by instance (see Section 8.1).

### 2.3 Site-Level Storage & Interface
- Sites have **no user interface** — they are headless collectors, forwarders, and script executors.
- Sites require local storage for: the current deployed (flattened) configurations, deployed scripts, shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, and SMTP configuration.
- After artifact deployment, sites are **fully self-contained** — all runtime configuration is read from local SQLite. Sites do **not** access the central configuration database at runtime.
- Store-and-forward buffers are persisted to a **local SQLite database on each node** and replicated between nodes via application-level replication (see 1.3).

### 2.4 Data Connection Protocols
- The system supports **OPC UA** and **LmxProxy** (a gRPC-based custom protocol with an existing client SDK).
- Both protocols implement a **common interface** supporting: connect, subscribe to tag paths, receive value updates, and write values.
- Additional protocols can be added by implementing the common interface.
- The Data Connection Layer is a **clean data pipe** — it publishes tag value updates to Instance Actors but performs no evaluation of triggers or alarm conditions.
- **Initial attribute quality**: Attributes bound to a data connection start with **uncertain** quality when the Instance Actor initializes. The quality remains uncertain until the first value update is received from the Data Connection Layer. This distinguishes "never received a value" from "received a known-good value" or "connection lost" (bad quality).
- Data connections support optional **backup endpoints** with automatic failover after a configurable retry count. On failover, all subscriptions are transparently re-created on the new endpoint.

### 2.5 Scale
- Approximately **10 sites**.
- **50–500 machines per site**.
- **25–75 live data point tags per machine**.

## 3. Template & Machine Modeling

### 3.1 Template Structure
- Machines are modeled as **instances of templates**.
- Templates define a set of **attributes**.
- Each attribute has a **lock flag** that controls whether it can be overridden downstream.

### 3.2 Attribute Definition
Each attribute carries the following metadata:
- **Name**: Identifier for the attribute.
- **Value**: The default or configured value. May be empty if intended to be set at the instance level.
- **Data Type**: The value's type. Fixed set: Boolean, Integer, Float, String.
- **Lock Flag**: Controls whether the attribute can be overridden downstream.
- **Description**: Human-readable explanation of the attribute's purpose.
- **Data Source Reference** *(optional)*: A **relative path** within a data connection (e.g., `/Motor/Speed`). The template defines *what* to read — the path relative to a data connection. The template does **not** specify which data connection to use; that is an instance-level concern (see Section 3.3). Attributes without a data source reference are static configuration values.

### 3.3 Data Connections
- **Data connections** are reusable, named resources defined centrally and then **assigned to specific sites** (e.g., an OPC server, a PLC endpoint). Data connection definitions are deployed to sites as part of **artifact deployment** (see Section 1.5) and stored in local SQLite.
- A data connection encapsulates the details needed to communicate with a data source (protocol, address, credentials, etc.).
- Attributes with a data source reference must be **bound to a data connection at instance creation** — the template defines *what* to read (the relative path), and the instance specifies *where* to read it from (the data connection assigned to the site).
- **Binding is per-attribute**: Each attribute with a data source reference individually selects its data connection. Different attributes on the same instance may use different data connections. The Central UI supports bulk assignment (selecting multiple attributes and assigning a data connection to all of them at once) to reduce tedium.
- Templates do **not** specify a default connection. The connection binding is an instance-level concern.
- The flattened configuration sent to a site resolves connection references into concrete connection details paired with attribute relative paths.
- Data connection names are **not** standardized across sites — different sites may have different data connection names for equivalent devices.

### 3.4 Alarm Definitions
Alarms are **first-class template members** alongside attributes and scripts, following the same **inheritance, override, and lock rules**.

Each alarm has:
- **Name**: Identifier for the alarm.
- **Description**: Human-readable explanation of the alarm condition.
- **Priority Level**: Numeric value from 0–1000.
- **Lock Flag**: Controls whether the alarm can be overridden downstream.
- **Trigger Definition**: One of the following trigger types:
  - **Value Match**: Triggers when a monitored attribute equals a predefined value.
  - **Range Violation**: Triggers when a monitored attribute value falls outside an allowed range.
  - **Rate of Change**: Triggers when a monitored attribute value changes faster than a defined threshold.
- **On-Trigger Script** *(optional)*: A script to execute when the alarm triggers. The alarm on-trigger script executes in the context of the instance and can call instance scripts, but instance scripts **cannot** call alarm on-trigger scripts. The call direction is one-way.

### 3.4.1 Alarm State
- Alarm state (active/normal) is **managed at the site level** per instance, held **in memory** by the Alarm Actor.
- When the alarm condition clears, the alarm **automatically returns to normal state** — no acknowledgment workflow is required.
- Alarm state is **not persisted** — on restart, alarm states are re-evaluated from incoming values.
- Alarm state changes are published to the site-wide Akka stream as `[InstanceUniqueName].[AlarmName]`, alarm state (active/normal), priority, timestamp.

### 3.5 Template Relationships

Templates participate in two distinct relationship types:

- **Inheritance (is-a)**: A child template extends a parent template. The child inherits all attributes, alarms, scripts, and composed feature modules from the parent. The child can:
  - Override the **values** of non-locked inherited attributes, alarms, and scripts.
  - **Add** new attributes, alarms, or scripts not present in the parent.
  - **Not** remove attributes, alarms, or scripts defined by the parent.
- **Composition (has-a)**: A template can nest an instance of another template as a **feature module** (e.g., embedding a RecipeSystem module inside a base machine template). Feature modules can themselves compose other feature modules **recursively**.
- **Naming collisions**: If a template composes two feature modules that each define an attribute, alarm, or script with the same name, this is a **design-time error**. The system must detect and report the collision, and the template cannot be saved until the conflict is resolved.

### 3.6 Locking
- Locking applies to **attributes, alarms, and scripts** uniformly.
- Any of these can be **locked** at the level where it is defined or overridden.
- A locked attribute **cannot** be overridden by any downstream level (child templates, composing templates, or instances).
- An unlocked attribute **can** be overridden by any downstream level.
- **Intermediate locking**: Any level in the chain can lock an attribute that was unlocked upstream. Once locked, it remains locked for all levels below — a downstream level **cannot** unlock an attribute locked above it.

### 3.6 Attribute Resolution Order
Attributes are resolved from most-specific to least-specific. The first value encountered wins:

1. **Instance** (site-deployed machine)
2. **Child Template** (most derived first, walking up the inheritance chain)
3. **Composing Template** (the template that embeds a feature module can override the module's attributes)
4. **Composed Module** (the original feature module definition, recursively resolved if modules nest other modules)

At any level, an override is only permitted if the attribute has **not been locked** at a higher-priority level.

### 3.7 Override Scope
- **Inheritance**: Child templates can override non-locked attributes from their parent, including attributes originating from composed feature modules.
- **Composition**: A template that composes a feature module can override non-locked attributes within that module.
- Overrides can "pierce" into composed modules — a child template can override attributes inside a feature module it inherited from its parent.

### 3.8 Instance Rules
- An instance is a deployed occurrence of a template at a site.
- Instances **can** override the values of non-locked attributes.
- Instances **cannot** add new attributes.
- Instances **cannot** remove attributes.
- The instance's structure (which attributes exist, which feature modules are composed) is strictly defined by its template.
- Each instance is **assigned to an area** within its site (see 3.10).

### 3.8.1 Instance Lifecycle
- Instances can be in one of two states: **enabled** or **disabled**.
- **Enabled**: The instance is active at the site — data subscriptions, script triggers, and alarm evaluation are all running.
- **Disabled**: The site **stops** script triggers, data subscriptions (no live data collection), and alarm evaluation. The deployed configuration is **retained** on the site so the instance can be re-enabled without redeployment. Store-and-forward messages for a disabled instance **continue to drain** (deliver pending messages).
- **Deletion**: Instances can be deleted. Deletion removes the running configuration from the site, stops subscriptions, and destroys the Instance Actor and its children. Store-and-forward messages are **not** cleared on deletion — they continue to be delivered or can be managed (retried/discarded) via parked message management. If the site is unreachable when a delete is triggered, the deletion **fails** (same behavior as a failed deployment). The central side does not mark it as deleted until the site confirms.
- Templates **cannot** be deleted if any instances or child templates reference them. The user must remove all references first.

### 3.9 Template Deployment & Change Propagation
- Template changes are **not** automatically propagated to deployed instances.
- The system maintains two views of each instance:
  - **Deployed Configuration**: The currently active configuration on the instance, as it was last explicitly deployed.
  - **Template-Derived Configuration**: The configuration the instance *would* have based on the current state of its template (including resolved inheritance, composition, and overrides).
- Deployment is performed at the **individual instance level** — an engineer explicitly commands the system to update a specific instance.
- The system must be able to **show differences** between the deployed configuration and the current template-derived configuration, allowing engineers to see what would change before deploying.
- **No rollback** support is required. The system only needs to track the current deployed state, not a history of prior deployments.
- **Concurrent editing**: Template editing uses a **last-write-wins** model. No pessimistic locking or optimistic concurrency conflict detection is required.

### 3.10 Areas
- Areas are **predefined hierarchical groupings** associated with a site, stored in the configuration database.
- Areas support **parent-child relationships** (e.g., Plant → Building → Production Line → Cell).
- Each instance is assigned to an area within its site.
- Areas are used for **filtering and finding instances** in the central UI.
- Area definitions are managed by users with the **Design** role.

### 3.11 Pre-Deployment Validation

Before any deployment is sent to a site, the central cluster performs **comprehensive validation**. Validation covers:

- **Flattening**: The full template hierarchy is resolved and flattened successfully.
- **Naming collision detection**: No duplicate attribute, alarm, or script names exist in the flattened configuration.
- **Script compilation**: All instance scripts and alarm on-trigger scripts are test-compiled and must compile without errors.
- **Alarm trigger references**: Alarm trigger definitions reference attributes that exist in the flattened configuration.
- **Script trigger references**: Script triggers (value change, conditional) reference attributes that exist in the flattened configuration.
- **Data connection binding completeness**: Every attribute with a data source reference has a data connection binding assigned on the instance, and the bound data connection name exists as a defined connection at the instance's site.
- **Exception**: Validation does **not** verify that data source relative paths resolve to real tags on physical devices — that is a runtime concern that can only be determined at the site.

Validation is also available **on demand in the Central UI** for Design users during template authoring, providing early feedback without requiring a deployment attempt.

For **shared scripts**, pre-compilation validation is performed before deployment to sites. Since shared scripts have no instance context, validation is limited to C# syntax and structural correctness.

## 4. Scripting

### 4.1 Script Definitions
- Scripts are **C#** and are defined at the **template level** as first-class template members.
- Scripts follow the same **inheritance, override, and lock rules** as attributes. A parent template can define a script, a child template can override it (if not locked), and any level can lock a script to prevent downstream changes.
- Scripts are deployed to sites as part of the flattened instance configuration.
- Scripts are **compiled at the site** when a deployment is received. Pre-compilation validation occurs at central before deployment (see Section 3.11), but the site performs the actual compilation for execution.
- Scripts can optionally define **input parameters** (name and data type per parameter). Scripts without parameter definitions accept no arguments.
- Scripts can optionally define a **return value definition** (field names and data types). Return values support **single objects** and **lists of objects**. Scripts without a return definition return void.
- Return values are used when scripts are called explicitly by other scripts (via `Instance.CallScript()` or `Scripts.CallShared()`) or by the Inbound API (via `Route.To().Call()`). When invoked by a trigger (interval, value change, conditional, alarm), any return value is discarded.

### 4.2 Script Triggers
Scripts can be triggered by:
- **Interval**: Execute on a recurring time schedule.
- **Value Change**: Execute when a specific instance attribute value changes.
- **Conditional**: Execute when an instance attribute value equals or does not equal a given value.

Scripts have an optional **minimum time between runs** setting. If a trigger fires before the minimum interval has elapsed since the last execution, the invocation is skipped.

### 4.3 Script Error Handling
- If a script fails (unhandled exception, timeout, etc.), the failure is **logged locally** at the site.
- The script is **not disabled** — it remains active and will fire on the next qualifying trigger event.
- Script failures are **not reported to central**. Diagnostics are local only.
- For external system call failures within scripts, store-and-forward handling (Section 5.3) applies independently of script error handling.

### 4.4 Script Capabilities
Scripts executing on a site for a given instance can:
- **Read** attribute values on that instance (live data points and static config).
- **Write** attribute values on that instance. For attributes with a data source reference, the write goes to the Data Connection Layer which writes to the physical device; the in-memory value updates when the device confirms the new value via the existing subscription. For static attributes, the write updates the in-memory value and **persists the override to local SQLite** — the value survives restart and failover. Persisted overrides are reset when the instance is redeployed.
- **Call other scripts** on that instance via `Instance.CallScript("scriptName", params)`. Calls use the Akka ask pattern and return the called script's return value. Script-to-script calls support concurrent execution.
- **Call shared scripts** via `Scripts.CallShared("scriptName", params)`. Shared scripts execute **inline** in the calling Script Actor's context — they are compiled code libraries, not separate actors.
- **Call external system API methods** in two modes: `ExternalSystem.Call()` for synchronous request/response, or `ExternalSystem.CachedCall()` for fire-and-forget with store-and-forward on transient failure (see Section 5).
- **Send notifications** (see Section 6).
- **Access databases** by requesting an MS SQL client connection by name (see Section 5.5).

Scripts **cannot** access other instances' attributes or scripts.

### 4.4.1 Script Call Recursion Limit
- Script-to-script calls (via `Instance.CallScript` and `Scripts.CallShared`) enforce a **maximum recursion depth** to prevent infinite loops.
- The default maximum depth is a reasonable limit (e.g., 10 levels).
- The current call depth is tracked and incremented with each nested call. If the limit is reached, the call fails with an error logged to the site event log.
- This applies to all script call chains including alarm on-trigger scripts calling instance scripts.

### 4.5 Shared Scripts
- Shared scripts are **not associated with any template** — they are a **system-wide library** of reusable C# scripts.
- Shared scripts can optionally define **input parameters** and **return value definitions**, following the same rules as template-level scripts.
- Managed by users with the **Design** role.
- Deployed to **all sites** for use by any instance script (deployment requires explicit action by a user with the Deployment role).
- Shared scripts execute **inline** in the calling Script Actor's context as compiled code. They are not separate actors. This avoids serialization bottlenecks and messaging overhead.
- Shared scripts are **not available on the central cluster** — Inbound API scripts cannot call them directly. To execute shared script logic, route to a site instance via `Route.To().Call()`.

### 4.6 Alarm On-Trigger Scripts
- Alarm on-trigger scripts are defined as part of the alarm definition and execute when the alarm activates.
- They execute directly in the Alarm Actor's context (via a short-lived Alarm Execution Actor), similar to how shared scripts execute inline.
- Alarm on-trigger scripts **can** call instance scripts via `Instance.CallScript()`, which sends an ask message to the appropriate sibling Script Actor.
- Instance scripts **cannot** call alarm on-trigger scripts — the call direction is one-way.
- The recursion depth limit applies to alarm-to-instance script call chains.

## 5. External System Integrations

### 5.1 External System Definitions
- External systems are **predefined contracts** created by users with the **Design** role.
- Each definition includes:
  - **Connection details**: Endpoint URL, authentication, protocol information.
  - **Method definitions**: Available API methods with defined parameters and return types.
- Definitions are deployed **uniformly to all sites** — no per-site connection detail overrides.
- Deployment of definition changes requires **explicit action** by a user with the Deployment role.
- At the site, external system definitions are read from **local SQLite** (populated by artifact deployment), not from the central config DB.

### 5.2 Site-to-External-System Communication
- Sites communicate with external systems **directly** (not routed through central).
- Scripts invoke external system methods by referencing the predefined definitions.

### 5.3 Store-and-Forward for External Calls
- If an external system is unavailable when a script invokes a method, the message is **buffered locally at the site**.
- Retry is performed **per message** — individual failed messages retry independently.
- Each external system definition includes configurable retry settings:
  - **Max retry count**: Maximum number of retry attempts before giving up.
  - **Time between retries**: Fixed interval between retry attempts (no exponential backoff).
- After max retries are exhausted, the message is **parked** (dead-lettered) for manual review.
- There is **no maximum buffer size** — messages accumulate until delivery succeeds or retries are exhausted.

### 5.4 Parked Message Management
- Parked messages are **stored at the site** where they originated.
- The **central UI** can **query sites** for parked messages and manage them remotely.
- Operators can **retry** or **discard** parked messages from the central UI.
- Parked message management covers **external system calls**, **notifications**, and **cached database writes**.

### 5.5 Database Connections
- Database connections are **predefined, named resources** created by users with the **Design** role.
- Each definition includes the connection details needed to connect to an MS SQL database (server, database name, credentials, etc.).
- Each definition includes configurable retry settings (same pattern as external systems): **max retry count** and **time between retries** (fixed interval).
- Definitions are deployed **uniformly to all sites** — no per-site overrides.
- Deployment of definition changes requires **explicit action** by a user with the Deployment role.
- At the site, database connection definitions are read from **local SQLite** (populated by artifact deployment), not from the central config DB.

### 5.6 Database Access Modes
Scripts can interact with databases in two modes:

- **Real-time (synchronous)**: Scripts request a **raw MS SQL client connection by name** (e.g., `Database.Connection("MES_DB")`), giving script authors full ADO.NET-level control for immediate queries and updates.
- **Cached write (store-and-forward)**: Scripts submit a write operation for deferred, reliable delivery. The cached entry stores the **database connection name**, the **SQL statement to execute**, and **parameter values**. If the database is unavailable, the write is buffered locally at the site and retried per the connection's retry settings. After max retries are exhausted, the write is **parked** for manual review (managed via central UI alongside other parked messages).

## 6. Notifications

### 6.1 Notification Lists
- Notification lists are **system-wide**, managed by users with the **Design** role.
- Each list has a **name** and contains one or more **recipients**.
- Each recipient has a **name** and an **email address**.
- Notification lists are deployed to **all sites** (deployment requires explicit action by a user with the Deployment role).
- At the site, notification lists and recipients are read from **local SQLite** (populated by artifact deployment), not from the central config DB.

### 6.2 Email Support
- The system has **predefined support for sending email** as the notification delivery mechanism.
- Email server configuration (SMTP settings) is defined centrally and deployed to all sites as part of **artifact deployment** (see Section 1.5). Sites read SMTP configuration from **local SQLite**.

### 6.3 Script API
- Scripts send notifications using a simplified API: `Notify.To("list name").Send("subject", "message")`
- This API is available to instance scripts, alarm on-trigger scripts, and shared scripts.

### 6.4 Store-and-Forward for Notifications
- If the email server is unavailable, notifications are **buffered locally at the site**.
- Follows the same retry pattern as external system calls: configurable **max retry count** and **time between retries** (fixed interval).
- After max retries are exhausted, the notification is **parked** for manual review (managed via central UI alongside external system parked messages).
- There is **no maximum buffer size** for notification messages.

## 7. Inbound API (Central)

### 7.1 Purpose
The system exposes a **web API on the central cluster** for external systems to call into the SCADA system. This is the counterpart to the outbound External System Integrations (Section 5) — where Section 5 defines how the system calls out, this section defines how external systems call in.

### 7.2 API Key Management
- API keys are stored in the **configuration database**.
- Each API key has a **name/label** (for identification), the **key value**, and an **enabled/disabled** flag.
- API keys are managed by users with the **Admin** role.

### 7.3 Authentication
- Inbound API requests are authenticated via **API key** (not LDAP/AD).
- The API key must be included with each request.
- Invalid or disabled keys are rejected.

### 7.4 API Method Definitions
- API methods are **predefined** and managed by users with the **Design** role.
- Each method definition includes:
  - **Method name**: Unique identifier for the endpoint.
  - **Approved API keys**: List of API keys authorized to call this method.
  - **Parameter definitions**: Name and data type for each input parameter.
  - **Return value definition**: Data type and structure of the response. Supports **single objects** and **lists of objects**.
  - **Timeout**: Configurable per method. Maximum execution time including routed calls to sites.
- The implementation of each method is a **C# script stored inline** in the method definition. It executes on the central cluster. No template inheritance — API scripts are standalone.
- API scripts can route calls to any instance at any site via `Route.To("instanceCode").Call("scriptName", parameters)`, read/write attributes in batch, and access databases directly.
- API scripts **cannot** call shared scripts directly (shared scripts are site-only). To invoke site logic, use `Route.To().Call()`.

### 7.5 Availability
- The inbound API is hosted **only on the central cluster** (active node).
- On central failover, the API becomes available on the new active node.

## 8. Central UI

The central cluster hosts a **configuration and management UI** (no live machine data visualization, except on-demand debug views). The UI supports the following workflows:

- **Template Authoring**: Create, edit, and manage templates including hierarchy (inheritance) and composition (feature modules). Author and manage scripts within templates. **Design-time validation** available on demand to check flattening, naming collisions, and script compilation without deploying.
- **Shared Script Management**: Create, edit, and manage the system-wide shared script library.
- **Notification List Management**: Create, edit, and manage notification lists and recipients.
- **External System Management**: Define external system contracts (connection details, API method definitions).
- **Database Connection Management**: Define named database connections for script use.
- **Inbound API Management**: Manage API keys (create, enable/disable, delete). Define API methods (name, parameters, return values, approved keys, implementation script). *(Admin role for keys, Design role for methods.)*
- **Instance Management**: Create instances from templates, bind data connections (per-attribute, with **bulk assignment** UI for selecting multiple attributes and assigning a data connection at once), set instance-level attribute overrides, assign instances to areas. **Disable** or **delete** instances.
- **Site & Data Connection Management**: Define sites (including optional NodeAAddress and NodeBAddress fields for Akka remoting paths, and optional GrpcNodeAAddress and GrpcNodeBAddress fields for gRPC streaming endpoints), manage data connections and assign them to sites.
- **Area Management**: Define hierarchical area structures per site for organizing instances.
- **Deployment**: View diffs between deployed and current template-derived configurations, deploy updates to individual instances. Filter instances by area. Pre-deployment validation runs automatically before any deployment is sent.
- **System-Wide Artifact Deployment**: Explicitly deploy shared scripts, external system definitions, database connection definitions, data connection definitions, notification lists, and SMTP configuration to all sites or to an individual site (requires Deployment role). Per-site deployment is available via the Sites admin page.
- **Deployment Status Monitoring**: Track whether deployments were successfully applied at site level.
- **Debug View**: On-demand real-time view of a specific instance's tag values and alarm states for troubleshooting (see 8.1).
- **Parked Message Management**: Query sites for parked messages (external system calls, notifications, and cached database writes), retry or discard them.
- **Health Monitoring Dashboard**: View site cluster health, node status, data connection health, script error rates, alarm evaluation errors, and store-and-forward buffer depths (see Section 11).
- **Site Event Log Viewer**: Query and view operational event logs from site clusters (see Section 12).

### 8.1 Debug View
- **Subscribe-on-demand**: When an engineer opens a debug view for an instance, central opens a **gRPC server-streaming subscription** to the site's `SiteStreamGrpcServer` for the instance, then requests a **snapshot** of all current attribute values and alarm states via ClusterClient. The gRPC stream delivers subsequent attribute value and alarm state changes directly from the site's `SiteStreamManager`.
- Attribute value stream messages are structured as: `[InstanceUniqueName].[AttributePath].[AttributeName]`, attribute value, attribute quality, attribute change timestamp.
- Alarm state stream messages are structured as: `[InstanceUniqueName].[AlarmName]`, alarm state (active/normal), priority, timestamp.
- The stream continues until the engineer **closes the debug view**, at which point central unsubscribes and the site stops streaming.
- No attribute/alarm selection — the debug view always shows all tag values and alarm states for the instance.
- No special concurrency limits are required.

## 9. Security & Access Control

### 9.1 Authentication
- **UI users** authenticate via **username/password** validated directly against **LDAP/Active Directory**. Sessions are maintained via JWT tokens.
- **External system API callers** authenticate via **API key** (see Section 7).

### 9.2 Authorization
- Authorization is **role-based**, with roles assigned by **LDAP group membership**.
- Roles are **independent** — they can be mixed and matched per user (via group membership). There is no implied hierarchy between roles.
- A user may hold multiple roles simultaneously (e.g., both Design and Deployment) by being a member of the corresponding LDAP groups.
- Inbound API authorization is per-method, based on **approved API key lists** (see Section 7.4).

### 9.3 Roles
- **Admin**: System-wide permission to manage sites, data connections, LDAP group-to-role mappings, API keys, and system-level configuration.
- **Design**: System-wide permission to author and edit templates, scripts, shared scripts, external system definitions, notification lists, inbound API method definitions, and area definitions.
- **Deployment**: Permission to manage instances (create, set overrides, bind connections, disable, delete) and deploy configurations to sites. Also triggers system-wide artifact deployment. Can be scoped **per site**.

### 9.4 Role Scoping
- Admin is always **system-wide**.
- Design is always **system-wide**.
- Deployment can be **system-wide** or **site-scoped**, controlled by LDAP group membership (e.g., `Deploy-SiteA`, `Deploy-SiteB`, or `Deploy-All`).

## 10. Audit Logging

Audit logging is implemented as part of the **Configuration Database** component via the `IAuditService` interface.

### 10.1 Storage
- Audit logs are stored in the **configuration MS SQL database** alongside system config data, enabling direct querying.
- Entries are **append-only** — never modified or deleted. No retention policy — retained indefinitely.

### 10.2 Scope
All system-modifying actions are logged, including:
- **Template changes**: Create, edit, delete templates.
- **Script changes**: Template script and shared script create, edit, delete.
- **Alarm changes**: Create, edit, delete alarm definitions.
- **Instance changes**: Create, override values, bind connections, area assignment, disable, enable, delete.
- **Deployments**: Who deployed what to which instance, and the result (success/failure).
- **System-wide artifact deployments**: Who deployed shared scripts / external system definitions / DB connections / data connections / notification lists / SMTP config, to which site(s), and the result.
- **External system definition changes**: Create, edit, delete.
- **Database connection changes**: Create, edit, delete.
- **Notification list changes**: Create, edit, delete lists and recipients.
- **Inbound API changes**: API key create, enable/disable, delete. API method create, edit, delete.
- **Area changes**: Create, edit, delete area definitions.
- **Site & data connection changes**: Create, edit, delete.
- **Security/admin changes**: Role mapping changes, site permission changes.

### 10.3 Detail Level
- Each audit log entry records the **state of the entity after the change**, serialized as JSON. Only the after-state is stored — change history is reconstructed by comparing consecutive entries for the same entity at query time.
- Each entry includes: **who** (authenticated user), **what** (action, entity type, entity ID, entity name), **when** (timestamp), and **state** (JSON after-state, null for deletes).
- **One entry per save operation** — when a user edits a template and changes multiple attributes in one save, a single entry captures the full entity state.

### 10.4 Transactional Guarantee
- Audit entries are written **synchronously** within the same database transaction as the change (via the unit-of-work pattern). If the change succeeds, the audit entry is guaranteed to be recorded. If the change rolls back, the audit entry rolls back too.

## 11. Health Monitoring

### 11.1 Monitored Metrics
The central cluster monitors the health of each site cluster, including:
- **Site cluster online/offline status**: Whether the site is reachable.
- **Active vs. standby node status**: Which node is active and which is standby.
- **Data connection health**: Connected/disconnected status per data connection at the site.
- **Script error rates**: Frequency of script failures at the site.
- **Alarm evaluation errors**: Frequency of alarm evaluation failures at the site.
- **Store-and-forward buffer depth**: Number of messages currently queued (broken down by external system calls, notifications, and cached database writes).

### 11.2 Reporting
- Site clusters **report health metrics to central** periodically.
- Health status is **visible in the central UI** — no automated alerting/notifications for now.

## 12. Site-Level Event Logging

### 12.1 Events Logged
Sites log operational events locally, including:
- **Script executions**: Start, complete, error (with error details).
- **Alarm events**: Alarm activated, alarm cleared (which alarm, which instance, when). Alarm evaluation errors.
- **Deployment applications**: Configuration received from central, applied successfully or failed. Script compilation results.
- **Data connection status changes**: Connected, disconnected, reconnected per connection.
- **Store-and-forward activity**: Message queued, delivered, retried, parked.
- **Instance lifecycle**: Instance enabled, disabled, deleted.

### 12.2 Storage
- Event logs are stored in **local SQLite** on each site node.
- **Retention policy**: 30 days. Events older than 30 days are automatically purged.

### 12.3 Central Access
- The central UI can **query site event logs remotely**, following the same pattern as parked message management — central requests data from the site over Akka.NET remoting.

## 13. Management Service & CLI

### 13.1 Management Service
- The central cluster exposes a **ManagementActor** that provides programmatic access to all administrative operations — the same operations available through the Central UI.
- The ManagementActor registers with Akka.NET **ClusterClientReceptionist** for cross-cluster access, and is also exposed via an HTTP Management API endpoint (`POST /management`) with Basic Auth, LDAP authentication, and role resolution — enabling external tools like the CLI to interact without Akka.NET dependencies.
- The ManagementActor enforces the **same role-based authorization** as the Central UI. Every incoming message carries the authenticated user's identity and roles.
- All mutating operations performed through the Management Service are **audit logged** via IAuditService, identical to operations performed through the Central UI.
- The ManagementActor runs on **every central node** (stateless). For HTTP API access, any central node can handle any request without sticky sessions.

### 13.2 CLI
- The system provides a standalone **command-line tool** (`scadalink`) for scripting and automating administrative operations.
- The CLI connects to the Central Host's HTTP Management API (`POST /management`) — it sends commands as JSON with HTTP Basic Auth credentials. The server handles LDAP authentication, role resolution, and ManagementActor dispatch.
- The CLI sends user credentials via HTTP Basic Auth. The server authenticates against **LDAP/AD** and resolves roles before dispatching commands to the ManagementActor.
- CLI commands mirror all Management Service operations: templates, instances, sites, data connections, deployments, external systems, notifications, security (API keys and role mappings), audit log queries, and health status.
- Output is **JSON by default** (machine-readable, suitable for scripting) with an optional `--format table` flag for human-readable tabular output.
- Configuration is resolved from command-line options, **environment variables** (`SCADALINK_MANAGEMENT_URL`, `SCADALINK_FORMAT`), or a **configuration file** (`~/.scadalink/config.json`).
- The CLI is a separate executable from the Host binary — it is deployed on any machine with HTTP access to a central node.

## 14. General Conventions

### 14.1 Timestamps
- All timestamps throughout the system are stored, transmitted, and processed in **UTC**.
- This applies to: attribute value timestamps, alarm state change timestamps, audit log entries, event log entries, deployment records, health reports, store-and-forward message timestamps, and all inter-node messages.
- Local time conversion for display is a **Central UI concern only** — no other component performs timezone conversion.

---

*All initial high-level requirements have been captured. This document will continue to be updated as the design evolves.*