# Component: Site Runtime

## Purpose

The Site Runtime component manages the execution of deployed machine instances at site clusters. It encompasses the actor hierarchy that represents running instances, their scripts, and their alarms. It owns the site-side deployment lifecycle (receiving configs from central, compiling scripts, creating actors), script execution, alarm evaluation, and the site-wide Akka stream for attribute and alarm state changes.

This component replaces the previously separate Script Engine and Alarm Engine concepts, unifying them under a single actor hierarchy rooted at the Deployment Manager singleton.

## Location

Site clusters only.

## Responsibilities

- Run the Deployment Manager singleton (Akka.NET cluster singleton) on the active site node.
- On startup (or failover), read all deployed configurations from local SQLite and re-create the full actor hierarchy.
- Receive deployment commands from central: new/updated instance configurations, instance lifecycle commands (disable, enable, delete), and system-wide artifact updates.
- Compile C# scripts when deployments are received.
- Manage the Instance Actor hierarchy (Instance Actors, Script Actors, Alarm Actors).
- Execute scripts via Script Actors with support for concurrent execution.
- Evaluate alarm conditions via Alarm Actors and manage alarm state.
- Maintain the site-wide Akka stream for attribute value and alarm state changes.
- Execute shared scripts inline as compiled code libraries (no separate actors).
- Enforce script call recursion limits.

---

## Actor Hierarchy

```mermaid
%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    DMS["Deployment Manager Singleton<br/>(Cluster Singleton)"]
    IA1["Instance Actor<br/>('MachineA-001')"]
    IA2["Instance Actor<br/>('MachineA-002')"]
    IAMORE["… more Instance Actors"]

    SA1["Script Actor ('MonitorSpeed')<br/>— coordinator"]
    SA2["Script Actor ('CalculateOEE')<br/>— coordinator"]
    AA1["Alarm Actor ('OverTemp')<br/>— coordinator (computed)"]
    AA2["Alarm Actor ('LowPressure')<br/>— coordinator (computed)"]
    NAA1["Native Alarm Actor ('OpcUaServer1')<br/>— read-only mirror, peer to Alarm Actor"]

    SEA1["Script Execution Actor<br/>— short-lived, per invocation"]
    SEA2["Script Execution Actor<br/>— short-lived, per invocation"]
    AEA1["Alarm Execution Actor<br/>— short-lived, per on-trigger invocation"]

    IA2CHILD["… (Script / Alarm Actors)"]

    DMS --> IA1
    DMS --> IA2
    DMS -.-> IAMORE

    IA1 --> SA1
    IA1 --> SA2
    IA1 --> AA1
    IA1 --> AA2
    IA1 --> NAA1

    SA1 --> SEA1
    SA2 --> SEA2
    AA1 --> AEA1

    IA2 -.-> IA2CHILD

    classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef warn fill:#ffe6cc,stroke:#d79b00,color:#111111;
    classDef bad fill:#f8cecc,stroke:#b85450,color:#111111;
    classDef alt fill:#e1d5e7,stroke:#9673a6,color:#111111;
    classDef muted fill:#f5f5f5,stroke:#999999,color:#666666;
    class DMS proc
    class IA1,IA2 start
    class SA1,SA2 dec
    class AA1,AA2 bad
    class NAA1 alt
    class SEA1,SEA2,AEA1 warn
    class IAMORE,IA2CHILD muted
```

---

## Deployment Manager Singleton

### Role
- Akka.NET **cluster singleton** — guaranteed to run on exactly one node in the site cluster (the active node).
- On failover, Akka.NET restarts the singleton on the new active node.

### Startup Behavior
1. Read all deployed configurations from local SQLite.
2. Read all shared scripts from local storage.
3. Compile all scripts (instance scripts, alarm on-trigger scripts, shared scripts).
4. Create Instance Actors for all deployed, **enabled** instances as child actors. Instance Actors are created in **staggered batches** (e.g., 20 at a time with a short delay between batches) to prevent a reconnection storm — 500 Instance Actors all registering data subscriptions simultaneously would overwhelm OPC UA servers and network capacity.
5. Make compiled shared script code available to all Script Actors.

### Deployment Handling
- Receives flattened instance configurations from central via the Communication Layer.
- Stores the new configuration in local SQLite.
- Compiles all scripts in the configuration.
- Creates a new Instance Actor (for new instances) or updates an existing one (for redeployments).
- For redeployments: the existing Instance Actor and all its children are stopped, then a new Instance Actor is created with the updated configuration. Subscriptions are re-established.
- Reports deployment result (success/failure) back to central.

### System-Wide Artifact Handling
- Receives updated shared scripts, external system definitions, database connection definitions, and data connection definitions from central. (Notification lists and SMTP configuration are central-only and are not deployed to sites — see Component-NotificationService.md.)
- Stores all artifacts in local SQLite. After artifact deployment, the site is fully self-contained — all runtime configuration is read from local SQLite with no access to the central configuration database.
- Recompiles shared scripts and makes updated code available to all Script Actors.

### Instance Lifecycle Commands
- **Disable**: Stops the Instance Actor and its children. Retains the deployed configuration in SQLite so the instance can be re-enabled without redeployment.
- **Enable**: Creates a new Instance Actor from the stored configuration (same as startup).
- **Delete**: Stops the Instance Actor and its children, removes the deployed configuration from local SQLite. Does **not** clear store-and-forward messages.

### Debug Snapshot Routing
- Receives `DebugSnapshotRequest` from the Communication Layer and forwards to the Instance Actor by unique name (same lookup as `SubscribeDebugViewRequest`).
- Returns an error response if no Instance Actor exists for the requested unique name (instance not deployed or not enabled).

### Server Certificate Trust (Per-Node Broadcast)
- The OPC UA trusted-peer certificate store is **node-local** — it lives in each node's PKI directory, so trusting a cert on only the active node would silently fail certificate validation after failover. To keep both nodes consistent, certificate trust is handled by a per-node actor and a broadcast.
- **`CertStoreActor`** is a per-node, **non-singleton** actor — it runs on **every** site node (unlike the Deployment Manager singleton). It owns its node's OPC UA PKI trusted-peer / rejected stores and handles three commands:
  - **`TrustServerCertCommand`** — write a base64/DER server certificate into the node's trusted-peer store (path-traversal-guarded thumbprints; the cert is decoded and persisted as a `.der`).
  - **`RemoveServerCertCommand`** — delete a certificate (by thumbprint) from the node's stores.
  - **`ListServerCertsCommand`** — enumerate the node's trusted-peer + rejected store contents (Subject / Issuer / Thumbprint / validity / status).
- The Deployment Manager singleton receives the central cert-trust verbs (relayed via the Communication Layer) and **broadcasts** `TrustServerCertCommand` / `RemoveServerCertCommand` to the `CertStoreActor` on **both** site nodes (via `ActorSelection`), so node-a and node-b PKI stores stay in sync. List reads from the local node. Trust is **site-local** — there is no central persistence of trusted certs (logged as a follow-up). The captured-but-untrusted server cert that seeds this flow comes from the DCL verify-endpoint probe (see Component-DataConnectionLayer.md), which never trusts on its own; the Central UI surfaces Trust / Remove (Admin-gated — see Component-CentralUI.md).

---

## Instance Actor

### Role
- **Single source of truth** for all runtime state of a deployed instance.
- Holds all attribute values (both static configuration values and live values from data connections).
- Holds current alarm states (active/normal), updated by child Alarm Actors.
- Publishes attribute value changes and alarm state changes to the site-wide Akka stream.

### Initialization
1. Load all attribute values from the flattened configuration (static defaults).
2. Set quality to **uncertain** for all attributes that have a data source reference. Static attributes (no data source reference) have quality **good**. The uncertain quality persists until the first value update arrives from the Data Connection Layer, distinguishing "not yet received" from "known good" or "connection lost."
3. Register data source references with the Data Connection Layer for subscriptions.
4. Create child Script Actors (one per script defined on the instance).
5. Create child Alarm Actors (one per alarm defined on the instance).

### Attribute Value Updates
- Receives tag value updates from the Data Connection Layer for attributes with data source references.
- Updates the in-memory attribute value.
- Notifies subscribed child Script Actors and Alarm Actors of the change.
- Publishes the change to the site-wide Akka stream.

### Stream Message Format
- **Attribute changes**: `[InstanceUniqueName].[AttributePath].[AttributeName]`, attribute value, attribute quality, attribute change timestamp.
- **Alarm state changes**: `[InstanceUniqueName].[AlarmName]`, alarm state (active/normal), priority, timestamp.

### GetAttribute / SetAttribute
- **GetAttribute**: Returns the current in-memory value for the requested attribute.
- **SetAttribute** (for attributes with data source reference): Sends a write request to the Data Connection Layer. The DCL writes to the physical device. If the write fails (connection down, device rejection, timeout), the error is returned synchronously to the calling script for handling. On success, the existing subscription picks up the confirmed value from the device and sends it back as a value update, which then updates the in-memory value. The in-memory value is **not** optimistically updated.
- **SetAttribute** (for static attributes): Updates the in-memory value and **persists the override to local SQLite**. On restart (or failover), the Instance Actor loads persisted overrides on top of the deployed configuration, preserving runtime-modified values. A redeployment of the instance resets all persisted overrides to the new deployed configuration values.

### Debug View Support
- On request from central (via Communication Layer), the Instance Actor provides a **snapshot** of all current attribute values and alarm states.
- Subsequent changes are delivered via the **SiteStreamManager** → **SiteStreamGrpcServer** → gRPC stream to central. The Instance Actor publishes attribute value and alarm state changes to the SiteStreamManager; it does not forward events directly to the Communication Layer.
- The Instance Actor also handles one-shot `DebugSnapshotRequest` messages: it builds the same snapshot (attribute values and alarm states) and replies directly to the sender. Unlike `SubscribeDebugViewRequest`, no subscriber is registered and no stream is established.

### Supervision Strategy

The Instance Actor supervises all child Script and Alarm Actors with explicit strategies:

| Child Actor | Exception Type | Strategy | Rationale |
|-------------|---------------|----------|-----------|
| Script Actor | Any exception | Resume | Script Actor is a coordinator — its state (trigger timers, last execution time) should survive child failures. Script Execution Actor failures are isolated. |
| Alarm Actor | Any exception | Resume | Alarm Actor holds alarm state. Resume preserves state and continues evaluation on next value update. |
| Script Execution Actor | Unhandled exception | Stop | Short-lived, per-invocation. Failure is logged; the Script Actor coordinator remains active for future triggers. |
| Alarm Execution Actor | Unhandled exception | Stop | Short-lived, per on-trigger invocation. Same as Script Execution Actor. |

The Deployment Manager singleton supervises Instance Actors with a **OneForOneStrategy** — one Instance Actor's failure does not affect other instances.

When the Instance Actor is stopped (due to disable, delete, or redeployment), Akka.NET automatically stops all child actors.

---

## Script Actor

### Role
- **Coordinator** for a single script definition on an instance.
- Holds the compiled script code and trigger configuration.
- Manages trigger evaluation (interval timer, value change detection, conditional evaluation).
- Spawns short-lived Script Execution Actors for each invocation.

### Trigger Management
- **Interval**: The Script Actor manages an internal timer. When the timer fires, it spawns a Script Execution Actor.
- **Value Change**: The Script Actor subscribes to attribute change notifications from its parent Instance Actor for the specific monitored attribute. When the attribute changes, it spawns a Script Execution Actor.
- **Conditional**: The Script Actor subscribes to attribute change notifications for the monitored attribute. On each update, it evaluates the condition (compares the attribute against a threshold). Firing depends on the **fire mode** (see below).
- **Expression**: The Script Actor evaluates a compiled boolean expression against an attribute snapshot on each attribute change. Firing depends on the **fire mode** (see below).
- **Fire mode (Conditional + Expression)**:
  - **OnTrue** (default): Conditional fires on each matching attribute change; Expression fires once per `false → true` transition (edge-triggered). This is the original behavior — a trigger configuration with no mode field is treated as OnTrue.
  - **WhileTrue**: On the `false → true` edge the script fires once, then re-fires on a periodic timer while the condition stays true; on the `true → false` edge the timer stops. The re-fire cadence is the script's **minimum time between runs**; with none configured the trigger degrades to the single edge fire and logs a warning.
- **Minimum time between runs**: If configured, the Script Actor tracks the last execution time and skips trigger invocations that fire before the minimum interval has elapsed. For a WhileTrue trigger it doubles as the re-fire cadence.

### Concurrent Execution
- Each invocation spawns a **new Script Execution Actor** as a child.
- Multiple Script Execution Actors can run concurrently (e.g., a trigger fires while a previous `Instance.CallScript` invocation is still running).
- The Script Actor coordinates but does not block on child completion.

### Script Execution Actor
- **Short-lived** child actor created per invocation.
- Receives: compiled script code, input parameters, reference to the parent Instance Actor, current call depth.
- Executes the script in the Akka actor context.
- Has access to the full Script Runtime API (see below).
- Returns the script's return value (if defined) to the caller, then stops.

### Handling `Instance.CallScript`
- When an external caller (another Script Execution Actor, an Alarm Execution Actor, or a routed call from the Inbound API) sends a `CallScript` message to the Script Actor, it spawns a Script Execution Actor to handle the call.
- The caller uses the **Akka ask pattern** and receives the return value when the execution completes.

---

## Alarm Actor

### Role
- **Coordinator** for a single alarm definition on an instance.
- Evaluates alarm trigger conditions against attribute value updates.
- Manages alarm state (active/normal) in memory.
- Executes on-trigger scripts when the alarm activates.

### Alarm Evaluation
- Subscribes to attribute change notifications from its parent Instance Actor for the attribute(s) referenced by its trigger definition.
- On each value update, evaluates the trigger condition:
  - **Value Match**: Incoming value equals the predefined target. Supports `"!=X"` prefix for not-equals semantics.
  - **Range Violation**: Value is outside the allowed min/max range.
  - **Rate of Change**: Value change rate exceeds the defined threshold over a configurable time window. Direction filter (rising / falling / either) restricts which side of the rate triggers.
  - **HiLo**: Multi-setpoint level alarm with up to four configurable setpoints (LoLo, Lo, Hi, HiHi). Any subset may be configured. Each setpoint may carry its own priority that overrides the alarm-level priority for that band.
- For binary trigger types (ValueMatch / RangeViolation / RateOfChange), when the condition is met and the alarm is currently in **normal** state, the alarm transitions to **active**:
  - Updates the alarm state on the parent Instance Actor (which publishes to the Akka stream).
  - If an on-trigger script is defined, spawns an Alarm Execution Actor to execute it.
- When the condition clears and the alarm is in **active** state, the alarm transitions to **normal**.
- For HiLo triggers, the actor tracks the current `AlarmLevel` (None / Low / LowLow / High / HighHigh). Each level transition emits a fresh `AlarmStateChanged` with the new level and its priority; level escalations (e.g., High → HighHigh) and de-escalations (HighHigh → High) both produce events. The on-trigger script fires only on the Normal → non-None edge, not on escalations between alarm bands.
- No script execution on clear in any trigger type.

### Alarm State
- Held **in memory** only — not persisted to SQLite. State comprises `AlarmState` (Active / Normal) and `AlarmLevel` (None for binary triggers; the active band for HiLo).
- On restart (or failover), alarm states are re-evaluated from incoming values. All alarms start in normal state with level None and transition based on incoming values.

### Alarm Execution Actor
- **Short-lived** child actor created when an on-trigger script needs to execute.
- Same pattern as Script Execution Actor — receives compiled code, executes, returns, and stops.
- Has access to the Instance Actor for `GetAttribute`/`SetAttribute`.
- **Can** call instance scripts via `Instance.CallScript()` — sends an ask message to the appropriate sibling Script Actor.
- Instance scripts **cannot** call alarm on-trigger scripts — the call direction is one-way.

---

## Native Alarm Actor

### Role
- **Read-only mirror** of alarms raised natively by an external source — OPC UA Alarms & Conditions (A&C) servers and the MxAccess Gateway — surfaced into the Site Runtime alongside the alarms ScadaBridge computes itself.
- Created as a child of the **Instance Actor** and is a **peer to the computed Alarm Actor** (not a child of it). One `NativeAlarmActor` is spawned per resolved native alarm source binding on the instance.
- Mirrors source-of-truth condition state into the Instance Actor's view and onto the site-wide stream; it **does not** acknowledge, clear, or otherwise write back to the source. There is no ack-back path — the external source remains authoritative.

### Construction
- Constructed with `(ResolvedNativeAlarmSource source, string instanceName, IActorRef instanceActor, IActorRef dclManager, SiteStorageService storage, SiteRuntimeOptions options, ILogger logger, AlarmKind nativeKind = NativeOpcUa)`.
- `nativeKind` distinguishes the two native flavors and stamps the `Kind` on every emitted `AlarmStateChanged`. The Instance Actor selects it from the bound connection's protocol (see **Instance Actor wiring** below).

### Lifecycle & Subscription
- **PreStart**: rehydrates any previously mirrored conditions for this source from the site SQLite `native_alarm_state` table, then subscribes to the source through the Data Connection Layer by sending a `SubscribeAlarmsRequest` to the DCL manager. The DCL routes the subscription to the bound connection's `IAlarmSubscribableConnection` implementation.
- **Failed subscribe**: schedules a retry timer at `NativeAlarmRetryIntervalMs` and re-attempts until the subscription is established. Rehydrated state remains visible in the meantime.
- **`NativeAlarmSourceUnavailable`**: the source connection has dropped. The actor **retains its last-known mirrored conditions** but marks them uncertain rather than purging them, so a transient disconnect does not flap every condition to normal. The set is reconciled against truth by the next reconnect snapshot.

### Transition Handling (`NativeAlarmTransitionUpdate`)
- **Snapshot / SnapshotComplete (reconnect reconciliation)**: `Snapshot` updates buffer into a staging set; `SnapshotComplete` performs an **atomic swap** of the mirrored set with the staged set. Any condition that was previously mirrored but is **not present** in the new snapshot emits a return-to-normal `AlarmStateChanged` and drops out. This is how the mirror self-corrects after an outage.
- **Live transitions** (`Raise` / `Ack` / `Clear` / `Retrigger` / `StateChange`): upsert the condition by `SourceReference`. Updates carrying a `TransitionTime` **older** than the currently held transition are ignored (out-of-order protection). Accepted transitions persist to SQLite and emit an enriched `AlarmStateChanged` upward to the Instance Actor.
- **Retention**: a mirrored condition is dropped once it is both inactive **and** acknowledged (`!Active && Acknowledged`) — the alarm has fully run its course at the source and no longer needs mirroring. The drop emits a final state change and deletes the SQLite row.
- **Per-source cap**: at most `MirroredAlarmCapPerSource` conditions are retained per source. When the cap is exceeded the **oldest** condition is dropped and the eviction is **logged** — there is no silent truncation.

### Persistence
- Mirrored condition state is persisted to the site SQLite `native_alarm_state` table on every accepted transition and removed on drop-out.
- Persistence is **best-effort / fire-and-forget**: a persistence failure is logged but never blocks the actor's mailbox and never aborts the upward `AlarmStateChanged` emit. The in-memory mirror remains authoritative for the running actor; SQLite exists to survive failover.

### Supervision & Restart
- Supervised by the Instance Actor under the same **OneForOneStrategy** as the computed Alarm Actor — a native source fault is isolated to its own actor.
- On site restart or failover, the actor rehydrates its mirror from `native_alarm_state` in PreStart, then reconciles against the source via the reconnect snapshot. Native mirror state therefore **survives failover** (unlike computed alarm state, which is re-evaluated from values).
- Mirrored native state **is cleared on redeploy/undeploy** of the instance (mirroring the static-override reset): the stale rows for the instance are removed and the fresh actor re-subscribes from a clean slate.

---

## Instance Actor — Native Alarm Wiring

The Instance Actor owns native-alarm setup alongside its computed Script and Alarm Actors:

- **Spawning**: for each entry in `_configuration.NativeAlarmSources`, the Instance Actor spawns a `NativeAlarmActor`. Spawning is **skipped when there is no DCL manager** (e.g., debug/test contexts with no data connections), since native alarms require a live source subscription.
- **Kind derivation**: the `AlarmKind` passed to each `NativeAlarmActor` is derived from the bound connection's protocol — `Mx*` protocols → `NativeMxAccess`, otherwise → `NativeOpcUa`.
- **Latest-event retention**: the Instance Actor retains the latest enriched `AlarmStateChanged` per alarm name in `_latestAlarmEvents`. The DebugView snapshot is built from this map so it carries the **unified condition view plus native metadata** for both computed and native alarms. Computed alarms that have not yet produced an event fall back to a **Normal projection** so the snapshot is complete.
- **Idle native source binding placeholders**: `BuildAlarmStatesSnapshot()` additionally emits one placeholder `AlarmStateChanged` for each configured native alarm source binding that currently has **no live conditions** in `_latestAlarmEvents`. The placeholder has `IsConfiguredPlaceholder = true` and carries the binding's canonical name in `NativeSourceCanonicalName`. The Instance Actor maintains a `_nativeAlarmKinds` map (`sourceCanonicalName → AlarmKind`) populated at spawn time to stamp the correct `Kind` on each placeholder. This ensures the Debug View Alarms tab shows every configured binding as a tree node even when quiet.
- **`NativeSourceCanonicalName` on live events**: every `AlarmStateChanged` emitted by a `NativeAlarmActor` stamps its source binding's canonical name in `NativeSourceCanonicalName`. The Debug View uses this field to place each live condition under the correct native-source binding node in the tree.
- **Reset semantics**: `_latestAlarmEvents` and the mirrored native state are cleared on redeploy/undeploy (same trigger as static-override reset) but rehydrate from SQLite on failover.

---

## Native Alarm State Persistence (Site SQLite)

`SiteStorageService` gains a `native_alarm_state` table backing the native mirror:

- **Primary key**: `(instance_unique_name, source_canonical_name, source_reference)` — one row per mirrored condition.
- **Columns**: `condition_json` (the serialized `AlarmConditionState`) and `last_transition_at` (the accepted `TransitionTime`).
- **Operations**: `Upsert` (on accepted transition), `Delete` (on condition drop-out), `Get` (PreStart rehydrate, scoped to instance + source), and `ClearForInstance` (redeploy/undeploy reset).
- This is a **peer SQLite store** to the existing deployed-configuration, store-and-forward, operation-tracking, and `AuditLog` stores. Unlike computed alarm state, native mirror state is intentionally persisted so it survives failover.

---

## Enriched `AlarmStateChanged` Message

The `AlarmStateChanged` message published by both Alarm Actors and Native Alarm Actors was extended **additively** (existing consumers keep working with computed defaults):

- **`Kind`** (`AlarmKind`): `Computed` for ScadaBridge-evaluated alarms; `NativeOpcUa` / `NativeMxAccess` for mirrored native alarms.
- **`Condition`** (`AlarmConditionState`): the unified condition view. Computed alarms supply a computed default; native alarms carry the mirrored source condition.
- **Native metadata** (populated for native alarms; defaulted/empty for computed): `SourceReference`, `AlarmTypeName`, `Category`, `OperatorUser`, `OperatorComment`, `OriginalRaiseTime`, `CurrentValue`, `LimitValue`.
- **Computed-alarm projection**: computed alarms are surfaced as **auto-acknowledged** with `Severity = Priority`, so a single enriched shape carries both computed and native alarms onto the stream and into the DebugView snapshot.
- **`NativeSourceCanonicalName`** (`string?`): the canonical name of the native alarm source binding a native condition belongs to (e.g. `Motor1.NativeAlarms`). `null` for computed alarms. Stamped by the `NativeAlarmActor` on every emitted event; also carried on idle-binding placeholder rows emitted by the Instance Actor.
- **`IsConfiguredPlaceholder`** (`bool`): `true` for a synthetic row emitted by the Instance Actor to represent a configured native source binding that currently has no live conditions. These rows are never persisted or forwarded as operational alarms — they exist solely so the Debug View can render every configured binding as a tree node. `false` on all other rows.

Both fields are **additive-only** (init-only with defaults: `null` / `false`) and are mirrored additively on the gRPC `AlarmStateUpdate` proto as fields 22 (`native_source_canonical_name`, `string`) and 23 (`is_configured_placeholder`, `bool`). `StreamRelayActor` and `SiteStreamGrpcClient` pack/unpack these fields without touching existing fields, preserving backward compatibility across version-skew boundaries.

The enriched message flows Instance Actor → site-wide Akka stream → `SiteStreamManager` → `SiteStreamGrpcServer` and is streamed to central as the gRPC `AlarmStateUpdate` event (see [Component-Communication.md](Component-Communication.md)).

---

## Shared Script Library

- Shared scripts are compiled at the site when received from central.
- Compiled code is stored in memory and made available to all Script Actors.
- When a Script Execution Actor calls `Scripts.CallShared("scriptName", params)`, the shared script code executes **inline** in the Script Execution Actor's context — it is a direct method invocation, not an actor message.
- This avoids serialization bottlenecks since there is no shared script actor to contend for.
- Shared scripts have access to the same runtime API as instance scripts (GetAttribute, SetAttribute, external systems, notifications, databases).

---

## Script Lifecycle

All script types can be updated without restarting the cluster, but the mechanism differs per type.

### Instance Scripts and Alarm On-Trigger Scripts

- Compiled at deployment time when the Deployment Manager receives a flattened configuration and creates the Instance Actor hierarchy.
- **To update**: modify the script in the template, then redeploy the instance (`instance deploy --id <N>`).
- Redeployment stops the existing Instance Actor and all its children, creates a new Instance Actor, and recompiles all scripts and alarms from the updated configuration.
- There is no way to hot-reload a single instance script without redeploying the entire instance.

### Shared Scripts

- Compiled at the site when received from central via artifact deployment (`deploy artifacts`).
- The `SharedScriptLibrary` replaces its in-memory compiled code dictionary under a lock, making updated code immediately available to all Script Actors.
- **To update**: modify the shared script via the CLI or UI, then run `deploy artifacts` to push the change to sites.
- No instance redeployment is required — running instances pick up the new shared script code on the next `Scripts.CallShared()` invocation.

### Inbound API Method Scripts

- See Component-InboundAPI.md for the compilation and hot-reload lifecycle of API method scripts.

> **Warning**: Editing scripts via direct SQL does not trigger recompilation. The in-memory compiled script will remain stale until the next node restart or redeployment. Always use the CLI, Management API, or Central UI to modify scripts.

---

## Script Runtime API

Available to all Script Execution Actors and Alarm Execution Actors:

### Instance Attributes
- `Instance.GetAttribute("name")` — Read an attribute value from the parent Instance Actor.
- `Instance.SetAttribute("name", value)` — Write an attribute value. For data-connected attributes, writes to the DCL; for static attributes, updates in-memory and persists to local SQLite (survives restart/failover, reset on redeployment).

### Other Scripts
- `Instance.CallScript("scriptName", parameters)` — Send an ask message to a sibling Script Actor. The target Script Actor spawns a Script Execution Actor, executes, and returns the result. The call includes the current recursion depth.
- `Scripts.CallShared("scriptName", parameters)` — Execute shared script code inline (direct method invocation). The call includes the current recursion depth.

### External Systems
- `ExternalSystem.Call("systemName", "methodName", params)` — Synchronous HTTP call. Blocks until response or timeout. All failures return to script. Use when the script needs the result.
- `ExternalSystem.CachedCall("systemName", "methodName", params)` — Deferred delivery. Returns a `TrackedOperationId` tracking handle immediately rather than the response; the call is attempted immediately and, on transient failure, store-and-forwarded for retry. Use for outbound data pushes where deferred delivery is acceptable.
- The returned `TrackedOperationId` can be passed to `Tracking.Status(id)` (see **Operation Tracking** below) to observe delivery progress.

### Notifications
- `Notify.To("listName").Send("subject", "message")` — Send a notification via a named notification list. Generates a `TrackedOperationId` (GUID) locally and returns it immediately; the notification is store-and-forwarded to the central cluster, which owns delivery. The script never contacts SMTP. (`NotificationId` is the notification-domain name for this same `TrackedOperationId` type.)
- `Notify.Status("trackedOperationId")` — A thin alias of `Tracking.Status(id)` retained for the notification domain. Returns a status record (status, retry count, last error, key timestamps). While the notification is still in the site store-and-forward buffer the site answers locally (status `Forwarding`); once forwarded the query round-trips to central.

### Database Access
- `Database.Connection("connectionName")` — Obtain a raw MS SQL client connection (ADO.NET) for synchronous read/write.
- `Database.CachedWrite("connectionName", "sql", parameters)` — Submit a write operation for store-and-forward delivery. Returns a `TrackedOperationId` tracking handle immediately; pass it to `Tracking.Status(id)` to observe delivery progress.

### Operation Tracking
- `Tracking.Status("trackedOperationId")` — Returns a status record (status, retry count, last error, key timestamps) for any tracked operation: a cached external system call, a cached database write, or a notification. For cached calls and writes the answer is always site-local and authoritative — the site owns the operation tracking table. (`Notify.Status(...)` is a thin alias scoped to the notification domain.)

### Parameter Access
- `Parameters["key"]` — Raw dictionary access (returns `object?`, requires manual casting).
- `Parameters.Get<T>("key")` — Typed access with descriptive error messages. Throws `ScriptParameterException` if parameter is missing, null, or cannot be converted to `T`.
- `Parameters.Get<T?>("key")` — Nullable typed access. Returns `null` if parameter is missing, null, or cannot be converted.
- `Parameters.Get<T[]>("key")` — Array access. Converts a list parameter to a typed array. Throws on first unconvertible element.
- `Parameters.Get<List<T>>("key")` — List access. Converts a list parameter to a typed `List<T>`.
- Supported types: `bool`, `int`, `long`, `float`, `double`, `string`, `DateTime`.

### Recursion Limit
- Every script call (`Instance.CallScript` and `Scripts.CallShared`) increments a call depth counter.
- If the counter exceeds the maximum recursion depth (default: 10), the call fails with an error.
- The error is logged to the site event log.

---

## Script Trust Model

Scripts execute **in-process** with constrained access. The following restrictions are enforced at compilation and runtime:

- **Allowed**: Access to the Script Runtime API (GetAttribute, SetAttribute, CallScript, CallShared, ExternalSystem, Notify, Database, Tracking), standard C# language features, basic .NET types (collections, string manipulation, math, date/time). `System.Diagnostics.Stopwatch`, `Debug`, and `Activity` are permitted.
- **Forbidden**: File system access (`System.IO`), process spawning (`System.Diagnostics.Process`), threading (`System.Threading` — except `Tasks`, `CancellationToken`, and `CancellationTokenSource`), reflection (`System.Reflection`), all raw network access (`System.Net` — must use `ExternalSystem.Call`), native interop (`System.Runtime.InteropServices`, `Microsoft.Win32`), assembly loading, unsafe code, `dynamic`, `Activator`.
- **Execution timeout**: Configurable per-script maximum execution time. Exceeding the timeout cancels the script and logs an error.
- **Memory**: Scripts share the host process memory. No per-script memory limit, but the execution timeout prevents runaway allocations.

The forbidden-API policy is defined authoritatively in `ScriptTrustPolicy` (Script Analysis component, #25). `ScriptCompilationService.ValidateTrustModel` delegates to `ScriptTrustValidator.FindViolations` for the trust verdict; the Site Runtime also performs a `CSharpScript.Compile` against the real `ScriptGlobals` for execution. This is defence-in-depth static enforcement, not a true runtime sandbox.

### Script Trust Boundary Auditing

Every script-trust-boundary call (External System Gateway, Database layer, Notify) emits an `AuditEvent` to `IAuditWriter` (site-local SQLite append). Hot path; never fails the calling action; failures logged via the `SiteAuditWriteFailures` health metric (see [Component-HealthMonitoring.md](Component-HealthMonitoring.md)). The central audit mirror and event schema live in [Component-AuditLog.md](Component-AuditLog.md).

## Script Scoping Rules

- Scripts can only read/write attributes on **their own instance** (via the parent Instance Actor).
- Scripts can call other scripts on **their own instance** (via sibling Script Actors).
- Scripts can call **shared scripts** (inline execution).
- Scripts **cannot** access other instances' attributes or scripts.
- Alarm on-trigger scripts **can** call instance scripts; instance scripts **cannot** call alarm on-trigger scripts.

---

## Tell vs. Ask Usage

Per Akka.NET best practices, internal actor communication uses **Tell** (fire-and-forget with reply-to) for the hot path:

- **Tag value updates** (DCL → Instance Actor): Tell. High-frequency, no response needed.
- **Attribute change notifications** (Instance Actor → Script/Alarm Actors): Tell. Fan-out notifications.
- **Stream publishing** (Instance Actor → Akka stream): Tell. Fire-and-forget.

**Ask** is reserved for system boundaries where a synchronous response is needed:

- **`Instance.CallScript()`**: Ask pattern from Script Execution Actor to sibling Script Actor. The caller needs the return value. Acceptable because script calls are infrequent relative to tag updates.
- **`Route.To().Call()`**: Ask from Inbound API to site Instance Actor via Communication Layer. External caller needs a response.
- **Debug view snapshot**: Ask from Communication Layer to Instance Actor for initial state.

## Concurrency & Serialization

- The Instance Actor processes messages **sequentially** (standard Akka actor model). This means `SetAttribute` calls from concurrent Script Execution Actors are serialized at the Instance Actor, preventing race conditions on attribute state.
- Script Execution Actors may run concurrently, but all state mutations (attribute reads/writes, alarm state updates) are mediated through the parent Instance Actor's message queue.
- External side effects (external system calls, notifications, database writes) are not serialized — concurrent scripts may produce interleaved side effects. This is acceptable because each side effect is independent.

## SiteStreamManager and gRPC Integration

- The `SiteStreamManager` implements the `ISiteStreamSubscriber` interface, allowing the Communication Layer's `SiteStreamGrpcServer` to subscribe to the stream for cross-cluster delivery via gRPC.
- When a gRPC `SubscribeInstance` call arrives, the `SiteStreamGrpcServer` creates a `StreamRelayActor` and subscribes it to `SiteStreamManager` for the requested instance. Events flow from `SiteStreamManager` → `StreamRelayActor` → `Channel<SiteStreamEvent>` → gRPC response stream to central.
- The `SiteStreamManager` filters events by instance unique name and forwards matching events to all registered subscribers (both local debug consumers and gRPC relay actors).

## Site-Wide Stream Backpressure

- The site-wide Akka stream uses **per-subscriber buffering** with bounded buffers. Each subscriber (gRPC relay actors, future consumers) gets an independent buffer.
- If a subscriber falls behind (e.g., slow network on gRPC stream), its buffer fills and oldest events are dropped. This does not affect other subscribers or the publishing Instance Actors.
- Instance Actors publish to the stream with **fire-and-forget** semantics — publishing never blocks the actor.

## Error Handling

### Script Errors
- Unhandled exceptions and timeouts in Script Execution Actors are **logged locally** to the site event log.
- The Script Actor (coordinator) is **not affected** — it remains active for future trigger events.
- Script failures are **not reported to central** (except as aggregated error rate metrics via Health Monitoring).

### Alarm Evaluation Errors
- Errors during alarm condition evaluation are **logged locally** to the site event log.
- The Alarm Actor remains active and continues evaluating on subsequent value updates.
- Alarm evaluation error rates are reported to central via Health Monitoring.

### Script Compilation Errors
- If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied.
- The failure is reported back to central as a failed deployment.
- Note: Pre-deployment validation at central should catch compilation errors before they reach the site. Site-side compilation failures indicate an unexpected issue.

---

## Dependencies

- **Script Analysis (#25)**: `ScriptCompilationService.ValidateTrustModel` delegates the forbidden-API verdict to `ScriptTrustValidator.FindViolations`. The Site Runtime retains its own `CSharpScript.Compile` against the real `ScriptGlobals` for execution.
- **Data Connection Layer**: Provides tag value updates to Instance Actors. Receives write requests from Instance Actors. Also feeds Native Alarm Actors: connections implementing `IAlarmSubscribableConnection` (OPC UA A&C servers, MxAccess Gateway) deliver `NativeAlarmTransitionUpdate` events in response to a `SubscribeAlarmsRequest`, and signal `NativeAlarmSourceUnavailable` on connection loss.
- **Store-and-Forward Engine**: Handles reliable delivery for external system calls, cached database writes, and notifications submitted by scripts. For the notification category specifically, it forwards to the central cluster for delivery (not directly to SMTP). Owns the site-local operation tracking table that backs `Tracking.Status(id)`.
- **External System Gateway**: Provides external system method invocations for scripts.
- **Communication Layer**: Receives deployments and lifecycle commands from central. Handles debug view requests. Reports deployment results.
- **Site Event Logging**: Records script executions, alarm events, deployment events, instance lifecycle events.
- **Health Monitoring**: Reports script error rates and alarm evaluation error rates.
- **Local SQLite**: Persists deployed configurations, system-wide artifacts (external system definitions, database connection definitions, data connection definitions). Sites also maintain peer SQLite stores for the Store-and-Forward buffer, the site event log, the operation tracking table, and the site-local `AuditLog` (see [Component-AuditLog.md](Component-AuditLog.md)). The `AuditLog` file is purged on the same daily cadence as the others but respects the hard `ForwardState` invariant — rows still `Pending` forward are never purged, regardless of age.

## Interactions

- **Deployment Manager (central)**: Receives flattened configurations, system-wide artifact updates, and instance lifecycle commands.
- **Data Connection Layer**: Bidirectional — receives value updates, sends write-back commands.
- **Communication Layer**: Receives commands from central, sends deployment results, serves debug view data.
- **Store-and-Forward Engine**: Scripts route cached writes, notifications, and external system calls here.
- **Health Monitoring**: Periodically reports error rate metrics.
