# Component: Site Runtime

## Purpose

The Site Runtime component manages the execution of deployed machine instances at site clusters. It encompasses the actor hierarchy that represents running instances, their scripts, and their alarms. It owns the site-side deployment lifecycle (receiving configs from central, compiling scripts, creating actors), script execution, alarm evaluation, and the site-wide Akka stream for attribute and alarm state changes.

This component replaces the previously separate Script Engine and Alarm Engine concepts, unifying them under a single actor hierarchy rooted at the Deployment Manager singleton.

## Location

Site clusters only.

## Responsibilities

- Run the Deployment Manager singleton (Akka.NET cluster singleton) on the active site node.
- On startup (or failover), read all deployed configurations from local SQLite and re-create the full actor hierarchy.
- Receive deployment commands from central: new/updated instance configurations, instance lifecycle commands (disable, enable, delete), and system-wide artifact updates.
- Compile C# scripts when deployments are received.
- Manage the Instance Actor hierarchy (Instance Actors, Script Actors, Alarm Actors).
- Execute scripts via Script Actors with support for concurrent execution.
- Evaluate alarm conditions via Alarm Actors and manage alarm state.
- Maintain the site-wide Akka stream for attribute value and alarm state changes.
- Execute shared scripts inline as compiled code libraries (no separate actors).
- Enforce script call recursion limits.

---

## Actor Hierarchy

```mermaid
%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    DMS["Deployment Manager Singleton<br/>(Cluster Singleton)"]
    IA1["Instance Actor<br/>('MachineA-001')"]
    IA2["Instance Actor<br/>('MachineA-002')"]
    IAMORE["… more Instance Actors"]

    SA1["Script Actor ('MonitorSpeed')<br/>— coordinator"]
    SA2["Script Actor ('CalculateOEE')<br/>— coordinator"]
    AA1["Alarm Actor ('OverTemp')<br/>— coordinator (computed)"]
    AA2["Alarm Actor ('LowPressure')<br/>— coordinator (computed)"]
    NAA1["Native Alarm Actor ('OpcUaServer1')<br/>— read-only mirror, peer to Alarm Actor"]

    SEA1["script run<br/>— launched task, not an actor"]
    SEA2["script run<br/>— launched task, not an actor"]
    AEA1["alarm on-trigger run<br/>— launched task, not an actor"]

    IA2CHILD["… (Script / Alarm Actors)"]

    DMS --> IA1
    DMS --> IA2
    DMS -.-> IAMORE

    IA1 --> SA1
    IA1 --> SA2
    IA1 --> AA1
    IA1 --> AA2
    IA1 --> NAA1

    SA1 -.-> SEA1
    SA2 -.-> SEA2
    AA1 -.-> AEA1

    IA2 -.-> IA2CHILD

    classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef warn fill:#ffe6cc,stroke:#d79b00,color:#111111;
    classDef bad fill:#f8cecc,stroke:#b85450,color:#111111;
    classDef alt fill:#e1d5e7,stroke:#9673a6,color:#111111;
    classDef muted fill:#f5f5f5,stroke:#999999,color:#666666;
    class DMS proc
    class IA1,IA2 start
    class SA1,SA2 dec
    class AA1,AA2 bad
    class NAA1 alt
    class SEA1,SEA2,AEA1 warn
    class IAMORE,IA2CHILD muted
```

---

## Deployment Manager Singleton

### Role
- Akka.NET **cluster singleton** — guaranteed to run on exactly one node in the site cluster (the active node).
- On failover, Akka.NET restarts the singleton on the new active node.

### Startup Behavior
1. Read all deployed configurations from local SQLite.
2. Read all shared scripts from local storage.
3. Compile all scripts (instance scripts, alarm on-trigger scripts, shared scripts).
4. Create Instance Actors for all deployed, **enabled** instances as child actors. Instance Actors are created in **staggered batches** (e.g., 20 at a time with a short delay between batches) to prevent a reconnection storm — 500 Instance Actors all registering data subscriptions simultaneously would overwhelm OPC UA servers and network capacity.
5. Make compiled shared script code available to all Script Actors.

> **Startup config load is retried (S5)**: step 1's deployed-config read is best-effort against local SQLite, which can transiently fail (locked/busy). A failed load must **not** leave the site as a silent zero-instance node — it is re-attempted on a fixed interval until it succeeds, then the retry self-cancels.

> **Per-instance compilation during staggered startup — CLOSED (WP3.1)**: this was a deferred optimization (P6) affecting failover time-to-recover; each Instance Actor Roslyn-compiled its own scripts inside its own start, serialising a site's whole recovery behind compilation. Each staggered startup batch now runs a **pre-warm step**: before the batch's Instance Actors are created, the Deployment Manager compiles that batch's distinct script bodies and trigger expressions on a background task and pipes `BatchCompileWarmed` back, so every `PreStart` compile is a cache hit. Instance Actor start still calls the compilation service — it remains the correctness backstop — but in every warmed path those calls are memoised lookups. Cost: one extra message per batch.

> **The Roslyn `ScriptOptions` are process-static, and must stay that way**: the reference set backing every site-side compile is built once per process, never per compile. `ScriptOptions.WithReferences(Assembly[])` resolves each assembly through `MetadataReference.CreateFromFile`, which does **not** cache — each call mints a fresh `MetadataReference` owning an `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock`, an unmanaged copy of the assembly metadata that nothing disposes. Building the options per compile leaks native memory permanently: no GC reclaims it, and it is invisible to gcdump and to the managed allocation counters, so the node's working set grows without the GC heap growing. This was a live defect (a Site node at 2,885 MB working set with 150 MB of live GC heap, ~2,469 MB of it on the default process heap across ~6,700 undisposed `AssemblyMetadata` instances). Note this is orthogonal to the compile cache above — the cache dedupes *identical* script bodies, so it bounds nothing when the bodies differ. (It no longer clears wholesale on overflow: WP3.1 replaced that latency cliff with approximate-LRU batch eviction, see **Compile cache eviction** below.) Pinned by a reference-equality regression test rather than by watching memory.

> **Compile cache eviction is approximate LRU, not a wholesale clear (WP3.1)**: `SiteScriptCompileCache` is bounded at 1024 entries — deliberately smaller than the TemplateEngine verdict cache's 4096, because entries pin compiled assemblies rather than verdict strings. It used to `Clear()` **everything** on overflow. Instance scripts *and* trigger expressions share this cache, so on a site large enough to cross the bound every overflow discarded up to 1023 live compiled scripts and the next deploy or Instance-Actor start paid a full recompile storm **on actor threads**. Overflow now evicts only the oldest ⅛ (128 entries) by last-access stamp: one 1024-element scan instead of 1023 future recompiles, and hot entries survive. Recency is an `Interlocked` access sequence rather than a clock — deterministic for tests and immune to clock steps; hits update it lock-free, and only the (rare) eviction sweep takes a lock, double-checked so concurrent inserts do not stampede it. The sweep snapshots through `ConcurrentDictionary.ToArray()` and **not** LINQ over the dictionary: LINQ's `ToArray` picks the `ICollection.CopyTo` fast path, which throws when a concurrent insert lands between its Count read and its copy, and inserts are deliberately lock-free.

> **Static options were only half the fix — the compile must also use the shared caching metadata resolver**: the static `ScriptOptions` carry only the *direct* API-surface references (5 assemblies here), and each `script.Compile()` still binds their **transitive closure** (~105 assemblies on a site node). Every transitively-referenced assembly is resolved through the options' `MetadataReferenceResolver`, and Roslyn's default one has no cross-compilation cache — it calls `MetadataReference.CreateFromFile` afresh on **every compile**, minting the same undisposed `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock` triple per assembly per compiled script, pinned for the process lifetime by the compile cache. That per-compile *re-resolution* — not the per-compile options construction — was the dominant term (measured live: 21 site scripts holding 6,640 metadata objects, counts bit-identical before and after the static-options fix). `ScriptCompilationService` therefore also attaches the process-wide `CachingScriptMetadataResolver.Instance` (owned by the Script Analysis component, see `Component-ScriptAnalysis.md`) via `ScriptOptions.WithMetadataResolver`, so each distinct assembly is materialized once per process. The decorator returns exactly what the undecorated resolver returned — only object identity is de-duplicated — so compile results and the trust gate are unchanged.

### Deployment Handling
- Receives flattened instance configurations from central via the Communication Layer.
- Stores the new configuration in local SQLite.
- Compiles all scripts in the configuration.
- Creates a new Instance Actor (for new instances) or updates an existing one (for redeployments).
- For redeployments: the existing Instance Actor and all its children are stopped, then a new Instance Actor is created with the updated configuration. Subscriptions are re-established.
- Reports deployment result (success/failure) back to central.

> **Dead-child bookkeeping + two-signal deploy join (S6).** All Instance Actors are `Watch`ed; an unexpected `Terminated` evicts the actor ref from the singleton's map. **Deviation from the naive design (2026-07-09, see master-tracker "Plan deviations during Wave 1 PLAN-03 execution"):** deploy-success is **NOT** reported on persistence alone — the config store can commit before the Instance Actor's asynchronous `PreStart` throws. The design therefore uses a **two-signal join**: the Instance Actor sends a new `InstanceActorInitialized` readiness message at the end of a successful `PreStart`, and the Deployment Manager reports `Success` only once **both** persistence has committed **and** that readiness signal has arrived. An actor that dies during init never signals readiness → the `Terminated` fallback replies `Failed` and rolls the optimistic in-memory state back (the persisted-row rollback is deferred until the store commits so it cannot race the write).

### System-Wide Artifact Handling
- Receives updated shared scripts, external system definitions, database connection definitions, and data connection definitions from central. (Notification lists and SMTP configuration are central-only and are not deployed to sites — see Component-NotificationService.md.)
- Stores all artifacts in local SQLite. After artifact deployment, the site is fully self-contained — all runtime configuration is read from local SQLite with no access to the central configuration database.
- Recompiles shared scripts and makes updated code available to all Script Actors.
- **Reconciles deletions**: each non-null artifact list is the complete set for its class, so after upserting, any stored row absent from the set is deleted (the artifact was deleted centrally — there is no per-artifact delete command). A removed shared script is also unregistered from the compiled script library, and a removed data connection's live DCL connection actor is stopped. A `null` list means "field not shipped" and reconciles nothing.

### Instance Lifecycle Commands
- **Disable**: Stops the Instance Actor and its children. Retains the deployed configuration in SQLite so the instance can be re-enabled without redeployment.
- **Enable**: Creates a new Instance Actor from the stored configuration (same as startup).
- **Delete**: Stops the Instance Actor and its children, removes the deployed configuration from local SQLite. Does **not** clear store-and-forward messages.

### Debug Snapshot Routing
- Receives `DebugSnapshotRequest` from the Communication Layer and forwards to the Instance Actor by unique name (same lookup as `SubscribeDebugViewRequest`).
- Returns an error response if no Instance Actor exists for the requested unique name (instance not deployed or not enabled).

### Server Certificate Trust (Per-Node Broadcast)
- The OPC UA trusted-peer certificate store is **node-local** — it lives in each node's PKI directory, so trusting a cert on only the active node would silently fail certificate validation after failover. To keep both nodes consistent, certificate trust is handled by a per-node actor and a broadcast.
- **`CertStoreActor`** is a per-node, **non-singleton** actor — it runs on **every** site node (unlike the Deployment Manager singleton). It owns its node's OPC UA PKI trusted-peer / rejected stores and handles three commands:
  - **`TrustServerCertCommand`** — write a base64/DER server certificate into the node's trusted-peer store (path-traversal-guarded thumbprints; the cert is decoded and persisted as a `.der`).
  - **`RemoveServerCertCommand`** — delete a certificate (by thumbprint) from the node's stores.
  - **`ListServerCertsCommand`** — enumerate the node's trusted-peer + rejected store contents (Subject / Issuer / Thumbprint / validity / status).
- The Deployment Manager singleton receives the central cert-trust verbs (relayed via the Communication Layer) and **broadcasts** `TrustServerCertCommand` / `RemoveServerCertCommand` to the `CertStoreActor` on **both** site nodes (via `ActorSelection`), so node-a and node-b PKI stores stay in sync. List reads from the local node. Trust is **site-local** — there is no central persistence of trusted certs (logged as a follow-up). The captured-but-untrusted server cert that seeds this flow comes from the DCL verify-endpoint probe (see Component-DataConnectionLayer.md), which never trusts on its own; the Central UI surfaces Trust / Remove (Admin-gated — see Component-CentralUI.md).
- **Reconcile-on-join (UA1)**: a broadcast only reaches nodes that are Up at the time, so a node that was down when a cert was trusted would stay divergent forever. To close that gap, the Deployment Manager singleton subscribes to cluster `MemberUp` events; when a site node (re)joins, it reads its own trusted-peer store (`ExportLocalTrustedCerts` → `CertStoreActor`) and pushes each certificate to the joining node's `CertStoreActor` (additive union; each `WriteCertToLocalStore` is idempotent by thumbprint). **Removal** reconciliation and central persistence of trust decisions remain the documented follow-up — only additions converge automatically.

---

## Instance Actor

### Role
- **Single source of truth** for all runtime state of a deployed instance.
- Holds all attribute values (both static configuration values and live values from data connections).
- Holds current alarm states (active/normal), updated by child Alarm Actors.
- Publishes attribute value changes and alarm state changes to the site-wide Akka stream.

### Initialization
1. Load all attribute values from the flattened configuration (static defaults).
2. Set quality to **uncertain** for all attributes that have a data source reference. Static attributes (no data source reference) have quality **good**. The uncertain quality persists until the first value update arrives from the Data Connection Layer, distinguishing "not yet received" from "known good" or "connection lost."
3. Register data source references with the Data Connection Layer for subscriptions.
4. Create child Script Actors (one per script defined on the instance).
5. Create child Alarm Actors (one per alarm defined on the instance).

### Attribute Value Updates
- Receives tag value updates from the Data Connection Layer for attributes with data source references.
- Updates the in-memory attribute value.
- Notifies subscribed child Script Actors and Alarm Actors of the change.
- Publishes the change to the site-wide Akka stream.

### Stream Message Format
- **Attribute changes**: `[InstanceUniqueName].[AttributePath].[AttributeName]`, attribute value, attribute quality, attribute change timestamp.
- **Alarm state changes**: `[InstanceUniqueName].[AlarmName]`, alarm state (active/normal), priority, timestamp.

### GetAttribute / SetAttribute
- **GetAttribute**: Returns the current in-memory value for the requested attribute.
- **SetAttribute** (for attributes with data source reference): Sends a write request to the Data Connection Layer. The DCL writes to the physical device. If the write fails (connection down, device rejection, timeout), the error is returned synchronously to the calling script for handling. On success, the existing subscription picks up the confirmed value from the device and sends it back as a value update, which then updates the in-memory value. The in-memory value is **not** optimistically updated.
- **SetAttribute** (for static attributes): Updates the in-memory value and **persists the override to local SQLite**. On restart (or failover), the Instance Actor loads persisted overrides on top of the deployed configuration, preserving runtime-modified values. A redeployment of the instance resets all persisted overrides to the new deployed configuration values.

### Debug View Support
- On request from central (via Communication Layer), the Instance Actor provides a **snapshot** of all current attribute values and alarm states.
- Subsequent changes are delivered via the **SiteStreamManager** → **SiteStreamGrpcServer** → gRPC stream to central. The Instance Actor publishes attribute value and alarm state changes to the SiteStreamManager; it does not forward events directly to the Communication Layer.
- The Instance Actor also handles one-shot `DebugSnapshotRequest` messages: it builds the same snapshot (attribute values and alarm states) and replies directly to the sender. Unlike `SubscribeDebugViewRequest`, no subscriber is registered and no stream is established. When the request sets `AlarmsOnly` (WP2.3 — the central live alarm cache's seed/reconcile fan-out, which discards attribute rows), only the alarm half is built and `AttributeValues` comes back empty; the flag defaults to `false`, so the Debug View path is unchanged.

### Supervision Strategy

The Instance Actor supervises all child Script and Alarm Actors with explicit strategies:

| Child Actor | Exception Type | Strategy | Rationale |
|-------------|---------------|----------|-----------|
| Script Actor | Any exception | Resume | Script Actor is a coordinator — its state (trigger timers, last execution time) should survive run failures. |
| Alarm Actor | Any exception | Resume | Alarm Actor holds alarm state. Resume preserves state and continues evaluation on next value update. |

Script Actor and Alarm Actor have **no children of their own** since WP3.1 removed the per-run execution actors, so neither declares a supervision strategy any more. A run's own exceptions were always contained inside the run's try/catch (they never reached supervision); a failure of the *launch* is caught by the coordinator, which answers the caller and releases the run slot rather than letting the throw escalate to the Instance Actor.

The Deployment Manager singleton supervises Instance Actors with a **OneForOneStrategy** — one Instance Actor's failure does not affect other instances.

When the Instance Actor is stopped (due to disable, delete, or redeployment), Akka.NET automatically stops all child actors.

---

## Script Actor

### Role
- **Coordinator** for a single script definition on an instance.
- Holds the compiled script code and trigger configuration.
- Manages trigger evaluation (interval timer, value change detection, conditional evaluation).
- Launches script runs directly for each invocation (no per-run child actor since WP3.1).

### Trigger Management
- **Interval**: The Script Actor manages an internal timer. When the timer fires, it launches a run.
- **Value Change**: The Script Actor subscribes to attribute change notifications from its parent Instance Actor for the specific monitored attribute. When the attribute changes, it launches a run.
- **Conditional**: The Script Actor subscribes to attribute change notifications for the monitored attribute. On each update, it evaluates the condition (compares the attribute against a threshold). Firing depends on the **fire mode** (see below).
- **Expression**: The Script Actor evaluates a compiled boolean expression against an attribute snapshot on each attribute change. Firing depends on the **fire mode** (see below).
- **Fire mode (Conditional + Expression)**:
  - **OnTrue** (default): Conditional fires on each matching attribute change; Expression fires once per `false → true` transition (edge-triggered). This is the original behavior — a trigger configuration with no mode field is treated as OnTrue.
  - **WhileTrue**: On the `false → true` edge the script fires once, then re-fires on a periodic timer while the condition stays true; on the `true → false` edge the timer stops. The re-fire cadence is the script's **minimum time between runs**; with none configured the trigger degrades to the single edge fire and logs a warning.
- **Minimum time between runs**: If configured, the Script Actor tracks the last execution time and skips trigger invocations that fire before the minimum interval has elapsed. For a WhileTrue trigger it doubles as the re-fire cadence.

> **Per-attribute change routing (P2).** Rather than broadcasting every attribute change to all child Script/Alarm Actors, the Instance Actor routes each change only to the children that monitor that attribute (built from their trigger configs at spawn time); **Expression-trigger children, which read an attribute snapshot, receive all changes**. Each child still keeps its own trigger gate as defense-in-depth — the routing is an optimization, not the authority.

### Concurrent Execution
- Each invocation launches a **new run** on the script-execution scheduler.
- Multiple runs can execute concurrently (e.g., a trigger fires while a previous `Instance.CallScript` invocation is still running), up to `MaxConcurrentRunsPerScript` — beyond which the newest is shed (see below).
- The Script Actor coordinates but does not block on run completion.

### Script Run Launch (WP3.1 — replaces the Script Execution Actor)
- A run is launched **directly by the Script Actor** through `ScriptRunLauncher`. There is no per-run child actor.
  - The former `ScriptExecutionActor` / `AlarmExecutionActor` were already inert shells: neither declared a `Receive` handler (they executed from their constructor), neither had a `PostStop`, state, or stash, and neither's `IActorRef` was ever a message target — the whole lifecycle lived inside a detached task the actor never observed. What they cost was an actor cell, mailbox, and name registration **per run**, plus a per-spawn expression-tree `Props.Create`. Removing them changed no semantics; the run body moved verbatim into the shared launcher.
  - Everything the shells provided is preserved: exception and timeout containment, one DI scope per run disposed on every path, the site-event/health telemetry, the Ask reply, the completion notification (now the coordinator's own `Self`), and the audit `ExecutionId` / `ParentExecutionId` threading. A run still in flight when its Script Actor is stopped runs to completion and its completion message dead-letters, exactly as before — **stopping does NOT cancel in-flight runs**; redeploy/undeploy semantics are unchanged.
  - **Per-run Started/Completed site events are sampled, not unconditional (WP3.2).** The two Info rows an instance script run used to emit unconditionally were the dominant `site_events` writer under load (design memo: `docs/plans/2026-08-15-site-events-policy-design.md`). They are now **off by default** (`ScadaBridge:SiteRuntime:PerRunScriptEvents`, default `false`); a `ScriptRunSummaryRecorder` accumulates per-`(instance, script)` run counters from the same call sites and a site-only hosted flush service emits **one aggregate "script" Info row per interval** (`ScriptRunSummaryIntervalSeconds`, default 300s) with a top-50-scripts breakdown, capped so a pathological deployment cannot mint an oversized row — an idle interval emits nothing. `ScadaBridge:SiteRuntime:PerRunScriptEventScripts` restores the legacy per-run rows for named scripts (`"Instance/Script"` exact or `"Instance/*"` wildcard) for live debugging. All three keys are read from `IOptionsMonitor<SiteRuntimeOptions>` per run, so they are hot-togglable without a restart. **Error-level rows (timeout, failure, stuck-watchdog, recursion-limit) are unaffected — always emitted, regardless of this policy.** See Component-SiteEventLogging.md → Volume Policy for the full design.
  - One deliberate improvement: a failure of the *launch itself* (e.g. queueing onto a disposed scheduler) is caught by the Script Actor, which replies to the Ask caller and releases the run slot. The old per-run child's constructor throw was handled by a Stop supervision directive that sent no reply, leaving the caller to hang to its Ask timeout.
- The script body runs on the **dedicated `ScriptExecutionScheduler`** (a bounded set of dedicated threads), not the shared .NET thread pool, so blocking script I/O cannot starve the global pool or stall Akka dispatchers. The scheduler is **process-wide by default** (one pool per host), but each script/alarm actor takes it through an **optional injection seam** rather than reaching for the static directly: the Host injects nothing and gets the shared pool, while tests (or a future multi-site host) can hand an actor its own instance. The shared accessor also recreates a disposed pool rather than returning it, so a disposed scheduler can never silently poison later executions.
- **Pool sizing is instance-scaled and grow-only (WP3.1).** The pool was a fixed 8 threads regardless of load. It is now `clamp(max(ScriptExecutionThreadCount, ceil(enabledInstances / 8)), 1, ScriptExecutionMaxThreadCount)` — the existing `ScriptExecutionThreadCount` (default 8) becomes the **floor**, so configurations at or below 64 instances behave exactly as before, and the new `ScriptExecutionMaxThreadCount` (default 32) is the ceiling. Beyond the ceiling the per-script cap below is the real regulator. `DeploymentManagerActor.UpdateInstanceCounts` calls `EnsureCapacity` on every deploy / undeploy / enable / disable and once per staggered startup batch. Growth only: undeploying leaves idle threads, which cost nothing measurable and avoid drain/steal complexity.
- **The deadline is armed at enqueue, not at dequeue (WP3.1).** The run's timeout `CancellationTokenSource` is created on the actor thread *before* the body is queued, so queue wait consumes the script's own budget. A body that dequeues past its deadline **skips execution entirely** and takes the existing timeout path (site event, script-error counter, error reply, completion message): a saturated pool sheds stale work instead of running it late with a fresh full budget.
- **Concurrent runs per script are capped (WP3.1).** `MaxConcurrentRunsPerScript` (default 4) bounds runs in flight — queued or executing — for any one script or alarm on-trigger script. Over the cap the **newest** run is shed: the four already in flight are closest to their own deadlines and already charged against them, so nothing is ever reordered and no extra queue is needed (the scheduler's FIFO already is the queue). A shed increments `ISiteHealthCollector.IncrementScriptRunShed` (surfaced as `ScriptRunShedCount`), emits a `script`/`Warning` site event **rate-limited to one per script per minute** so a hot trigger cannot flood `site_events`, and — for an Ask-based `CallScript` — replies with an explicit error so a nested call or inbound-API route fails fast instead of hanging.
  - **The cap gates only depth-0 launches (review fix).** A trigger fire or a depth-0 `CallScript`/inbound-API route counts against `MaxConcurrentRunsPerScript`; a *nested* `Instance.CallScript` self-recursion (`callDepth > 0`) is exempt — it is already bounded by `MaxScriptCallDepth` instead, and the calling run is holding one of the four cap slots itself while it awaits the callee, so counting the nested launch too would spuriously shed legitimate self-recursion.

### Handling `Instance.CallScript`
- When an external caller (another script run, an alarm on-trigger run, or a routed call from the Inbound API) sends a `CallScript` message to the Script Actor, it launches a run to handle the call.
- The caller uses the **Akka ask pattern** and receives the return value when the execution completes.

---

## Alarm Actor

### Role
- **Coordinator** for a single alarm definition on an instance.
- Evaluates alarm trigger conditions against attribute value updates.
- Manages alarm state (active/normal) in memory.
- Executes on-trigger scripts when the alarm activates.

### Alarm Evaluation
- Subscribes to attribute change notifications from its parent Instance Actor for the attribute(s) referenced by its trigger definition.
- On each value update, evaluates the trigger condition:
  - **Value Match**: Incoming value equals the predefined target. Supports `"!=X"` prefix for not-equals semantics.
  - **Range Violation**: Value is outside the allowed min/max range.
  - **Rate of Change**: Value change rate exceeds the defined threshold over a configurable time window. Direction filter (rising / falling / either) restricts which side of the rate triggers.
  - **HiLo**: Multi-setpoint level alarm with up to four configurable setpoints (LoLo, Lo, Hi, HiHi). Any subset may be configured. Each setpoint may carry its own priority that overrides the alarm-level priority for that band.
- **Expression** trigger evaluation runs on the shared .NET thread pool behind a process-wide concurrency gate (`TriggerEvalGate`, sized by `TriggerEvalMaxConcurrency`, default `max(2, ProcessorCount)`) — **not** on the bounded script-execution pool. This is the WP3.1 fix for arch-review finding #4 (High): when evaluation shared that pool, N script bodies blocked in synchronous I/O stalled *every* Expression trigger on the node — scripts and alarms alike — for an unbounded time, and the evaluation's own 2 s timeout was constructed inside the queued body, so it did not start ticking until dequeue. An alarm that should have raised in milliseconds simply never raised, with neither a raise nor a timeout visible to the operator. Trigger expressions are non-blocking **by construction** (`TriggerExpressionGlobals` exposes only reads over an in-memory snapshot, and the script trust gate has already denied I/O, network, threading, and reflection), so the shared pool is where they belong; a second dedicated pool was considered and rejected as adding threads, gauges, and a second starvation surface for no isolation gain. The evaluation deadline (`TriggerEvalTimeoutSeconds`, default 2, previously hardcoded) is now armed **at enqueue**, so gate-wait time burns the same budget and a saturated gate yields a timely `false` rather than an unbounded stall. Per-actor coalescing (one evaluation in flight, one pending) is unchanged and caps waiters at one per Expression trigger, so the gate queue is bounded by trigger count.
- For binary trigger types (ValueMatch / RangeViolation / RateOfChange), when the condition is met and the alarm is currently in **normal** state, the alarm transitions to **active**:
  - Updates the alarm state on the parent Instance Actor (which publishes to the Akka stream).
  - If an on-trigger script is defined, launches its run via `ScriptRunLauncher` (see Alarm On-Trigger Run below) — no per-run child actor.
- When the condition clears and the alarm is in **active** state, the alarm transitions to **normal**.
- For HiLo triggers, the actor tracks the current `AlarmLevel` (None / Low / LowLow / High / HighHigh). Each level transition emits a fresh `AlarmStateChanged` with the new level and its priority; level escalations (e.g., High → HighHigh) and de-escalations (HighHigh → High) both produce events. The on-trigger script fires only on the Normal → non-None edge, not on escalations between alarm bands.
- No script execution on clear in any trigger type.

### Alarm State
- Held **in memory** only — not persisted to SQLite. State comprises `AlarmState` (Active / Normal) and `AlarmLevel` (None for binary triggers; the active band for HiLo).
- On restart (or failover), alarm states are re-evaluated from incoming values. All alarms start in normal state with level None and transition based on incoming values.

### Alarm On-Trigger Run
- Launched directly by the Alarm Actor when an on-trigger script needs to execute — the same `ScriptRunLauncher` path as an instance script run, with the firing alarm's name/level/priority/message exposed through the `Alarm` global. Bounded by the same `MaxConcurrentRunsPerScript` cap.
- Has access to the Instance Actor for `GetAttribute`/`SetAttribute`.
- **Can** call instance scripts via `Instance.CallScript()` — sends an ask message to the appropriate sibling Script Actor.
- Instance scripts **cannot** call alarm on-trigger scripts — the call direction is one-way.

### Audit correlation of an on-trigger run (`ParentExecutionId` tag-cascade)

An alarm fired by a **script- or inbound-API-initiated static attribute write**
is a genuine execution spawn, and the on-trigger run records the writing
execution as its parent in the central Audit Log (#23). The originating id rides
site-locally through three additive, nullable fields — no wire, proto or schema
change:

| Hop | Field |
|---|---|
| `ScriptRuntimeContext.SetAttribute` (this run's `ExecutionId`) / `RouteToSetAttributesRequest.ParentExecutionId` (the inbound request's) | → `SetStaticAttributeCommand.SourceExecutionId` |
| Instance Actor static-write path (`HandleSetStaticAttributeCore`) | → `AttributeValueChanged.SourceExecutionId` |
| Alarm Actor trigger evaluation → `SpawnAlarmExecution` | → `ScriptRunLauncher.LaunchAlarmScript` → `ScriptRuntimeContext.ParentExecutionId` |

All four computed trigger types participate. `Expression` triggers evaluate a
whole attribute snapshot **off the dispatcher**, so the firing change is no
longer in scope when the boolean returns: the writer of the newest value folded
into the snapshot is captured *with* the snapshot and echoed back on the
evaluation result, so a change arriving mid-flight cannot mis-attribute the
raise.

The on-trigger run is correctly a **tree root** (`ParentExecutionId` NULL) when
the firing value has no originating execution:

- Values from the **Data Connection Layer** — external device data has no
  spawning execution. This includes the confirmed value of a write to a
  **data-sourced** attribute: that write is forwarded to the device and the echo
  arrives on the subscription long after the writing execution ended, so only
  **static** attribute writes cascade.
- Deploy-time seeding and central Test Run / tooling writes.

`AttributeValueChanged.SourceExecutionId` is deliberately **not** projected onto
the gRPC debug/stream wire shapes — the cascade is resolved inside the site node
that owns the Instance Actor, and central reads the linkage from audit rows.

---

## Native Alarm Actor

### Role
- **Read-only mirror** of alarms raised natively by an external source — OPC UA Alarms & Conditions (A&C) servers and the MxAccess Gateway — surfaced into the Site Runtime alongside the alarms ScadaBridge computes itself.
- Created as a child of the **Instance Actor** and is a **peer to the computed Alarm Actor** (not a child of it). One `NativeAlarmActor` is spawned per resolved native alarm source binding on the instance.
- Mirrors source-of-truth condition state into the Instance Actor's view and onto the site-wide stream; it **does not** acknowledge, clear, or otherwise write back to the source. There is no ack-back path — the external source remains authoritative.

### Construction
- Constructed with `(ResolvedNativeAlarmSource source, string instanceName, IActorRef instanceActor, IActorRef dclManager, SiteStorageService storage, SiteRuntimeOptions options, ILogger logger, AlarmKind nativeKind = NativeOpcUa)`.
- `nativeKind` distinguishes the two native flavors and stamps the `Kind` on every emitted `AlarmStateChanged`. The Instance Actor selects it from the bound connection's protocol (see **Instance Actor wiring** below).

### Lifecycle & Subscription
- **PreStart**: rehydrates any previously mirrored conditions for this source from the site SQLite `native_alarm_state` table, then subscribes to the source through the Data Connection Layer by sending a `SubscribeAlarmsRequest` to the DCL manager. The DCL routes the subscription to the bound connection's `IAlarmSubscribableConnection` implementation.
- **Failed subscribe**: schedules a retry timer at `NativeAlarmRetryIntervalMs` and re-attempts until the subscription is established. Rehydrated state remains visible in the meantime.
- **`NativeAlarmSourceUnavailable`**: the source connection has dropped. The actor **retains its last-known mirrored conditions** but marks them uncertain rather than purging them, so a transient disconnect does not flap every condition to normal. The set is reconciled against truth by the next reconnect snapshot.

### Transition Handling (`NativeAlarmTransitionUpdate`)
- **Snapshot / SnapshotComplete (reconnect reconciliation)**: `Snapshot` updates buffer into a staging set; `SnapshotComplete` performs an **atomic swap** of the mirrored set with the staged set. Any condition that was previously mirrored but is **not present** in the new snapshot emits a return-to-normal `AlarmStateChanged` and drops out. This is how the mirror self-corrects after an outage.
- **Live transitions** (`Raise` / `Ack` / `Clear` / `Retrigger` / `StateChange`): upsert the condition by `SourceReference`. Updates carrying a `TransitionTime` **older** than the currently held transition are ignored (out-of-order protection). Accepted transitions persist to SQLite and emit an enriched `AlarmStateChanged` upward to the Instance Actor.
- **Retention**: a mirrored condition is dropped once it is both inactive **and** acknowledged (`!Active && Acknowledged`) — the alarm has fully run its course at the source and no longer needs mirroring. The drop emits a final state change and deletes the SQLite row.
- **Per-source cap**: at most `MirroredAlarmCapPerSource` conditions are retained per source. When the cap is exceeded the **oldest** condition is dropped and the eviction is **logged** — there is no silent truncation. If the evicted condition is still **active**, a final return-to-normal `AlarmStateChanged` is emitted before it is dropped (mirroring the retention drop above), so a capped-out active condition is never left stuck-active downstream.

### Persistence
- Mirrored condition state is persisted to the site SQLite `native_alarm_state` table on every accepted transition and removed on drop-out.
- Persistence is **best-effort / fire-and-forget**: a persistence failure is logged but never blocks the actor's mailbox and never aborts the upward `AlarmStateChanged` emit. The in-memory mirror remains authoritative for the running actor; SQLite exists to survive failover.

### Supervision & Restart
- Supervised by the Instance Actor under the same **OneForOneStrategy** as the computed Alarm Actor — a native source fault is isolated to its own actor.
- On site restart or failover, the actor rehydrates its mirror from `native_alarm_state` in PreStart, then reconciles against the source via the reconnect snapshot. Native mirror state therefore **survives failover** (unlike computed alarm state, which is re-evaluated from values).
- Mirrored native state **is cleared on redeploy/undeploy** of the instance (mirroring the static-override reset): the stale rows for the instance are removed and the fresh actor re-subscribes from a clean slate.

---

## Instance Actor — Native Alarm Wiring

The Instance Actor owns native-alarm setup alongside its computed Script and Alarm Actors:

- **Spawning**: for each entry in `_configuration.NativeAlarmSources`, the Instance Actor spawns a `NativeAlarmActor`. Spawning is **skipped when there is no DCL manager** (e.g., debug/test contexts with no data connections), since native alarms require a live source subscription.
- **Kind derivation**: the `AlarmKind` passed to each `NativeAlarmActor` is derived from the bound connection's protocol — `Mx*` protocols → `NativeMxAccess`, otherwise → `NativeOpcUa`.
- **Latest-event retention**: the Instance Actor retains the latest enriched `AlarmStateChanged` per alarm name in `_latestAlarmEvents`. The DebugView snapshot is built from this map so it carries the **unified condition view plus native metadata** for both computed and native alarms. Computed alarms that have not yet produced an event fall back to a **Normal projection** so the snapshot is complete.
- **Idle native source binding placeholders**: `BuildAlarmStatesSnapshot()` additionally emits one placeholder `AlarmStateChanged` for each configured native alarm source binding that currently has **no live conditions** in `_latestAlarmEvents`. The placeholder has `IsConfiguredPlaceholder = true` and carries the binding's canonical name in `NativeSourceCanonicalName`. The Instance Actor maintains a `_nativeAlarmKinds` map (`sourceCanonicalName → AlarmKind`) populated at spawn time to stamp the correct `Kind` on each placeholder. This ensures the Debug View Alarms tab shows every configured binding as a tree node even when quiet.
- **`NativeSourceCanonicalName` on live events**: every `AlarmStateChanged` emitted by a `NativeAlarmActor` stamps its source binding's canonical name in `NativeSourceCanonicalName`. The Debug View uses this field to place each live condition under the correct native-source binding node in the tree.
- **Reset semantics**: `_latestAlarmEvents` and the mirrored native state are cleared on redeploy/undeploy (same trigger as static-override reset) but rehydrate from SQLite on failover. In addition, a native condition's `_latestAlarmEvents` key is **evicted per-condition** whenever that condition leaves the mirror — snapshot-swap drop, retention drop, or cap eviction — driven by a `NativeAlarmDropped` signal from the `NativeAlarmActor`; this prevents unbounded `_latestAlarmEvents` growth for sources that mint a fresh `SourceReference` per occurrence.

---

## Native Alarm State Persistence (Site SQLite)

`SiteStorageService` gains a `native_alarm_state` table backing the native mirror:

- **Primary key**: `(instance_unique_name, source_canonical_name, source_reference)` — one row per mirrored condition.
- **Columns**: `condition_json` (the serialized `AlarmConditionState`), `metadata_json` (display metadata — type/category/message/current+limit, UA4), and `last_transition_at` (the accepted `TransitionTime`).
- **Operations**: `Upsert` (on accepted transition), `Delete` (on condition drop-out), `Get` (PreStart rehydrate, scoped to instance + source), and `ClearForInstance` (redeploy/undeploy reset).
- **Display metadata persistence (UA4)**: persisted rows include the condition's display metadata, so **rehydrated conditions render fully** (type / category / message / current + limit values) before the first fresh source snapshot arrives; a pre-UA4 row with no metadata rehydrates with empty display fields (unchanged behavior).
- **Ack timestamp persistence**: the condition's `AckTime` rides the same `metadata_json` blob, so an acknowledged alarm keeps its ack instant across restart/failover rather than reappearing as if it had never been acknowledged. It is deliberately **not** a new physical column: `native_alarm_state` is `RegisterReplicated` in `SiteLocalDbSetup`, and LocalDb builds that table's CDC triggers from the column list at registration time — an additive JSON property changes no schema, no triggers and no replication contract, and `metadata_json` is exactly the extension point UA4 introduced for this. A row written before the field existed deserializes it as `null` ("ack time unknown"), which is the correct fallback. See `docs/plans/2026-06-30-mes-alarm-status-api.md` §6.4.
- **Coalesced writes (P4)**: per-transition upserts are coalesced (latest-per-source-reference) and flushed on a short single-shot timer rather than one SQLite write per transition, so an alarm storm collapses to one batched write; a pending upsert is dropped if the condition is deleted before the flush (no resurrection). Durability window is at most one flush interval on crash, reconciled by the next re-subscribe snapshot.
- This is a **peer SQLite store** to the existing deployed-configuration, store-and-forward, operation-tracking, and `AuditLog` stores. Unlike computed alarm state, native mirror state is intentionally persisted so it survives failover.

---

## Enriched `AlarmStateChanged` Message

The `AlarmStateChanged` message published by both Alarm Actors and Native Alarm Actors was extended **additively** (existing consumers keep working with computed defaults):

- **`Kind`** (`AlarmKind`): `Computed` for ScadaBridge-evaluated alarms; `NativeOpcUa` / `NativeMxAccess` for mirrored native alarms.
- **`Condition`** (`AlarmConditionState`): the unified condition view. Computed alarms supply a computed default; native alarms carry the mirrored source condition.
- **Native metadata** (populated for native alarms; defaulted/empty for computed): `SourceReference`, `AlarmTypeName`, `Category`, `OperatorUser`, `OperatorComment`, `OriginalRaiseTime`, `CurrentValue`, `LimitValue`.
- **Computed-alarm projection**: computed alarms are surfaced as **auto-acknowledged** with `Severity = Priority`, so a single enriched shape carries both computed and native alarms onto the stream and into the DebugView snapshot.
- **`NativeSourceCanonicalName`** (`string?`): the canonical name of the native alarm source binding a native condition belongs to (e.g. `Motor1.NativeAlarms`). `null` for computed alarms. Stamped by the `NativeAlarmActor` on every emitted event; also carried on idle-binding placeholder rows emitted by the Instance Actor.
- **`IsConfiguredPlaceholder`** (`bool`): `true` for a synthetic row emitted by the Instance Actor to represent a configured native source binding that currently has no live conditions. These rows are never persisted or forwarded as operational alarms — they exist solely so the Debug View can render every configured binding as a tree node. `false` on all other rows.
- **`AckTime`** (`DateTimeOffset?`): the instant the condition was acknowledged; `null` while unacknowledged, `null` on computed alarms (auto-acked but with no operator ack event), and cleared on re-raise. Stamped by the DCL adapter — the source's own ack instant where the protocol supplies one (OPC UA A&C), the DCL's observation time of the ack transition where it does not (MxAccess Gateway) — and carried verbatim by the `NativeAlarmActor`, which never invents or suppresses one. See [Component-DataConnectionLayer.md](Component-DataConnectionLayer.md) → *Ack Timestamp*.

These fields are **additive-only** (init-only with defaults: `null` / `false`) and are mirrored additively on the gRPC `AlarmStateUpdate` proto as fields 22 (`native_source_canonical_name`, `string`), 23 (`is_configured_placeholder`, `bool`) and 24 (`ack_time`, `google.protobuf.Timestamp`, absent ⇒ `null`). `StreamRelayActor` and `SiteStreamGrpcClient` pack/unpack these fields without touching existing fields, preserving backward compatibility across version-skew boundaries.

The enriched message flows Instance Actor → site-wide Akka stream → `SiteStreamManager` → `SiteStreamGrpcServer` and is streamed to central as the gRPC `AlarmStateUpdate` event (see [Component-Communication.md](Component-Communication.md)).

---

## Shared Script Library

- Shared scripts are compiled at the site when received from central.
- Compiled code is stored in memory and made available to all Script Actors.
- When a script run calls `Scripts.CallShared("scriptName", params)`, the shared script code executes **inline** in that run's context — it is a direct method invocation, not an actor message.
- This avoids serialization bottlenecks since there is no shared script actor to contend for.
- Shared scripts have access to the same runtime API as instance scripts (GetAttribute, SetAttribute, external systems, notifications, databases).
- **Shared scripts always execute with Root scope (C5)**: `Attributes[...]` inside a shared script addresses **root** attributes even when the shared script is invoked from a composed-module script. Callers needing module-scoped access must pass the values in as parameters.

---

## Script Lifecycle

All script types can be updated without restarting the cluster, but the mechanism differs per type.

### Instance Scripts and Alarm On-Trigger Scripts

- Compiled at deployment time when the Deployment Manager receives a flattened configuration and creates the Instance Actor hierarchy.
- **To update**: modify the script in the template, then redeploy the instance (`instance deploy --id <N>`).
- Redeployment stops the existing Instance Actor and all its children, creates a new Instance Actor, and recompiles all scripts and alarms from the updated configuration.
- There is no way to hot-reload a single instance script without redeploying the entire instance.

### Shared Scripts

- Compiled at the site when received from central via artifact deployment (`deploy artifacts`).
- The `SharedScriptLibrary` replaces its in-memory compiled code dictionary under a lock, making updated code immediately available to all Script Actors.
- **To update**: modify the shared script via the CLI or UI, then run `deploy artifacts` to push the change to sites.
- No instance redeployment is required — running instances pick up the new shared script code on the next `Scripts.CallShared()` invocation.

### Inbound API Method Scripts

- See Component-InboundAPI.md for the compilation and hot-reload lifecycle of API method scripts.

> **Warning**: Editing scripts via direct SQL does not trigger recompilation. The in-memory compiled script will remain stale until the next node restart or redeployment. Always use the CLI, Management API, or Central UI to modify scripts.

---

## Script Runtime API

Available to all script runs and alarm on-trigger runs:

### Instance Attributes
- `Instance.GetAttribute("name")` — Read an attribute value from the parent Instance Actor.
- `Instance.SetAttribute("name", value)` — Write an attribute value. For data-connected attributes, writes to the DCL; for static attributes, updates in-memory and persists to local SQLite (survives restart/failover, reset on redeployment).

### Other Scripts
- `Instance.CallScript("scriptName", parameters)` — Send an ask message to a sibling Script Actor. The target Script Actor launches a run, executes, and returns the result (or an explicit shed error if that script is already at its concurrent-run cap). The call includes the current recursion depth.
- `Scripts.CallShared("scriptName", parameters)` — Execute shared script code inline (direct method invocation). The call includes the current recursion depth.

### External Systems
- `ExternalSystem.Call("systemName", "methodName", params)` — Synchronous HTTP call. Blocks until response or timeout. All failures return to script. Use when the script needs the result.
- `ExternalSystem.CachedCall("systemName", "methodName", params)` — Deferred delivery. Returns a `TrackedOperationId` tracking handle immediately rather than the response; the call is attempted immediately and, on transient failure, store-and-forwarded for retry. Use for outbound data pushes where deferred delivery is acceptable.
- The returned `TrackedOperationId` can be passed to `Tracking.Status(id)` (see **Operation Tracking** below) to observe delivery progress.

### Notifications
- `Notify.To("listName").Send("subject", "message")` — Send a notification via a named notification list. Generates a `TrackedOperationId` (GUID) locally and returns it immediately; the notification is store-and-forwarded to the central cluster, which owns delivery. The script never contacts SMTP. (`NotificationId` is the notification-domain name for this same `TrackedOperationId` type.)
- `Notify.Status("trackedOperationId")` — A thin alias of `Tracking.Status(id)` retained for the notification domain. Returns a status record (status, retry count, last error, key timestamps). While the notification is still in the site store-and-forward buffer the site answers locally (status `Forwarding`); once forwarded the query round-trips to central.

### Database Access
- `Database.Connection("connectionName")` — Obtain a raw MS SQL client connection (ADO.NET) for synchronous read/write.
- `Database.CachedWrite("connectionName", "sql", parameters)` — Submit a write operation for store-and-forward delivery. Returns a `TrackedOperationId` tracking handle immediately; pass it to `Tracking.Status(id)` to observe delivery progress.

### Operation Tracking
- `Tracking.Status("trackedOperationId")` — Returns a status record (status, retry count, last error, key timestamps) for any tracked operation: a cached external system call, a cached database write, or a notification. For cached calls and writes the answer is always site-local and authoritative — the site owns the operation tracking table. (`Notify.Status(...)` is a thin alias scoped to the notification domain.)

### Alarm State (read-only)
- `Alarms.CurrentAsync()` — Returns a snapshot of the instance's **current** alarm conditions as `IReadOnlyList<ScriptAlarm>` (`Commons.Types.Scripts`), covering computed alarms and mirrored native (OPC UA A&C / MxAccess Gateway) conditions alike. Served by a **local Ask** (`GetAlarmSnapshotRequest`/`GetAlarmSnapshotResponse`) to the script's own Instance Actor — the same mechanism attribute reads use, never a cross-cluster hop.
- Available to **every** script, unlike the `Alarm` global, which exists only inside an on-trigger handler and describes the single alarm that fired. This accessor is what lets an inbound-API-routed `Call` script answer "what is in alarm on this machine right now?"; before it, native mirrored conditions were reachable only from the Debug View.
- **Read-only by design** — native alarms are a read-only mirror of the source (no ack-back), so the accessor exposes no acknowledge/shelve operation.
- **Not scope-prefixed**, unlike `Attributes`: alarm identity is not a scope-relative attribute name (computed alarms are keyed by configured name, native conditions by a source-supplied reference), so every scope sees the same list rather than a silently truncated one.
- The snapshot is built by the same `BuildAlarmStatesSnapshot()` the Debug View uses, so it **includes placeholder rows** for configured-but-quiet native source bindings (`IsConfiguredPlaceholder`). Callers enumerating real alarms filter on `Active && !IsConfiguredPlaceholder`.
- `ScriptAlarm` is a flat projection of the retained `AlarmStateChanged`: `Name`, `SourceReference`, `NativeSourceCanonicalName`, `Active`, `Acknowledged`, `Confirmed?`, `Shelved` (all shelve sub-states collapse to one boolean), `Suppressed`, `Severity` (0–1000), `Kind`, `Message`, `AlarmTypeName`, `Category`, `OperatorUser`, `OperatorComment`, `OriginalRaiseTime?`, `Timestamp`, `AckTime?`, `CurrentValue`, `LimitValue`, `IsConfiguredPlaceholder`. The `Condition` is the authority for active/acked/severity, so computed and native alarms read identically from one filter expression.
- The type lives in Commons so the runtime accessor and the Script Analysis compile-only surface (`ScriptCompileSurface.CompileAlarmsAccessor`) project to the **same** type — a script binding at the design-time gate binds identically at the site. Parity is pinned by `CompileSurfaceParityTests`.
- Design record: `docs/plans/2026-06-30-mes-alarm-status-api.md` §5.2.

### Parameter Access
- `Parameters["key"]` — Raw dictionary access (returns `object?`, requires manual casting).
- `Parameters.Get<T>("key")` — Typed access with descriptive error messages. Throws `ScriptParameterException` if parameter is missing, null, or cannot be converted to `T`.
- `Parameters.Get<T?>("key")` — Nullable typed access. Returns `null` if parameter is missing, null, or cannot be converted.
- `Parameters.Get<T[]>("key")` — Array access. Converts a list parameter to a typed array. Throws on first unconvertible element.
- `Parameters.Get<List<T>>("key")` — List access. Converts a list parameter to a typed `List<T>`.
- Supported types: `bool`, `int`, `long`, `float`, `double`, `string`, `DateTime`.

### Recursion Limit
- Every script call (`Instance.CallScript` and `Scripts.CallShared`) increments a call depth counter.
- If the counter exceeds the maximum recursion depth (default: 10), the call fails with an error.
- The error is logged to the site event log.

---

## Script Trust Model

Scripts execute **in-process** with constrained access. The following restrictions are enforced at compilation and runtime:

- **Allowed**: Access to the Script Runtime API (GetAttribute, SetAttribute, CallScript, CallShared, ExternalSystem, Notify, Database, Tracking, Alarms), standard C# language features, basic .NET types (collections, string manipulation, math, date/time). `System.Diagnostics.Stopwatch`, `Debug`, and `Activity` are permitted.
- **Forbidden**: File system access (`System.IO`), process spawning (`System.Diagnostics.Process`), threading (`System.Threading` — except `Tasks`, `CancellationToken`, and `CancellationTokenSource`), reflection (`System.Reflection`), all raw network access (`System.Net` — must use `ExternalSystem.Call`), native interop (`System.Runtime.InteropServices`, `Microsoft.Win32`), assembly loading, unsafe code, `dynamic`, `Activator`.
- **Execution timeout**: Configurable per-script maximum execution time, **measured from enqueue** (WP3.1) — a run that spends its whole budget queued behind blocked scripts is shed at dequeue without executing, rather than being granted a fresh full budget for work whose triggering condition is already stale. Exceeding the timeout cancels the script **cooperatively** (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A **watchdog names the script** once its thread has not returned within a grace period after the timeout — and, as of WP3.1, **detaches and replaces that worker thread**, so a wedged script costs the pool a thread only until the watchdog fires rather than for the rest of the process. The detached worker exits (instead of pulling more work) when its body finally returns, so capacity never silently doubles. Live detached workers are capped at the pool size — at the cap no replacement is started and an Error site event says so, because bounded starvation is preferable to unbounded thread growth when scripts wedge en masse. The scheduler's **queue depth / busy-thread / oldest-busy-age / detached-thread gauges** and the **per-interval shed count** are reported through Health Monitoring so a saturated, stuck, or shedding scheduler is visible.
- **Memory**: Scripts share the host process memory. No per-script memory limit, but the execution timeout prevents runaway allocations.

The forbidden-API policy is defined authoritatively in `ScriptTrustPolicy` (Script Analysis component, #25). `ScriptCompilationService.ValidateTrustModel` delegates to `ScriptTrustValidator.FindViolations` for the trust verdict; the Site Runtime also performs a `CSharpScript.Compile` against the real `ScriptGlobals` for execution. This is defence-in-depth static enforcement, not a true runtime sandbox.

### Script Trust Boundary Auditing

Every script-trust-boundary call (External System Gateway, Database layer, Notify) emits an `AuditEvent` to `IAuditWriter` (site-local SQLite append). Hot path; never fails the calling action; failures logged via the `SiteAuditWriteFailures` health metric (see [Component-HealthMonitoring.md](Component-HealthMonitoring.md)). The central audit mirror and event schema live in [Component-AuditLog.md](Component-AuditLog.md).

## Script Scoping Rules

- Scripts can only read/write attributes on **their own instance** (via the parent Instance Actor).
- Scripts can call other scripts on **their own instance** (via sibling Script Actors).
- Scripts can call **shared scripts** (inline execution).
- Scripts **cannot** access other instances' attributes or scripts.
- Alarm on-trigger scripts **can** call instance scripts; instance scripts **cannot** call alarm on-trigger scripts.

---

## Tell vs. Ask Usage

Per Akka.NET best practices, internal actor communication uses **Tell** (fire-and-forget with reply-to) for the hot path:

- **Tag value updates** (DCL → Instance Actor): Tell. High-frequency, no response needed.
- **Attribute change notifications** (Instance Actor → Script/Alarm Actors): Tell. Fan-out notifications.
- **Stream publishing** (Instance Actor → Akka stream): Tell. Fire-and-forget.

**Ask** is reserved for system boundaries where a synchronous response is needed:

- **`Instance.CallScript()`**: Ask pattern from the running script (via `ScriptRuntimeContext.CallScript`) to sibling Script Actor. The caller needs the return value. Acceptable because script calls are infrequent relative to tag updates.
- **`Route.To().Call()`**: Ask from Inbound API to site Instance Actor via Communication Layer. External caller needs a response.
- **Debug view snapshot**: Ask from Communication Layer to Instance Actor for initial state.

## Concurrency & Serialization

- The Instance Actor processes messages **sequentially** (standard Akka actor model). This means `SetAttribute` calls from concurrent script runs are serialized at the Instance Actor, preventing race conditions on attribute state.
- Script runs (launched via `ScriptRunLauncher`, no per-run child actor) may run concurrently, but all state mutations (attribute reads/writes, alarm state updates) are mediated through the parent Instance Actor's message queue.
- External side effects (external system calls, notifications, database writes) are not serialized — concurrent scripts may produce interleaved side effects. This is acceptable because each side effect is independent.

## SiteStreamManager and gRPC Integration

- The `SiteStreamManager` implements the `ISiteStreamSubscriber` interface, allowing the Communication Layer's `SiteStreamGrpcServer` to subscribe to the stream for cross-cluster delivery via gRPC.
- When a gRPC `SubscribeInstance` call arrives, the `SiteStreamGrpcServer` creates a `StreamRelayActor` and subscribes it to `SiteStreamManager` for the requested instance. Events flow from `SiteStreamManager` → `StreamRelayActor` → `Channel<SiteStreamEvent>` → gRPC response stream to central.
- The `SiteStreamManager` filters events by instance unique name and forwards matching events to all registered subscribers (both local debug consumers and gRPC relay actors).

## Site-Wide Stream Backpressure

- The site-wide Akka stream uses **per-subscriber buffering** with bounded buffers. Each subscriber (gRPC relay actors, future consumers) gets an independent buffer.
- If a subscriber falls behind (e.g., slow network on gRPC stream), its buffer fills and oldest events are dropped. This does not affect other subscribers or the publishing Instance Actors.
- Instance Actors publish to the stream with **fire-and-forget** semantics — publishing never blocks the actor.

## Error Handling

### Script Errors
- Unhandled exceptions and timeouts in a script run are **logged locally** to the site event log — unconditionally, regardless of the WP3.2 per-run Started/Completed sampling policy (see Script Run Launch above).
- The Script Actor (coordinator) is **not affected** — it remains active for future trigger events.
- Script failures are **not reported to central** (except as aggregated error rate metrics via Health Monitoring).

### Alarm Evaluation Errors
- Errors during alarm condition evaluation are **logged locally** to the site event log.
- The Alarm Actor remains active and continues evaluating on subsequent value updates.
- Alarm evaluation error rates are reported to central via Health Monitoring.

### Script Compilation Errors
- If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied.
- The failure is reported back to central as a failed deployment.
- **Enforced by a site-side pre-compile validation gate (S3)**: before the Instance Actor is created or the config persisted, the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **synchronously, as a pure prefix step of the deploy application on the singleton's actor thread** — the verdict is never derived off-thread, because piping a verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering. WP3.1 made it **warm-then-gate**: the same compile is first run off-thread purely to populate the process-wide compile cache (`DeployCompileWarmed` pipes back), so the authoritative synchronous gate is then all cache hits and no longer holds the singleton for a Roslyn compile. Ordering is preserved by a **per-instance in-flight guard** — while a warm is in flight for instance X, further mutating commands for X (deploy / delete / disable / enable) are queued and re-dispatched in arrival order once the deploy applies; a second deploy with nothing queued behind it supersedes the pending one last-write-wins and its displaced deployer is answered `Failed`-superseded rather than left to Ask-timeout. Commands for **other** instances flow freely, which is the point, and is safe because cross-instance ordering was never guaranteed to callers. Unchanged script bodies hit the compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors. This makes the deployment result honest even if central's pre-deployment validation was bypassed or skew let a bad artifact through.
- Note: Pre-deployment validation at central should still catch compilation errors before they reach the site; the site gate is the authoritative backstop.

---

## Dependencies

- **Script Analysis (#25)**: `ScriptCompilationService.ValidateTrustModel` delegates the forbidden-API verdict to `ScriptTrustValidator.FindViolations`. The Site Runtime retains its own `CSharpScript.Compile` against the real `ScriptGlobals` for execution.
- **Data Connection Layer**: Provides tag value updates to Instance Actors. Receives write requests from Instance Actors. Also feeds Native Alarm Actors: connections implementing `IAlarmSubscribableConnection` (OPC UA A&C servers, MxAccess Gateway) deliver `NativeAlarmTransitionUpdate` events in response to a `SubscribeAlarmsRequest`, and signal `NativeAlarmSourceUnavailable` on connection loss. **Subscription-handshake retry (S4/UA6)**: a tag-subscription handshake that either failed or whose response was lost is re-sent on a fixed interval per connection (default 5 s) — the retry is armed on every send and cancelled on the first success reply, closing both the failed-response and lost-response gaps; the `NativeAlarmActor` subscribe uses the same retry pattern.
- **Store-and-Forward Engine**: Handles reliable delivery for external system calls, cached database writes, and notifications submitted by scripts. For the notification category specifically, it forwards to the central cluster for delivery (not directly to SMTP). Owns the site-local operation tracking table that backs `Tracking.Status(id)`.
- **External System Gateway**: Provides external system method invocations for scripts.
- **Communication Layer**: Receives deployments and lifecycle commands from central. Handles debug view requests. Reports deployment results.
- **Site Event Logging**: Records script executions, alarm events, deployment events, instance lifecycle events.
- **Health Monitoring**: Reports script error rates and alarm evaluation error rates.
- **Local SQLite**: Persists deployed configurations, system-wide artifacts (external system definitions, database connection definitions, data connection definitions). Sites also maintain peer SQLite stores for the Store-and-Forward buffer, the site event log, the operation tracking table, and the site-local `AuditLog` (see [Component-AuditLog.md](Component-AuditLog.md)). The `AuditLog` file is purged on the same daily cadence as the others but respects the hard `ForwardState` invariant — rows still `Pending` forward are never purged, regardless of age.

## Interactions

- **Deployment Manager (central)**: Receives flattened configurations, system-wide artifact updates, and instance lifecycle commands.
- **Data Connection Layer**: Bidirectional — receives value updates, sends write-back commands.
- **Communication Layer**: Receives commands from central, sends deployment results, serves debug view data.
- **Store-and-Forward Engine**: Scripts route cached writes, notifications, and external system calls here.
- **Health Monitoring**: Periodically reports error rate metrics.
