# Component: Site Runtime

## Purpose

The Site Runtime component manages the execution of deployed machine instances at site clusters. It encompasses the actor hierarchy that represents running instances, their scripts, and their alarms. It owns the site-side deployment lifecycle (receiving configs from central, compiling scripts, creating actors), script execution, alarm evaluation, and the site-wide Akka stream for attribute and alarm state changes.

This component replaces the previously separate Script Engine and Alarm Engine concepts, unifying them under a single actor hierarchy rooted at the Deployment Manager singleton.

## Location

Site clusters only.

## Responsibilities

- Run the Deployment Manager singleton (Akka.NET cluster singleton) on the active site node.
- On startup (or failover), read all deployed configurations from local SQLite and re-create the full actor hierarchy.
- Receive deployment commands from central: new/updated instance configurations, instance lifecycle commands (disable, enable, delete), and system-wide artifact updates.
- Compile C# scripts when deployments are received.
- Manage the Instance Actor hierarchy (Instance Actors, Script Actors, Alarm Actors).
- Execute scripts via Script Actors with support for concurrent execution.
- Evaluate alarm conditions via Alarm Actors and manage alarm state.
- Maintain the site-wide Akka stream for attribute value and alarm state changes.
- Execute shared scripts inline as compiled code libraries (no separate actors).
- Enforce script call recursion limits.

---

## Actor Hierarchy

```
Deployment Manager Singleton (Cluster Singleton)
├── Instance Actor ("MachineA-001")
│   ├── Script Actor ("MonitorSpeed") — coordinator
│   │   └── Script Execution Actor — short-lived, per invocation
│   ├── Script Actor ("CalculateOEE") — coordinator
│   │   └── Script Execution Actor — short-lived, per invocation
│   ├── Alarm Actor ("OverTemp") — coordinator
│   │   └── Alarm Execution Actor — short-lived, per on-trigger invocation
│   └── Alarm Actor ("LowPressure") — coordinator
├── Instance Actor ("MachineA-002")
│   └── ...
└── ...
```

---

## Deployment Manager Singleton

### Role
- Akka.NET **cluster singleton** — guaranteed to run on exactly one node in the site cluster (the active node).
- On failover, Akka.NET restarts the singleton on the new active node.

### Startup Behavior
1. Read all deployed configurations from local SQLite.
2. Read all shared scripts from local storage.
3. Compile all scripts (instance scripts, alarm on-trigger scripts, shared scripts).
4. Create Instance Actors for all deployed, **enabled** instances as child actors. Instance Actors are created in **staggered batches** (e.g., 20 at a time with a short delay between batches) to prevent a reconnection storm — 500 Instance Actors all registering data subscriptions simultaneously would overwhelm OPC UA servers and network capacity.
5. Make compiled shared script code available to all Script Actors.

### Deployment Handling
- Receives flattened instance configurations from central via the Communication Layer.
- Stores the new configuration in local SQLite.
- Compiles all scripts in the configuration.
- Creates a new Instance Actor (for new instances) or updates an existing one (for redeployments).
- For redeployments: the existing Instance Actor and all its children are stopped, then a new Instance Actor is created with the updated configuration. Subscriptions are re-established.
- Reports deployment result (success/failure) back to central.

### System-Wide Artifact Handling
- Receives updated shared scripts, external system definitions, database connection definitions, and data connection definitions from central. (Notification lists and SMTP configuration are central-only and are not deployed to sites — see Component-NotificationService.md.)
- Stores all artifacts in local SQLite. After artifact deployment, the site is fully self-contained — all runtime configuration is read from local SQLite with no access to the central configuration database.
- Recompiles shared scripts and makes updated code available to all Script Actors.

### Instance Lifecycle Commands
- **Disable**: Stops the Instance Actor and its children. Retains the deployed configuration in SQLite so the instance can be re-enabled without redeployment.
- **Enable**: Creates a new Instance Actor from the stored configuration (same as startup).
- **Delete**: Stops the Instance Actor and its children, removes the deployed configuration from local SQLite. Does **not** clear store-and-forward messages.

### Debug Snapshot Routing
- Receives `DebugSnapshotRequest` from the Communication Layer and forwards to the Instance Actor by unique name (same lookup as `SubscribeDebugViewRequest`).
- Returns an error response if no Instance Actor exists for the requested unique name (instance not deployed or not enabled).

---

## Instance Actor

### Role
- **Single source of truth** for all runtime state of a deployed instance.
- Holds all attribute values (both static configuration values and live values from data connections).
- Holds current alarm states (active/normal), updated by child Alarm Actors.
- Publishes attribute value changes and alarm state changes to the site-wide Akka stream.

### Initialization
1. Load all attribute values from the flattened configuration (static defaults).
2. Set quality to **uncertain** for all attributes that have a data source reference. Static attributes (no data source reference) have quality **good**. The uncertain quality persists until the first value update arrives from the Data Connection Layer, distinguishing "not yet received" from "known good" or "connection lost."
3. Register data source references with the Data Connection Layer for subscriptions.
4. Create child Script Actors (one per script defined on the instance).
5. Create child Alarm Actors (one per alarm defined on the instance).

### Attribute Value Updates
- Receives tag value updates from the Data Connection Layer for attributes with data source references.
- Updates the in-memory attribute value.
- Notifies subscribed child Script Actors and Alarm Actors of the change.
- Publishes the change to the site-wide Akka stream.

### Stream Message Format
- **Attribute changes**: `[InstanceUniqueName].[AttributePath].[AttributeName]`, attribute value, attribute quality, attribute change timestamp.
- **Alarm state changes**: `[InstanceUniqueName].[AlarmName]`, alarm state (active/normal), priority, timestamp.

### GetAttribute / SetAttribute
- **GetAttribute**: Returns the current in-memory value for the requested attribute.
- **SetAttribute** (for attributes with data source reference): Sends a write request to the Data Connection Layer. The DCL writes to the physical device. If the write fails (connection down, device rejection, timeout), the error is returned synchronously to the calling script for handling. On success, the existing subscription picks up the confirmed value from the device and sends it back as a value update, which then updates the in-memory value. The in-memory value is **not** optimistically updated.
- **SetAttribute** (for static attributes): Updates the in-memory value and **persists the override to local SQLite**. On restart (or failover), the Instance Actor loads persisted overrides on top of the deployed configuration, preserving runtime-modified values. A redeployment of the instance resets all persisted overrides to the new deployed configuration values.

### Debug View Support
- On request from central (via Communication Layer), the Instance Actor provides a **snapshot** of all current attribute values and alarm states.
- Subsequent changes are delivered via the **SiteStreamManager** → **SiteStreamGrpcServer** → gRPC stream to central. The Instance Actor publishes attribute value and alarm state changes to the SiteStreamManager; it does not forward events directly to the Communication Layer.
- The Instance Actor also handles one-shot `DebugSnapshotRequest` messages: it builds the same snapshot (attribute values and alarm states) and replies directly to the sender. Unlike `SubscribeDebugViewRequest`, no subscriber is registered and no stream is established.

### Supervision Strategy

The Instance Actor supervises all child Script and Alarm Actors with explicit strategies:

| Child Actor | Exception Type | Strategy | Rationale |
|-------------|---------------|----------|-----------|
| Script Actor | Any exception | Resume | Script Actor is a coordinator — its state (trigger timers, last execution time) should survive child failures. Script Execution Actor failures are isolated. |
| Alarm Actor | Any exception | Resume | Alarm Actor holds alarm state. Resume preserves state and continues evaluation on next value update. |
| Script Execution Actor | Unhandled exception | Stop | Short-lived, per-invocation. Failure is logged; the Script Actor coordinator remains active for future triggers. |
| Alarm Execution Actor | Unhandled exception | Stop | Short-lived, per on-trigger invocation. Same as Script Execution Actor. |

The Deployment Manager singleton supervises Instance Actors with a **OneForOneStrategy** — one Instance Actor's failure does not affect other instances.

When the Instance Actor is stopped (due to disable, delete, or redeployment), Akka.NET automatically stops all child actors.

---

## Script Actor

### Role
- **Coordinator** for a single script definition on an instance.
- Holds the compiled script code and trigger configuration.
- Manages trigger evaluation (interval timer, value change detection, conditional evaluation).
- Spawns short-lived Script Execution Actors for each invocation.

### Trigger Management
- **Interval**: The Script Actor manages an internal timer. When the timer fires, it spawns a Script Execution Actor.
- **Value Change**: The Script Actor subscribes to attribute change notifications from its parent Instance Actor for the specific monitored attribute. When the attribute changes, it spawns a Script Execution Actor.
- **Conditional**: The Script Actor subscribes to attribute change notifications for the monitored attribute. On each update, it evaluates the condition (compares the attribute against a threshold). Firing depends on the **fire mode** (see below).
- **Expression**: The Script Actor evaluates a compiled boolean expression against an attribute snapshot on each attribute change. Firing depends on the **fire mode** (see below).
- **Fire mode (Conditional + Expression)**:
  - **OnTrue** (default): Conditional fires on each matching attribute change; Expression fires once per `false → true` transition (edge-triggered). This is the original behavior — a trigger configuration with no mode field is treated as OnTrue.
  - **WhileTrue**: On the `false → true` edge the script fires once, then re-fires on a periodic timer while the condition stays true; on the `true → false` edge the timer stops. The re-fire cadence is the script's **minimum time between runs**; with none configured the trigger degrades to the single edge fire and logs a warning.
- **Minimum time between runs**: If configured, the Script Actor tracks the last execution time and skips trigger invocations that fire before the minimum interval has elapsed. For a WhileTrue trigger it doubles as the re-fire cadence.

### Concurrent Execution
- Each invocation spawns a **new Script Execution Actor** as a child.
- Multiple Script Execution Actors can run concurrently (e.g., a trigger fires while a previous `Instance.CallScript` invocation is still running).
- The Script Actor coordinates but does not block on child completion.

### Script Execution Actor
- **Short-lived** child actor created per invocation.
- Receives: compiled script code, input parameters, reference to the parent Instance Actor, current call depth.
- Executes the script in the Akka actor context.
- Has access to the full Script Runtime API (see below).
- Returns the script's return value (if defined) to the caller, then stops.

### Handling `Instance.CallScript`
- When an external caller (another Script Execution Actor, an Alarm Execution Actor, or a routed call from the Inbound API) sends a `CallScript` message to the Script Actor, it spawns a Script Execution Actor to handle the call.
- The caller uses the **Akka ask pattern** and receives the return value when the execution completes.

---

## Alarm Actor

### Role
- **Coordinator** for a single alarm definition on an instance.
- Evaluates alarm trigger conditions against attribute value updates.
- Manages alarm state (active/normal) in memory.
- Executes on-trigger scripts when the alarm activates.

### Alarm Evaluation
- Subscribes to attribute change notifications from its parent Instance Actor for the attribute(s) referenced by its trigger definition.
- On each value update, evaluates the trigger condition:
  - **Value Match**: Incoming value equals the predefined target. Supports `"!=X"` prefix for not-equals semantics.
  - **Range Violation**: Value is outside the allowed min/max range.
  - **Rate of Change**: Value change rate exceeds the defined threshold over a configurable time window. Direction filter (rising / falling / either) restricts which side of the rate triggers.
  - **HiLo**: Multi-setpoint level alarm with up to four configurable setpoints (LoLo, Lo, Hi, HiHi). Any subset may be configured. Each setpoint may carry its own priority that overrides the alarm-level priority for that band.
- For binary trigger types (ValueMatch / RangeViolation / RateOfChange), when the condition is met and the alarm is currently in **normal** state, the alarm transitions to **active**:
  - Updates the alarm state on the parent Instance Actor (which publishes to the Akka stream).
  - If an on-trigger script is defined, spawns an Alarm Execution Actor to execute it.
- When the condition clears and the alarm is in **active** state, the alarm transitions to **normal**.
- For HiLo triggers, the actor tracks the current `AlarmLevel` (None / Low / LowLow / High / HighHigh). Each level transition emits a fresh `AlarmStateChanged` with the new level and its priority; level escalations (e.g., High → HighHigh) and de-escalations (HighHigh → High) both produce events. The on-trigger script fires only on the Normal → non-None edge, not on escalations between alarm bands.
- No script execution on clear in any trigger type.

### Alarm State
- Held **in memory** only — not persisted to SQLite. State comprises `AlarmState` (Active / Normal) and `AlarmLevel` (None for binary triggers; the active band for HiLo).
- On restart (or failover), alarm states are re-evaluated from incoming values. All alarms start in normal state with level None and transition based on incoming values.

### Alarm Execution Actor
- **Short-lived** child actor created when an on-trigger script needs to execute.
- Same pattern as Script Execution Actor — receives compiled code, executes, returns, and stops.
- Has access to the Instance Actor for `GetAttribute`/`SetAttribute`.
- **Can** call instance scripts via `Instance.CallScript()` — sends an ask message to the appropriate sibling Script Actor.
- Instance scripts **cannot** call alarm on-trigger scripts — the call direction is one-way.

---

## Shared Script Library

- Shared scripts are compiled at the site when received from central.
- Compiled code is stored in memory and made available to all Script Actors.
- When a Script Execution Actor calls `Scripts.CallShared("scriptName", params)`, the shared script code executes **inline** in the Script Execution Actor's context — it is a direct method invocation, not an actor message.
- This avoids serialization bottlenecks since there is no shared script actor to contend for.
- Shared scripts have access to the same runtime API as instance scripts (GetAttribute, SetAttribute, external systems, notifications, databases).

---

## Script Lifecycle

All script types can be updated without restarting the cluster, but the mechanism differs per type.

### Instance Scripts and Alarm On-Trigger Scripts

- Compiled at deployment time when the Deployment Manager receives a flattened configuration and creates the Instance Actor hierarchy.
- **To update**: modify the script in the template, then redeploy the instance (`instance deploy --id <N>`).
- Redeployment stops the existing Instance Actor and all its children, creates a new Instance Actor, and recompiles all scripts and alarms from the updated configuration.
- There is no way to hot-reload a single instance script without redeploying the entire instance.

### Shared Scripts

- Compiled at the site when received from central via artifact deployment (`deploy artifacts`).
- The `SharedScriptLibrary` replaces its in-memory compiled code dictionary under a lock, making updated code immediately available to all Script Actors.
- **To update**: modify the shared script via the CLI or UI, then run `deploy artifacts` to push the change to sites.
- No instance redeployment is required — running instances pick up the new shared script code on the next `Scripts.CallShared()` invocation.

### Inbound API Method Scripts

- See Component-InboundAPI.md for the compilation and hot-reload lifecycle of API method scripts.

> **Warning**: Editing scripts via direct SQL does not trigger recompilation. The in-memory compiled script will remain stale until the next node restart or redeployment. Always use the CLI, Management API, or Central UI to modify scripts.

---

## Script Runtime API

Available to all Script Execution Actors and Alarm Execution Actors:

### Instance Attributes
- `Instance.GetAttribute("name")` — Read an attribute value from the parent Instance Actor.
- `Instance.SetAttribute("name", value)` — Write an attribute value. For data-connected attributes, writes to the DCL; for static attributes, updates in-memory and persists to local SQLite (survives restart/failover, reset on redeployment).

### Other Scripts
- `Instance.CallScript("scriptName", parameters)` — Send an ask message to a sibling Script Actor. The target Script Actor spawns a Script Execution Actor, executes, and returns the result. The call includes the current recursion depth.
- `Scripts.CallShared("scriptName", parameters)` — Execute shared script code inline (direct method invocation). The call includes the current recursion depth.

### External Systems
- `ExternalSystem.Call("systemName", "methodName", params)` — Synchronous HTTP call. Blocks until response or timeout. All failures return to script. Use when the script needs the result.
- `ExternalSystem.CachedCall("systemName", "methodName", params)` — Deferred delivery. Returns a `TrackedOperationId` tracking handle immediately rather than the response; the call is attempted immediately and, on transient failure, store-and-forwarded for retry. Use for outbound data pushes where deferred delivery is acceptable.
- The returned `TrackedOperationId` can be passed to `Tracking.Status(id)` (see **Operation Tracking** below) to observe delivery progress.

### Notifications
- `Notify.To("listName").Send("subject", "message")` — Send a notification via a named notification list. Generates a `TrackedOperationId` (GUID) locally and returns it immediately; the notification is store-and-forwarded to the central cluster, which owns delivery. The script never contacts SMTP. (`NotificationId` is the notification-domain name for this same `TrackedOperationId` type.)
- `Notify.Status("trackedOperationId")` — A thin alias of `Tracking.Status(id)` retained for the notification domain. Returns a status record (status, retry count, last error, key timestamps). While the notification is still in the site store-and-forward buffer the site answers locally (status `Forwarding`); once forwarded the query round-trips to central.

### Database Access
- `Database.Connection("connectionName")` — Obtain a raw MS SQL client connection (ADO.NET) for synchronous read/write.
- `Database.CachedWrite("connectionName", "sql", parameters)` — Submit a write operation for store-and-forward delivery. Returns a `TrackedOperationId` tracking handle immediately; pass it to `Tracking.Status(id)` to observe delivery progress.

### Operation Tracking
- `Tracking.Status("trackedOperationId")` — Returns a status record (status, retry count, last error, key timestamps) for any tracked operation: a cached external system call, a cached database write, or a notification. For cached calls and writes the answer is always site-local and authoritative — the site owns the operation tracking table. (`Notify.Status(...)` is a thin alias scoped to the notification domain.)

### Parameter Access
- `Parameters["key"]` — Raw dictionary access (returns `object?`, requires manual casting).
- `Parameters.Get<T>("key")` — Typed access with descriptive error messages. Throws `ScriptParameterException` if parameter is missing, null, or cannot be converted to `T`.
- `Parameters.Get<T?>("key")` — Nullable typed access. Returns `null` if parameter is missing, null, or cannot be converted.
- `Parameters.Get<T[]>("key")` — Array access. Converts a list parameter to a typed array. Throws on first unconvertible element.
- `Parameters.Get<List<T>>("key")` — List access. Converts a list parameter to a typed `List<T>`.
- Supported types: `bool`, `int`, `long`, `float`, `double`, `string`, `DateTime`.

### Recursion Limit
- Every script call (`Instance.CallScript` and `Scripts.CallShared`) increments a call depth counter.
- If the counter exceeds the maximum recursion depth (default: 10), the call fails with an error.
- The error is logged to the site event log.

---

## Script Trust Model

Scripts execute **in-process** with constrained access. The following restrictions are enforced at compilation and runtime:

- **Allowed**: Access to the Script Runtime API (GetAttribute, SetAttribute, CallScript, CallShared, ExternalSystem, Notify, Database, Tracking), standard C# language features, basic .NET types (collections, string manipulation, math, date/time).
- **Forbidden**: File system access (`System.IO`), process spawning (`System.Diagnostics.Process`), threading (`System.Threading` — except async/await), reflection (`System.Reflection`), raw network access (`System.Net.Sockets`, `System.Net.Http` — must use `ExternalSystem.Call`), assembly loading, unsafe code.
- **Execution timeout**: Configurable per-script maximum execution time. Exceeding the timeout cancels the script and logs an error.
- **Memory**: Scripts share the host process memory. No per-script memory limit, but the execution timeout prevents runaway allocations.

These constraints are enforced by restricting the set of assemblies and namespaces available to the script compilation context.

### Script Trust Boundary Auditing

Every script-trust-boundary call (External System Gateway, Database layer, Notify) emits an `AuditEvent` to `IAuditWriter` (site-local SQLite append). Hot path; never fails the calling action; failures logged via the `SiteAuditWriteFailures` health metric (see [Component-HealthMonitoring.md](Component-HealthMonitoring.md)). The central audit mirror and event schema live in [Component-AuditLog.md](Component-AuditLog.md).

## Script Scoping Rules

- Scripts can only read/write attributes on **their own instance** (via the parent Instance Actor).
- Scripts can call other scripts on **their own instance** (via sibling Script Actors).
- Scripts can call **shared scripts** (inline execution).
- Scripts **cannot** access other instances' attributes or scripts.
- Alarm on-trigger scripts **can** call instance scripts; instance scripts **cannot** call alarm on-trigger scripts.

---

## Tell vs. Ask Usage

Per Akka.NET best practices, internal actor communication uses **Tell** (fire-and-forget with reply-to) for the hot path:

- **Tag value updates** (DCL → Instance Actor): Tell. High-frequency, no response needed.
- **Attribute change notifications** (Instance Actor → Script/Alarm Actors): Tell. Fan-out notifications.
- **Stream publishing** (Instance Actor → Akka stream): Tell. Fire-and-forget.

**Ask** is reserved for system boundaries where a synchronous response is needed:

- **`Instance.CallScript()`**: Ask pattern from Script Execution Actor to sibling Script Actor. The caller needs the return value. Acceptable because script calls are infrequent relative to tag updates.
- **`Route.To().Call()`**: Ask from Inbound API to site Instance Actor via Communication Layer. External caller needs a response.
- **Debug view snapshot**: Ask from Communication Layer to Instance Actor for initial state.

## Concurrency & Serialization

- The Instance Actor processes messages **sequentially** (standard Akka actor model). This means `SetAttribute` calls from concurrent Script Execution Actors are serialized at the Instance Actor, preventing race conditions on attribute state.
- Script Execution Actors may run concurrently, but all state mutations (attribute reads/writes, alarm state updates) are mediated through the parent Instance Actor's message queue.
- External side effects (external system calls, notifications, database writes) are not serialized — concurrent scripts may produce interleaved side effects. This is acceptable because each side effect is independent.

## SiteStreamManager and gRPC Integration

- The `SiteStreamManager` implements the `ISiteStreamSubscriber` interface, allowing the Communication Layer's `SiteStreamGrpcServer` to subscribe to the stream for cross-cluster delivery via gRPC.
- When a gRPC `SubscribeInstance` call arrives, the `SiteStreamGrpcServer` creates a `StreamRelayActor` and subscribes it to `SiteStreamManager` for the requested instance. Events flow from `SiteStreamManager` → `StreamRelayActor` → `Channel<SiteStreamEvent>` → gRPC response stream to central.
- The `SiteStreamManager` filters events by instance unique name and forwards matching events to all registered subscribers (both local debug consumers and gRPC relay actors).

## Site-Wide Stream Backpressure

- The site-wide Akka stream uses **per-subscriber buffering** with bounded buffers. Each subscriber (gRPC relay actors, future consumers) gets an independent buffer.
- If a subscriber falls behind (e.g., slow network on gRPC stream), its buffer fills and oldest events are dropped. This does not affect other subscribers or the publishing Instance Actors.
- Instance Actors publish to the stream with **fire-and-forget** semantics — publishing never blocks the actor.

## Error Handling

### Script Errors
- Unhandled exceptions and timeouts in Script Execution Actors are **logged locally** to the site event log.
- The Script Actor (coordinator) is **not affected** — it remains active for future trigger events.
- Script failures are **not reported to central** (except as aggregated error rate metrics via Health Monitoring).

### Alarm Evaluation Errors
- Errors during alarm condition evaluation are **logged locally** to the site event log.
- The Alarm Actor remains active and continues evaluating on subsequent value updates.
- Alarm evaluation error rates are reported to central via Health Monitoring.

### Script Compilation Errors
- If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied.
- The failure is reported back to central as a failed deployment.
- Note: Pre-deployment validation at central should catch compilation errors before they reach the site. Site-side compilation failures indicate an unexpected issue.

---

## Dependencies

- **Data Connection Layer**: Provides tag value updates to Instance Actors. Receives write requests from Instance Actors.
- **Store-and-Forward Engine**: Handles reliable delivery for external system calls, cached database writes, and notifications submitted by scripts. For the notification category specifically, it forwards to the central cluster for delivery (not directly to SMTP). Owns the site-local operation tracking table that backs `Tracking.Status(id)`.
- **External System Gateway**: Provides external system method invocations for scripts.
- **Communication Layer**: Receives deployments and lifecycle commands from central. Handles debug view requests. Reports deployment results.
- **Site Event Logging**: Records script executions, alarm events, deployment events, instance lifecycle events.
- **Health Monitoring**: Reports script error rates and alarm evaluation error rates.
- **Local SQLite**: Persists deployed configurations, system-wide artifacts (external system definitions, database connection definitions, data connection definitions). Sites also maintain peer SQLite stores for the Store-and-Forward buffer, the site event log, the operation tracking table, and the site-local `AuditLog` (see [Component-AuditLog.md](Component-AuditLog.md)). The `AuditLog` file is purged on the same daily cadence as the others but respects the hard `ForwardState` invariant — rows still `Pending` forward are never purged, regardless of age.

## Interactions

- **Deployment Manager (central)**: Receives flattened configurations, system-wide artifact updates, and instance lifecycle commands.
- **Data Connection Layer**: Bidirectional — receives value updates, sends write-back commands.
- **Communication Layer**: Receives commands from central, sends deployment results, serves debug view data.
- **Store-and-Forward Engine**: Scripts route cached writes, notifications, and external system calls here.
- **Health Monitoring**: Periodically reports error rate metrics.
