docs(requirements): sync SiteRuntime + DCL specs with hardening changes (C3/C4/C5 + S1-S6, P2/P4/P6, UA1/UA4 behavior notes)

This commit is contained in:
Joseph Doherty
2026-07-09 02:13:13 -04:00
parent c9ac075e39
commit 1db0c32115
2 changed files with 21 additions and 6 deletions
@@ -255,6 +255,7 @@ The `DataConnectionActor` opens **one alarm feed per connection** (not per subsc
- **State gating**: `SubscribeAlarmsRequest` is handled only in the **Connected** state; requests arriving while **Connecting**/**Reconnecting** are stashed (standard Become/Stash) and processed on entering Connected. - **State gating**: `SubscribeAlarmsRequest` is handled only in the **Connected** state; requests arriving while **Connecting**/**Reconnecting** are stashed (standard Become/Stash) and processed on entering Connected.
- **Capability check**: if `_adapter is not IAlarmSubscribableConnection`, the actor replies `SubscribeAlarmsResponse(Success = false, ...)`. - **Capability check**: if `_adapter is not IAlarmSubscribableConnection`, the actor replies `SubscribeAlarmsResponse(Success = false, ...)`.
- **Reconnect handling**: on entering **Reconnecting**, the actor pushes a `NativeAlarmSourceUnavailable` to every alarm subscriber (consumers mark mirrored alarms uncertain rather than clearing them). On successful reconnection it re-subscribes the feed; the adapter re-emits a snapshot, reconciling state. - **Reconnect handling**: on entering **Reconnecting**, the actor pushes a `NativeAlarmSourceUnavailable` to every alarm subscriber (consumers mark mirrored alarms uncertain rather than clearing them). On successful reconnection it re-subscribes the feed; the adapter re-emits a snapshot, reconciling state.
- **Shared condition filter (last-subscriber-wins)**: because the feed is opened once per source, it carries a single condition filter. A second subscriber that registers a *different* filter for the same source overwrites it (last writer wins) and the actor logs a warning — co-subscribers to one source are expected to agree on the filter.
### Protocol-Neutral Types & Messages ### Protocol-Neutral Types & Messages
@@ -299,10 +300,11 @@ This pattern ensures no messages are lost during connection transitions and is t
The DCL manages connection lifecycle automatically: The DCL manages connection lifecycle automatically:
1. **Connection drop detection**: When a connection to a data source is lost, the DCL immediately pushes a value update with quality `bad` for **every tag subscribed on that connection**. Instance Actors and their downstream consumers (alarms, scripts checking quality) see the staleness immediately. 1. **Connection drop detection**: When a connection to a data source is lost, the DCL immediately pushes a value update with quality `bad` for **every tag subscribed on that connection**. Instance Actors see the staleness immediately and publish it to the site stream / debug view; **script and alarm actors are deliberately NOT re-notified on a quality-only change** (no value changed — re-evaluating triggers would cause spurious firings). Scripts observe quality on their next attribute read; a computed alarm holds its last-known state until fresh values arrive.
2. **Auto-reconnect with fixed interval**: The DCL retries the connection at a configurable fixed interval (e.g., every 5 seconds). The retry interval is defined **per data connection**. This is consistent with the fixed-interval retry philosophy used throughout the system. Individual gRPC/OPC UA operations (reads, writes) fail immediately to the caller on error; there is no operation-level retry within the adapter. 2. **Auto-reconnect with fixed interval**: The DCL retries the connection at a configurable fixed interval (e.g., every 5 seconds). The retry interval is a **shared setting for all data connections** (`ReconnectInterval` in the Shared Settings table). This is consistent with the fixed-interval retry philosophy used throughout the system. Individual gRPC/OPC UA operations (reads, writes) fail immediately to the caller on error; there is no operation-level retry within the adapter.
3. **Connection state transitions**: The DCL tracks each connection's state as `connected`, `disconnected`, or `reconnecting`. All transitions are logged to Site Event Logging. 3. **Connection state transitions**: The DCL tracks each connection's state as `connected`, `disconnected`, or `reconnecting`. All transitions are logged to Site Event Logging.
4. **Transparent re-subscribe**: On successful reconnection, the DCL automatically re-establishes all previously active subscriptions for that connection. Instance Actors require no action — they simply see quality return to `good` as fresh values arrive from restored subscriptions. 4. **Transparent re-subscribe**: On successful reconnection, the DCL automatically re-establishes all previously active subscriptions for that connection. Instance Actors require no action — they simply see quality return to `good` as fresh values arrive from restored subscriptions.
5. **Clean client replacement on (re)connect**: On every (re)connect the adapter detaches and disposes the previous protocol client (session, keep-alive timers, event loop) before creating a new one — a stale client can neither leak nor signal a disconnect against the new session.
### Disconnect Detection Pattern ### Disconnect Detection Pattern
+17 -4
View File
@@ -94,6 +94,10 @@ flowchart TD
4. Create Instance Actors for all deployed, **enabled** instances as child actors. Instance Actors are created in **staggered batches** (e.g., 20 at a time with a short delay between batches) to prevent a reconnection storm — 500 Instance Actors all registering data subscriptions simultaneously would overwhelm OPC UA servers and network capacity. 4. Create Instance Actors for all deployed, **enabled** instances as child actors. Instance Actors are created in **staggered batches** (e.g., 20 at a time with a short delay between batches) to prevent a reconnection storm — 500 Instance Actors all registering data subscriptions simultaneously would overwhelm OPC UA servers and network capacity.
5. Make compiled shared script code available to all Script Actors. 5. Make compiled shared script code available to all Script Actors.
> **Startup config load is retried (S5)**: step 1's deployed-config read is best-effort against local SQLite, which can transiently fail (locked/busy). A failed load must **not** leave the site as a silent zero-instance node — it is re-attempted on a fixed interval until it succeeds, then the retry self-cancels.
> **Per-instance compilation during staggered startup (P6, follow-up)**: the Roslyn compile of each instance's scripts still runs inside Instance Actor start during staggered startup; moving it off-thread is a deferred optimization (it affects failover time-to-recover only, not correctness).
### Deployment Handling ### Deployment Handling
- Receives flattened instance configurations from central via the Communication Layer. - Receives flattened instance configurations from central via the Communication Layer.
- Stores the new configuration in local SQLite. - Stores the new configuration in local SQLite.
@@ -102,6 +106,8 @@ flowchart TD
- For redeployments: the existing Instance Actor and all its children are stopped, then a new Instance Actor is created with the updated configuration. Subscriptions are re-established. - For redeployments: the existing Instance Actor and all its children are stopped, then a new Instance Actor is created with the updated configuration. Subscriptions are re-established.
- Reports deployment result (success/failure) back to central. - Reports deployment result (success/failure) back to central.
> **Dead-child bookkeeping + two-signal deploy join (S6).** All Instance Actors are `Watch`ed; an unexpected `Terminated` evicts the actor ref from the singleton's map. **Deviation from the naive design (2026-07-09, see master-tracker "Plan deviations during Wave 1 PLAN-03 execution"):** deploy-success is **NOT** reported on persistence alone — the config store can commit before the Instance Actor's asynchronous `PreStart` throws. The design therefore uses a **two-signal join**: the Instance Actor sends a new `InstanceActorInitialized` readiness message at the end of a successful `PreStart`, and the Deployment Manager reports `Success` only once **both** persistence has committed **and** that readiness signal has arrived. An actor that dies during init never signals readiness → the `Terminated` fallback replies `Failed` and rolls the optimistic in-memory state back (the persisted-row rollback is deferred until the store commits so it cannot race the write).
### System-Wide Artifact Handling ### System-Wide Artifact Handling
- Receives updated shared scripts, external system definitions, database connection definitions, and data connection definitions from central. (Notification lists and SMTP configuration are central-only and are not deployed to sites — see Component-NotificationService.md.) - Receives updated shared scripts, external system definitions, database connection definitions, and data connection definitions from central. (Notification lists and SMTP configuration are central-only and are not deployed to sites — see Component-NotificationService.md.)
- Stores all artifacts in local SQLite. After artifact deployment, the site is fully self-contained — all runtime configuration is read from local SQLite with no access to the central configuration database. - Stores all artifacts in local SQLite. After artifact deployment, the site is fully self-contained — all runtime configuration is read from local SQLite with no access to the central configuration database.
@@ -123,6 +129,7 @@ flowchart TD
- **`RemoveServerCertCommand`** — delete a certificate (by thumbprint) from the node's stores. - **`RemoveServerCertCommand`** — delete a certificate (by thumbprint) from the node's stores.
- **`ListServerCertsCommand`** — enumerate the node's trusted-peer + rejected store contents (Subject / Issuer / Thumbprint / validity / status). - **`ListServerCertsCommand`** — enumerate the node's trusted-peer + rejected store contents (Subject / Issuer / Thumbprint / validity / status).
- The Deployment Manager singleton receives the central cert-trust verbs (relayed via the Communication Layer) and **broadcasts** `TrustServerCertCommand` / `RemoveServerCertCommand` to the `CertStoreActor` on **both** site nodes (via `ActorSelection`), so node-a and node-b PKI stores stay in sync. List reads from the local node. Trust is **site-local** — there is no central persistence of trusted certs (logged as a follow-up). The captured-but-untrusted server cert that seeds this flow comes from the DCL verify-endpoint probe (see Component-DataConnectionLayer.md), which never trusts on its own; the Central UI surfaces Trust / Remove (Admin-gated — see Component-CentralUI.md). - The Deployment Manager singleton receives the central cert-trust verbs (relayed via the Communication Layer) and **broadcasts** `TrustServerCertCommand` / `RemoveServerCertCommand` to the `CertStoreActor` on **both** site nodes (via `ActorSelection`), so node-a and node-b PKI stores stay in sync. List reads from the local node. Trust is **site-local** — there is no central persistence of trusted certs (logged as a follow-up). The captured-but-untrusted server cert that seeds this flow comes from the DCL verify-endpoint probe (see Component-DataConnectionLayer.md), which never trusts on its own; the Central UI surfaces Trust / Remove (Admin-gated — see Component-CentralUI.md).
- **Reconcile-on-join (UA1)**: a broadcast only reaches nodes that are Up at the time, so a node that was down when a cert was trusted would stay divergent forever. To close that gap, the Deployment Manager singleton subscribes to cluster `MemberUp` events; when a site node (re)joins, it reads its own trusted-peer store (`ExportLocalTrustedCerts``CertStoreActor`) and pushes each certificate to the joining node's `CertStoreActor` (additive union; each `WriteCertToLocalStore` is idempotent by thumbprint). **Removal** reconciliation and central persistence of trust decisions remain the documented follow-up — only additions converge automatically.
--- ---
@@ -196,6 +203,8 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
- **WhileTrue**: On the `false → true` edge the script fires once, then re-fires on a periodic timer while the condition stays true; on the `true → false` edge the timer stops. The re-fire cadence is the script's **minimum time between runs**; with none configured the trigger degrades to the single edge fire and logs a warning. - **WhileTrue**: On the `false → true` edge the script fires once, then re-fires on a periodic timer while the condition stays true; on the `true → false` edge the timer stops. The re-fire cadence is the script's **minimum time between runs**; with none configured the trigger degrades to the single edge fire and logs a warning.
- **Minimum time between runs**: If configured, the Script Actor tracks the last execution time and skips trigger invocations that fire before the minimum interval has elapsed. For a WhileTrue trigger it doubles as the re-fire cadence. - **Minimum time between runs**: If configured, the Script Actor tracks the last execution time and skips trigger invocations that fire before the minimum interval has elapsed. For a WhileTrue trigger it doubles as the re-fire cadence.
> **Per-attribute change routing (P2).** Rather than broadcasting every attribute change to all child Script/Alarm Actors, the Instance Actor routes each change only to the children that monitor that attribute (built from their trigger configs at spawn time); **Expression-trigger children, which read an attribute snapshot, receive all changes**. Each child still keeps its own trigger gate as defense-in-depth — the routing is an optimization, not the authority.
### Concurrent Execution ### Concurrent Execution
- Each invocation spawns a **new Script Execution Actor** as a child. - Each invocation spawns a **new Script Execution Actor** as a child.
- Multiple Script Execution Actors can run concurrently (e.g., a trigger fires while a previous `Instance.CallScript` invocation is still running). - Multiple Script Execution Actors can run concurrently (e.g., a trigger fires while a previous `Instance.CallScript` invocation is still running).
@@ -300,8 +309,10 @@ The Instance Actor owns native-alarm setup alongside its computed Script and Ala
`SiteStorageService` gains a `native_alarm_state` table backing the native mirror: `SiteStorageService` gains a `native_alarm_state` table backing the native mirror:
- **Primary key**: `(instance_unique_name, source_canonical_name, source_reference)` — one row per mirrored condition. - **Primary key**: `(instance_unique_name, source_canonical_name, source_reference)` — one row per mirrored condition.
- **Columns**: `condition_json` (the serialized `AlarmConditionState`) and `last_transition_at` (the accepted `TransitionTime`). - **Columns**: `condition_json` (the serialized `AlarmConditionState`), `metadata_json` (display metadata — type/category/message/current+limit, UA4), and `last_transition_at` (the accepted `TransitionTime`).
- **Operations**: `Upsert` (on accepted transition), `Delete` (on condition drop-out), `Get` (PreStart rehydrate, scoped to instance + source), and `ClearForInstance` (redeploy/undeploy reset). - **Operations**: `Upsert` (on accepted transition), `Delete` (on condition drop-out), `Get` (PreStart rehydrate, scoped to instance + source), and `ClearForInstance` (redeploy/undeploy reset).
- **Display metadata persistence (UA4)**: persisted rows include the condition's display metadata, so **rehydrated conditions render fully** (type / category / message / current + limit values) before the first fresh source snapshot arrives; a pre-UA4 row with no metadata rehydrates with empty display fields (unchanged behavior).
- **Coalesced writes (P4)**: per-transition upserts are coalesced (latest-per-source-reference) and flushed on a short single-shot timer rather than one SQLite write per transition, so an alarm storm collapses to one batched write; a pending upsert is dropped if the condition is deleted before the flush (no resurrection). Durability window is at most one flush interval on crash, reconciled by the next re-subscribe snapshot.
- This is a **peer SQLite store** to the existing deployed-configuration, store-and-forward, operation-tracking, and `AuditLog` stores. Unlike computed alarm state, native mirror state is intentionally persisted so it survives failover. - This is a **peer SQLite store** to the existing deployed-configuration, store-and-forward, operation-tracking, and `AuditLog` stores. Unlike computed alarm state, native mirror state is intentionally persisted so it survives failover.
--- ---
@@ -330,6 +341,7 @@ The enriched message flows Instance Actor → site-wide Akka stream → `SiteStr
- When a Script Execution Actor calls `Scripts.CallShared("scriptName", params)`, the shared script code executes **inline** in the Script Execution Actor's context — it is a direct method invocation, not an actor message. - When a Script Execution Actor calls `Scripts.CallShared("scriptName", params)`, the shared script code executes **inline** in the Script Execution Actor's context — it is a direct method invocation, not an actor message.
- This avoids serialization bottlenecks since there is no shared script actor to contend for. - This avoids serialization bottlenecks since there is no shared script actor to contend for.
- Shared scripts have access to the same runtime API as instance scripts (GetAttribute, SetAttribute, external systems, notifications, databases). - Shared scripts have access to the same runtime API as instance scripts (GetAttribute, SetAttribute, external systems, notifications, databases).
- **Shared scripts always execute with Root scope (C5)**: `Attributes[...]` inside a shared script addresses **root** attributes even when the shared script is invoked from a composed-module script. Callers needing module-scoped access must pass the values in as parameters.
--- ---
@@ -408,7 +420,7 @@ Scripts execute **in-process** with constrained access. The following restrictio
- **Allowed**: Access to the Script Runtime API (GetAttribute, SetAttribute, CallScript, CallShared, ExternalSystem, Notify, Database, Tracking), standard C# language features, basic .NET types (collections, string manipulation, math, date/time). `System.Diagnostics.Stopwatch`, `Debug`, and `Activity` are permitted. - **Allowed**: Access to the Script Runtime API (GetAttribute, SetAttribute, CallScript, CallShared, ExternalSystem, Notify, Database, Tracking), standard C# language features, basic .NET types (collections, string manipulation, math, date/time). `System.Diagnostics.Stopwatch`, `Debug`, and `Activity` are permitted.
- **Forbidden**: File system access (`System.IO`), process spawning (`System.Diagnostics.Process`), threading (`System.Threading` — except `Tasks`, `CancellationToken`, and `CancellationTokenSource`), reflection (`System.Reflection`), all raw network access (`System.Net` — must use `ExternalSystem.Call`), native interop (`System.Runtime.InteropServices`, `Microsoft.Win32`), assembly loading, unsafe code, `dynamic`, `Activator`. - **Forbidden**: File system access (`System.IO`), process spawning (`System.Diagnostics.Process`), threading (`System.Threading` — except `Tasks`, `CancellationToken`, and `CancellationTokenSource`), reflection (`System.Reflection`), all raw network access (`System.Net` — must use `ExternalSystem.Call`), native interop (`System.Runtime.InteropServices`, `Microsoft.Win32`), assembly loading, unsafe code, `dynamic`, `Activator`.
- **Execution timeout**: Configurable per-script maximum execution time. Exceeding the timeout cancels the script and logs an error. - **Execution timeout**: Configurable per-script maximum execution time. Exceeding the timeout cancels the script **cooperatively** (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A **watchdog logs the script by name** once its thread has not returned within a grace period after the timeout, and the script-execution scheduler's **queue depth / busy-thread / oldest-busy-age gauges** are reported through Health Monitoring so a saturated or stuck scheduler is visible.
- **Memory**: Scripts share the host process memory. No per-script memory limit, but the execution timeout prevents runaway allocations. - **Memory**: Scripts share the host process memory. No per-script memory limit, but the execution timeout prevents runaway allocations.
The forbidden-API policy is defined authoritatively in `ScriptTrustPolicy` (Script Analysis component, #25). `ScriptCompilationService.ValidateTrustModel` delegates to `ScriptTrustValidator.FindViolations` for the trust verdict; the Site Runtime also performs a `CSharpScript.Compile` against the real `ScriptGlobals` for execution. This is defence-in-depth static enforcement, not a true runtime sandbox. The forbidden-API policy is defined authoritatively in `ScriptTrustPolicy` (Script Analysis component, #25). `ScriptCompilationService.ValidateTrustModel` delegates to `ScriptTrustValidator.FindViolations` for the trust verdict; the Site Runtime also performs a `CSharpScript.Compile` against the real `ScriptGlobals` for execution. This is defence-in-depth static enforcement, not a true runtime sandbox.
@@ -474,14 +486,15 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
### Script Compilation Errors ### Script Compilation Errors
- If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied. - If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied.
- The failure is reported back to central as a failed deployment. - The failure is reported back to central as a failed deployment.
- Note: Pre-deployment validation at central should catch compilation errors before they reach the site. Site-side compilation failures indicate an unexpected issue. - **Enforced by a site-side pre-compile validation gate (S3)**: before the Instance Actor is created or the config persisted, the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **off-thread**; any compile failure rejects the deployment with the collected compile errors. This makes the deployment result honest even if central's pre-deployment validation was bypassed or skew let a bad artifact through.
- Note: Pre-deployment validation at central should still catch compilation errors before they reach the site; the site gate is the authoritative backstop.
--- ---
## Dependencies ## Dependencies
- **Script Analysis (#25)**: `ScriptCompilationService.ValidateTrustModel` delegates the forbidden-API verdict to `ScriptTrustValidator.FindViolations`. The Site Runtime retains its own `CSharpScript.Compile` against the real `ScriptGlobals` for execution. - **Script Analysis (#25)**: `ScriptCompilationService.ValidateTrustModel` delegates the forbidden-API verdict to `ScriptTrustValidator.FindViolations`. The Site Runtime retains its own `CSharpScript.Compile` against the real `ScriptGlobals` for execution.
- **Data Connection Layer**: Provides tag value updates to Instance Actors. Receives write requests from Instance Actors. Also feeds Native Alarm Actors: connections implementing `IAlarmSubscribableConnection` (OPC UA A&C servers, MxAccess Gateway) deliver `NativeAlarmTransitionUpdate` events in response to a `SubscribeAlarmsRequest`, and signal `NativeAlarmSourceUnavailable` on connection loss. - **Data Connection Layer**: Provides tag value updates to Instance Actors. Receives write requests from Instance Actors. Also feeds Native Alarm Actors: connections implementing `IAlarmSubscribableConnection` (OPC UA A&C servers, MxAccess Gateway) deliver `NativeAlarmTransitionUpdate` events in response to a `SubscribeAlarmsRequest`, and signal `NativeAlarmSourceUnavailable` on connection loss. **Subscription-handshake retry (S4/UA6)**: a tag-subscription handshake that either failed or whose response was lost is re-sent on a fixed interval per connection (default 5 s) — the retry is armed on every send and cancelled on the first success reply, closing both the failed-response and lost-response gaps; the `NativeAlarmActor` subscribe uses the same retry pattern.
- **Store-and-Forward Engine**: Handles reliable delivery for external system calls, cached database writes, and notifications submitted by scripts. For the notification category specifically, it forwards to the central cluster for delivery (not directly to SMTP). Owns the site-local operation tracking table that backs `Tracking.Status(id)`. - **Store-and-Forward Engine**: Handles reliable delivery for external system calls, cached database writes, and notifications submitted by scripts. For the notification category specifically, it forwards to the central cluster for delivery (not directly to SMTP). Owns the site-local operation tracking table that backs `Tracking.Status(id)`.
- **External System Gateway**: Provides external system method invocations for scripts. - **External System Gateway**: Provides external system method invocations for scripts.
- **Communication Layer**: Receives deployments and lifecycle commands from central. Handles debug view requests. Reports deployment results. - **Communication Layer**: Receives deployments and lifecycle commands from central. Handles debug view requests. Reports deployment results.