Joseph Doherty
693 lines
117 KiB
Markdown
693 lines
117 KiB
Markdown
# Component: Site Runtime
|
||
|
||
## Purpose
|
||
|
||
The Site Runtime component manages the execution of deployed machine instances at site clusters. It encompasses the actor hierarchy that represents running instances, their scripts, and their alarms. It owns the site-side deployment lifecycle (receiving configs from central, compiling scripts, creating actors), script execution, alarm evaluation, and the site-wide Akka stream for attribute and alarm state changes.
|
||
|
||
This component replaces the previously separate Script Engine and Alarm Engine concepts, unifying them under a single actor hierarchy rooted at the Deployment Manager singleton.
|
||
|
||
## Location
|
||
|
||
Site clusters only.
|
||
|
||
## Responsibilities
|
||
|
||
- Run the Deployment Manager singleton (Akka.NET cluster singleton) on the active site node.
|
||
- On startup (or failover), read all deployed configurations from local SQLite and re-create the full actor hierarchy.
|
||
- Receive deployment commands from central: new/updated instance configurations, instance lifecycle commands (disable, enable, delete), and system-wide artifact updates.
|
||
- Compile C# scripts when deployments are received.
|
||
- Manage the Instance Actor hierarchy (Instance Actors, Script Actors, Alarm Actors).
|
||
- Execute scripts via Script Actors with support for concurrent execution.
|
||
- Evaluate alarm conditions via Alarm Actors and manage alarm state.
|
||
- Maintain the site-wide Akka stream for attribute value and alarm state changes.
|
||
- Execute shared scripts inline as compiled code libraries (no separate actors).
|
||
- Enforce script call recursion limits.
|
||
|
||
---
|
||
|
||
## Actor Hierarchy
|
||
|
||
```mermaid
|
||
%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
|
||
flowchart TD
|
||
DMS["Deployment Manager Singleton<br/>(Cluster Singleton)"]
|
||
IA1["Instance Actor<br/>('MachineA-001')"]
|
||
IA2["Instance Actor<br/>('MachineA-002')"]
|
||
IAMORE["… more Instance Actors"]
|
||
|
||
SA1["Script Actor ('MonitorSpeed')<br/>— coordinator"]
|
||
SA2["Script Actor ('CalculateOEE')<br/>— coordinator"]
|
||
AA1["Alarm Actor ('OverTemp')<br/>— coordinator (computed)"]
|
||
AA2["Alarm Actor ('LowPressure')<br/>— coordinator (computed)"]
|
||
NAA1["Native Alarm Actor ('OpcUaServer1')<br/>— read-only mirror, peer to Alarm Actor"]
|
||
|
||
SEA1["script run<br/>— launched task, not an actor"]
|
||
SEA2["script run<br/>— launched task, not an actor"]
|
||
AEA1["alarm on-trigger run<br/>— launched task, not an actor"]
|
||
|
||
IA2CHILD["… (Script / Alarm Actors)"]
|
||
|
||
DMS --> IA1
|
||
DMS --> IA2
|
||
DMS -.-> IAMORE
|
||
|
||
IA1 --> SA1
|
||
IA1 --> SA2
|
||
IA1 --> AA1
|
||
IA1 --> AA2
|
||
IA1 --> NAA1
|
||
|
||
SA1 -.-> SEA1
|
||
SA2 -.-> SEA2
|
||
AA1 -.-> AEA1
|
||
|
||
IA2 -.-> IA2CHILD
|
||
|
||
classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
|
||
classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
|
||
classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
|
||
classDef warn fill:#ffe6cc,stroke:#d79b00,color:#111111;
|
||
classDef bad fill:#f8cecc,stroke:#b85450,color:#111111;
|
||
classDef alt fill:#e1d5e7,stroke:#9673a6,color:#111111;
|
||
classDef muted fill:#f5f5f5,stroke:#999999,color:#666666;
|
||
class DMS proc
|
||
class IA1,IA2 start
|
||
class SA1,SA2 dec
|
||
class AA1,AA2 bad
|
||
class NAA1 alt
|
||
class SEA1,SEA2,AEA1 warn
|
||
class IAMORE,IA2CHILD muted
|
||
```
|
||
|
||
---
|
||
|
||
## Deployment Manager Singleton
|
||
|
||
### Role
|
||
- Akka.NET **cluster singleton** — guaranteed to run on exactly one node in the site cluster (the active node).
|
||
- On failover, Akka.NET restarts the singleton on the new active node.
|
||
|
||
### Startup Behavior
|
||
1. Read all deployed configurations from local SQLite.
|
||
2. Read all shared scripts from local storage.
|
||
3. Compile all scripts (instance scripts, alarm on-trigger scripts, shared scripts).
|
||
4. Create Instance Actors for all deployed, **enabled** instances as child actors. Instance Actors are created in **staggered batches** (e.g., 20 at a time with a short delay between batches) to prevent a reconnection storm — 500 Instance Actors all registering data subscriptions simultaneously would overwhelm OPC UA servers and network capacity.
|
||
5. Make compiled shared script code available to all Script Actors.
|
||
|
||
> **Startup config load is retried (S5)**: step 1's deployed-config read is best-effort against local SQLite, which can transiently fail (locked/busy). A failed load must **not** leave the site as a silent zero-instance node — it is re-attempted on a fixed interval until it succeeds, then the retry self-cancels.
|
||
|
||
> **Startup reconcile heals the RUNTIME, not just the config file (register row 67)**: `SiteReconciliationActor` runs on **every** site node (not a singleton) and performs one best-effort pass shortly after boot — it reports the node's local deployed inventory to central and fetches + stores any config central reports as a gap. That pass fires **after** step 1's deployed-config read has already run, so storing a config started nothing: Instance Actors are created only by the boot-time load above and by an explicit deploy/enable. An instance whose config was missing at boot (lost volume, first boot after a restore, rehomed data dir) therefore stayed **dead until a second restart**, whose load finally picked it up — silently, with nothing surfacing "config stored, actor not running". The pass now raises a fire-and-forget `EnsureInstanceStartedCommand` for every **enabled** instance it healed, routed through the **deployment-manager singleton proxy** so Instance Actors are only ever created under the singleton regardless of which node's pass found the gap. The handler is deliberately advisory: it **no-ops when the actor is already running** (both nodes legitimately raise it for the same instance), **re-reads the durable row** — which is the authority on both the config body and enabled/disabled state, so a disable landing after the fetch still wins and the instance stays stopped — **defers to any in-flight deploy/redeploy** for that instance, preserving the deploy gate's ordering guarantees and the all-or-nothing-per-instance rule (it never applies a config, only starts one already fully persisted), and treats a missing row as a quiet no-op (the normal case on an **unreplicated** site whose peer node healed its own file). A successful reconcile start is **surfaced**, not silent: a Warning log line plus an operational `deployment` site event, because reaching it means the node booted without a config for an instance that should have been running.
|
||
|
||
> **Instance reconcile sweep — the store and the runtime are kept in agreement, and disagreement is surfaced (register rows 78 + 72)**: the reconcile self-heal above covers configs the reconcile pass *itself* fetched. Two sibling divergences were not covered, and neither surfaced anything.
|
||
> - **Row 78 — a config that arrives by replication starts nothing.** `deployed_configurations` is one of LocalDb's ten replicated tables, so on a replicated pair a config can appear in a node's file with **no local actor involved at all** — the CDC sync applier writes the row directly. Proven live 2026-08-15: a fresh site-a-a was seeded from its peer by snapshot replication within ~5 s of boot, i.e. **after** its boot-time load had read an empty table and **invisibly** to its own reconcile (central rightly reported no gap, because central's view of the site was correct). Enabled instances sat dead with their configs present until another restart.
|
||
> - **Row 72 — a refreshed config does not reach the running actor.** For a *stale* `ReconcileGapItem` the pass writes the newer config via `StoreDeployedConfigIfNewerAsync`, a pure file write; the running Instance Actor keeps executing the previously compiled config until the next restart or redeploy.
|
||
>
|
||
> Both are closed by one **periodic sweep owned by the Deployment Manager singleton** (`ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds`, default 60, validated `>= 0`; `0` disables it). Each tick re-reads the durable rows off-thread and, on the actor thread, for every **enabled** stored config: starts the Instance Actor if none is running (row 78), or — if one is running at a revision the store has since replaced — **re-applies the stored config through the exact per-instance apply/restart path a central redeploy uses** (row 72). A sweep is chosen over a LocalDb change hook deliberately: a hook would cover only the replication arrival path, only on replicated sites, and `ILocalDb` exposes no change-notification API at all, whereas the sweep is arrival-path-agnostic (replication seeding, post-boot snapshot resync, a hand-restored file, any future path). See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`.
|
||
>
|
||
> The **row-72 decision is to converge, not merely warn**: the stored row *is* what central successfully deployed, so an actor serving the config central believes was replaced is the actual fault. Reusing `HandleDeploy` means the compile gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm reset and all-or-nothing-per-instance are the production ones — and a stop still does **not** cancel an in-flight script run, exactly as for a central redeploy.
|
||
>
|
||
> Guards, all shared with the row-67 handler: a **disabled** instance is never started; the sweep **defers to any deploy pipeline that owns the instance** (compile warm in flight, predecessor still terminating, deployment awaiting its persistence/init join) so it can never interleave with a real redeploy; it is **inert until boot-time Instance Actor creation has finished**, so a tick between staggered batches does not "heal" instances the boot load is about to create; it makes **one repair attempt per (instance, revision) pair**, which is what stops a periodic self-heal becoming a restart loop for an actor that dies during init or a stored config the compile gate rejects (a rejection leaves the running instance untouched — the gate returns before anything is stopped); and reads never overlap. One guard is **not** in that list because it needed adding later: deferring to an in-flight deploy pipeline says nothing about a snapshot that landed *after* the pipeline finished, which is the stale-snapshot race — a tick whose read raced this node's own deploy/enable/disable/delete is discarded whole by the store-write generation check (see **Stale-snapshot guard** below). A sweep that finds nothing is **completely silent**. A repair emits the same Warning `deployment` site-event pattern as row 67, naming the trigger (`deployment-manager sweep self-heal`) so it is distinguishable from a startup-reconcile heal, and a stale-revision restart names **both** revisions; a rejected restart adds an Error event. The synthetic redeploy replies to the singleton itself rather than to `NoSender`, because dead letters are a monitored site health metric.
|
||
|
||
> **Starting an Instance Actor also creates its DCL connections — the ensure lives in the creation funnel, not at call sites (register row 85)**: a DCL connection actor is **never** created from the `data_connection_definitions` table — that stayed true when rows 89/90 gave the table a reader, because the overlay it feeds is an *arbiter*, not a creation source, and at the time of row 85 nothing read the table at all. A connection actor exists only as a side effect of starting an instance, built from the `Connections` map inside that instance's **own flattened config** via `EnsureDclConnections` — plus the separate system-wide artifact-deploy path. That ensure used to be each *caller's* responsibility, and only two of the four callers made it (the boot-time staggered batch and `ApplyDeployment`). The two that did not were **`StartStoredInstance`** — the path shared by the row-67 ensure-started handler and the row-78 sweep — and **`HandleEnableResult`**. Both therefore produced a running instance whose every tag subscribe was answered `Unknown connection: <name>` by the DCL manager, indefinitely, because nothing else on the node would ever create that connection actor. Proven live 2026-08-16: a replication-seeded node's instances were correctly started by the sweep and then failed every subscribe for 6+ minutes until a second pair restart let the boot load create the connections. **It was never replication-specific** — the boot-time load reads only *enabled* configs, so a node that boots with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. `EnsureDclConnections` now sits at the top of **`CreateInstanceActor`**, already the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), so all four paths get it and a fifth cannot forget. It is placed *before* the already-exists guard so the two paths that already called it keep their exact previous semantics, and it stays silent in steady state because `EnsureDclConnection` is hash-guarded on the last config pushed per connection name. Deliberately **not** done at the time: a sweep pass over `data_connection_definitions` — it would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. See `docs/plans/2026-08-15-instance-sweep-self-heal-design.md` §7. That pass is now present, in a form that resolves the conflict rather than creating it — see **DCL connection arbitration** immediately below.
|
||
|
||
> **DCL connection arbitration — the definitions store wins wherever it has an opinion (register rows 89 + 90)**: `data_connection_definitions` now has a production reader. The rule is: **a connection name with a row in that table is configured from that row; a name with no row is configured from the instance's flattened `Connections` entry, exactly as before.** Rule 2 is not a fallback — it is the correct answer for a deploy-only site, where the table is legitimately empty while every connection that matters lives inline in the instance configs.
|
||
>
|
||
> **The register's premise needed two corrections, and they change what the fix is.** First, the peer node's DCL is not "serving the old endpoint" — it is **empty**. The Deployment Manager is a `site-{SiteId}`-scoped singleton and the *only* sender of `CreateConnectionCommand`/`RemoveConnectionCommand`, sending to its own node's `dcl-manager` (which exists on every node but is populated only where the singleton runs). The real defect is that `_createdConnections`, the per-name hash cache, is **in-memory and per singleton incarnation**: a restart or failover rebuilds every connection from the `Connections` snapshot frozen into each instance's flattened config at *that instance's* deploy time, and a newer connection-only artifact deploy is silently discarded. **That makes row 89 create-time, not message-delivery, and not replication-specific** — the identical loss happens on an unreplicated single-node restart. Second, `EnsureDclConnection`'s documented "a changed config re-issues the create so the DCL adopts it" was **false**: `DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and returns for a name it already holds), so the artifact deploy's live push has never worked on the deploy-handling node either. That second correction was left as a residual by rows 89/90 and is closed by **row 97** — see *In-place reconfiguration* below.
|
||
>
|
||
> **Mechanism.** `data_connection_definitions` is read in the *same* off-thread task as `deployed_configurations` (`ReadStoreSnapshotAsync`), by both the boot load and every sweep tick, and applied to an actor-thread-confined **arbitration overlay** *before* the boot batch creates any Instance Actor and *before* the sweep's repair loop — so a connection is built from the winning config on its **first** push rather than created stale and corrected a tick later. The overlay is an **arbiter, not a creation source**: a definition row naming a connection no deployed instance references creates nothing. A CDC-delivered definition change therefore converges within one sweep tick on whichever node runs the singleton, with **no new cross-node message**. A definitions read that *fails* yields `null` and leaves the overlay untouched — deliberately distinct from an empty list, which is a real answer; clearing on a transient failure would reintroduce the ping-pong through the error path.
|
||
>
|
||
> **The ping-pong is killed by keying the hash on the winner.** The conflict §7.4 identified is a property of *where the hash is computed*, not of the sweep: every ensure path now resolves the winning configuration **before** hashing or pushing, so for any name only one configuration can ever occupy the hash slot regardless of which path calls. The flip is not guarded against — it is unrepresentable. `HandleApplyArtifactDataConnectionsToDcl` writes its rows into the overlay in the same pass it pushes them, closing the same flip arriving through the artifact-deploy door.
|
||
>
|
||
> **Stale-snapshot guard — covers the WHOLE snapshot, both halves and both consumers (row 101).** `ReadStoreSnapshotAsync` reads `deployed_configurations` and `data_connection_definitions` off the actor thread but is consumed only when its piped message lands, so a read dispatched before a mutation can land after the actor's own state has already moved past it. Every read therefore captures a single monotonic **store-write generation** on the actor thread at dispatch (`_storeWriteGeneration`), and a snapshot carrying a stale generation is discarded **whole**. One counter covers both tables on purpose: the snapshot is one consistent read and is discarded whole anyway, so a per-table counter would only buy the chance to apply half a snapshot the other half's mutation already invalidated.
|
||
>
|
||
> Each half would do real damage. **Definitions half:** stale absences "prove" just-applied names disappeared — tombstoned, live connection actors stopped, and because the removal evicts the hash slot the overlay's push is gated on, **nothing re-creates them until an instance redeploy**; stale rows push superseded configs at live connections and flip-flop with the next fresh read. **Configs half, row-72 leg:** a snapshot carrying the revision a deploy just replaced makes the stale-revision repair run **backward**, re-applying the snapshot's own superseded config JSON through the redeploy path — and `InstanceIsOwnedByADeployPipeline` is no defence, because it only holds while a deploy is *in flight* and the stale snapshot can land after the pipeline released the instance. **Configs half, row-78 leg:** a snapshot read before an undeploy/delete or a disable committed still shows an enabled row while the actor is already stopped, so the start repair **resurrects a just-stopped instance** — and delete specifically clears `_sweepAttemptedRevisions`, so the once-per-revision claim is freshly available and does not stand in the way either. All three were reproduced by neutralising the guard and re-running the regression tests.
|
||
>
|
||
> **The generation is bumped twice per mutation — when the off-thread write is dispatched and again when its completion message lands on the actor thread.** The pair is what leaves no window: an initiation-only bump would let a read dispatched *between* the decision and the commit compare equal and be applied even though it predates the write, and a completion-only bump would miss a read dispatched before the handler ran. With both, a read dispatched before the mutation is discarded, a read dispatched during it is discarded, and a read dispatched after the completion bump necessarily sees the committed row. Bump sites are every actor-thread path that mutates either table: deploy persist (dispatch + `DeployPersistenceResult`), enable (dispatch + `EnableResult`), disable and delete and the init-failure rollback (dispatch + a `StoreWriteCompleted` self-message that exists purely to bump after the commit), and the artifact apply (dispatch + `ApplyArtifactDataConnectionsToDcl`).
|
||
>
|
||
> **Writes that must NOT bump**, and this is load-bearing: rows arriving by LocalDb CDC replication, by a snapshot resync, by an operator restoring the file, or by `SiteReconciliationActor`'s startup fetch never pass through this actor — and discovering exactly those is the sweep's entire reason to exist (rows 72 + 78). They are not stale relative to actor state; they are the state the actor has yet to learn about.
|
||
>
|
||
> **The two consumers differ.** A stale **sweep** result is dropped whole at Debug — no overlay apply, no start, no restart — because there is nothing to complete and the next tick is already a fresh read. A stale **boot** load cannot be dropped (a node whose startup read is discarded stays a silent zero-instance node), so it **re-dispatches `LoadDeployedConfigs()`**, capturing the current generation. Staleness requires a fresh mutation inside each attempt's window, so this converges in practice; as a livelock backstop the fifth consecutive superseded boot read is applied anyway with a Warning, on the grounds that a boot that never finishes is worse than one starting from a slightly superseded view that the sweep converges on its next tick. A superseded boot snapshot arriving *after* the boot has completed (the S5 retry timer racing a success) is dropped rather than re-read.
|
||
>
|
||
> Surfaced originally as a test flake (a raced sweep removed `PlcA` where the test expected `PlcGone`). Deterministic reproductions: `DclConnectionArbitrationTests.StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections` for the definitions half, and in `InstanceSweepSelfHealTests` — `StaleSweepSnapshot_CarryingASupersededRevision_DoesNotRestartBackward`, `StaleSweepSnapshot_ShowingADeletedInstance_DoesNotResurrectIt`, `StaleBootSnapshot_IsReReadRatherThanApplied`, each paired with a `Fresh…` twin pinning that the legitimate row-72 restart and row-78 start still happen.
|
||
>
|
||
> **Removal (row 90).** Both removal paths share one helper and now record a **tombstone** for the incarnation: the artifact reconcile's `RemovedConnectionNames`, and the sweep noticing that a name it had adopted no longer has a row (a removal it can *prove* — the store owned it a tick ago and does not now). `EnsureDclConnections` skips a tombstoned name, so a removal survives the next instance start instead of being undone by it — a pre-existing bug, since the removal evicted the hash slot but left no memory of why. Tombstones are **in-memory only** by decision: a durable marker would be a second divergent authority, and flattened configs already resurrect a removed connection across a restart. A definition row reappearing clears the tombstone.
|
||
>
|
||
> **In-place reconfiguration — a changed config now reaches the LIVE connection (register row 97)**: resolving the winner decides *what* to push; it took a third command to make a push against an already-live connection mean anything. `EnsureDclConnection` has three outcomes, all keyed off the one per-name hash slot: **unchanged → nothing at all** (no message, no log — this is what keeps a healthy site silent across every sweep tick); **new name → `CreateConnectionCommand`**; **changed → `ReconfigureConnectionCommand`**, which the DCL manager forwards to the live connection actor. The actor swaps its adapter through the very machinery the primary→backup failover uses and re-issues every subscription it holds — no redeployment, no restart. The winner-resolution and tombstone logic above are untouched; only what happens *after* a change is detected changed. The log line is `Reconfigured DCL connection {Connection} (protocol={Protocol}) in place` at Information, replacing the Warning rows 89/90 added to make the gap honest. See `Component-DataConnectionLayer.md` → *In-place reconfiguration* for what the connection actor does, and why stop-and-recreate was rejected.
|
||
>
|
||
> **Residuals still open**, recorded precisely in the decision memo: a removal is not durable across a restart; and a delete is undetectable on a node that never adopted the row. See `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md`.
|
||
|
||
> **Per-instance compilation during staggered startup — CLOSED (WP3.1)**: this was a deferred optimization (P6) affecting failover time-to-recover; each Instance Actor Roslyn-compiled its own scripts inside its own start, serialising a site's whole recovery behind compilation. Each staggered startup batch now runs a **pre-warm step**: before the batch's Instance Actors are created, the Deployment Manager compiles that batch's distinct script bodies and trigger expressions on a background task and pipes `BatchCompileWarmed` back, so every `PreStart` compile is a cache hit. Instance Actor start still calls the compilation service — it remains the correctness backstop — but in every warmed path those calls are memoised lookups. Cost: one extra message per batch.
|
||
|
||
> **The Roslyn `ScriptOptions` are process-static, and must stay that way**: the reference set backing every site-side compile is built once per process, never per compile. `ScriptOptions.WithReferences(Assembly[])` resolves each assembly through `MetadataReference.CreateFromFile`, which does **not** cache — each call mints a fresh `MetadataReference` owning an `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock`, an unmanaged copy of the assembly metadata that nothing disposes. Building the options per compile leaks native memory permanently: no GC reclaims it, and it is invisible to gcdump and to the managed allocation counters, so the node's working set grows without the GC heap growing. This was a live defect (a Site node at 2,885 MB working set with 150 MB of live GC heap, ~2,469 MB of it on the default process heap across ~6,700 undisposed `AssemblyMetadata` instances). Note this is orthogonal to the compile cache above — the cache dedupes *identical* script bodies, so it bounds nothing when the bodies differ. (It no longer clears wholesale on overflow: WP3.1 replaced that latency cliff with approximate-LRU batch eviction, see **Compile cache eviction** below.) Pinned by a reference-equality regression test rather than by watching memory.
|
||
|
||
> **The compile cache is owned by the `ScriptCompilationService` singleton, not static (register row 100)**: `SiteScriptCompileCache` is an ordinary instance held by `ScriptCompilationService`, and "one Roslyn compile per script body per process" (N4) is exactly the statement that `AddSiteRuntime` registers that service as a **DI singleton** — the deploy gate (via `DeployCompileValidator`), every Instance Actor's `PreStart`, and `SharedScriptLibrary` all reach the one cache through the one service the Deployment Manager threads to them. The singleton lifetime is therefore load-bearing rather than incidental, and is pinned by a registration test (`SiteRuntimeServiceRegistrationTests`); downgrading it to Scoped/Transient would hand every consumer an empty cache and silently reintroduce WP3.1's recompile storms with no behavioural test failing. The class was previously `static`, which made its entry table and its `Hits`/`Count` counters one process-global blob every test class in the assembly wrote into; the `Clear()` method that existed only so tests could reset that blob is **removed** — it never had a production caller, and constructing a cache is the honest reset.
|
||
|
||
> **Compile cache eviction is approximate LRU, not a wholesale clear (WP3.1)**: `SiteScriptCompileCache` is bounded at 1024 entries — deliberately smaller than the TemplateEngine verdict cache's 4096, because entries pin compiled assemblies rather than verdict strings. It used to clear **everything** on overflow. Instance scripts *and* trigger expressions share this cache, so on a site large enough to cross the bound every overflow discarded up to 1023 live compiled scripts and the next deploy or Instance-Actor start paid a full recompile storm **on actor threads**. Overflow now evicts only the oldest ⅛ (128 entries) by last-access stamp: one 1024-element scan instead of 1023 future recompiles, and hot entries survive. Recency is an `Interlocked` access sequence rather than a clock — deterministic for tests and immune to clock steps; hits update it lock-free, and only the (rare) eviction sweep takes a lock, double-checked so concurrent inserts do not stampede it. The sweep snapshots through `ConcurrentDictionary.ToArray()` and **not** LINQ over the dictionary: LINQ's `ToArray` picks the `ICollection.CopyTo` fast path, which throws when a concurrent insert lands between its Count read and its copy, and inserts are deliberately lock-free.
|
||
|
||
> **Static options were only half the fix — the compile must also use the shared caching metadata resolver**: the static `ScriptOptions` carry only the *direct* API-surface references (5 assemblies here), and each `script.Compile()` still binds their **transitive closure** (~105 assemblies on a site node). Every transitively-referenced assembly is resolved through the options' `MetadataReferenceResolver`, and Roslyn's default one has no cross-compilation cache — it calls `MetadataReference.CreateFromFile` afresh on **every compile**, minting the same undisposed `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock` triple per assembly per compiled script, pinned for the process lifetime by the compile cache. That per-compile *re-resolution* — not the per-compile options construction — was the dominant term (measured live: 21 site scripts holding 6,640 metadata objects, counts bit-identical before and after the static-options fix). `ScriptCompilationService` therefore also attaches the process-wide `CachingScriptMetadataResolver.Instance` (owned by the Script Analysis component, see `Component-ScriptAnalysis.md`) via `ScriptOptions.WithMetadataResolver`, so each distinct assembly is materialized once per process. The decorator returns exactly what the undecorated resolver returned — only object identity is de-duplicated — so compile results and the trust gate are unchanged.
|
||
|
||
### Deployment Handling
|
||
- Receives flattened instance configurations from central via the Communication Layer.
|
||
- Stores the new configuration in local SQLite.
|
||
- Compiles all scripts in the configuration.
|
||
- Creates a new Instance Actor (for new instances) or updates an existing one (for redeployments).
|
||
- For redeployments: the existing Instance Actor and all its children are stopped, then a new Instance Actor is created with the updated configuration. Subscriptions are re-established.
|
||
- Reports deployment result (success/failure) back to central.
|
||
|
||
> **Dead-child bookkeeping + two-signal deploy join (S6).** All Instance Actors are `Watch`ed; an unexpected `Terminated` evicts the actor ref from the singleton's map. **Deviation from the naive design (2026-07-09, see master-tracker "Plan deviations during Wave 1 PLAN-03 execution"):** deploy-success is **NOT** reported on persistence alone — the config store can commit before the Instance Actor's asynchronous `PreStart` throws. The design therefore uses a **two-signal join**: the Instance Actor sends a new `InstanceActorInitialized` readiness message at the end of a successful `PreStart`, and the Deployment Manager reports `Success` only once **both** persistence has committed **and** that readiness signal has arrived. An actor that dies during init never signals readiness → the `Terminated` fallback replies `Failed` and rolls the optimistic in-memory state back (the persisted-row rollback is deferred until the store commits so it cannot race the write).
|
||
|
||
### System-Wide Artifact Handling
|
||
- Receives updated shared scripts, external system definitions, database connection definitions, and data connection definitions from central. (Notification lists and SMTP configuration are central-only and are not deployed to sites — see Component-NotificationService.md.)
|
||
- Stores all artifacts in local SQLite. After artifact deployment, the site is fully self-contained — all runtime configuration is read from local SQLite with no access to the central configuration database.
|
||
- Recompiles shared scripts and makes updated code available to all Script Actors.
|
||
- **Reconciles deletions**: each non-null artifact list is the complete set for its class, so after upserting, any stored row absent from the set is deleted (the artifact was deleted centrally — there is no per-artifact delete command). A removed shared script is also unregistered from the compiled script library, and a removed data connection's live DCL connection actor is stopped. A `null` list means "field not shipped" and reconciles nothing.
|
||
|
||
### Deployed Secrets (2026-09-28)
|
||
- `HandleDeployArtifacts` additionally accepts the command's `Secrets` list (`SecretArtifact`, ciphertext only) and writes the full shipped set to the `deployed_secrets` table via `SiteStorageService.UpsertDeployedSecretsAsync` — upsert every shipped name, soft-remove (`is_removed = 1`) any stored name absent from the set. A `null` `Secrets` field (pre-feature central) touches nothing and answers with no `SecretsApplied`. A current central always ships a list — **empty** when the site references no secrets — so dropping a site's last `secret:` reference removes the secret from both nodes. Storage detail: *Deployed Secrets (Site SQLite / LocalDb)* below.
|
||
- After writing the table, the active node runs `DeployedSecretsMirror.ReconcileAsync` **synchronously**, before replying — so the deploy response's `SecretsApplied` count and any apply/verify error reflect this deploy's own outcome, not a value left over from the mirror's background schedule. `SecretsApplied` counts only the names *this deploy shipped* that applied and verified cleanly; a mirror failure for an unrelated, already-broken name (a different deploy's leftover, or a stale removal) is logged at Warning and excluded — it neither fails this deploy nor reduces its `SecretsApplied`.
|
||
- No `DeployedSecretsMirror` registered (secrets deployment not composed on this node) with a non-empty `Secrets` list fails the deploy with `Secret deployment is not enabled on this site node`; with an empty `Secrets` list it still succeeds (`SecretsApplied = 0`).
|
||
- **Ordering against the DCL.** The shipped secrets are written and mirrored **before** the deploy's data-connection definitions are pushed to the DCL (the definitions are still stored in SQLite at the same point, and the push still goes out if the secrets step fails), so a connection created or reconfigured by the same deploy resolves a newly shipped secret on its first connect. A rotation with **unchanged** connection config does not recreate the connection (the DCL push is hash-guarded) — the new value takes effect on that connection's next (re)connect. No forced reconnect is performed.
|
||
- See *Deployed Secrets (Site SQLite / LocalDb)* below for the mirror itself.
|
||
|
||
### Instance Lifecycle Commands
|
||
- **Disable**: Stops the Instance Actor and its children. Retains the deployed configuration in SQLite so the instance can be re-enabled without redeployment.
|
||
- **Enable**: Creates a new Instance Actor from the stored configuration (same as startup).
|
||
- **Delete**: Stops the Instance Actor and its children, removes the deployed configuration from local SQLite. Does **not** clear store-and-forward messages.
|
||
- **Ensure Started** (`EnsureInstanceStartedCommand`, site-internal — never sent by central): advisory, idempotent request to make sure an already-persisted, enabled instance is actually running. Raised by the startup reconcile self-heal; see the note under **Startup Behavior**. Unlike the three commands above it has **no reply** and applies no configuration — it only starts an Instance Actor for a config that is already fully stored, and no-ops when the actor is running, the stored row says disabled, the row is absent, or a deploy for that instance is in flight.
|
||
- **Instance reconcile sweep** (internal timer, not a command): the periodic singleton-owned check that the running Instance Actors still agree with the durable config rows — starting an enabled instance that has none, and re-applying a stored config a running instance has fallen behind. See the note under **Startup Behavior**; the start half shares its implementation (and therefore its bookkeeping and site events) with **Ensure Started**, and the restart half goes through the ordinary redeploy path.
|
||
|
||
### Debug Snapshot Routing
|
||
- Receives `DebugSnapshotRequest` from the Communication Layer and forwards to the Instance Actor by unique name (same lookup as `SubscribeDebugViewRequest`).
|
||
- Returns an error response if no Instance Actor exists for the requested unique name (instance not deployed or not enabled).
|
||
|
||
### Server Certificate Trust (Per-Node Broadcast)
|
||
- The OPC UA trusted-peer certificate store is **node-local** — it lives in each node's PKI directory, so trusting a cert on only the active node would silently fail certificate validation after failover. To keep both nodes consistent, certificate trust is handled by a per-node actor and a broadcast.
|
||
- **`CertStoreActor`** is a per-node, **non-singleton** actor — it runs on **every** site node (unlike the Deployment Manager singleton). It owns its node's OPC UA PKI trusted-peer / rejected stores and handles three commands:
|
||
- **`TrustServerCertCommand`** — write a base64/DER server certificate into the node's trusted-peer store (path-traversal-guarded thumbprints; the cert is decoded and persisted as a `.der`).
|
||
- **`RemoveServerCertCommand`** — delete a certificate (by thumbprint) from the node's stores.
|
||
- **`ListServerCertsCommand`** — enumerate the node's trusted-peer + rejected store contents (Subject / Issuer / Thumbprint / validity / status).
|
||
- The Deployment Manager singleton receives the central cert-trust verbs (relayed via the Communication Layer) and **broadcasts** `TrustServerCertCommand` / `RemoveServerCertCommand` to the `CertStoreActor` on **both** site nodes (via `ActorSelection`), so node-a and node-b PKI stores stay in sync. List reads from the local node. Trust is **site-local** — there is no central persistence of trusted certs (logged as a follow-up). The captured-but-untrusted server cert that seeds this flow comes from the DCL verify-endpoint probe (see Component-DataConnectionLayer.md), which never trusts on its own; the Central UI surfaces Trust / Remove (Admin-gated — see Component-CentralUI.md).
|
||
- **Reconcile-on-join (UA1)**: a broadcast only reaches nodes that are Up at the time, so a node that was down when a cert was trusted would stay divergent forever. To close that gap, the Deployment Manager singleton subscribes to cluster `MemberUp` events; when a site node (re)joins, it reads its own trusted-peer store (`ExportLocalTrustedCerts` → `CertStoreActor`) and pushes each certificate to the joining node's `CertStoreActor` (additive union; each `WriteCertToLocalStore` is idempotent by thumbprint). **Removal** reconciliation and central persistence of trust decisions remain the documented follow-up — only additions converge automatically.
|
||
|
||
---
|
||
|
||
## Instance Actor
|
||
|
||
### Role
|
||
- **Single source of truth** for all runtime state of a deployed instance.
|
||
- Holds all attribute values (both static configuration values and live values from data connections).
|
||
- Holds current alarm states (active/normal), updated by child Alarm Actors.
|
||
- Publishes attribute value changes and alarm state changes to the site-wide Akka stream.
|
||
|
||
### Trace arm (Flow Debugger, 2026-09-05)
|
||
- A `SetStaticAttributeCommand` / `SetDataAttribute` carrying `Trace = true` and a `SourceExecutionId` **arms** the attribute on the Instance Actor for `TraceArmWindowSeconds` (default 30, validated `> 0`). A static write publishes `AttributeValueChanged` with the writer's `SourceExecutionId` and `Trace` immediately; a **data-sourced** write is a real tag write, and the next device echo of that attribute inside the window consumes the arm and is published with the same ids — so the script run the echo triggers is parented to the writer's execution and runs traced. An echo outside the window, or a second echo, is an ordinary untraced change. Batch-write echoes are not armed (design §5.3); only the single-attribute write path arms.
|
||
|
||
### Initialization
|
||
1. Load all attribute values from the flattened configuration (static defaults).
|
||
2. Set quality to **uncertain** for all attributes that have a data source reference. Static attributes (no data source reference) have quality **good**. The uncertain quality persists until the first value update arrives from the Data Connection Layer, distinguishing "not yet received" from "known good" or "connection lost."
|
||
3. Register data source references with the Data Connection Layer for subscriptions.
|
||
|
||
> **Tag-subscribe retry (S4/UA6, hardened by Lever 1 of the MxGateway subscribe-storm design, 2026-08-28).** The Instance Actor groups its data-sourced tag paths by connection and sends one `SubscribeTagsRequest` per connection, arming a per-connection watchdog *on send* so both a failed reply and a lost reply are retried. **At most one request per connection is outstanding:** a watchdog tick while the in-flight request is younger than `TagSubscribeInFlightCeilingMs` (45 s, sized above the gateway's 30 s command deadline) only re-arms — it never issues a duplicate — so a gateway that is slow to answer is not handed a fresh N-tag subscribe every interval on top of the one it is still processing (the amplifier behind the 2026-08-26 QA freeze). Past the ceiling the reply is presumed lost and a new request goes out; a late reply for the superseded id is ignored. The retry interval is exponential: floor `TagSubscribeRetryIntervalMs` (5 s), ×2 with ±20 % jitter per failed/lost attempt, capped at `TagSubscribeRetryMaxIntervalMs` (60 s); a `Success` reply or the connection returning to `Good` quality resets it to the floor — but the `Good` reset does **not** release an outstanding request (it still owns the connection until the ceiling), so it shortens the wait only when nothing is in flight; and once the backed-off interval exceeds the ceiling the guard is inert on ticks and the interval itself is the rate limit (one re-send per 60 s at the defaults). All three keys are startup-validated (`SiteRuntimeOptionsValidator`). Design: `docs/plans/2026-08-28-mxgateway-subscribe-storm-prevention-design.md` §3; plan: `docs/plans/2026-08-28-tag-subscribe-inflight-guard-backoff.md`.
|
||
|
||
4. Create child Script Actors (one per script defined on the instance).
|
||
5. Create child Alarm Actors (one per alarm defined on the instance).
|
||
|
||
### Attribute Value Updates
|
||
- Receives tag value updates from the Data Connection Layer for attributes with data source references.
|
||
- Updates the in-memory attribute value.
|
||
- Notifies subscribed child Script Actors and Alarm Actors of the change.
|
||
- Publishes the change to the site-wide Akka stream.
|
||
|
||
### Stream Message Format
|
||
- **Attribute changes**: `[InstanceUniqueName].[AttributePath].[AttributeName]`, attribute value, attribute quality, attribute change timestamp.
|
||
- **Alarm state changes**: `[InstanceUniqueName].[AlarmName]`, alarm state (active/normal), priority, timestamp.
|
||
|
||
### GetAttribute / SetAttribute
|
||
- **GetAttribute**: Returns the current in-memory value for the requested attribute. Attribute read/write/wait helper continuations do not capture the calling scheduler (`ConfigureAwait(false)`), so synchronous `Attributes[key]` access can complete even when every script worker is blocked. AsyncLocal trace attribution still flows with execution context.
|
||
- **SetAttribute** (for attributes with data source reference): Sends a write request to the Data Connection Layer. The DCL writes to the physical device. If the write fails (connection down, device rejection, timeout), the error is returned synchronously to the calling script for handling. On success, the existing subscription picks up the confirmed value from the device and sends it back as a value update, which then updates the in-memory value. The in-memory value is **not** optimistically updated.
|
||
- **SetAttribute** (for static attributes): Updates the in-memory value and **persists the override to local SQLite**. On restart (or failover), the Instance Actor loads persisted overrides on top of the deployed configuration, preserving runtime-modified values. A redeployment of the instance resets all persisted overrides to the new deployed configuration values.
|
||
|
||
### Debug View Support
|
||
- On request from central (via Communication Layer), the Instance Actor provides a **snapshot** of all current attribute values and alarm states.
|
||
- Subsequent changes are delivered via the **SiteStreamManager** → **SiteStreamGrpcServer** → gRPC stream to central. The Instance Actor publishes attribute value and alarm state changes to the SiteStreamManager; it does not forward events directly to the Communication Layer.
|
||
- The Instance Actor also handles one-shot `DebugSnapshotRequest` messages: it builds the same snapshot (attribute values and alarm states) and replies directly to the sender. Unlike `SubscribeDebugViewRequest`, no subscriber is registered and no stream is established. When the request sets `AlarmsOnly` (WP2.3 — the central live alarm cache's seed/reconcile fan-out, which discards attribute rows), only the alarm half is built and `AttributeValues` comes back empty; the flag defaults to `false`, so the Debug View path is unchanged.
|
||
|
||
### Supervision Strategy
|
||
|
||
The Instance Actor supervises all child Script and Alarm Actors with explicit strategies:
|
||
|
||
| Child Actor | Exception Type | Strategy | Rationale |
|
||
|-------------|---------------|----------|-----------|
|
||
| Script Actor | Any exception | Resume | Script Actor is a coordinator — its state (trigger timers, last execution time) should survive run failures. |
|
||
| Alarm Actor | Any exception | Resume | Alarm Actor holds alarm state. Resume preserves state and continues evaluation on next value update. |
|
||
|
||
Script Actor and Alarm Actor have **no children of their own** since WP3.1 removed the per-run execution actors, so neither declares a supervision strategy any more. A run's own exceptions were always contained inside the run's try/catch (they never reached supervision); a failure of the *launch* is caught by the coordinator, which answers the caller and releases the run slot rather than letting the throw escalate to the Instance Actor.
|
||
|
||
The Deployment Manager singleton supervises Instance Actors with a **OneForOneStrategy** — one Instance Actor's failure does not affect other instances.
|
||
|
||
When the Instance Actor is stopped (due to disable, delete, or redeployment), Akka.NET automatically stops all child actors.
|
||
|
||
---
|
||
|
||
## Script Actor
|
||
|
||
### Role
|
||
- **Coordinator** for a single script definition on an instance.
|
||
- Holds the compiled script code and trigger configuration.
|
||
- Manages trigger evaluation (interval timer, value change detection, conditional evaluation).
|
||
- Launches script runs directly for each invocation (no per-run child actor since WP3.1).
|
||
|
||
### Trigger Management
|
||
- **Interval**: The Script Actor manages an internal timer. When the timer fires, it launches a run.
|
||
- **Value Change**: The Script Actor subscribes to attribute change notifications from its parent Instance Actor for the specific monitored attribute. When the attribute changes, it launches a run.
|
||
- **Conditional**: The Script Actor subscribes to attribute change notifications for the monitored attribute. On each update, it evaluates the condition (compares the attribute against a threshold). Firing depends on the **fire mode** (see below).
|
||
- **Expression**: The Script Actor evaluates a compiled boolean expression against an attribute snapshot on each attribute change. Firing depends on the **fire mode** (see below).
|
||
- **Fire mode (Conditional + Expression)**:
|
||
- **OnTrue** (default): Conditional fires on each matching attribute change; Expression fires once per `false → true` transition (edge-triggered). This is the original behavior — a trigger configuration with no mode field is treated as OnTrue.
|
||
- **WhileTrue**: On the `false → true` edge the script fires once, then re-fires on a periodic timer while the condition stays true; on the `true → false` edge the timer stops. The re-fire cadence is the script's **minimum time between runs**; with none configured the trigger degrades to the single edge fire and logs a warning.
|
||
- **Minimum time between runs**: If configured, the Script Actor tracks the last execution time and skips trigger invocations that fire before the minimum interval has elapsed. For a WhileTrue trigger it doubles as the re-fire cadence.
|
||
|
||
> **Per-attribute change routing (P2).** Rather than broadcasting every attribute change to all child Script/Alarm Actors, the Instance Actor routes each change only to the children that monitor that attribute (built from their trigger configs at spawn time); **Expression-trigger children, which read an attribute snapshot, receive all changes**. Each child still keeps its own trigger gate as defense-in-depth — the routing is an optimization, not the authority.
|
||
|
||
### Concurrent Execution
|
||
- Each invocation launches a **new run** on the script-execution scheduler.
|
||
- Multiple runs can execute concurrently (e.g., a trigger fires while a previous `Instance.CallScript` invocation is still running), up to `MaxConcurrentRunsPerScript` — beyond which the newest is shed (see below).
|
||
- The Script Actor coordinates but does not block on run completion.
|
||
|
||
### Script Run Launch (WP3.1 — replaces the Script Execution Actor)
|
||
- A run is launched **directly by the Script Actor** through `ScriptRunLauncher`. There is no per-run child actor.
|
||
- The former `ScriptExecutionActor` / `AlarmExecutionActor` were already inert shells: neither declared a `Receive` handler (they executed from their constructor), neither had a `PostStop`, state, or stash, and neither's `IActorRef` was ever a message target — the whole lifecycle lived inside a detached task the actor never observed. What they cost was an actor cell, mailbox, and name registration **per run**, plus a per-spawn expression-tree `Props.Create`. Removing them changed no semantics; the run body moved verbatim into the shared launcher.
|
||
- Everything the shells provided is preserved: exception and timeout containment, one DI scope per run disposed on every path, the site-event/health telemetry, the Ask reply, the completion notification (now the coordinator's own `Self`), and the audit `ExecutionId` / `ParentExecutionId` threading. A run still in flight when its Script Actor is stopped runs to completion and its completion message dead-letters, exactly as before — **stopping does NOT cancel in-flight runs**; redeploy/undeploy semantics are unchanged.
|
||
- **Per-run Started/Completed site events are sampled, not unconditional (WP3.2).** The two Info rows an instance script run used to emit unconditionally were the dominant `site_events` writer under load (design memo: `docs/plans/2026-08-15-site-events-policy-design.md`). They are now **off by default** (`ScadaBridge:SiteRuntime:PerRunScriptEvents`, default `false`); a `ScriptRunSummaryRecorder` accumulates per-`(instance, script)` run counters from the same call sites and a site-only hosted flush service emits **one aggregate "script" Info row per interval** (`ScriptRunSummaryIntervalSeconds`, default 300s) with a top-50-scripts breakdown, capped so a pathological deployment cannot mint an oversized row — an idle interval emits nothing. `ScadaBridge:SiteRuntime:PerRunScriptEventScripts` restores the legacy per-run rows for named scripts (`"Instance/Script"` exact or `"Instance/*"` wildcard) for live debugging. All three keys are read from `IOptionsMonitor<SiteRuntimeOptions>` per run, so they are hot-togglable without a restart. **Error-level rows (timeout, failure, stuck-watchdog, recursion-limit) are unaffected — always emitted, regardless of this policy.** See Component-SiteEventLogging.md → Volume Policy for the full design.
|
||
- One deliberate improvement: a failure of the *launch itself* (e.g. queueing onto a disposed scheduler) is caught by the Script Actor, which replies to the Ask caller and releases the run slot. The old per-run child's constructor throw was handled by a Stop supervision directive that sent no reply, leaving the caller to hang to its Ask timeout.
|
||
- The script body runs on the **dedicated `ScriptExecutionScheduler`** (a bounded set of dedicated threads), not the shared .NET thread pool, so blocking script I/O cannot starve the global pool or stall Akka dispatchers. The scheduler is **process-wide by default** (one pool per host), but each script/alarm actor takes it through an **optional injection seam** rather than reaching for the static directly: the Host injects nothing and gets the shared pool, while tests (or a future multi-site host) can hand an actor its own instance. The shared accessor also recreates a disposed pool rather than returning it, so a disposed scheduler can never silently poison later executions.
|
||
- **Pool sizing is instance-scaled and grow-only (WP3.1).** The pool was a fixed 8 threads regardless of load. It is now `clamp(max(ScriptExecutionThreadCount, ceil(enabledInstances / 8)), 1, ScriptExecutionMaxThreadCount)` — the existing `ScriptExecutionThreadCount` (default 8) becomes the **floor**, so configurations at or below 64 instances behave exactly as before, and the new `ScriptExecutionMaxThreadCount` (default 32) is the ceiling. Beyond the ceiling the per-script cap below is the real regulator. `DeploymentManagerActor.UpdateInstanceCounts` calls `EnsureCapacity` on every deploy / undeploy / enable / disable and once per staggered startup batch. Growth only: undeploying leaves idle threads, which cost nothing measurable and avoid drain/steal complexity.
|
||
- **The deadline is armed at enqueue, not at dequeue (WP3.1).** The run's timeout `CancellationTokenSource` is created on the actor thread *before* the body is queued, so queue wait consumes the script's own budget. A body that dequeues past its deadline **skips execution entirely** and takes the existing timeout path (site event, script-error counter, error reply, completion message): a saturated pool sheds stale work instead of running it late with a fresh full budget.
|
||
- **Concurrent runs per script are capped (WP3.1).** `MaxConcurrentRunsPerScript` (default 4) bounds runs in flight — queued or executing — for any one script or alarm on-trigger script. Over the cap the **newest** run is shed: the four already in flight are closest to their own deadlines and already charged against them, so nothing is ever reordered and no extra queue is needed (the scheduler's FIFO already is the queue). A shed increments `ISiteHealthCollector.IncrementScriptRunShed` (surfaced as `ScriptRunShedCount`), emits a `script`/`Warning` site event **rate-limited to one per script per minute** so a hot trigger cannot flood `site_events`, and — for an Ask-based `CallScript` — replies with an explicit error so a nested call or inbound-API route fails fast instead of hanging.
|
||
- **The cap gates only depth-0 launches (review fix).** A trigger fire or a depth-0 `CallScript`/inbound-API route counts against `MaxConcurrentRunsPerScript`; a *nested* `Instance.CallScript` self-recursion (`callDepth > 0`) is exempt — it is already bounded by `MaxScriptCallDepth` instead, and the calling run is holding one of the four cap slots itself while it awaits the callee, so counting the nested launch too would spuriously shed legitimate self-recursion.
|
||
|
||
#### Traced runs (Flow Debugger, 2026-09-05)
|
||
- A run is **traced** when its `ScriptCallRequest.Trace` flag is set (a routed inbound call with `TraceAllCalls` or a Flow Debugger Live run), or when it was spawned by a value change whose `AttributeValueChanged.Trace` was set (see *Trace arm* under the Instance Actor). `ScriptRunLauncher` passes `isTraced` + a per-run `FlowTraceCollector` (cap 500 events, `Truncated` flag) into `ScriptRuntimeContext`, and on completion — success, timeout or failure — emits one `ScriptTrace.ScriptRun` audit row (occurred-at = run start, `Extra.triggerKind` = `call` or `valuechange` with the trigger attribute/value/source) through the site `IAuditWriter`.
|
||
- Every trace row is **also** collected in the run's collector and returned inline on `ScriptCallResult.TraceEvents` (with `ScriptCallResult.ExecutionId`), which `DeploymentManagerActor` maps onto `RouteToCallResponse.ChildExecutionId` / `TraceEvents` — Live gets its steps in the same reply; Replay reads the same rows from the Audit Log after the 5 s telemetry push.
|
||
- A value-change run spawned by a traced write inherits the writer's execution id as `ParentExecutionId` and runs traced itself, so a Flow Debugger value-change entry sees the run the write caused.
|
||
- Untraced runs emit nothing new — the flag defaults to `false` on every record and the collector is null.
|
||
|
||
### Handling `Instance.CallScript`
|
||
- When an external caller (another script run, an alarm on-trigger run, or a routed call from the Inbound API) sends a `CallScript` message to the Script Actor, it launches a run to handle the call.
|
||
- The caller uses the **Akka ask pattern** and receives the return value when the execution completes.
|
||
- **Trace flag inheritance:** `ScriptRuntimeContext.CallScript` forwards the calling run's `IsTraced` on `ScriptCallRequest.Trace`, and `CreateChildContextForSharedScript` hands the same flag + collector to an inline shared-script context, so a nested run's `ScriptRun` / `AttributeWrite` / `AttributeWait` rows land in the same flow.
|
||
|
||
---
|
||
|
||
## Alarm Actor
|
||
|
||
### Role
|
||
- **Coordinator** for a single alarm definition on an instance.
|
||
- Evaluates alarm trigger conditions against attribute value updates.
|
||
- Manages alarm state (active/normal) in memory.
|
||
- Executes on-trigger scripts when the alarm activates.
|
||
|
||
### Alarm Evaluation
|
||
- Subscribes to attribute change notifications from its parent Instance Actor for the attribute(s) referenced by its trigger definition.
|
||
- On each value update, evaluates the trigger condition:
|
||
- **Value Match**: Incoming value equals the predefined target. Supports `"!=X"` prefix for not-equals semantics.
|
||
- **Range Violation**: Value is outside the allowed min/max range.
|
||
- **Rate of Change**: Value change rate exceeds the defined threshold over a configurable time window. Direction filter (rising / falling / either) restricts which side of the rate triggers.
|
||
- **HiLo**: Multi-setpoint level alarm with up to four configurable setpoints (LoLo, Lo, Hi, HiHi). Any subset may be configured. Each setpoint may carry its own priority that overrides the alarm-level priority for that band.
|
||
- **Expression** trigger evaluation runs on the shared .NET thread pool behind a process-wide concurrency gate (`TriggerEvalGate`, sized by `TriggerEvalMaxConcurrency`, default `max(2, ProcessorCount)`) — **not** on the bounded script-execution pool. This is the WP3.1 fix for arch-review finding #4 (High): when evaluation shared that pool, N script bodies blocked in synchronous I/O stalled *every* Expression trigger on the node — scripts and alarms alike — for an unbounded time, and the evaluation's own 2 s timeout was constructed inside the queued body, so it did not start ticking until dequeue. An alarm that should have raised in milliseconds simply never raised, with neither a raise nor a timeout visible to the operator. Trigger expressions are non-blocking **by construction** (`TriggerExpressionGlobals` exposes only reads over an in-memory snapshot, and the script trust gate has already denied I/O, network, threading, and reflection), so the shared pool is where they belong; a second dedicated pool was considered and rejected as adding threads, gauges, and a second starvation surface for no isolation gain. The evaluation deadline (`TriggerEvalTimeoutSeconds`, default 2, previously hardcoded) is now armed **at enqueue**, so gate-wait time burns the same budget and a saturated gate yields a timely `false` rather than an unbounded stall. Per-actor coalescing (one evaluation in flight, one pending) is unchanged and caps waiters at one per Expression trigger, so the gate queue is bounded by trigger count.
|
||
- For binary trigger types (ValueMatch / RangeViolation / RateOfChange), when the condition is met and the alarm is currently in **normal** state, the alarm transitions to **active**:
|
||
- Updates the alarm state on the parent Instance Actor (which publishes to the Akka stream).
|
||
- If an on-trigger script is defined, launches its run via `ScriptRunLauncher` (see Alarm On-Trigger Run below) — no per-run child actor.
|
||
- When the condition clears and the alarm is in **active** state, the alarm transitions to **normal**.
|
||
- For HiLo triggers, the actor tracks the current `AlarmLevel` (None / Low / LowLow / High / HighHigh). Each level transition emits a fresh `AlarmStateChanged` with the new level and its priority; level escalations (e.g., High → HighHigh) and de-escalations (HighHigh → High) both produce events. The on-trigger script fires only on the Normal → non-None edge, not on escalations between alarm bands.
|
||
- No script execution on clear in any trigger type.
|
||
|
||
### Alarm State
|
||
- Held **in memory** only — not persisted to SQLite. State comprises `AlarmState` (Active / Normal) and `AlarmLevel` (None for binary triggers; the active band for HiLo).
|
||
- On restart (or failover), alarm states are re-evaluated from incoming values. All alarms start in normal state with level None and transition based on incoming values.
|
||
|
||
### Alarm On-Trigger Run
|
||
- Launched directly by the Alarm Actor when an on-trigger script needs to execute — the same `ScriptRunLauncher` path as an instance script run, with the firing alarm's name/level/priority/message exposed through the `Alarm` global. Bounded by the same `MaxConcurrentRunsPerScript` cap.
|
||
- Has access to the Instance Actor for `GetAttribute`/`SetAttribute`.
|
||
- **Can** call instance scripts via `Instance.CallScript()` — sends an ask message to the appropriate sibling Script Actor.
|
||
- Instance scripts **cannot** call alarm on-trigger scripts — the call direction is one-way.
|
||
|
||
### Audit correlation of an on-trigger run (`ParentExecutionId` tag-cascade)
|
||
|
||
An alarm fired by a **script- or inbound-API-initiated static attribute write**
|
||
is a genuine execution spawn, and the on-trigger run records the writing
|
||
execution as its parent in the central Audit Log (#23). The originating id rides
|
||
site-locally through three additive, nullable fields — no wire, proto or schema
|
||
change:
|
||
|
||
| Hop | Field |
|
||
|---|---|
|
||
| `ScriptRuntimeContext.SetAttribute` (this run's `ExecutionId`) / `RouteToSetAttributesRequest.ParentExecutionId` (the inbound request's) | → `SetStaticAttributeCommand.SourceExecutionId` |
|
||
| Instance Actor static-write path (`HandleSetStaticAttributeCore`) | → `AttributeValueChanged.SourceExecutionId` |
|
||
| Alarm Actor trigger evaluation → `SpawnAlarmExecution` | → `ScriptRunLauncher.LaunchAlarmScript` → `ScriptRuntimeContext.ParentExecutionId` |
|
||
|
||
All four computed trigger types participate. `Expression` triggers evaluate a
|
||
whole attribute snapshot **off the dispatcher**, so the firing change is no
|
||
longer in scope when the boolean returns: the writer of the newest value folded
|
||
into the snapshot is captured *with* the snapshot and echoed back on the
|
||
evaluation result, so a change arriving mid-flight cannot mis-attribute the
|
||
raise.
|
||
|
||
The on-trigger run is correctly a **tree root** (`ParentExecutionId` NULL) when
|
||
the firing value has no originating execution:
|
||
|
||
- Values from the **Data Connection Layer** — external device data has no
|
||
spawning execution. This includes the confirmed value of a write to a
|
||
**data-sourced** attribute: that write is forwarded to the device and the echo
|
||
arrives on the subscription long after the writing execution ended, so only
|
||
**static** attribute writes cascade.
|
||
- Deploy-time seeding and central Test Run / tooling writes.
|
||
|
||
`AttributeValueChanged.SourceExecutionId` is deliberately **not** projected onto
|
||
the gRPC debug/stream wire shapes — the cascade is resolved inside the site node
|
||
that owns the Instance Actor, and central reads the linkage from audit rows.
|
||
|
||
---
|
||
|
||
## Native Alarm Actor
|
||
|
||
### Role
|
||
- **Read-only mirror** of alarms raised natively by an external source — OPC UA Alarms & Conditions (A&C) servers and the MxAccess Gateway — surfaced into the Site Runtime alongside the alarms ScadaBridge computes itself.
|
||
- Created as a child of the **Instance Actor** and is a **peer to the computed Alarm Actor** (not a child of it). One `NativeAlarmActor` is spawned per resolved native alarm source binding on the instance.
|
||
- Mirrors source-of-truth condition state into the Instance Actor's view and onto the site-wide stream; it **does not** acknowledge, clear, or otherwise write back to the source. There is no ack-back path — the external source remains authoritative.
|
||
|
||
### Construction
|
||
- Constructed with `(ResolvedNativeAlarmSource source, string instanceName, IActorRef instanceActor, IActorRef dclManager, SiteStorageService storage, SiteRuntimeOptions options, ILogger logger, AlarmKind nativeKind = NativeOpcUa)`.
|
||
- `nativeKind` distinguishes the two native flavors and stamps the `Kind` on every emitted `AlarmStateChanged`. The Instance Actor selects it from the bound connection's protocol (see **Instance Actor wiring** below).
|
||
|
||
### Lifecycle & Subscription
|
||
- **PreStart**: rehydrates any previously mirrored conditions for this source from the site SQLite `native_alarm_state` table, then subscribes to the source through the Data Connection Layer by sending a `SubscribeAlarmsRequest` to the DCL manager. The DCL routes the subscription to the bound connection's `IAlarmSubscribableConnection` implementation.
|
||
- **Failed subscribe**: schedules a retry timer at `NativeAlarmRetryIntervalMs` and re-attempts until the subscription is established. Rehydrated state remains visible in the meantime.
|
||
- **`NativeAlarmSourceUnavailable`**: the source connection has dropped. The actor **retains its last-known mirrored conditions** but marks them uncertain rather than purging them, so a transient disconnect does not flap every condition to normal. The set is reconciled against truth by the next reconnect snapshot.
|
||
|
||
### Transition Handling (`NativeAlarmTransitionUpdate`)
|
||
- **Owned subscription cleanup**: Instance Actors send tag unsubscribes for each bound connection in `PostStop`; Native Alarm Actors send their alarm-source unsubscribe with their own actor identity. DCL DeathWatch provides cleanup if an owner terminates without a graceful message. Late cleanup from an old actor cannot remove its same-name replacement.
|
||
- **OPC UA refresh scope**: snapshot start, entries and completion preserve the exact source binding and refresh epoch. Native Alarm Actors ignore unrelated feeds and mismatched/duplicate completion epochs, and discard a partial refresh on source disconnect. MxGateway keeps its connection-wide snapshot framing.
|
||
- **Snapshot / SnapshotComplete (reconnect reconciliation)**: `Snapshot` updates buffer into a staging set; `SnapshotComplete` performs an **atomic swap** of the mirrored set with the staged set. Any condition that was previously mirrored but is **not present** in the new snapshot emits a return-to-normal `AlarmStateChanged` and drops out. This is how the mirror self-corrects after an outage.
|
||
- **Live transitions** (`Raise` / `Ack` / `Clear` / `Retrigger` / `StateChange`): upsert the condition by `SourceReference`. Updates carrying a `TransitionTime` **older** than the currently held transition are ignored (out-of-order protection). Accepted transitions persist to SQLite and emit an enriched `AlarmStateChanged` upward to the Instance Actor.
|
||
- **Retention**: a mirrored condition is dropped once it is both inactive **and** acknowledged (`!Active && Acknowledged`) — the alarm has fully run its course at the source and no longer needs mirroring. The drop emits a final state change and deletes the SQLite row.
|
||
- **Per-source cap**: at most `MirroredAlarmCapPerSource` conditions are retained per source. When the cap is exceeded the **oldest** condition is dropped and the eviction is **logged** — there is no silent truncation. If the evicted condition is still **active**, a final return-to-normal `AlarmStateChanged` is emitted before it is dropped (mirroring the retention drop above), so a capped-out active condition is never left stuck-active downstream.
|
||
|
||
### Persistence
|
||
- Mirrored condition state is persisted to the site SQLite `native_alarm_state` table on every accepted transition and removed on drop-out.
|
||
- Persistence is **best-effort / fire-and-forget**: a persistence failure is logged but never blocks the actor's mailbox and never aborts the upward `AlarmStateChanged` emit. The in-memory mirror remains authoritative for the running actor; SQLite exists to survive failover.
|
||
|
||
### Supervision & Restart
|
||
- Supervised by the Instance Actor under the same **OneForOneStrategy** as the computed Alarm Actor — a native source fault is isolated to its own actor.
|
||
- On site restart or failover, the actor rehydrates its mirror from `native_alarm_state` in PreStart, then reconciles against the source via the reconnect snapshot. Native mirror state therefore **survives failover** (unlike computed alarm state, which is re-evaluated from values).
|
||
- Mirrored native state **is cleared on redeploy/undeploy** of the instance (mirroring the static-override reset): the stale rows for the instance are removed and the fresh actor re-subscribes from a clean slate.
|
||
|
||
---
|
||
|
||
## Instance Actor — Native Alarm Wiring
|
||
|
||
The Instance Actor owns native-alarm setup alongside its computed Script and Alarm Actors:
|
||
|
||
- **Spawning**: for each entry in `_configuration.NativeAlarmSources`, the Instance Actor spawns a `NativeAlarmActor`. Spawning is **skipped when there is no DCL manager** (e.g., debug/test contexts with no data connections), since native alarms require a live source subscription.
|
||
- **Kind derivation**: the `AlarmKind` passed to each `NativeAlarmActor` is derived from the bound connection's protocol — `Mx*` protocols → `NativeMxAccess`, otherwise → `NativeOpcUa`.
|
||
- **Latest-event retention**: the Instance Actor retains the latest enriched `AlarmStateChanged` per alarm name in `_latestAlarmEvents`. The DebugView snapshot is built from this map so it carries the **unified condition view plus native metadata** for both computed and native alarms. Computed alarms that have not yet produced an event fall back to a **Normal projection** so the snapshot is complete.
|
||
- **Idle native source binding placeholders**: `BuildAlarmStatesSnapshot()` additionally emits one placeholder `AlarmStateChanged` for each configured native alarm source binding that currently has **no live conditions** in `_latestAlarmEvents`. The placeholder has `IsConfiguredPlaceholder = true` and carries the binding's canonical name in `NativeSourceCanonicalName`. The Instance Actor maintains a `_nativeAlarmKinds` map (`sourceCanonicalName → AlarmKind`) populated at spawn time to stamp the correct `Kind` on each placeholder. This ensures the Debug View Alarms tab shows every configured binding as a tree node even when quiet.
|
||
- **`NativeSourceCanonicalName` on live events**: every `AlarmStateChanged` emitted by a `NativeAlarmActor` stamps its source binding's canonical name in `NativeSourceCanonicalName`. The Debug View uses this field to place each live condition under the correct native-source binding node in the tree.
|
||
- **Reset semantics**: `_latestAlarmEvents` and the mirrored native state are cleared on redeploy/undeploy (same trigger as static-override reset) but rehydrate from SQLite on failover. In addition, a native condition's `_latestAlarmEvents` key is **evicted per-condition** whenever that condition leaves the mirror — snapshot-swap drop, retention drop, or cap eviction — driven by a `NativeAlarmDropped` signal from the `NativeAlarmActor`; this prevents unbounded `_latestAlarmEvents` growth for sources that mint a fresh `SourceReference` per occurrence.
|
||
|
||
---
|
||
|
||
## Native Alarm State Persistence (Site SQLite)
|
||
|
||
`SiteStorageService` gains a `native_alarm_state` table backing the native mirror:
|
||
|
||
- **Primary key**: `(instance_unique_name, source_canonical_name, source_reference)` — one row per mirrored condition.
|
||
- **Columns**: `condition_json` (the serialized `AlarmConditionState`), `metadata_json` (display metadata — type/category/message/current+limit, UA4), and `last_transition_at` (the accepted `TransitionTime`).
|
||
- **Operations**: `Upsert` (on accepted transition), `Delete` (on condition drop-out), `Get` (PreStart rehydrate, scoped to instance + source), and `ClearForInstance` (redeploy/undeploy reset).
|
||
- **Display metadata persistence (UA4)**: persisted rows include the condition's display metadata, so **rehydrated conditions render fully** (type / category / message / current + limit values) before the first fresh source snapshot arrives; a pre-UA4 row with no metadata rehydrates with empty display fields (unchanged behavior).
|
||
- **Ack timestamp persistence**: the condition's `AckTime` rides the same `metadata_json` blob, so an acknowledged alarm keeps its ack instant across restart/failover rather than reappearing as if it had never been acknowledged. It is deliberately **not** a new physical column: `native_alarm_state` is `RegisterReplicated` in `SiteLocalDbSetup`, and LocalDb builds that table's CDC triggers from the column list at registration time — an additive JSON property changes no schema, no triggers and no replication contract, and `metadata_json` is exactly the extension point UA4 introduced for this. A row written before the field existed deserializes it as `null` ("ack time unknown"), which is the correct fallback. See `docs/plans/2026-06-30-mes-alarm-status-api.md` §6.4.
|
||
- **Coalesced writes (P4)**: per-transition upserts are coalesced (latest-per-source-reference) and flushed on a short single-shot timer rather than one SQLite write per transition, so an alarm storm collapses to one batched write; a pending upsert is dropped if the condition is deleted before the flush (no resurrection). Durability window is at most one flush interval on crash, reconciled by the next re-subscribe snapshot.
|
||
- This is a **peer SQLite store** to the existing deployed-configuration, store-and-forward, operation-tracking, and `AuditLog` stores. Unlike computed alarm state, native mirror state is intentionally persisted so it survives failover.
|
||
|
||
---
|
||
|
||
## Deployed Secrets (Site SQLite / LocalDb)
|
||
|
||
Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see that doc's *Implementation notes (as built)* for deviations from the original design). Three tables, two different replication postures:
|
||
|
||
- **`deployed_secrets`** — the **11th LocalDb-replicated table** (`SiteLocalDbSetup.ReplicatedTables`, up from ten). Columns: `name` (PK), `row_json` (the shipped `SecretArtifact` — ciphertext, nonce, tag, wrapped DEK, `KekId`, revision; never plaintext), `revision`, `updated_utc`, `is_removed` (soft-remove flag, default 0), `deployment_id`, `applied_at`. No FKs (the standing FK guard — `SiteLocalDbReplicatedTableFkGuardTests` — applies to it like every other replicated table). `UpsertDeployedSecretsAsync` is full-set upsert-then-soft-remove, never a hard `DELETE`, so a removal is itself a replicated row change the standby node can react to.
|
||
- **`secret_mirror_status`** — the **12th LocalDb-replicated table** (review finding M1). One row per site node: `node_name` (PK — `INodeIdentityProvider.NodeName`, i.e. `node-a`/`node-b`, falling back to the machine name when unconfigured), `failures_json` (a JSON array of `{SecretName, Message}` — the node's *current* mirror failures; names and messages only, never values or ciphertext), `checked_at`. No FKs. It is per-node *data* but replicated on purpose: each node writes only its own row, and replication is what lets the **active** node read the **standby's** row onto the site health report (only the active node sends one — see *Mirror status on the health report* below).
|
||
- **`mirrored_secrets`** — a **node-local, unreplicated** ledger (`name` PK, `revision`, `mirrored_at`) recording which names *this specific node* has applied into its own `ISecretStore`. Deliberately not registered with LocalDb: it is a per-node fact (which names this node mirrored), not site-wide state, and replicating it would make one node's mirroring progress masquerade as the other's.
|
||
|
||
**`DeployedSecretsMirror`** (`SiteRuntime/Secrets/DeployedSecretsMirror.cs`) is a per-node `BackgroundService` — registered on **every** site node, not a singleton — that reconciles `deployed_secrets` into the node's own local `ISecretStore` (site nodes never run a replication-decorated store; see Component-Host.md → *Secrets*):
|
||
|
||
- **Schedule**: runs once immediately at start, then every `SiteRuntime:DeployedSecretsMirrorInterval` (default 15s, validated `> 0`), and synchronously on demand right after an active-node artifact apply (see *Deployed Secrets* under Deployment Manager Singleton above). This 15s interval is the **standby node's own convergence path** — it never receives the active node's synchronous call, only the replicated `deployed_secrets` rows via LocalDb, so it picks them up on its own next tick. **A pair that uses deployed secrets must therefore run LocalDb replication:** on an unreplicated pair only the node that was active at deploy time holds the secrets, and after a failover the new active node has none until the next *Deploy Artifacts* (topology guide → *Secrets: shared store, per-site deploy, and the KEK*).
|
||
- **Apply is confirm-then-ledger, not blind.** `ApplyReplicatedAsync` is strict last-writer-wins on `(UpdatedUtc, Revision)` (a tie is a no-op) and `DeleteAsync` tombstones rather than hard-deletes — so the mirror always reads the row back after applying and only ledgers it in `mirrored_secrets` when the local ciphertext matches what was shipped. A tombstoned or clock-skewed local row that the ledger already owns is safely re-stamped (`UpdatedUtc`/`Revision` bumped past the local value) and re-applied; a live row with **different** ciphertext that the ledger does **not** already own is left completely untouched — not overwritten, not ledgered, not verified — and reported as an out-of-band conflict (`"exists on this node with a newer local value; the deployed revision was not applied"`), so an operator's genuinely local secret of the same name is never silently destroyed by a later central deploy or removal.
|
||
- **Ownership, not tombstone-state, decides re-stamping.** Once a name is ledgered in `mirrored_secrets` at any revision, this mirror treats that name as its own going forward — including re-stamping a mirror-owned name that drifted back to the deployed value on the next pass. This holds even when the name is already ledgered at the shipped revision: the mirror reads the local row back every pass, and one that is missing, tombstoned (an operator deleted it locally) or live with different ciphertext (an operator edited it) is healed through the same bumped-stamp re-apply, read-back and cache invalidation before the verify — a local delete is repaired, never misreported as a master-key mismatch. The drift is logged at Warning once per occurrence. Central is authoritative for every name it deploys.
|
||
- **Unreadable rows are isolated.** A `deployed_secrets` row whose `row_json` cannot be parsed is reported as a failure for that name (in `Failed`/`FailedNames`) instead of aborting the pass; an unreadable **removed** row is still honoured by name.
|
||
- **Removal** deletes from the local store **only if the name was previously ledgered** (present in `mirrored_secrets`) — an out-of-band local secret sharing a deployed name that was never mirrored is never touched by a central-side removal.
|
||
- **Verify.** Every pass trial-resolves every live row through `ISecretResolver` (even one not re-applied this pass, so a later KEK change is still caught) — a decrypt failure names the secret and its `KekId` and is reported as a distinct error (de-spammed: Error the first time a given `(name, revision, KekId)` fails, Debug on identical repeats, Error again after an intervening success).
|
||
- **Isolation.** One row failing to apply/verify does not block any other row in the same pass.
|
||
- **Cache invalidation.** A successful apply or delete invalidates the shared `ISecretResolver` cache before verifying, so the verify step (and any script/connection reading the secret) sees the fresh value immediately rather than waiting out the resolver's cache TTL.
|
||
- **Never logs a secret value** — log lines and the reported `MirrorResult` carry only names, revisions, and `KekId`s.
|
||
- **Mirror status on the health report (review finding M1).** A failure on the **standby** — above all a secret that will not decrypt because the standby's master key differs from central's — used to live only in that node's log, invisible until a failover broke its connections. After every pass the mirror records its node's current failure set in `secret_mirror_status`, but **only when the set (sorted name+message pairs) differs from what this process last wrote**, plus once on the first pass after start-up so a restart refreshes a stale row — never on every 15 s tick, because every write is a replicated CDC change. The write is best-effort: a failure logs a Warning (repeats at Debug), never affects the pass result, and is retried next pass. `SecretMirrorStatusReporter` (`SiteRuntime/Secrets/`, every site node, 30 s, immediate first probe) reads every node's row into `ISiteHealthCollector.SetSecretMirrorStatuses`, so the active node's report carries both nodes' status as `SiteHealthReport.SecretMirrorStatuses` (see Component-HealthMonitoring.md). On an **unreplicated** pair only the active node's own row is visible — the same replication prerequisite deployed secrets already have.
|
||
|
||
---
|
||
|
||
## Enriched `AlarmStateChanged` Message
|
||
|
||
The `AlarmStateChanged` message published by both Alarm Actors and Native Alarm Actors was extended **additively** (existing consumers keep working with computed defaults):
|
||
|
||
- **`Kind`** (`AlarmKind`): `Computed` for ScadaBridge-evaluated alarms; `NativeOpcUa` / `NativeMxAccess` for mirrored native alarms.
|
||
- **`Condition`** (`AlarmConditionState`): the unified condition view. Computed alarms supply a computed default; native alarms carry the mirrored source condition.
|
||
- **Native metadata** (populated for native alarms; defaulted/empty for computed): `SourceReference`, `AlarmTypeName`, `Category`, `OperatorUser`, `OperatorComment`, `OriginalRaiseTime`, `CurrentValue`, `LimitValue`.
|
||
- **Computed-alarm projection**: computed alarms are surfaced as **auto-acknowledged** with `Severity = Priority`, so a single enriched shape carries both computed and native alarms onto the stream and into the DebugView snapshot.
|
||
- **`NativeSourceCanonicalName`** (`string?`): the canonical name of the native alarm source binding a native condition belongs to (e.g. `Motor1.NativeAlarms`). `null` for computed alarms. Stamped by the `NativeAlarmActor` on every emitted event; also carried on idle-binding placeholder rows emitted by the Instance Actor.
|
||
- **`IsConfiguredPlaceholder`** (`bool`): `true` for a synthetic row emitted by the Instance Actor to represent a configured native source binding that currently has no live conditions. These rows are never persisted or forwarded as operational alarms — they exist solely so the Debug View can render every configured binding as a tree node. `false` on all other rows.
|
||
- **`AckTime`** (`DateTimeOffset?`): the instant the condition was acknowledged; `null` while unacknowledged, `null` on computed alarms (auto-acked but with no operator ack event), and cleared on re-raise. Stamped by the DCL adapter — the source's own ack instant where the protocol supplies one (OPC UA A&C), the DCL's observation time of the ack transition where it does not (MxAccess Gateway) — and carried verbatim by the `NativeAlarmActor`, which never invents or suppresses one. See [Component-DataConnectionLayer.md](Component-DataConnectionLayer.md) → *Ack Timestamp*.
|
||
|
||
These fields are **additive-only** (init-only with defaults: `null` / `false`) and are mirrored additively on the gRPC `AlarmStateUpdate` proto as fields 22 (`native_source_canonical_name`, `string`), 23 (`is_configured_placeholder`, `bool`) and 24 (`ack_time`, `google.protobuf.Timestamp`, absent ⇒ `null`). `StreamRelayActor` and `SiteStreamGrpcClient` pack/unpack these fields without touching existing fields, preserving backward compatibility across version-skew boundaries.
|
||
|
||
The enriched message flows Instance Actor → site-wide Akka stream → `SiteStreamManager` → `SiteStreamGrpcServer` and is streamed to central as the gRPC `AlarmStateUpdate` event (see [Component-Communication.md](Component-Communication.md)).
|
||
|
||
---
|
||
|
||
## Shared Script Library
|
||
|
||
- Shared scripts are compiled at the site when received from central.
|
||
- Compiled code is stored in memory and made available to all Script Actors.
|
||
- When a script run calls `Scripts.CallShared("scriptName", params)`, the shared script code executes **inline** in that run's context — it is a direct method invocation, not an actor message.
|
||
- This avoids serialization bottlenecks since there is no shared script actor to contend for.
|
||
- Shared scripts have access to the same runtime API as instance scripts (GetAttribute, SetAttribute, external systems, notifications, databases).
|
||
- **Shared scripts always execute with Root scope (C5)**: `Attributes[...]` inside a shared script addresses **root** attributes even when the shared script is invoked from a composed-module script. Callers needing module-scoped access must pass the values in as parameters.
|
||
|
||
---
|
||
|
||
## Script Lifecycle
|
||
|
||
All script types can be updated without restarting the cluster, but the mechanism differs per type.
|
||
|
||
### Instance Scripts and Alarm On-Trigger Scripts
|
||
|
||
- Compiled at deployment time when the Deployment Manager receives a flattened configuration and creates the Instance Actor hierarchy.
|
||
- **To update**: modify the script in the template, then redeploy the instance (`instance deploy --id <N>`).
|
||
- Redeployment stops the existing Instance Actor and all its children, creates a new Instance Actor, and recompiles all scripts and alarms from the updated configuration.
|
||
- There is no way to hot-reload a single instance script without redeploying the entire instance.
|
||
|
||
### Shared Scripts
|
||
|
||
- Compiled at the site when received from central via artifact deployment (`deploy artifacts`).
|
||
- The `SharedScriptLibrary` replaces its in-memory compiled code dictionary under a lock, making updated code immediately available to all Script Actors.
|
||
- **To update**: modify the shared script via the CLI or UI, then run `deploy artifacts` to push the change to sites.
|
||
- No instance redeployment is required — running instances pick up the new shared script code on the next `Scripts.CallShared()` invocation.
|
||
|
||
### Inbound API Method Scripts
|
||
|
||
- See Component-InboundAPI.md for the compilation and hot-reload lifecycle of API method scripts.
|
||
|
||
> **Warning**: Editing scripts via direct SQL does not trigger recompilation. The in-memory compiled script will remain stale until the next node restart or redeployment. Always use the CLI, Management API, or Central UI to modify scripts.
|
||
|
||
---
|
||
|
||
## Script Runtime API
|
||
|
||
Available to all script runs and alarm on-trigger runs:
|
||
|
||
### Instance Attributes
|
||
- `Instance.GetAttribute("name")` — Read an attribute value from the parent Instance Actor.
|
||
- `Instance.SetAttribute("name", value)` — Write an attribute value. For data-connected attributes, writes to the DCL; for static attributes, updates in-memory and persists to local SQLite (survives restart/failover, reset on redeployment).
|
||
|
||
### Other Scripts
|
||
- `Instance.CallScript("scriptName", parameters)` — Send an ask message to a sibling Script Actor. The target Script Actor launches a run, executes, and returns the result (or an explicit shed error if that script is already at its concurrent-run cap). The call includes the current recursion depth.
|
||
- `Scripts.CallShared("scriptName", parameters)` — Execute shared script code inline (direct method invocation). The call includes the current recursion depth.
|
||
|
||
### External Systems
|
||
- `ExternalSystem.Call("systemName", "methodName", params)` — Synchronous HTTP call. Blocks until response or timeout. All failures return to script. Use when the script needs the result.
|
||
- `ExternalSystem.CachedCall("systemName", "methodName", params)` — Deferred delivery. Returns a `TrackedOperationId` tracking handle immediately rather than the response; the call is attempted immediately and, on transient failure, store-and-forwarded for retry. Use for outbound data pushes where deferred delivery is acceptable.
|
||
- The returned `TrackedOperationId` can be passed to `Tracking.Status(id)` (see **Operation Tracking** below) to observe delivery progress.
|
||
|
||
### Notifications
|
||
- `Notify.To("listName").Send("subject", "message")` — Send a notification via a named notification list. Generates a `TrackedOperationId` (GUID) locally and returns it immediately; the notification is store-and-forwarded to the central cluster, which owns delivery. The script never contacts SMTP. (`NotificationId` is the notification-domain name for this same `TrackedOperationId` type.)
|
||
- `Notify.Status("trackedOperationId")` — A thin alias of `Tracking.Status(id)` retained for the notification domain. Returns a status record (status, retry count, last error, key timestamps). While the notification is still in the site store-and-forward buffer the site answers locally (status `Forwarding`); once forwarded the query round-trips to central.
|
||
|
||
### Database Access
|
||
- `Database.Connection("connectionName")` — Obtain an audited ADO.NET connection for read/write. Commands and synchronous/asynchronous transactions retain the auditing connection on public readback; provider transactions are unwrapped only internally for command assignment. Each command execution emits one audit event, including commands reached through `transaction.Connection` or `command.Connection`. Commit, rollback, savepoints and disposal retain provider semantics.
|
||
- `Database.CachedWrite("connectionName", "sql", parameters)` — Submit a write operation for store-and-forward delivery. Returns a `TrackedOperationId` tracking handle immediately; pass it to `Tracking.Status(id)` to observe delivery progress.
|
||
|
||
### Operation Tracking
|
||
- `Tracking.Status("trackedOperationId")` — Returns a status record (status, retry count, last error, key timestamps) for any tracked operation: a cached external system call, a cached database write, or a notification. For cached calls and writes the answer is always site-local and authoritative — the site owns the operation tracking table. (`Notify.Status(...)` is a thin alias scoped to the notification domain.)
|
||
- **Cached-call telemetry is off the script thread (2026-09-06).** A `Database.CachedWrite` / `ExternalSystem.CachedCall` emits three `CachedCallTelemetry` packets (submit, attempted, resolve) through `CachedCallTelemetryForwarder`, each an audit row plus a tracking row. Both halves used to be awaited to commit on the script thread; the audit half was the real cost — `SqliteAuditWriter.WriteAsync` returns the batch completion, and the writer coalesces for `FlushIntervalMs` (50 ms) before committing — so a cached call paid ~160 ms of a 176 ms run for bookkeeping (measured on the rig; now 17 ms). The forwarder now **enqueues the audit row and observes its completion on a fault-logging continuation** (the same pattern every other script-path audit emission already used), and `OperationTrackingStore` is a **single-writer queue**: `RecordEnqueue/Attempt/TerminalAsync` return once the row is *queued*, and one background pump applies each drained batch in one SQLite transaction. Preserved: FIFO order across the three lifecycle rows; **read-your-writes** — `GetStatusAsync` (behind `Tracking.Status`, and the telemetry actor's status join) and `ReadChangedSinceAsync` (central's `PullSiteCalls`) wait for every write queued before them; timestamps are stamped when recorded, not when applied; `PurgeTerminalAsync` returns after the purge lands; Both `DisposeAsync` and sync `Dispose()` drain the queue before closing the writer; host shutdown uses the async path. Accepted: a row queued but unapplied is lost on a hard crash (milliseconds), leaving the tracking row at its previous state until reconciled; the S&F buffer and audit rows are unaffected. The queue is bounded (8 192) with wait backpressure — it never drops. `FlushAsync()` is the explicit barrier for callers that go to the file directly. **Trap for new audit emitters on a script path:** never `await` `IAuditWriter.WriteAsync` there — it waits for the writer's flush, not the enqueue. Design + measurements: `docs/plans/2026-09-06-tracking-store-async-writes-design.md`.
|
||
|
||
### Alarm State (read-only)
|
||
- `Alarms.CurrentAsync()` — Returns a snapshot of the instance's **current** alarm conditions as `IReadOnlyList<ScriptAlarm>` (`Commons.Types.Scripts`), covering computed alarms and mirrored native (OPC UA A&C / MxAccess Gateway) conditions alike. Served by a **local Ask** (`GetAlarmSnapshotRequest`/`GetAlarmSnapshotResponse`) to the script's own Instance Actor — the same mechanism attribute reads use, never a cross-cluster hop.
|
||
- `Alarms.MesReportableAsync()` — `CurrentAsync()` filtered to rows where `ScriptAlarm.IsMesReportable` is true (2026-09-11). Native alarms carry no per-alarm config row, so they never satisfy the filter — they still convey MES relevance via the 900–999 severity band (see `docs/plans/2026-06-30-mes-alarm-status-api.md`).
|
||
- Available to **every** script, unlike the `Alarm` global, which exists only inside an on-trigger handler and describes the single alarm that fired. This accessor is what lets an inbound-API-routed `Call` script answer "what is in alarm on this machine right now?"; before it, native mirrored conditions were reachable only from the Debug View.
|
||
- **Read-only by design** — native alarms are a read-only mirror of the source (no ack-back), so the accessor exposes no acknowledge/shelve operation.
|
||
- **Not scope-prefixed**, unlike `Attributes`: alarm identity is not a scope-relative attribute name (computed alarms are keyed by configured name, native conditions by a source-supplied reference), so every scope sees the same list rather than a silently truncated one.
|
||
- The snapshot is built by the same `BuildAlarmStatesSnapshot()` the Debug View uses, so it **includes placeholder rows** for configured-but-quiet native source bindings (`IsConfiguredPlaceholder`). Callers enumerating real alarms filter on `Active && !IsConfiguredPlaceholder`.
|
||
- `ScriptAlarm` is a flat projection of the retained `AlarmStateChanged`: `Name`, `SourceReference`, `NativeSourceCanonicalName`, `Active`, `Acknowledged`, `Confirmed?`, `Shelved` (all shelve sub-states collapse to one boolean), `Suppressed`, `Severity` (0–1000), `Kind`, `Message`, `AlarmTypeName`, `Category`, `OperatorUser`, `OperatorComment`, `OriginalRaiseTime?`, `Timestamp`, `AckTime?`, `CurrentValue`, `LimitValue`, `IsConfiguredPlaceholder`, `IsMesReportable` (trailing, 2026-09-11 — always `false` on native rows). The `Condition` is the authority for active/acked/severity, so computed and native alarms read identically from one filter expression.
|
||
- **`IsMesReportable` is stamped in one place.** `InstanceActor.BuildAlarmStatesSnapshot()` sets `AlarmStateChanged.IsMesReportable` from `ResolvedAlarm.IsMesReportable` on every row it produces — retained (fired/active) rows and the fallback/idle rows alike — so `Alarms.CurrentAsync()`/`MesReportableAsync()` and the Debug View snapshot always agree regardless of whether the alarm has fired. The flag is **not** carried on the gRPC stream or `DebugAlarmStateDto` (no proto change in this slice — deferred-work register row 122), so central-side surfaces (Debug View live stream, Alarm Summary) see `false` until that follow-up lands.
|
||
- The type lives in Commons so the runtime accessor and the Script Analysis compile-only surface (`ScriptCompileSurface.CompileAlarmsAccessor`) project to the **same** type — a script binding at the design-time gate binds identically at the site. Parity is pinned by `CompileSurfaceParityTests`.
|
||
- Design record: `docs/plans/2026-06-30-mes-alarm-status-api.md` §5.2.
|
||
|
||
### File Access (`Files`, read-only, allow-listed — 2026-09-28)
|
||
- `Files.ReadTextAsync(path)` → `string`, `Files.ReadLinesAsync(path)` → `IReadOnlyList<string>` (`\r\n`/`\n`/`\r`; a trailing line break adds no empty last line), `Files.ExistsAsync(path)` → `bool`. Also reachable as `Instance.Files`. The use case that motivated it: a recipe/NC file whose path a Delmia/MES call hands to the machine, which a script parses and loads into attributes.
|
||
- **The one sanctioned exception to the `System.IO` ban, and it is host-mediated.** Scripts still cannot name any `System.IO` type — the trust policy is unchanged. The runtime `FilesAccessor` delegates to the site-singleton `ScriptFileReader` (`SiteRuntime/Scripts/`), which does every check on the host side.
|
||
- **Off by default.** `ScadaBridge:SiteRuntime:ScriptFileAccess:AllowedRoots` (list of absolute local/UNC directories, default empty) enables it; with no roots every call throws `ScriptFileAccessException(Disabled)`, so an unconfigured site behaves exactly as before. `ScriptFileAccess:MaxFileBytes` (default 1 MiB, validated 1…64 MiB) caps one read. Both validated at boot (`SiteRuntimeOptionsValidator` — a relative root is rejected). Read per call through `IOptionsMonitor`.
|
||
- **Guard, in order:** roots configured → path non-empty, fully qualified, not a device path (`\\?\`, `\\.\`), no NTFS alternate data stream → canonical path (`Path.GetFullPath`, so `..` is resolved first) **strictly inside** a root with a directory-boundary match (root `C:\recipes` does not admit `C:\recipes2\x`, and the root itself is not a file) → no symbolic link / junction / reparse point on the file **or on any directory between the root and the file** (the root itself may be a link — an administrator chose it) → size under the cap, re-checked while reading in case the file grows. Files are opened `FileShare.ReadWrite | Delete` so a file still being written by its producer can be read; text is BOM-detected, UTF-8 by default.
|
||
- **Errors are one Commons type.** Every refusal and I/O failure is `ScriptFileAccessException` (`Commons.Types.Scripts`) with a `Failure` of `Disabled`, `InvalidPath`, `OutsideAllowedRoots`, `LinkNotAllowed`, `NotFound`, `TooLarge` or `IoError` — scripts cannot name `System.IO` exception types, so this is what they `catch`. `Commons.Types.Scripts` is now one of the **default script imports** on every surface (runtime, deploy gate, Test Run editor), so the short name binds.
|
||
- **Bounded by the run.** Reads are cancelled by the script's own execution deadline as well as any token the script passes. Refusals (outside roots, links) log a Warning on the node; successful reads log at Debug. There is no audit row — file reads are not an audited channel (the `AuditLog` scope is outbound API/DB, notifications and inbound API).
|
||
- **The node's service account still needs read permission** on the path; the allow-list only ever narrows access. A time-of-check/time-of-use window exists between the link check and the open (a writer inside an allowed root could swap a file for a link in between) — acceptable because only the directories an administrator lists are reachable at all, and read-only.
|
||
- **Site scripts only.** Inbound API method scripts (central) have no `Files` global. The Central UI Test Run sandbox mirrors the member so scripts bind, but every call throws `ScriptFileAccessException(Disabled)` — central cannot reach a site's filesystem — so a script's own error handling runs as it would on an unconfigured site. The design-time `ScriptCompileSurface.CompileFiles` mirror is pinned by `CompileSurfaceParityTests`.
|
||
- Design record: `docs/plans/2026-09-28-script-file-access-design.md`.
|
||
|
||
### Logging (`Log`)
|
||
- `Log.Debug(message, exception?)` / `Log.Info` / `Log.Warn` / `Log.Error` — leveled, per-run logging available to **every** script, alongside the `Alarms` accessor above (design `docs/plans/2026-09-09-flow-debugger-step-logs-design.md` D4/D5). Synchronous, `void`, and it **never throws into the script**: a failing sink is swallowed, because a debug observer must not fail a production run.
|
||
- **Every line always reaches the node's `ILogger`** (structured `Instance` / `Script` / `ExecutionId` / message), traced or not — an untraced run is an ordinary logging surface and pays nothing else. A `ScriptTrace` / `ScriptLog` **audit row** is emitted **only when the run is traced** (a Flow Debugger Live run, or a real call to a method with `TraceAllCalls`), so a timer or value-change script logging at high rate costs no audit volume. There is no site-event-log sink for `Log.*`.
|
||
- The implementation is `ScriptLogger` (`SiteRuntime/Scripts/ScriptLogger.cs`), created by `ScriptRuntimeContext` and exposed as `ScriptGlobals.Log`. **One logger instance is one run's counter** — `CallShared` shares the calling context, so an inline shared script shares the caller's counter, execution id and anchor row rather than starting its own.
|
||
- **Runtime lines land on the step that produced them.** While a run is traced, `ScriptRuntimeContext` wraps its own `ILogger` in a `TraceCapturingLogger`, which forwards everything to the inner logger unchanged and additionally copies **Information and above** into `ScriptLog` rows with `logSource = "runtime"` (Debug/Trace are deliberately not captured — internal chatter). `IsEnabled` returns true for Information and above regardless of the node's minimum level, because those entries are trace data rather than node log output. Attribution rides an ambient `TraceStepScope` (`AsyncLocal<Guid?>`): `SetAttribute`, `WriteBatchAndWait`, `WaitInternal` and the synchronous `ExternalSystem.Call` pre-mint their step row's `EventId`, open the scope around their body and pass the id to `EmitTrace`, so a warning the runtime logs inside a write shows on that write's node. Everything else falls back to the run's anchor `ScriptRun` row. **Build `ScriptLogger` over the UNDECORATED logger** — `TraceCapturingLogger`'s constructor rejects an inner `TraceCapturingLogger`, since double wrapping would emit two rows per line.
|
||
- **Bounded (D7).** `ScadaBridge:SiteRuntime:MaxLogLinesPerRun` (default 200, validated `> 0`) caps the rows one run may emit; messages are clipped at 2 000 characters. Lines past the cap are counted and dropped, and a single closing `warn` / `runtime` row naming the dropped count is flushed at run end (it bypasses the counter — it is the one row that must always land).
|
||
- **Exception detail.** A failed run's `ScriptRun` row now carries `errorDetail = ex.ToString()` alongside the existing one-line `errorMessage`, so the Flow Debugger can show type, message, stack and inner exception. A timeout emits no detail — an `OperationCanceledException` stack is noise.
|
||
- **`Console.WriteLine` is not the logging surface on a site.** `System` is imported, so it compiles, and the Central UI Test Run sandbox does capture it; but on a site node `Console.Out` is process-global and shared by every concurrent script run, so per-run capture is unsound. `Log.*` is the leveled, per-run replacement, and the sandbox's mirror writes to the same console pane.
|
||
|
||
### Parameter Access
|
||
- `Parameters["key"]` — Raw dictionary access (returns `object?`, requires manual casting).
|
||
- `Parameters.Get<T>("key")` — Typed access with descriptive error messages. Throws `ScriptParameterException` if parameter is missing, null, or cannot be converted to `T`.
|
||
- `Parameters.Get<T?>("key")` — Nullable typed access. Returns `null` if parameter is missing, null, or cannot be converted.
|
||
- `Parameters.Get<T[]>("key")` — Array access. Converts a list parameter to a typed array. Throws on first unconvertible element.
|
||
- `Parameters.Get<List<T>>("key")` — List access. Converts a list parameter to a typed `List<T>`.
|
||
- Supported types: `bool`, `int`, `long`, `float`, `double`, `string`, `DateTime`.
|
||
|
||
### Recursion Limit
|
||
- Every script call (`Instance.CallScript` and `Scripts.CallShared`) increments a call depth counter.
|
||
- If the counter exceeds the maximum recursion depth (default: 10), the call fails with an error.
|
||
- The error is logged to the site event log.
|
||
|
||
---
|
||
|
||
## Script Trust Model
|
||
|
||
Scripts execute **in-process** with constrained access. The following restrictions are enforced at compilation and runtime:
|
||
|
||
- **Allowed**: Access to the Script Runtime API (GetAttribute, SetAttribute, CallScript, CallShared, ExternalSystem, Notify, Database, Tracking, Alarms, Log), standard C# language features, basic .NET types (collections, string manipulation, math, date/time). `System.Diagnostics.Stopwatch`, `Debug`, and `Activity` are permitted.
|
||
- **Forbidden**: File system access (`System.IO`), process spawning (`System.Diagnostics.Process`), threading (`System.Threading` — except `Tasks`, `CancellationToken`, and `CancellationTokenSource`), reflection (`System.Reflection`), all raw network access (`System.Net` — must use `ExternalSystem.Call`), native interop (`System.Runtime.InteropServices`, `Microsoft.Win32`), assembly loading, unsafe code, `dynamic`, `Activator`.
|
||
- **Execution timeout**: Configurable per-script maximum execution time, **measured from enqueue** (WP3.1) — a run that spends its whole budget queued behind blocked scripts is shed at dequeue without executing, rather than being granted a fresh full budget for work whose triggering condition is already stale. Exceeding the timeout cancels the script **cooperatively** (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A **watchdog names the script** once its thread has not returned within a grace period after the timeout — and, as of WP3.1, **detaches and replaces that worker thread**, so a wedged script costs the pool a thread only until the watchdog fires rather than for the rest of the process. The detached worker exits (instead of pulling more work) when its body finally returns, so capacity never silently doubles. Live detached workers are capped at the pool size — at the cap no replacement is started and an Error site event says so, because bounded starvation is preferable to unbounded thread growth when scripts wedge en masse. The scheduler's **queue depth / busy-thread / oldest-busy-age / detached-thread gauges** and the **per-interval shed count** are reported through Health Monitoring so a saturated, stuck, or shedding scheduler is visible.
|
||
- **Memory**: Scripts share the host process memory. No per-script memory limit, but the execution timeout prevents runaway allocations.
|
||
|
||
The forbidden-API policy is defined authoritatively in `ScriptTrustPolicy` (Script Analysis component, #25). `ScriptCompilationService.ValidateTrustModel` delegates to `ScriptTrustValidator.FindViolations` for the trust verdict; the Site Runtime also performs a `CSharpScript.Compile` against the real `ScriptGlobals` for execution. This is defence-in-depth static enforcement, not a true runtime sandbox.
|
||
|
||
### Script Trust Boundary Auditing
|
||
|
||
Every script-trust-boundary call (External System Gateway, Database layer, Notify) emits an `AuditEvent` to `IAuditWriter` (site-local SQLite append). Hot path; never fails the calling action; failures logged via the `SiteAuditWriteFailures` health metric (see [Component-HealthMonitoring.md](Component-HealthMonitoring.md)). The central audit mirror and event schema live in [Component-AuditLog.md](Component-AuditLog.md).
|
||
|
||
## Script Scoping Rules
|
||
|
||
- Scripts can only read/write attributes on **their own instance** (via the parent Instance Actor).
|
||
- Scripts can call other scripts on **their own instance** (via sibling Script Actors).
|
||
- Scripts can call **shared scripts** (inline execution).
|
||
- Scripts **cannot** access other instances' attributes or scripts.
|
||
- Alarm on-trigger scripts **can** call instance scripts; instance scripts **cannot** call alarm on-trigger scripts.
|
||
|
||
---
|
||
|
||
## Tell vs. Ask Usage
|
||
|
||
Per Akka.NET best practices, internal actor communication uses **Tell** (fire-and-forget with reply-to) for the hot path:
|
||
|
||
- **Tag value updates** (DCL → Instance Actor): Tell. High-frequency, no response needed.
|
||
- **Attribute change notifications** (Instance Actor → Script/Alarm Actors): Tell. Fan-out notifications.
|
||
- **Stream publishing** (Instance Actor → Akka stream): Tell. Fire-and-forget.
|
||
|
||
**Ask** is reserved for system boundaries where a synchronous response is needed:
|
||
|
||
- **`Instance.CallScript()`**: Ask pattern from the running script (via `ScriptRuntimeContext.CallScript`) to sibling Script Actor. The caller needs the return value. Acceptable because script calls are infrequent relative to tag updates.
|
||
- **`Route.To().Call()`**: Ask from Inbound API to site Instance Actor via Communication Layer. External caller needs a response.
|
||
- **Debug view snapshot**: Ask from Communication Layer to Instance Actor for initial state.
|
||
|
||
## Concurrency & Serialization
|
||
|
||
- The Instance Actor processes messages **sequentially** (standard Akka actor model). This means `SetAttribute` calls from concurrent script runs are serialized at the Instance Actor, preventing race conditions on attribute state.
|
||
- Script runs (launched via `ScriptRunLauncher`, no per-run child actor) may run concurrently, but all state mutations (attribute reads/writes, alarm state updates) are mediated through the parent Instance Actor's message queue.
|
||
- External side effects (external system calls, notifications, database writes) are not serialized — concurrent scripts may produce interleaved side effects. This is acceptable because each side effect is independent.
|
||
|
||
## SiteStreamManager and gRPC Integration
|
||
|
||
- The `SiteStreamManager` implements the `ISiteStreamSubscriber` interface, allowing the Communication Layer's `SiteStreamGrpcServer` to subscribe to the stream for cross-cluster delivery via gRPC.
|
||
- When a gRPC `SubscribeInstance` call arrives, the `SiteStreamGrpcServer` creates a `StreamRelayActor` and subscribes it to `SiteStreamManager` for the requested instance. Events flow from `SiteStreamManager` → `StreamRelayActor` → `Channel<SiteStreamEvent>` → gRPC response stream to central.
|
||
- The `SiteStreamManager` filters events by instance unique name and forwards matching events to all registered subscribers (both local debug consumers and gRPC relay actors).
|
||
|
||
## Site-Wide Stream Backpressure
|
||
|
||
- The site-wide Akka stream uses **per-subscriber buffering** with bounded buffers. Each subscriber (gRPC relay actors, future consumers) gets an independent buffer.
|
||
- If a subscriber falls behind (e.g., slow network on gRPC stream), its buffer fills and oldest events are dropped. This does not affect other subscribers or the publishing Instance Actors.
|
||
- Instance Actors publish to the stream with **fire-and-forget** semantics — publishing never blocks the actor.
|
||
|
||
## Error Handling
|
||
|
||
### Script Errors
|
||
- Unhandled exceptions and timeouts in a script run are **logged locally** to the site event log — unconditionally, regardless of the WP3.2 per-run Started/Completed sampling policy (see Script Run Launch above).
|
||
- The Script Actor (coordinator) is **not affected** — it remains active for future trigger events.
|
||
- Script failures are **not reported to central** (except as aggregated error rate metrics via Health Monitoring).
|
||
|
||
### Alarm Evaluation Errors
|
||
- Errors during alarm condition evaluation are **logged locally** to the site event log.
|
||
- The Alarm Actor remains active and continues evaluating on subsequent value updates.
|
||
- Alarm evaluation error rates are reported to central via Health Monitoring.
|
||
|
||
### Script Compilation Errors
|
||
- If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied.
|
||
- The failure is reported back to central as a failed deployment.
|
||
- **Enforced by a site-side pre-compile validation gate (S3)**: before the Instance Actor is created or the config persisted, the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **synchronously, as a pure prefix step of the deploy application on the singleton's actor thread** — the verdict is never derived off-thread, because piping a verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering. WP3.1 made it **warm-then-gate**: the same compile is first run off-thread purely to populate the process-wide compile cache (`DeployCompileWarmed` pipes back), so the authoritative synchronous gate is then all cache hits and no longer holds the singleton for a Roslyn compile. Ordering is preserved by a **per-instance in-flight guard** — while a warm is in flight for instance X, further mutating commands for X (deploy / delete / disable / enable) are queued and re-dispatched in arrival order once the deploy applies; a second deploy with nothing queued behind it supersedes the pending one last-write-wins and its displaced deployer is answered `Failed`-superseded rather than left to Ask-timeout. Commands for **other** instances flow freely, which is the point, and is safe because cross-instance ordering was never guaranteed to callers. Unchanged script bodies hit the compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors. This makes the deployment result honest even if central's pre-deployment validation was bypassed or skew let a bad artifact through.
|
||
- Note: Pre-deployment validation at central should still catch compilation errors before they reach the site; the site gate is the authoritative backstop.
|
||
|
||
---
|
||
|
||
## Dependencies
|
||
|
||
- **Script Analysis (#25)**: `ScriptCompilationService.ValidateTrustModel` delegates the forbidden-API verdict to `ScriptTrustValidator.FindViolations`. The Site Runtime retains its own `CSharpScript.Compile` against the real `ScriptGlobals` for execution.
|
||
- **Data Connection Layer**: Provides tag value updates to Instance Actors. Receives write requests from Instance Actors. Also feeds Native Alarm Actors: connections implementing `IAlarmSubscribableConnection` (OPC UA A&C servers, MxAccess Gateway) deliver `NativeAlarmTransitionUpdate` events in response to a `SubscribeAlarmsRequest`, and signal `NativeAlarmSourceUnavailable` on connection loss. **Subscription-handshake retry (S4/UA6)**: a tag-subscription handshake that either failed or whose response was lost is re-sent on a fixed interval per connection (default 5 s) — the retry is armed on every send and cancelled on the first success reply, closing both the failed-response and lost-response gaps; the `NativeAlarmActor` subscribe uses the same retry pattern.
|
||
- **Store-and-Forward Engine**: Handles reliable delivery for external system calls, cached database writes, and notifications submitted by scripts. For the notification category specifically, it forwards to the central cluster for delivery (not directly to SMTP). Owns the site-local operation tracking table that backs `Tracking.Status(id)`.
|
||
- **External System Gateway**: Provides external system method invocations for scripts.
|
||
- **Communication Layer**: Receives deployments and lifecycle commands from central. Handles debug view requests. Reports deployment results.
|
||
- **Site Event Logging**: Records script executions, alarm events, deployment events, instance lifecycle events.
|
||
- **Health Monitoring**: Reports script error rates and alarm evaluation error rates.
|
||
- **Local SQLite**: Persists deployed configurations, system-wide artifacts (external system definitions, database connection definitions, data connection definitions). Sites also maintain peer SQLite stores for the Store-and-Forward buffer, the site event log, the operation tracking table, and the site-local `AuditLog` (see [Component-AuditLog.md](Component-AuditLog.md)). The `AuditLog` file is purged on the same daily cadence as the others but respects the hard `ForwardState` invariant — rows still `Pending` forward are never purged, regardless of age.
|
||
- **`ZB.MOM.WW.Secrets.Abstractions`** (`ISecretStore`, `ISecretResolver`, `ISecretCacheInvalidator`): `DeployedSecretsMirror` applies deployed secrets into, and verifies against, the node's local secret store (composed by the Host — see Component-Host.md → *Secrets*). The Site Runtime depends only on the abstractions package, never the concrete store implementation.
|
||
|
||
## Interactions
|
||
|
||
- **Deployment Manager (central)**: Receives flattened configurations, system-wide artifact updates (including a site's referenced secrets, ciphertext only), and instance lifecycle commands.
|
||
- **Data Connection Layer**: Bidirectional — receives value updates, sends write-back commands.
|
||
- **Communication Layer**: Receives commands from central, sends deployment results, serves debug view data.
|
||
- **Store-and-Forward Engine**: Scripts route cached writes, notifications, and external system calls here.
|
||
- **Health Monitoring**: Periodically reports error rate metrics.
|
||
|
||
|
||
### Medium architecture review lifecycle guarantees (2026-09-09)
|
||
|
||
- Alarm on-trigger runs resolve the same scoped integration services as ordinary scripts, including external systems, databases, notifications, operation tracking, auditing, and site/node provenance. The scope remains alive for the run and is disposed on every exit.
|
||
- Native alarm subscription retries retain the current correlation ID until its response deadline, reject stale replies, and back off to a bounded interval. Rehydration cannot overwrite a condition touched by a live event, including deletion, or a completed authoritative snapshot.
|
||
- Audited raw ADO connections expose no connection string and reject `ConnectionString` assignment and `ChangeDatabase`; their configured database name remains the audit target.
|
||
- Static override persistence runs in order off the actor dispatcher. Redeployment waits for both predecessor termination and successful final persistence acknowledgement before clearing old state; failed persistence fails the redeployment. At most four redeployment predecessor stops are in progress per deployment manager. Delete and disable retain the same-name stop barrier until the final flush and lifecycle persistence finish; an overlapping deploy/enable is rejected with a retryable busy diagnostic. Timer flushes keep coalescing while a previous SQLite write is pending.
|
||
- Synchronous operation-tracking disposal drains and joins its thread-pool writer before disposing SQLite resources, including when invoked under a synchronization context.
|