> **The compile cache is owned by the `ScriptCompilationService` singleton, not static (register row 100)**: `SiteScriptCompileCache` is an ordinary instance held by `ScriptCompilationService`, and "one Roslyn compile per script body per process" (N4) is exactly the statement that `AddSiteRuntime` registers that service as a **DI singleton** — the deploy gate (via `DeployCompileValidator`), every Instance Actor's `PreStart`, and `SharedScriptLibrary` all reach the one cache through the one service the Deployment Manager threads to them. The singleton lifetime is therefore load-bearing rather than incidental, and is pinned by a registration test (`SiteRuntimeServiceRegistrationTests`); downgrading it to Scoped/Transient would hand every consumer an empty cache and silently reintroduce WP3.1's recompile storms with no behavioural test failing. The class was previously `static`, which made its entry table and its `Hits`/`Count` counters one process-global blob every test class in the assembly wrote into; the `Clear()` method that existed only so tests could reset that blob is **removed** — it never had a production caller, and constructing a cache is the honest reset.
> **Compile cache eviction is approximate LRU, not a wholesale clear (WP3.1)**: `SiteScriptCompileCache` is bounded at 1024 entries — deliberately smaller than the TemplateEngine verdict cache's 4096, because entries pin compiled assemblies rather than verdict strings. It used to clear **everything** on overflow. Instance scripts *and* trigger expressions share this cache, so on a site large enough to cross the bound every overflow discarded up to 1023 live compiled scripts and the next deploy or Instance-Actor start paid a full recompile storm **on actor threads**. Overflow now evicts only the oldest ⅛ (128 entries) by last-access stamp: one 1024-element scan instead of 1023 future recompiles, and hot entries survive. Recency is an `Interlocked` access sequence rather than a clock — deterministic for tests and immune to clock steps; hits update it lock-free, and only the (rare) eviction sweep takes a lock, double-checked so concurrent inserts do not stampede it. The sweep snapshots through `ConcurrentDictionary.ToArray()` and **not** LINQ over the dictionary: LINQ's `ToArray` picks the `ICollection.CopyTo` fast path, which throws when a concurrent insert lands between its Count read and its copy, and inserts are deliberately lock-free.
>
> **Eviction is visible, the bound is configurable, and eviction never reclaims the assembly (review 2026-10-21 SR-M2)**: the bound is `ScadaBridge:SiteRuntime:ScriptCompileCacheMaxEntries` (default 1024, validated `>= 8`). Eviction discards only the cached `Script<object?>`; the assembly Roslyn loaded for it stays in the default `AssemblyLoadContext` for the process lifetime (Roslyn scripting offers no collectible loader), so a body that is evicted and later recompiled — every Instance Actor restart, sweep, enable or failover — loads a *second* permanent copy. A site whose distinct script + trigger-expression body count exceeds the bound therefore grows monotonically at the eviction × restart rate. Each sweep increments `SiteScriptCompileCache.Evictions` and logs one Warning (rate-limited to one per minute, carrying the count since the last report) naming the option to raise. Size it above the site's distinct-body count. **Faulted compiles are never cached**: only success and diagnostic failures (a verdict on the code) are memoised; an unexpected exception during compile returns a non-cacheable `Faulted` result so a transient fault is retried on the next call instead of sticking until eviction/restart.
> **Static options were only half the fix — the compile must also use the shared caching metadata resolver**: the static `ScriptOptions` carry only the *direct* API-surface references (5 assemblies here), and each `script.Compile()` still binds their **transitive closure** (~105 assemblies on a site node). Every transitively-referenced assembly is resolved through the options' `MetadataReferenceResolver`, and Roslyn's default one has no cross-compilation cache — it calls `MetadataReference.CreateFromFile` afresh on **every compile**, minting the same undisposed `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock` triple per assembly per compiled script, pinned for the process lifetime by the compile cache. That per-compile *re-resolution* — not the per-compile options construction — was the dominant term (measured live: 21 site scripts holding 6,640 metadata objects, counts bit-identical before and after the static-options fix). `ScriptCompilationService` therefore also attaches the process-wide `CachingScriptMetadataResolver.Instance` (owned by the Script Analysis component, see `Component-ScriptAnalysis.md`) via `ScriptOptions.WithMetadataResolver`, so each distinct assembly is materialized once per process. The decorator returns exactly what the undecorated resolver returned — only object identity is de-duplicated — so compile results and the trust gate are unchanged.
@@ -224,7 +226,7 @@ flowchart TD
- **Alarm state changes**: `[InstanceUniqueName].[AlarmName]`, alarm state (active/normal), priority, timestamp.
### GetAttribute / SetAttribute
- **GetAttribute**: Returns the current in-memory value for the requested attribute. Attribute read/write/wait helper continuations do not capture the calling scheduler (`ConfigureAwait(false)`), so synchronous `Attributes[key]` access can complete even when every script worker is blocked. AsyncLocal trace attribution still flows with execution context.
- **GetAttribute**: Returns the current in-memory value for the requested attribute. Attribute read/write/wait helper continuations do not capture the calling scheduler (`ConfigureAwait(false)`), so synchronous `Attributes[key]` access can complete even when every script worker is blocked. AsyncLocal trace attribution still flows with execution context. The same applies to the remaining script helper awaits (`CallScript`, `Alarms.CurrentAsync`, `ExternalSystem.Call`/`CachedCall`, `Database.Connection`, `Notify.Send`/`Status`, shared-script execution): their continuations run off the script scheduler, so a script that blocks on one with `.Result`/`.Wait()` cannot deadlock the pool against its own callee's completion.
- **SetAttribute** (for attributes with data source reference): Sends a write request to the Data Connection Layer. The DCL writes to the physical device. If the write fails (connection down, device rejection, timeout), the error is returned synchronously to the calling script for handling. On success, the existing subscription picks up the confirmed value from the device and sends it back as a value update, which then updates the in-memory value. The in-memory value is **not** optimistically updated.
- **SetAttribute** (for static attributes): Updates the in-memory value and **persists the override to local SQLite**. On restart (or failover), the Instance Actor loads persisted overrides on top of the deployed configuration, preserving runtime-modified values. A redeployment of the instance resets all persisted overrides to the new deployed configuration values.
@@ -284,7 +286,7 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
- The script body runs on the **dedicated `ScriptExecutionScheduler`** (a bounded set of dedicated threads), not the shared .NET thread pool, so blocking script I/O cannot starve the global pool or stall Akka dispatchers. The scheduler is **process-wide by default** (one pool per host), but each script/alarm actor takes it through an **optional injection seam** rather than reaching for the static directly: the Host injects nothing and gets the shared pool, while tests (or a future multi-site host) can hand an actor its own instance. The shared accessor also recreates a disposed pool rather than returning it, so a disposed scheduler can never silently poison later executions.
- **Pool sizing is instance-scaled and grow-only (WP3.1).** The pool was a fixed 8 threads regardless of load. It is now `clamp(max(ScriptExecutionThreadCount, ceil(enabledInstances / 8)), 1, ScriptExecutionMaxThreadCount)` — the existing `ScriptExecutionThreadCount` (default 8) becomes the **floor**, so configurations at or below 64 instances behave exactly as before, and the new `ScriptExecutionMaxThreadCount` (default 32) is the ceiling. Beyond the ceiling the per-script cap below is the real regulator. `DeploymentManagerActor.UpdateInstanceCounts` calls `EnsureCapacity` on every deploy / undeploy / enable / disable and once per staggered startup batch. Growth only: undeploying leaves idle threads, which cost nothing measurable and avoid drain/steal complexity.
- **The deadline is armed at enqueue, not at dequeue (WP3.1).** The run's timeout `CancellationTokenSource` is created on the actor thread *before* the body is queued, so queue wait consumes the script's own budget. A body that dequeues past its deadline **skips execution entirely** and takes the existing timeout path (site event, script-error counter, error reply, completion message): a saturated pool sheds stale work instead of running it late with a fresh full budget.
- **Concurrent runs per script are capped (WP3.1).** `MaxConcurrentRunsPerScript` (default 4) bounds runs in flight — queued or executing — for any one script or alarm on-trigger script. Over the cap the **newest** run is shed: the four already in flight are closest to their own deadlines and already charged against them, so nothing is ever reordered and no extra queue is needed (the scheduler's FIFO already is the queue). A shed increments `ISiteHealthCollector.IncrementScriptRunShed` (surfaced as `ScriptRunShedCount`), emits a `script`/`Warning` site event **rate-limited to one per script per minute** so a hot trigger cannot flood `site_events`, and — for an Ask-based `CallScript` — replies with an explicit error so a nested call or inbound-API route fails fast instead of hanging.
- **Concurrent runs per script are capped (WP3.1).** `MaxConcurrentRunsPerScript` (default 4) bounds runs in flight — queued or executing — for any one script or alarm on-trigger script. Over the cap the **newest** run is shed: the four already in flight are closest to their own deadlines and already charged against them, so nothing is ever reordered and no extra queue is needed (the scheduler's FIFO already is the queue). A shed increments `ISiteHealthCollector.IncrementScriptRunShed` (surfaced as `ScriptRunShedCount`), emits a `script`/`Warning` site event **and a node-log Warning, both rate-limited to one per script per minute** (the log line carries the number of sheds since the previous one, so a 100 ms interval trigger against a wedged body cannot write 10 lines/s) so a hot trigger cannot flood `site_events` or the log, and — for an Ask-based `CallScript` — replies with an explicit error so a nested call or inbound-API route fails fast instead of hanging.
- **The cap gates only depth-0 launches (review fix).** A trigger fire or a depth-0 `CallScript`/inbound-API route counts against `MaxConcurrentRunsPerScript`; a *nested*`Instance.CallScript` self-recursion (`callDepth > 0`) is exempt — it is already bounded by `MaxScriptCallDepth` instead, and the calling run is holding one of the four cap slots itself while it awaits the callee, so counting the nested launch too would spuriously shed legitimate self-recursion.
#### Traced runs (Flow Debugger, 2026-09-05)
@@ -427,7 +429,7 @@ The Instance Actor owns native-alarm setup alongside its computed Script and Ala
- **Operations**: `Upsert` (on accepted transition), `Delete` (on condition drop-out), `Get` (PreStart rehydrate, scoped to instance + source), and `ClearForInstance` (redeploy/undeploy reset).
- **Display metadata persistence (UA4)**: persisted rows include the condition's display metadata, so **rehydrated conditions render fully** (type / category / message / current + limit values) before the first fresh source snapshot arrives; a pre-UA4 row with no metadata rehydrates with empty display fields (unchanged behavior).
- **Ack timestamp persistence**: the condition's `AckTime` rides the same `metadata_json` blob, so an acknowledged alarm keeps its ack instant across restart/failover rather than reappearing as if it had never been acknowledged. It is deliberately **not** a new physical column: `native_alarm_state` is `RegisterReplicated` in `SiteLocalDbSetup`, and LocalDb builds that table's CDC triggers from the column list at registration time — an additive JSON property changes no schema, no triggers and no replication contract, and `metadata_json` is exactly the extension point UA4 introduced for this. A row written before the field existed deserializes it as `null` ("ack time unknown"), which is the correct fallback. See `docs/plans/2026-06-30-mes-alarm-status-api.md` §6.4.
- **Coalesced writes (P4)**: per-transition upserts are coalesced (latest-per-source-reference) and flushed on a short single-shot timer rather than one SQLite write per transition, so an alarm storm collapses to one batched write; a pending upsert is dropped if the condition is deleted before the flush (no resurrection). Durability window is at most one flush interval on crash, reconciled by the next re-subscribe snapshot.
- **Coalesced writes (P4)**: per-transition upserts are coalesced (latest-per-source-reference) and flushed on a short single-shot timer rather than one SQLite write per transition, so an alarm storm collapses to one batched write; deletes ride the same dirty set (an upsert/delete discriminator per reference) and are applied in that one batched transaction — a snapshot swap dropping N stale conditions is one commit, not N — and a delete replaces a pending upsert for the same reference (no resurrection), as a later upsert replaces a pending delete. Durability window is at most one flush interval on crash, reconciled by the next re-subscribe snapshot.
- This is a **peer SQLite store** to the existing deployed-configuration, store-and-forward, operation-tracking, and `AuditLog` stores. Unlike computed alarm state, native mirror state is intentionally persisted so it survives failover.
---
@@ -592,7 +594,7 @@ Scripts execute **in-process** with constrained access. The following restrictio
- **Allowed**: Access to the Script Runtime API (GetAttribute, SetAttribute, CallScript, CallShared, ExternalSystem, Notify, Database, Tracking, Alarms, Log), standard C# language features, basic .NET types (collections, string manipulation, math, date/time). `System.Diagnostics.Stopwatch`, `Debug`, and `Activity` are permitted.
- **Forbidden**: File system access (`System.IO`), process spawning (`System.Diagnostics.Process`), threading (`System.Threading` — except `Tasks`, `CancellationToken`, and `CancellationTokenSource`), reflection (`System.Reflection`), all raw network access (`System.Net` — must use `ExternalSystem.Call`), native interop (`System.Runtime.InteropServices`, `Microsoft.Win32`), assembly loading, unsafe code, `dynamic`, `Activator`.
- **Execution timeout**: Configurable per-script maximum execution time, **measured from enqueue** (WP3.1) — a run that spends its whole budget queued behind blocked scripts is shed at dequeue without executing, rather than being granted a fresh full budget for work whose triggering condition is already stale. Exceeding the timeout cancels the script **cooperatively** (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A **watchdog names the script** once its thread has not returned within a grace period after the timeout — and, as of WP3.1, **detaches and replaces that worker thread**, so a wedged script costs the pool a thread only until the watchdog fires rather than for the rest of the process. The detached worker exits (instead of pulling more work) when its body finally returns, so capacity never silently doubles. Live detached workers are capped at the pool size — at the cap no replacement is started and an Error site event says so, because bounded starvation is preferable to unbounded thread growth when scripts wedge en masse. The scheduler's **queue depth / busy-thread / oldest-busy-age / detached-thread gauges** and the **per-interval shed count** are reported through Health Monitoring so a saturated, stuck, or shedding scheduler is visible.
- **Execution timeout**: Configurable per-script maximum execution time, **measured from enqueue** (WP3.1) — a run that spends its whole budget queued behind blocked scripts is shed at dequeue without executing, rather than being granted a fresh full budget for work whose triggering condition is already stale. Exceeding the timeout cancels the script **cooperatively** (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A **watchdog names the script** once its thread has not returned within a grace period after the timeout — and, as of WP3.1, **detaches and replaces that worker thread**, so a wedged script costs the pool a thread only until the watchdog fires rather than for the rest of the process. The detached worker exits (instead of pulling more work) when its body finally returns, so capacity never silently doubles. Its slot is then handed back and reused by the next worker start, so a script that wedges on every run does not grow the scheduler's slot table (or start a fresh slot) for the process lifetime. Live detached workers are capped at the pool size — at the cap no replacement is started and an Error site event says so, because bounded starvation is preferable to unbounded thread growth when scripts wedge en masse. The scheduler's **queue depth / busy-thread / oldest-busy-age / detached-thread gauges** and the **per-interval shed count** are reported through Health Monitoring so a saturated, stuck, or shedding scheduler is visible.
- **Memory**: Scripts share the host process memory. No per-script memory limit, but the execution timeout prevents runaway allocations.
The forbidden-API policy is defined authoritatively in `ScriptTrustPolicy` (Script Analysis component, #25). `ScriptCompilationService.ValidateTrustModel` delegates to `ScriptTrustValidator.FindViolations` for the trust verdict; the Site Runtime also performs a `CSharpScript.Compile` against the real `ScriptGlobals` for execution. This is defence-in-depth static enforcement, not a true runtime sandbox.
/// <summary>A snapshot swap that drops several conditions deletes their rows via the batched flush, and a delete cancels a pending upsert (review 2026-10-21 01/SR-P2).</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
/// <summary>Verifies that clearing an instance removes all of its sources and leaves other instances untouched.</summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Joseph Doherty