perf(runtime): split trigger evals from the blocking pool; bounded, deadline-aware execution

docs/requirements/Component-SiteRuntime.md

@@ -41,9 +41,9 @@ flowchart TD
     AA2["Alarm Actor ('LowPressure')<br/>— coordinator (computed)"]
     NAA1["Native Alarm Actor ('OpcUaServer1')<br/>— read-only mirror, peer to Alarm Actor"]
 
-    SEA1["Script Execution Actor<br/>— short-lived, per invocation"]
-    SEA2["Script Execution Actor<br/>— short-lived, per invocation"]
-    AEA1["Alarm Execution Actor<br/>— short-lived, per on-trigger invocation"]
+    SEA1["script run<br/>— launched task, not an actor"]
+    SEA2["script run<br/>— launched task, not an actor"]
+    AEA1["alarm on-trigger run<br/>— launched task, not an actor"]
 
     IA2CHILD["… (Script / Alarm Actors)"]
 
@@ -57,9 +57,9 @@ flowchart TD
     IA1 --> AA2
     IA1 --> NAA1
 
-    SA1 --> SEA1
-    SA2 --> SEA2
-    AA1 --> AEA1
+    SA1 -.-> SEA1
+    SA2 -.-> SEA2
+    AA1 -.-> AEA1
 
     IA2 -.-> IA2CHILD
 
@@ -96,9 +96,11 @@ flowchart TD
 
 > **Startup config load is retried (S5)**: step 1's deployed-config read is best-effort against local SQLite, which can transiently fail (locked/busy). A failed load must **not** leave the site as a silent zero-instance node — it is re-attempted on a fixed interval until it succeeds, then the retry self-cancels.
 
-> **Per-instance compilation during staggered startup (P6, follow-up)**: the Roslyn compile of each instance's scripts still runs inside Instance Actor start during staggered startup; moving it off-thread is a deferred optimization (it affects failover time-to-recover only, not correctness). As of the round-2 hardening, a process-wide compile cache dedupes identical script bodies within a node's process lifetime (the deploy gate's compile is reused by Instance Actor start), shrinking the recompile cost; the *first* compile after process start still runs inside Instance Actor start, so the deferral stands.
+> **Per-instance compilation during staggered startup — CLOSED (WP3.1)**: this was a deferred optimization (P6) affecting failover time-to-recover; each Instance Actor Roslyn-compiled its own scripts inside its own start, serialising a site's whole recovery behind compilation. Each staggered startup batch now runs a **pre-warm step**: before the batch's Instance Actors are created, the Deployment Manager compiles that batch's distinct script bodies and trigger expressions on a background task and pipes `BatchCompileWarmed` back, so every `PreStart` compile is a cache hit. Instance Actor start still calls the compilation service — it remains the correctness backstop — but in every warmed path those calls are memoised lookups. Cost: one extra message per batch.
 
-> **The Roslyn `ScriptOptions` are process-static, and must stay that way**: the reference set backing every site-side compile is built once per process, never per compile. `ScriptOptions.WithReferences(Assembly[])` resolves each assembly through `MetadataReference.CreateFromFile`, which does **not** cache — each call mints a fresh `MetadataReference` owning an `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock`, an unmanaged copy of the assembly metadata that nothing disposes. Building the options per compile leaks native memory permanently: no GC reclaims it, and it is invisible to gcdump and to the managed allocation counters, so the node's working set grows without the GC heap growing. This was a live defect (a Site node at 2,885 MB working set with 150 MB of live GC heap, ~2,469 MB of it on the default process heap across ~6,700 undisposed `AssemblyMetadata` instances). Note this is orthogonal to the compile cache above — the cache dedupes *identical* script bodies, so it bounds nothing when the bodies differ, and it clears wholesale on overflow, after which every script recompiles. Pinned by a reference-equality regression test rather than by watching memory.
+> **The Roslyn `ScriptOptions` are process-static, and must stay that way**: the reference set backing every site-side compile is built once per process, never per compile. `ScriptOptions.WithReferences(Assembly[])` resolves each assembly through `MetadataReference.CreateFromFile`, which does **not** cache — each call mints a fresh `MetadataReference` owning an `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock`, an unmanaged copy of the assembly metadata that nothing disposes. Building the options per compile leaks native memory permanently: no GC reclaims it, and it is invisible to gcdump and to the managed allocation counters, so the node's working set grows without the GC heap growing. This was a live defect (a Site node at 2,885 MB working set with 150 MB of live GC heap, ~2,469 MB of it on the default process heap across ~6,700 undisposed `AssemblyMetadata` instances). Note this is orthogonal to the compile cache above — the cache dedupes *identical* script bodies, so it bounds nothing when the bodies differ. (It no longer clears wholesale on overflow: WP3.1 replaced that latency cliff with approximate-LRU batch eviction, see **Compile cache eviction** below.) Pinned by a reference-equality regression test rather than by watching memory.
+
+> **Compile cache eviction is approximate LRU, not a wholesale clear (WP3.1)**: `SiteScriptCompileCache` is bounded at 1024 entries — deliberately smaller than the TemplateEngine verdict cache's 4096, because entries pin compiled assemblies rather than verdict strings. It used to `Clear()` **everything** on overflow. Instance scripts *and* trigger expressions share this cache, so on a site large enough to cross the bound every overflow discarded up to 1023 live compiled scripts and the next deploy or Instance-Actor start paid a full recompile storm **on actor threads**. Overflow now evicts only the oldest ⅛ (128 entries) by last-access stamp: one 1024-element scan instead of 1023 future recompiles, and hot entries survive. Recency is an `Interlocked` access sequence rather than a clock — deterministic for tests and immune to clock steps; hits update it lock-free, and only the (rare) eviction sweep takes a lock, double-checked so concurrent inserts do not stampede it. The sweep snapshots through `ConcurrentDictionary.ToArray()` and **not** LINQ over the dictionary: LINQ's `ToArray` picks the `ICollection.CopyTo` fast path, which throws when a concurrent insert lands between its Count read and its copy, and inserts are deliberately lock-free.
 
 > **Static options were only half the fix — the compile must also use the shared caching metadata resolver**: the static `ScriptOptions` carry only the *direct* API-surface references (5 assemblies here), and each `script.Compile()` still binds their **transitive closure** (~105 assemblies on a site node). Every transitively-referenced assembly is resolved through the options' `MetadataReferenceResolver`, and Roslyn's default one has no cross-compilation cache — it calls `MetadataReference.CreateFromFile` afresh on **every compile**, minting the same undisposed `AssemblyMetadata` → `PEReader` → `NativeHeapMemoryBlock` triple per assembly per compiled script, pinned for the process lifetime by the compile cache. That per-compile *re-resolution* — not the per-compile options construction — was the dominant term (measured live: 21 site scripts holding 6,640 metadata objects, counts bit-identical before and after the static-options fix). `ScriptCompilationService` therefore also attaches the process-wide `CachingScriptMetadataResolver.Instance` (owned by the Script Analysis component, see `Component-ScriptAnalysis.md`) via `ScriptOptions.WithMetadataResolver`, so each distinct assembly is materialized once per process. The decorator returns exactly what the undecorated resolver returned — only object identity is de-duplicated — so compile results and the trust gate are unchanged.
 
@@ -179,10 +181,10 @@ The Instance Actor supervises all child Script and Alarm Actors with explicit st
 
 | Child Actor | Exception Type | Strategy | Rationale |
 |-------------|---------------|----------|-----------|
-| Script Actor | Any exception | Resume | Script Actor is a coordinator — its state (trigger timers, last execution time) should survive child failures. Script Execution Actor failures are isolated. |
+| Script Actor | Any exception | Resume | Script Actor is a coordinator — its state (trigger timers, last execution time) should survive run failures. |
 | Alarm Actor | Any exception | Resume | Alarm Actor holds alarm state. Resume preserves state and continues evaluation on next value update. |
-| Script Execution Actor | Unhandled exception | Stop | Short-lived, per-invocation. Failure is logged; the Script Actor coordinator remains active for future triggers. |
-| Alarm Execution Actor | Unhandled exception | Stop | Short-lived, per on-trigger invocation. Same as Script Execution Actor. |
+
+Script Actor and Alarm Actor have **no children of their own** since WP3.1 removed the per-run execution actors, so neither declares a supervision strategy any more. A run's own exceptions were always contained inside the run's try/catch (they never reached supervision); a failure of the *launch* is caught by the coordinator, which answers the caller and releases the run slot rather than letting the throw escalate to the Instance Actor.
 
 The Deployment Manager singleton supervises Instance Actors with a **OneForOneStrategy** — one Instance Actor's failure does not affect other instances.
 
@@ -196,11 +198,11 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
 - **Coordinator** for a single script definition on an instance.
 - Holds the compiled script code and trigger configuration.
 - Manages trigger evaluation (interval timer, value change detection, conditional evaluation).
-- Spawns short-lived Script Execution Actors for each invocation.
+- Launches script runs directly for each invocation (no per-run child actor since WP3.1).
 
 ### Trigger Management
-- **Interval**: The Script Actor manages an internal timer. When the timer fires, it spawns a Script Execution Actor.
-- **Value Change**: The Script Actor subscribes to attribute change notifications from its parent Instance Actor for the specific monitored attribute. When the attribute changes, it spawns a Script Execution Actor.
+- **Interval**: The Script Actor manages an internal timer. When the timer fires, it launches a run.
+- **Value Change**: The Script Actor subscribes to attribute change notifications from its parent Instance Actor for the specific monitored attribute. When the attribute changes, it launches a run.
 - **Conditional**: The Script Actor subscribes to attribute change notifications for the monitored attribute. On each update, it evaluates the condition (compares the attribute against a threshold). Firing depends on the **fire mode** (see below).
 - **Expression**: The Script Actor evaluates a compiled boolean expression against an attribute snapshot on each attribute change. Firing depends on the **fire mode** (see below).
 - **Fire mode (Conditional + Expression)**:
@@ -211,20 +213,22 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
 > **Per-attribute change routing (P2).** Rather than broadcasting every attribute change to all child Script/Alarm Actors, the Instance Actor routes each change only to the children that monitor that attribute (built from their trigger configs at spawn time); **Expression-trigger children, which read an attribute snapshot, receive all changes**. Each child still keeps its own trigger gate as defense-in-depth — the routing is an optimization, not the authority.
 
 ### Concurrent Execution
-- Each invocation spawns a **new Script Execution Actor** as a child.
-- Multiple Script Execution Actors can run concurrently (e.g., a trigger fires while a previous `Instance.CallScript` invocation is still running).
-- The Script Actor coordinates but does not block on child completion.
+- Each invocation launches a **new run** on the script-execution scheduler.
+- Multiple runs can execute concurrently (e.g., a trigger fires while a previous `Instance.CallScript` invocation is still running), up to `MaxConcurrentRunsPerScript` — beyond which the newest is shed (see below).
+- The Script Actor coordinates but does not block on run completion.
 
-### Script Execution Actor
-- **Short-lived** child actor created per invocation.
-- Receives: compiled script code, input parameters, reference to the parent Instance Actor, current call depth.
-- Executes the script in the Akka actor context.
-- Has access to the full Script Runtime API (see below).
-- Returns the script's return value (if defined) to the caller, then stops.
-- The script body itself runs on the **dedicated `ScriptExecutionScheduler`** (a bounded set of dedicated threads), not the shared .NET thread pool, so blocking script I/O cannot starve the global pool or stall Akka dispatchers. The scheduler is **process-wide by default** (one pool per host, sized from `SiteRuntimeOptions.ScriptExecutionThreadCount`), but each script/alarm actor takes it through an **optional injection seam** rather than reaching for the static directly: the Host injects nothing and gets the shared pool, while tests (or a future multi-site host) can hand an actor its own instance. The shared accessor also recreates a disposed pool rather than returning it, so a disposed scheduler can never silently poison later executions.
+### Script Run Launch (WP3.1 — replaces the Script Execution Actor)
+- A run is launched **directly by the Script Actor** through `ScriptRunLauncher`. There is no per-run child actor.
+  - The former `ScriptExecutionActor` / `AlarmExecutionActor` were already inert shells: neither declared a `Receive` handler (they executed from their constructor), neither had a `PostStop`, state, or stash, and neither's `IActorRef` was ever a message target — the whole lifecycle lived inside a detached task the actor never observed. What they cost was an actor cell, mailbox, and name registration **per run**, plus a per-spawn expression-tree `Props.Create`. Removing them changed no semantics; the run body moved verbatim into the shared launcher.
+  - Everything the shells provided is preserved: exception and timeout containment, one DI scope per run disposed on every path, the site-event/health telemetry, the Ask reply, the completion notification (now the coordinator's own `Self`), and the audit `ExecutionId` / `ParentExecutionId` threading. A run still in flight when its Script Actor is stopped runs to completion and its completion message dead-letters, exactly as before — **stopping does NOT cancel in-flight runs**; redeploy/undeploy semantics are unchanged.
+  - One deliberate improvement: a failure of the *launch itself* (e.g. queueing onto a disposed scheduler) is caught by the Script Actor, which replies to the Ask caller and releases the run slot. The old per-run child's constructor throw was handled by a Stop supervision directive that sent no reply, leaving the caller to hang to its Ask timeout.
+- The script body runs on the **dedicated `ScriptExecutionScheduler`** (a bounded set of dedicated threads), not the shared .NET thread pool, so blocking script I/O cannot starve the global pool or stall Akka dispatchers. The scheduler is **process-wide by default** (one pool per host), but each script/alarm actor takes it through an **optional injection seam** rather than reaching for the static directly: the Host injects nothing and gets the shared pool, while tests (or a future multi-site host) can hand an actor its own instance. The shared accessor also recreates a disposed pool rather than returning it, so a disposed scheduler can never silently poison later executions.
+- **Pool sizing is instance-scaled and grow-only (WP3.1).** The pool was a fixed 8 threads regardless of load. It is now `clamp(max(ScriptExecutionThreadCount, ceil(enabledInstances / 8)), 1, ScriptExecutionMaxThreadCount)` — the existing `ScriptExecutionThreadCount` (default 8) becomes the **floor**, so configurations at or below 64 instances behave exactly as before, and the new `ScriptExecutionMaxThreadCount` (default 32) is the ceiling. Beyond the ceiling the per-script cap below is the real regulator. `DeploymentManagerActor.UpdateInstanceCounts` calls `EnsureCapacity` on every deploy / undeploy / enable / disable and once per staggered startup batch. Growth only: undeploying leaves idle threads, which cost nothing measurable and avoid drain/steal complexity.
+- **The deadline is armed at enqueue, not at dequeue (WP3.1).** The run's timeout `CancellationTokenSource` is created on the actor thread *before* the body is queued, so queue wait consumes the script's own budget. A body that dequeues past its deadline **skips execution entirely** and takes the existing timeout path (site event, script-error counter, error reply, completion message): a saturated pool sheds stale work instead of running it late with a fresh full budget.
+- **Concurrent runs per script are capped (WP3.1).** `MaxConcurrentRunsPerScript` (default 4) bounds runs in flight — queued or executing — for any one script or alarm on-trigger script. Over the cap the **newest** run is shed: the four already in flight are closest to their own deadlines and already charged against them, so nothing is ever reordered and no extra queue is needed (the scheduler's FIFO already is the queue). A shed increments `ISiteHealthCollector.IncrementScriptRunShed` (surfaced as `ScriptRunShedCount`), emits a `script`/`Warning` site event **rate-limited to one per script per minute** so a hot trigger cannot flood `site_events`, and — for an Ask-based `CallScript` — replies with an explicit error so a nested call or inbound-API route fails fast instead of hanging.
 
 ### Handling `Instance.CallScript`
-- When an external caller (another Script Execution Actor, an Alarm Execution Actor, or a routed call from the Inbound API) sends a `CallScript` message to the Script Actor, it spawns a Script Execution Actor to handle the call.
+- When an external caller (another script run, an alarm on-trigger run, or a routed call from the Inbound API) sends a `CallScript` message to the Script Actor, it launches a run to handle the call.
 - The caller uses the **Akka ask pattern** and receives the return value when the execution completes.
 
 ---
@@ -244,7 +248,7 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
   - **Range Violation**: Value is outside the allowed min/max range.
   - **Rate of Change**: Value change rate exceeds the defined threshold over a configurable time window. Direction filter (rising / falling / either) restricts which side of the rate triggers.
   - **HiLo**: Multi-setpoint level alarm with up to four configurable setpoints (LoLo, Lo, Hi, HiHi). Any subset may be configured. Each setpoint may carry its own priority that overrides the alarm-level priority for that band.
-- **Expression** trigger evaluation runs on the bounded script-execution thread pool shared with script bodies (and Expression *script* triggers): scheduler saturation — e.g. stuck scripts occupying all threads — delays Expression alarm transitions site-wide until a thread frees. This is a deliberate trade-off (evaluation no longer blocks dispatcher threads; per-actor coalescing bounds queue growth) and the scheduler gauges + stuck-script watchdog make the saturated state visible.
+- **Expression** trigger evaluation runs on the shared .NET thread pool behind a process-wide concurrency gate (`TriggerEvalGate`, sized by `TriggerEvalMaxConcurrency`, default `max(2, ProcessorCount)`) — **not** on the bounded script-execution pool. This is the WP3.1 fix for arch-review finding #4 (High): when evaluation shared that pool, N script bodies blocked in synchronous I/O stalled *every* Expression trigger on the node — scripts and alarms alike — for an unbounded time, and the evaluation's own 2 s timeout was constructed inside the queued body, so it did not start ticking until dequeue. An alarm that should have raised in milliseconds simply never raised, with neither a raise nor a timeout visible to the operator. Trigger expressions are non-blocking **by construction** (`TriggerExpressionGlobals` exposes only reads over an in-memory snapshot, and the script trust gate has already denied I/O, network, threading, and reflection), so the shared pool is where they belong; a second dedicated pool was considered and rejected as adding threads, gauges, and a second starvation surface for no isolation gain. The evaluation deadline (`TriggerEvalTimeoutSeconds`, default 2, previously hardcoded) is now armed **at enqueue**, so gate-wait time burns the same budget and a saturated gate yields a timely `false` rather than an unbounded stall. Per-actor coalescing (one evaluation in flight, one pending) is unchanged and caps waiters at one per Expression trigger, so the gate queue is bounded by trigger count.
 - For binary trigger types (ValueMatch / RangeViolation / RateOfChange), when the condition is met and the alarm is currently in **normal** state, the alarm transitions to **active**:
   - Updates the alarm state on the parent Instance Actor (which publishes to the Akka stream).
   - If an on-trigger script is defined, spawns an Alarm Execution Actor to execute it.
@@ -256,9 +260,8 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
 - Held **in memory** only — not persisted to SQLite. State comprises `AlarmState` (Active / Normal) and `AlarmLevel` (None for binary triggers; the active band for HiLo).
 - On restart (or failover), alarm states are re-evaluated from incoming values. All alarms start in normal state with level None and transition based on incoming values.
 
-### Alarm Execution Actor
-- **Short-lived** child actor created when an on-trigger script needs to execute.
-- Same pattern as Script Execution Actor — receives compiled code, executes, returns, and stops.
+### Alarm On-Trigger Run
+- Launched directly by the Alarm Actor when an on-trigger script needs to execute — the same `ScriptRunLauncher` path as an instance script run, with the firing alarm's name/level/priority/message exposed through the `Alarm` global. Bounded by the same `MaxConcurrentRunsPerScript` cap.
 - Has access to the Instance Actor for `GetAttribute`/`SetAttribute`.
 - **Can** call instance scripts via `Instance.CallScript()` — sends an ask message to the appropriate sibling Script Actor.
 - Instance scripts **cannot** call alarm on-trigger scripts — the call direction is one-way.
@@ -275,7 +278,7 @@ change:
 |---|---|
 | `ScriptRuntimeContext.SetAttribute` (this run's `ExecutionId`) / `RouteToSetAttributesRequest.ParentExecutionId` (the inbound request's) | → `SetStaticAttributeCommand.SourceExecutionId` |
 | Instance Actor static-write path (`HandleSetStaticAttributeCore`) | → `AttributeValueChanged.SourceExecutionId` |
-| Alarm Actor trigger evaluation → `SpawnAlarmExecution` | → `AlarmExecutionActor` → `ScriptRuntimeContext.ParentExecutionId` |
+| Alarm Actor trigger evaluation → `SpawnAlarmExecution` | → `ScriptRunLauncher.LaunchAlarmScript` → `ScriptRuntimeContext.ParentExecutionId` |
 
 All four computed trigger types participate. `Expression` triggers evaluate a
 whole attribute snapshot **off the dispatcher**, so the firing change is no
@@ -382,7 +385,7 @@ The enriched message flows Instance Actor → site-wide Akka stream → `SiteStr
 
 - Shared scripts are compiled at the site when received from central.
 - Compiled code is stored in memory and made available to all Script Actors.
-- When a Script Execution Actor calls `Scripts.CallShared("scriptName", params)`, the shared script code executes **inline** in the Script Execution Actor's context — it is a direct method invocation, not an actor message.
+- When a script run calls `Scripts.CallShared("scriptName", params)`, the shared script code executes **inline** in that run's context — it is a direct method invocation, not an actor message.
 - This avoids serialization bottlenecks since there is no shared script actor to contend for.
 - Shared scripts have access to the same runtime API as instance scripts (GetAttribute, SetAttribute, external systems, notifications, databases).
 - **Shared scripts always execute with Root scope (C5)**: `Attributes[...]` inside a shared script addresses **root** attributes even when the shared script is invoked from a composed-module script. Callers needing module-scoped access must pass the values in as parameters.
@@ -417,14 +420,14 @@ All script types can be updated without restarting the cluster, but the mechanis
 
 ## Script Runtime API
 
-Available to all Script Execution Actors and Alarm Execution Actors:
+Available to all script runs and alarm on-trigger runs:
 
 ### Instance Attributes
 - `Instance.GetAttribute("name")` — Read an attribute value from the parent Instance Actor.
 - `Instance.SetAttribute("name", value)` — Write an attribute value. For data-connected attributes, writes to the DCL; for static attributes, updates in-memory and persists to local SQLite (survives restart/failover, reset on redeployment).
 
 ### Other Scripts
-- `Instance.CallScript("scriptName", parameters)` — Send an ask message to a sibling Script Actor. The target Script Actor spawns a Script Execution Actor, executes, and returns the result. The call includes the current recursion depth.
+- `Instance.CallScript("scriptName", parameters)` — Send an ask message to a sibling Script Actor. The target Script Actor launches a run, executes, and returns the result (or an explicit shed error if that script is already at its concurrent-run cap). The call includes the current recursion depth.
 - `Scripts.CallShared("scriptName", parameters)` — Execute shared script code inline (direct method invocation). The call includes the current recursion depth.
 
 ### External Systems
@@ -474,7 +477,7 @@ Scripts execute **in-process** with constrained access. The following restrictio
 
 - **Allowed**: Access to the Script Runtime API (GetAttribute, SetAttribute, CallScript, CallShared, ExternalSystem, Notify, Database, Tracking, Alarms), standard C# language features, basic .NET types (collections, string manipulation, math, date/time). `System.Diagnostics.Stopwatch`, `Debug`, and `Activity` are permitted.
 - **Forbidden**: File system access (`System.IO`), process spawning (`System.Diagnostics.Process`), threading (`System.Threading` — except `Tasks`, `CancellationToken`, and `CancellationTokenSource`), reflection (`System.Reflection`), all raw network access (`System.Net` — must use `ExternalSystem.Call`), native interop (`System.Runtime.InteropServices`, `Microsoft.Win32`), assembly loading, unsafe code, `dynamic`, `Activator`.
-- **Execution timeout**: Configurable per-script maximum execution time. Exceeding the timeout cancels the script **cooperatively** (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A **watchdog logs the script by name** once its thread has not returned within a grace period after the timeout, and the script-execution scheduler's **queue depth / busy-thread / oldest-busy-age gauges** are reported through Health Monitoring so a saturated or stuck scheduler is visible.
+- **Execution timeout**: Configurable per-script maximum execution time, **measured from enqueue** (WP3.1) — a run that spends its whole budget queued behind blocked scripts is shed at dequeue without executing, rather than being granted a fresh full budget for work whose triggering condition is already stale. Exceeding the timeout cancels the script **cooperatively** (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A **watchdog names the script** once its thread has not returned within a grace period after the timeout — and, as of WP3.1, **detaches and replaces that worker thread**, so a wedged script costs the pool a thread only until the watchdog fires rather than for the rest of the process. The detached worker exits (instead of pulling more work) when its body finally returns, so capacity never silently doubles. Live detached workers are capped at the pool size — at the cap no replacement is started and an Error site event says so, because bounded starvation is preferable to unbounded thread growth when scripts wedge en masse. The scheduler's **queue depth / busy-thread / oldest-busy-age / detached-thread gauges** and the **per-interval shed count** are reported through Health Monitoring so a saturated, stuck, or shedding scheduler is visible.
 - **Memory**: Scripts share the host process memory. No per-script memory limit, but the execution timeout prevents runaway allocations.
 
 The forbidden-API policy is defined authoritatively in `ScriptTrustPolicy` (Script Analysis component, #25). `ScriptCompilationService.ValidateTrustModel` delegates to `ScriptTrustValidator.FindViolations` for the trust verdict; the Site Runtime also performs a `CSharpScript.Compile` against the real `ScriptGlobals` for execution. This is defence-in-depth static enforcement, not a true runtime sandbox.
@@ -540,7 +543,7 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
 ### Script Compilation Errors
 - If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied.
 - The failure is reported back to central as a failed deployment.
-- **Enforced by a site-side pre-compile validation gate (S3)**: before the Instance Actor is created or the config persisted, the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **synchronously, as a pure prefix step of the deploy handler on the singleton's actor thread** — deliberately not off-thread, because piping the verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering (see the in-code rationale in `DeploymentManagerActor.HandleDeploy`); unchanged script bodies hit the process-wide compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors. This makes the deployment result honest even if central's pre-deployment validation was bypassed or skew let a bad artifact through.
+- **Enforced by a site-side pre-compile validation gate (S3)**: before the Instance Actor is created or the config persisted, the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **synchronously, as a pure prefix step of the deploy application on the singleton's actor thread** — the verdict is never derived off-thread, because piping a verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering. WP3.1 made it **warm-then-gate**: the same compile is first run off-thread purely to populate the process-wide compile cache (`DeployCompileWarmed` pipes back), so the authoritative synchronous gate is then all cache hits and no longer holds the singleton for a Roslyn compile. Ordering is preserved by a **per-instance in-flight guard** — while a warm is in flight for instance X, further mutating commands for X (deploy / delete / disable / enable) are queued and re-dispatched in arrival order once the deploy applies; a second deploy with nothing queued behind it supersedes the pending one last-write-wins and its displaced deployer is answered `Failed`-superseded rather than left to Ask-timeout. Commands for **other** instances flow freely, which is the point, and is safe because cross-instance ordering was never guaranteed to callers. Unchanged script bodies hit the compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors. This makes the deployment result honest even if central's pre-deployment validation was bypassed or skew let a bad artifact through.
 - Note: Pre-deployment validation at central should still catch compilation errors before they reach the site; the site gate is the authoritative backstop.
 
 ---